diff options
Diffstat (limited to 'Doc')
169 files changed, 5275 insertions, 1402 deletions
diff --git a/Doc/Makefile b/Doc/Makefile index ea3023157d..200a9f7d1f 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -15,11 +15,12 @@ ALLSPHINXOPTS = -b $(BUILDER) -d build/doctrees -D latex_paper_size=$(PAPER) \ .PHONY: help build html htmlhelp latex text changes linkcheck \ suspicious coverage doctest pydoc-topics htmlview clean dist check serve \ - autobuild-dev autobuild-stable + autobuild-dev autobuild-stable venv help: @echo "Please use \`make <target>' where <target> is one of" @echo " clean to remove build files" + @echo " venv to create a venv with necessary tools" @echo " html to make standalone HTML files" @echo " htmlview to open the index page built by the html target in your browser" @echo " htmlhelp to make HTML files and a HTML help project" @@ -102,7 +103,11 @@ htmlview: html $(PYTHON) -c "import webbrowser; webbrowser.open('build/html/index.html')" clean: - -rm -rf build/* + -rm -rf build/* venv/* + +venv: + $(PYTHON) -m venv venv + ./venv/bin/python3 -m pip install -U Sphinx dist: rm -rf dist @@ -172,4 +177,3 @@ autobuild-stable: exit 1;; \ esac @make autobuild-dev - diff --git a/Doc/README.txt b/Doc/README.txt index 58bda62697..580d10241d 100644 --- a/Doc/README.txt +++ b/Doc/README.txt @@ -3,7 +3,7 @@ Python Documentation README This directory contains the reStructuredText (reST) sources to the Python documentation. You don't need to build them yourself, prebuilt versions are -available at <https://docs.python.org/3.4/download.html>. +available at <https://docs.python.org/dev/download.html>. Documentation on authoring Python documentation, including information about both style and markup, is available in the "Documenting Python" chapter of the diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index 3c0f4b98cd..ed62deaec1 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -152,7 +152,7 @@ Unless otherwise stated, buffers are not NUL-terminated. any conversion. Raises :exc:`TypeError` if the object is not a Unicode object. The C variable may also be declared as :c:type:`PyObject\*`. -``w*`` (:class:`bytearray` or read-write byte-oriented buffer) [Py_buffer] +``w*`` (read-write :term:`bytes-like object`) [Py_buffer] This format accepts any object which implements the read-write buffer interface. It fills a :c:type:`Py_buffer` structure provided by the caller. The buffer may contain embedded null bytes. The caller have to call diff --git a/Doc/c-api/codec.rst b/Doc/c-api/codec.rst index 83252afbb7..dfe3d436e5 100644 --- a/Doc/c-api/codec.rst +++ b/Doc/c-api/codec.rst @@ -116,3 +116,8 @@ Registry API for Unicode encoding error handlers Replace the unicode encode error with backslash escapes (``\x``, ``\u`` and ``\U``). +.. c:function:: PyObject* PyCodec_NameReplaceErrors(PyObject *exc) + + Replace the unicode encode error with ``\N{...}`` escapes. + + .. versionadded:: 3.5 diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index 66b7752661..814317b359 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -9,13 +9,19 @@ Exception Handling The functions described in this chapter will let you handle and raise Python exceptions. It is important to understand some of the basics of Python -exception handling. It works somewhat like the Unix :c:data:`errno` variable: +exception handling. It works somewhat like the POSIX :c:data:`errno` variable: there is a global indicator (per thread) of the last error that occurred. Most -functions don't clear this on success, but will set it to indicate the cause of -the error on failure. Most functions also return an error indicator, usually -*NULL* if they are supposed to return a pointer, or ``-1`` if they return an -integer (exception: the :c:func:`PyArg_\*` functions return ``1`` for success and -``0`` for failure). +C API functions don't clear this on success, but will set it to indicate the +cause of the error on failure. Most C API functions also return an error +indicator, usually *NULL* if they are supposed to return a pointer, or ``-1`` +if they return an integer (exception: the :c:func:`PyArg_\*` functions +return ``1`` for success and ``0`` for failure). + +Concretely, the error indicator consists of three object pointers: the +exception's type, the exception's value, and the traceback object. Any +of those pointers can be NULL if non-set (although some combinations are +forbidden, for example you can't have a non-NULL traceback if the exception +type is NULL). When a function must fail because some function it called failed, it generally doesn't set the error indicator; the function it called already set it. It is @@ -27,12 +33,21 @@ the caller that an error has been set. If the error is not handled or carefully propagated, additional calls into the Python/C API may not behave as intended and may fail in mysterious ways. -The error indicator consists of three Python objects corresponding to the result -of ``sys.exc_info()``. API functions exist to interact with the error indicator -in various ways. There is a separate error indicator for each thread. +.. note:: + The error indicator is **not** the result of :func:`sys.exc_info()`. + The former corresponds to an exception that is not yet caught (and is + therefore still propagating), while the latter returns an exception after + it is caught (and has therefore stopped propagating). -.. XXX Order of these should be more thoughtful. - Either alphabetical or some kind of structure. + +Printing and clearing +===================== + + +.. c:function:: void PyErr_Clear() + + Clear the error indicator. If the error indicator is not set, there is no + effect. .. c:function:: void PyErr_PrintEx(int set_sys_last_vars) @@ -51,127 +66,24 @@ in various ways. There is a separate error indicator for each thread. Alias for ``PyErr_PrintEx(1)``. -.. c:function:: PyObject* PyErr_Occurred() - - Test whether the error indicator is set. If set, return the exception *type* - (the first argument to the last call to one of the :c:func:`PyErr_Set\*` - functions or to :c:func:`PyErr_Restore`). If not set, return *NULL*. You do not - own a reference to the return value, so you do not need to :c:func:`Py_DECREF` - it. - - .. note:: - - Do not compare the return value to a specific exception; use - :c:func:`PyErr_ExceptionMatches` instead, shown below. (The comparison could - easily fail since the exception may be an instance instead of a class, in the - case of a class exception, or it may be a subclass of the expected exception.) - - -.. c:function:: int PyErr_ExceptionMatches(PyObject *exc) - - Equivalent to ``PyErr_GivenExceptionMatches(PyErr_Occurred(), exc)``. This - should only be called when an exception is actually set; a memory access - violation will occur if no exception has been raised. - - -.. c:function:: int PyErr_GivenExceptionMatches(PyObject *given, PyObject *exc) - - Return true if the *given* exception matches the exception in *exc*. If - *exc* is a class object, this also returns true when *given* is an instance - of a subclass. If *exc* is a tuple, all exceptions in the tuple (and - recursively in subtuples) are searched for a match. - - -.. c:function:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) - - Under certain circumstances, the values returned by :c:func:`PyErr_Fetch` below - can be "unnormalized", meaning that ``*exc`` is a class object but ``*val`` is - not an instance of the same class. This function can be used to instantiate - the class in that case. If the values are already normalized, nothing happens. - The delayed normalization is implemented to improve performance. - - .. note:: - - This function *does not* implicitly set the ``__traceback__`` - attribute on the exception value. If setting the traceback - appropriately is desired, the following additional snippet is needed:: - - if (tb != NULL) { - PyException_SetTraceback(val, tb); - } - - -.. c:function:: void PyErr_Clear() - - Clear the error indicator. If the error indicator is not set, there is no - effect. - - -.. c:function:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) - - Retrieve the error indicator into three variables whose addresses are passed. - If the error indicator is not set, set all three variables to *NULL*. If it is - set, it will be cleared and you own a reference to each object retrieved. The - value and traceback object may be *NULL* even when the type object is not. - - .. note:: - - This function is normally only used by code that needs to handle exceptions or - by code that needs to save and restore the error indicator temporarily. - - -.. c:function:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback) - - Set the error indicator from the three objects. If the error indicator is - already set, it is cleared first. If the objects are *NULL*, the error - indicator is cleared. Do not pass a *NULL* type and non-*NULL* value or - traceback. The exception type should be a class. Do not pass an invalid - exception type or value. (Violating these rules will cause subtle problems - later.) This call takes away a reference to each object: you must own a - reference to each object before the call and after the call you no longer own - these references. (If you don't understand this, don't use this function. I - warned you.) - - .. note:: - - This function is normally only used by code that needs to save and restore the - error indicator temporarily; use :c:func:`PyErr_Fetch` to save the current - exception state. - - -.. c:function:: void PyErr_GetExcInfo(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) - - Retrieve the exception info, as known from ``sys.exc_info()``. This refers - to an exception that was already caught, not to an exception that was - freshly raised. Returns new references for the three objects, any of which - may be *NULL*. Does not modify the exception info state. - - .. note:: - - This function is not normally used by code that wants to handle exceptions. - Rather, it can be used when code needs to save and restore the exception - state temporarily. Use :c:func:`PyErr_SetExcInfo` to restore or clear the - exception state. - - .. versionadded:: 3.3 - +.. c:function:: void PyErr_WriteUnraisable(PyObject *obj) -.. c:function:: void PyErr_SetExcInfo(PyObject *type, PyObject *value, PyObject *traceback) + This utility function prints a warning message to ``sys.stderr`` when an + exception has been set but it is impossible for the interpreter to actually + raise the exception. It is used, for example, when an exception occurs in an + :meth:`__del__` method. - Set the exception info, as known from ``sys.exc_info()``. This refers - to an exception that was already caught, not to an exception that was - freshly raised. This function steals the references of the arguments. - To clear the exception state, pass *NULL* for all three arguments. - For general rules about the three arguments, see :c:func:`PyErr_Restore`. + The function is called with a single argument *obj* that identifies the context + in which the unraisable exception occurred. The repr of *obj* will be printed in + the warning message. - .. note:: - This function is not normally used by code that wants to handle exceptions. - Rather, it can be used when code needs to save and restore the exception - state temporarily. Use :c:func:`PyErr_GetExcInfo` to read the exception - state. +Raising exceptions +================== - .. versionadded:: 3.3 +These functions help you set the current thread's error indicator. +For convenience, some of these functions will always return a +NULL pointer for use in a ``return`` statement. .. c:function:: void PyErr_SetString(PyObject *type, const char *message) @@ -197,6 +109,14 @@ in various ways. There is a separate error indicator for each thread. string. +.. c:function:: PyObject* PyErr_FormatV(PyObject *exception, const char *format, va_list vargs) + + Same as :c:func:`PyErr_Format`, but taking a :c:type:`va_list` argument rather + than a variable number of arguments. + + .. versionadded:: 3.5 + + .. c:function:: void PyErr_SetNone(PyObject *type) This is a shorthand for ``PyErr_SetObject(type, Py_None)``. @@ -346,6 +266,22 @@ in various ways. There is a separate error indicator for each thread. use. +Issuing warnings +================ + +Use these functions to issue warnings from C code. They mirror similar +functions exported by the Python :mod:`warnings` module. They normally +print a warning message to *sys.stderr*; however, it is +also possible that the user has specified that warnings are to be turned into +errors, and in that case they will raise an exception. It is also possible that +the functions raise an exception because of a problem with the warning machinery. +The return value is ``0`` if no exception is raised, or ``-1`` if an exception +is raised. (It is not possible to determine whether a warning message is +actually printed, nor what the reason is for the exception; this is +intentional.) If an exception is raised, the caller should do its normal +exception handling (for example, :c:func:`Py_DECREF` owned references and return +an error value). + .. c:function:: int PyErr_WarnEx(PyObject *category, const char *message, Py_ssize_t stack_level) Issue a warning message. The *category* argument is a warning category (see @@ -355,18 +291,6 @@ in various ways. There is a separate error indicator for each thread. is the function calling :c:func:`PyErr_WarnEx`, 2 is the function above that, and so forth. - This function normally prints a warning message to *sys.stderr*; however, it is - also possible that the user has specified that warnings are to be turned into - errors, and in that case this will raise an exception. It is also possible that - the function raises an exception because of a problem with the warning machinery - (the implementation imports the :mod:`warnings` module to do the heavy lifting). - The return value is ``0`` if no exception is raised, or ``-1`` if an exception - is raised. (It is not possible to determine whether a warning message is - actually printed, nor what the reason is for the exception; this is - intentional.) If an exception is raised, the caller should do its normal - exception handling (for example, :c:func:`Py_DECREF` owned references and return - an error value). - Warning categories must be subclasses of :c:data:`Warning`; the default warning category is :c:data:`RuntimeWarning`. The standard Python warning categories are available as global variables whose names are ``PyExc_`` followed by the Python @@ -410,6 +334,139 @@ in various ways. There is a separate error indicator for each thread. .. versionadded:: 3.2 +Querying the error indicator +============================ + +.. c:function:: PyObject* PyErr_Occurred() + + Test whether the error indicator is set. If set, return the exception *type* + (the first argument to the last call to one of the :c:func:`PyErr_Set\*` + functions or to :c:func:`PyErr_Restore`). If not set, return *NULL*. You do not + own a reference to the return value, so you do not need to :c:func:`Py_DECREF` + it. + + .. note:: + + Do not compare the return value to a specific exception; use + :c:func:`PyErr_ExceptionMatches` instead, shown below. (The comparison could + easily fail since the exception may be an instance instead of a class, in the + case of a class exception, or it may be a subclass of the expected exception.) + + +.. c:function:: int PyErr_ExceptionMatches(PyObject *exc) + + Equivalent to ``PyErr_GivenExceptionMatches(PyErr_Occurred(), exc)``. This + should only be called when an exception is actually set; a memory access + violation will occur if no exception has been raised. + + +.. c:function:: int PyErr_GivenExceptionMatches(PyObject *given, PyObject *exc) + + Return true if the *given* exception matches the exception type in *exc*. If + *exc* is a class object, this also returns true when *given* is an instance + of a subclass. If *exc* is a tuple, all exception types in the tuple (and + recursively in subtuples) are searched for a match. + + +.. c:function:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) + + Retrieve the error indicator into three variables whose addresses are passed. + If the error indicator is not set, set all three variables to *NULL*. If it is + set, it will be cleared and you own a reference to each object retrieved. The + value and traceback object may be *NULL* even when the type object is not. + + .. note:: + + This function is normally only used by code that needs to catch exceptions or + by code that needs to save and restore the error indicator temporarily, e.g.:: + + { + PyObject **type, **value, **traceback; + PyErr_Fetch(&type, &value, &traceback); + + /* ... code that might produce other errors ... */ + + PyErr_Restore(type, value, traceback); + } + + +.. c:function:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback) + + Set the error indicator from the three objects. If the error indicator is + already set, it is cleared first. If the objects are *NULL*, the error + indicator is cleared. Do not pass a *NULL* type and non-*NULL* value or + traceback. The exception type should be a class. Do not pass an invalid + exception type or value. (Violating these rules will cause subtle problems + later.) This call takes away a reference to each object: you must own a + reference to each object before the call and after the call you no longer own + these references. (If you don't understand this, don't use this function. I + warned you.) + + .. note:: + + This function is normally only used by code that needs to save and restore the + error indicator temporarily. Use :c:func:`PyErr_Fetch` to save the current + error indicator. + + +.. c:function:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) + + Under certain circumstances, the values returned by :c:func:`PyErr_Fetch` below + can be "unnormalized", meaning that ``*exc`` is a class object but ``*val`` is + not an instance of the same class. This function can be used to instantiate + the class in that case. If the values are already normalized, nothing happens. + The delayed normalization is implemented to improve performance. + + .. note:: + + This function *does not* implicitly set the ``__traceback__`` + attribute on the exception value. If setting the traceback + appropriately is desired, the following additional snippet is needed:: + + if (tb != NULL) { + PyException_SetTraceback(val, tb); + } + + +.. c:function:: void PyErr_GetExcInfo(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) + + Retrieve the exception info, as known from ``sys.exc_info()``. This refers + to an exception that was *already caught*, not to an exception that was + freshly raised. Returns new references for the three objects, any of which + may be *NULL*. Does not modify the exception info state. + + .. note:: + + This function is not normally used by code that wants to handle exceptions. + Rather, it can be used when code needs to save and restore the exception + state temporarily. Use :c:func:`PyErr_SetExcInfo` to restore or clear the + exception state. + + .. versionadded:: 3.3 + + +.. c:function:: void PyErr_SetExcInfo(PyObject *type, PyObject *value, PyObject *traceback) + + Set the exception info, as known from ``sys.exc_info()``. This refers + to an exception that was *already caught*, not to an exception that was + freshly raised. This function steals the references of the arguments. + To clear the exception state, pass *NULL* for all three arguments. + For general rules about the three arguments, see :c:func:`PyErr_Restore`. + + .. note:: + + This function is not normally used by code that wants to handle exceptions. + Rather, it can be used when code needs to save and restore the exception + state temporarily. Use :c:func:`PyErr_GetExcInfo` to read the exception + state. + + .. versionadded:: 3.3 + + +Signal Handling +=============== + + .. c:function:: int PyErr_CheckSignals() .. index:: @@ -443,13 +500,21 @@ in various ways. There is a separate error indicator for each thread. .. c:function:: int PySignal_SetWakeupFd(int fd) - This utility function specifies a file descriptor to which a ``'\0'`` byte will - be written whenever a signal is received. It returns the previous such file - descriptor. The value ``-1`` disables the feature; this is the initial state. + This utility function specifies a file descriptor to which the signal number + is written as a single byte whenever a signal is received. *fd* must be + non-blocking. It returns the previous such file descriptor. + + The value ``-1`` disables the feature; this is the initial state. This is equivalent to :func:`signal.set_wakeup_fd` in Python, but without any error checking. *fd* should be a valid file descriptor. The function should only be called from the main thread. + .. versionchanged:: 3.5 + On Windows, the function now also supports socket handles. + + +Exception Classes +================= .. c:function:: PyObject* PyErr_NewException(char *name, PyObject *base, PyObject *dict) @@ -475,18 +540,6 @@ in various ways. There is a separate error indicator for each thread. .. versionadded:: 3.2 -.. c:function:: void PyErr_WriteUnraisable(PyObject *obj) - - This utility function prints a warning message to ``sys.stderr`` when an - exception has been set but it is impossible for the interpreter to actually - raise the exception. It is used, for example, when an exception occurs in an - :meth:`__del__` method. - - The function is called with a single argument *obj* that identifies the context - in which the unraisable exception occurred. The repr of *obj* will be printed in - the warning message. - - Exception Objects ================= diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst index 3641fc69b1..01e0deb304 100644 --- a/Doc/c-api/import.rst +++ b/Doc/c-api/import.rst @@ -183,9 +183,9 @@ Importing Modules .. c:function:: long PyImport_GetMagicNumber() - Return the magic number for Python bytecode files (a.k.a. :file:`.pyc` and - :file:`.pyo` files). The magic number should be present in the first four bytes - of the bytecode file, in little-endian byte order. Returns -1 on error. + Return the magic number for Python bytecode files (a.k.a. :file:`.pyc` file). + The magic number should be present in the first four bytes of the bytecode + file, in little-endian byte order. Returns -1 on error. .. versionchanged:: 3.3 Return value of -1 upon failure. diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 4d358ca757..fac9b478cf 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -134,6 +134,9 @@ Process-wide parameters change for the duration of the program's execution. No code in the Python interpreter will change the contents of this storage. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. c:function:: wchar* Py_GetProgramName() @@ -245,6 +248,9 @@ Process-wide parameters :data:`sys.exec_prefix` to be empty. It is up to the caller to modify these if required after calling :c:func:`Py_Initialize`. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + The path argument is copied internally, so the caller may free it after the call completes. @@ -344,6 +350,9 @@ Process-wide parameters :data:`sys.path`, which is the same as prepending the current working directory (``"."``). + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. note:: It is recommended that applications embedding the Python interpreter for purposes other than executing a single script pass 0 as *updatepath*, @@ -368,6 +377,9 @@ Process-wide parameters to 1 unless the :program:`python` interpreter was started with the :option:`-I`. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. versionchanged:: 3.4 The *updatepath* value depends on :option:`-I`. @@ -382,6 +394,9 @@ Process-wide parameters execution. No code in the Python interpreter will change the contents of this storage. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. c:function:: w_char* Py_GetPythonHome() diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index a82e1c2f2c..5d78f385ee 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -92,8 +92,8 @@ functions are thread-safe, the :term:`GIL <global interpreter lock>` does not need to be held. The default raw memory block allocator uses the following functions: -:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when -requesting zero bytes. +:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call +``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes. .. versionadded:: 3.4 @@ -106,6 +106,17 @@ requesting zero bytes. been initialized in any way. +.. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize) + + Allocates *nelem* elements each whose size in bytes is *elsize* and returns + a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the + request fails. The memory is initialized to zeros. Requesting zero elements + or elements of size zero bytes returns a distinct non-*NULL* pointer if + possible, as if ``PyMem_RawCalloc(1, 1)`` had been called instead. + + .. versionadded:: 3.5 + + .. c:function:: void* PyMem_RawRealloc(void *p, size_t n) Resizes the memory block pointed to by *p* to *n* bytes. The contents will @@ -136,8 +147,8 @@ behavior when requesting zero bytes, are available for allocating and releasing memory from the Python heap. The default memory block allocator uses the following functions: -:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when -requesting zero bytes. +:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call +``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes. .. warning:: @@ -152,6 +163,17 @@ requesting zero bytes. been called instead. The memory will not have been initialized in any way. +.. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize) + + Allocates *nelem* elements each whose size in bytes is *elsize* and returns + a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the + request fails. The memory is initialized to zeros. Requesting zero elements + or elements of size zero bytes returns a distinct non-*NULL* pointer if + possible, as if ``PyMem_Calloc(1, 1)`` had been called instead. + + .. versionadded:: 3.5 + + .. c:function:: void* PyMem_Realloc(void *p, size_t n) Resizes the memory block pointed to by *p* to *n* bytes. The contents will be @@ -210,7 +232,7 @@ Customize Memory Allocators .. versionadded:: 3.4 -.. c:type:: PyMemAllocator +.. c:type:: PyMemAllocatorEx Structure used to describe a memory block allocator. The structure has four fields: @@ -222,11 +244,19 @@ Customize Memory Allocators +----------------------------------------------------------+---------------------------------------+ | ``void* malloc(void *ctx, size_t size)`` | allocate a memory block | +----------------------------------------------------------+---------------------------------------+ + | ``void* calloc(void *ctx, size_t nelem, size_t elsize)`` | allocate a memory block initialized | + | | with zeros | + +----------------------------------------------------------+---------------------------------------+ | ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block | +----------------------------------------------------------+---------------------------------------+ | ``void free(void *ctx, void *ptr)`` | free a memory block | +----------------------------------------------------------+---------------------------------------+ + .. versionchanged:: 3.5 + The :c:type:`PyMemAllocator` structure was renamed to + :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. + + .. c:type:: PyMemAllocatorDomain Enum used to identify an allocator domain. Domains: @@ -239,12 +269,12 @@ Customize Memory Allocators :c:func:`PyObject_Realloc` and :c:func:`PyObject_Free` -.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator) +.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) Get the memory block allocator of the specified domain. -.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator) +.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) Set the memory block allocator of the specified domain. diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst index 21951c38c0..9bcb649c9d 100644 --- a/Doc/c-api/number.rst +++ b/Doc/c-api/number.rst @@ -30,6 +30,14 @@ Number Protocol the equivalent of the Python expression ``o1 * o2``. +.. c:function:: PyObject* PyNumber_MatrixMultiply(PyObject *o1, PyObject *o2) + + Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on + failure. This is the equivalent of the Python expression ``o1 @ o2``. + + .. versionadded:: 3.5 + + .. c:function:: PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2) Return the floor of *o1* divided by *o2*, or *NULL* on failure. This is @@ -146,6 +154,15 @@ Number Protocol the Python statement ``o1 *= o2``. +.. c:function:: PyObject* PyNumber_InPlaceMatrixMultiply(PyObject *o1, PyObject *o2) + + Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on + failure. The operation is done *in-place* when *o1* supports it. This is + the equivalent of the Python statement ``o1 @= o2``. + + .. versionadded:: 3.5 + + .. c:function:: PyObject* PyNumber_InPlaceFloorDivide(PyObject *o1, PyObject *o2) Returns the mathematical floor of dividing *o1* by *o2*, or *NULL* on failure. diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index 9760dca2df..a6a939ccc2 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -47,6 +47,60 @@ Operating System Utilities not call those functions directly! :c:type:`PyOS_sighandler_t` is a typedef alias for :c:type:`void (\*)(int)`. +.. c:function:: wchar_t* Py_DecodeLocale(const char* arg, size_t *size) + + Decode a byte string from the locale encoding with the :ref:`surrogateescape + error handler <surrogateescape>`: undecodable bytes are decoded as + characters in range U+DC80..U+DCFF. If a byte sequence can be decoded as a + surrogate character, escape the bytes using the surrogateescape error + handler instead of decoding them. + + Return a pointer to a newly allocated wide character string, use + :c:func:`PyMem_RawFree` to free the memory. If size is not ``NULL``, write + the number of wide characters excluding the null character into ``*size`` + + Return ``NULL`` on decoding error or memory allocation error. If *size* is + not ``NULL``, ``*size`` is set to ``(size_t)-1`` on memory error or set to + ``(size_t)-2`` on decoding error. + + Decoding errors should never happen, unless there is a bug in the C + library. + + Use the :c:func:`Py_EncodeLocale` function to encode the character string + back to a byte string. + + .. seealso:: + + The :c:func:`PyUnicode_DecodeFSDefaultAndSize` and + :c:func:`PyUnicode_DecodeLocaleAndSize` functions. + + .. versionadded:: 3.5 + + +.. c:function:: char* Py_EncodeLocale(const wchar_t *text, size_t *error_pos) + + Encode a wide character string to the locale encoding with the + :ref:`surrogateescape error handler <surrogateescape>`: surrogate characters + in the range U+DC80..U+DCFF are converted to bytes 0x80..0xFF. + + Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free` + to free the memory. Return ``NULL`` on encoding error or memory allocation + error + + If error_pos is not ``NULL``, ``*error_pos`` is set to the index of the + invalid character on encoding error, or set to ``(size_t)-1`` otherwise. + + Use the :c:func:`Py_DecodeLocale` function to decode the bytes string back + to a wide character string. + + .. seealso:: + + The :c:func:`PyUnicode_EncodeFSDefault` and + :c:func:`PyUnicode_EncodeLocale` functions. + + .. versionadded:: 3.5 + + .. _systemfunctions: System Functions diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index b43622a810..eb637051c6 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -1118,6 +1118,9 @@ Number Object Structures binaryfunc nb_inplace_true_divide; unaryfunc nb_index; + + binaryfunc nb_matrix_multiply; + binaryfunc nb_inplace_matrix_multiply; } PyNumberMethods; .. note:: diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index 03c284ad39..00063d0b5e 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -759,11 +759,13 @@ system. *errors* is ``NULL``. *str* must end with a null character but cannot contain embedded null characters. + Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` to decode a string from + :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at + Python startup). + .. seealso:: - Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` to decode a string from - :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at - Python startup). + The :c:func:`Py_DecodeLocale` function. .. versionadded:: 3.3 @@ -784,11 +786,13 @@ system. *errors* is ``NULL``. Return a :class:`bytes` object. *str* cannot contain embedded null characters. + Use :c:func:`PyUnicode_EncodeFSDefault` to encode a string to + :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at + Python startup). + .. seealso:: - Use :c:func:`PyUnicode_EncodeFSDefault` to encode a string to - :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at - Python startup). + The :c:func:`Py_EncodeLocale` function. .. versionadded:: 3.3 @@ -833,12 +837,14 @@ used, passing :c:func:`PyUnicode_FSDecoder` as the conversion function: If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the locale encoding. + :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the + locale encoding and cannot be modified later. If you need to decode a string + from the current locale encoding, use + :c:func:`PyUnicode_DecodeLocaleAndSize`. + .. seealso:: - :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the - locale encoding and cannot be modified later. If you need to decode a - string from the current locale encoding, use - :c:func:`PyUnicode_DecodeLocaleAndSize`. + The :c:func:`Py_DecodeLocale` function. .. versionchanged:: 3.2 Use ``"strict"`` error handler on Windows. @@ -868,12 +874,13 @@ used, passing :c:func:`PyUnicode_FSDecoder` as the conversion function: If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the locale encoding. + :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the + locale encoding and cannot be modified later. If you need to encode a string + to the current locale encoding, use :c:func:`PyUnicode_EncodeLocale`. + .. seealso:: - :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the - locale encoding and cannot be modified later. If you need to encode a - string to the current locale encoding, use - :c:func:`PyUnicode_EncodeLocale`. + The :c:func:`Py_EncodeLocale` function. .. versionadded:: 3.2 diff --git a/Doc/conf.py b/Doc/conf.py index f803de238e..28dd80f859 100644 --- a/Doc/conf.py +++ b/Doc/conf.py @@ -36,6 +36,9 @@ highlight_language = 'python3' # Require Sphinx 1.2 for build. needs_sphinx = '1.2' +# Ignore any .rst files in the venv/ directory. +exclude_patterns = ['venv/*'] + # Options for HTML output # ----------------------- diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat index 6025617a1b..d1c24e5fca 100644 --- a/Doc/data/refcounts.dat +++ b/Doc/data/refcounts.dat @@ -349,6 +349,11 @@ PyErr_Format:PyObject*:exception:+1: PyErr_Format:const char*:format:: PyErr_Format::...:: +PyErr_FormatV:PyObject*::null: +PyErr_FormatV:PyObject*:exception:+1: +PyErr_FormatV:const char*:format:: +PyErr_FormatV:va_list:vargs:: + PyErr_WarnEx:int::: PyErr_WarnEx:PyObject*:category:0: PyErr_WarnEx:const char*:message:: diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst index 82bed24983..087c6715ce 100644 --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -1099,13 +1099,13 @@ other utility module. during the build of Python), not the OS version of the current system. For universal binary builds on Mac OS X the architecture value reflects - the univeral binary status instead of the architecture of the current + the universal binary status instead of the architecture of the current processor. For 32-bit universal binaries the architecture is ``fat``, for 64-bit universal binaries the architecture is ``fat64``, and for 4-way universal binaries the architecture is ``universal``. Starting from Python 2.7 and Python 3.2 the architecture ``fat3`` is used for a 3-way universal build (ppc, i386, x86_64) and ``intel`` is used for - a univeral build with the i386 and x86_64 architectures + a universal build with the i386 and x86_64 architectures Examples of returned values on Mac OS X: @@ -1193,12 +1193,12 @@ other utility module. .. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None]) - Byte-compile a collection of Python source files to either :file:`.pyc` or - :file:`.pyo` files in a :file:`__pycache__` subdirectory (see :pep:`3147`). + Byte-compile a collection of Python source files to :file:`.pyc` files in a + :file:`__pycache__` subdirectory (see :pep:`3147` and :pep:`488`). *py_files* is a list of files to compile; any files that don't end in :file:`.py` are silently skipped. *optimize* must be one of the following: - * ``0`` - don't optimize (generate :file:`.pyc`) + * ``0`` - don't optimize * ``1`` - normal optimization (like ``python -O``) * ``2`` - extra optimization (like ``python -OO``) @@ -1222,10 +1222,13 @@ other utility module. doing, leave it set to ``None``. .. versionchanged:: 3.2.3 - Create ``.pyc`` or ``.pyo`` files with an :func:`import magic tag + Create ``.pyc`` files with an :func:`import magic tag <imp.get_tag>` in their name, in a :file:`__pycache__` subdirectory instead of files without tag in the current directory. + .. versionchanged: 3.5 + Create ``.pyc`` files according to :pep:`488`. + .. function:: rfc822_escape(header) diff --git a/Doc/distutils/builtdist.rst b/Doc/distutils/builtdist.rst index ac96c40300..c5827b63cf 100644 --- a/Doc/distutils/builtdist.rst +++ b/Doc/distutils/builtdist.rst @@ -355,7 +355,7 @@ support this option, so the command:: would create a 64bit installation executable on your 32bit version of Windows. To cross-compile, you must download the Python source code and cross-compile -Python itself for the platform you are targetting - it is not possible from a +Python itself for the platform you are targeting - it is not possible from a binary installation of Python (as the .lib etc file for other platforms are not included.) In practice, this means the user of a 32 bit operating system will need to use Visual Studio 2008 to open the diff --git a/Doc/distutils/introduction.rst b/Doc/distutils/introduction.rst index 0ece646c34..8f46bd74c5 100644 --- a/Doc/distutils/introduction.rst +++ b/Doc/distutils/introduction.rst @@ -156,8 +156,8 @@ module pure Python module a module written in Python and contained in a single :file:`.py` file (and - possibly associated :file:`.pyc` and/or :file:`.pyo` files). Sometimes referred - to as a "pure module." + possibly associated :file:`.pyc` files). Sometimes referred to as a + "pure module." extension module a module written in the low-level language of the Python implementation: C/C++ @@ -210,5 +210,3 @@ distribution root the top-level directory of your source tree (or source distribution); the directory where :file:`setup.py` exists. Generally :file:`setup.py` will be run from this directory. - - diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst index 6cb686ab09..acd60aef8c 100644 --- a/Doc/extending/embedding.rst +++ b/Doc/extending/embedding.rst @@ -58,12 +58,18 @@ perform some operation on a file. :: int main(int argc, char *argv[]) { - Py_SetProgramName(argv[0]); /* optional but recommended */ - Py_Initialize(); - PyRun_SimpleString("from time import time,ctime\n" - "print('Today is', ctime(time()))\n"); - Py_Finalize(); - return 0; + wchar_t *program = Py_DecodeLocale(argv[0], NULL); + if (program == NULL) { + fprintf(stderr, "Fatal error: cannot decode argv[0]\n"); + exit(1); + } + Py_SetProgramName(program); /* optional but recommended */ + Py_Initialize(); + PyRun_SimpleString("from time import time,ctime\n" + "print('Today is', ctime(time()))\n"); + Py_Finalize(); + PyMem_RawFree(program); + return 0; } The :c:func:`Py_SetProgramName` function should be called before @@ -160,7 +166,7 @@ for data conversion between Python and C, and for error reporting. The interesting part with respect to embedding Python starts with :: Py_Initialize(); - pName = PyUnicode_FromString(argv[1]); + pName = PyUnicode_DecodeFSDefault(argv[1]); /* Error checking of pName left out */ pModule = PyImport_Import(pName); diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst index c10efa976d..17cfdeb965 100644 --- a/Doc/extending/extending.rst +++ b/Doc/extending/extending.rst @@ -375,11 +375,17 @@ optionally followed by an import of the module:: int main(int argc, char *argv[]) { + wchar_t *program = Py_DecodeLocale(argv[0], NULL); + if (program == NULL) { + fprintf(stderr, "Fatal error: cannot decode argv[0]\n"); + exit(1); + } + /* Add a built-in module, before Py_Initialize */ PyImport_AppendInittab("spam", PyInit_spam); /* Pass argv[0] to the Python interpreter */ - Py_SetProgramName(argv[0]); + Py_SetProgramName(program); /* Initialize the Python interpreter. Required. */ Py_Initialize(); @@ -391,6 +397,10 @@ optionally followed by an import of the module:: ... + PyMem_RawFree(program); + return 0; + } + .. note:: Removing entries from ``sys.modules`` or importing compiled modules into diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 1de86ef8c0..57c6ddd725 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -88,10 +88,17 @@ Glossary bytes-like object An object that supports the :ref:`bufferobjects`, like :class:`bytes`, :class:`bytearray` or :class:`memoryview`. Bytes-like objects can - be used for various operations that expect binary data, such as - compression, saving to a binary file or sending over a socket. - Some operations need the binary data to be mutable, in which case - not all bytes-like objects can apply. + be used for various operations that work with binary data; these include + compression, saving to a binary file, and sending over a socket. + + Some operations need the binary data to be mutable. The documentation + often refers to these as "read-write bytes-like objects". Example + mutable buffer objects include :class:`bytearray` and a + :class:`memoryview` of a :class:`bytearray`. + Other operations require the binary data to be stored in + immutable objects ("read-only bytes-like objects"); examples + of these include :class:`bytes` and a :class:`memoryview` + of a :class:`bytes` object. bytecode Python source code is compiled into bytecode, the internal representation @@ -402,6 +409,19 @@ Glossary than compiled ones, though their programs generally also run more slowly. See also :term:`interactive`. + interpreter shutdown + When asked to shut down, the Python interpreter enters a special phase + where it gradually releases all allocated resources, such as modules + and various critical internal structures. It also makes several calls + to the :term:`garbage collector <garbage collection>`. This can trigger + the execution of code in user-defined destructors or weakref callbacks. + Code executed during the shutdown phase can encounter various + exceptions as the resources it relies on may not function anymore + (common examples are library modules or the warnings machinery). + + The main reason for interpreter shutdown is that the ``__main__`` module + or the script being run has finished executing. + iterable An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as :class:`list`, :class:`str`, @@ -444,12 +464,13 @@ Glossary A number of tools in Python accept key functions to control how elements are ordered or grouped. They include :func:`min`, :func:`max`, - :func:`sorted`, :meth:`list.sort`, :func:`heapq.nsmallest`, - :func:`heapq.nlargest`, and :func:`itertools.groupby`. + :func:`sorted`, :meth:`list.sort`, :func:`heapq.merge`, + :func:`heapq.nsmallest`, :func:`heapq.nlargest`, and + :func:`itertools.groupby`. There are several ways to create a key function. For example. the :meth:`str.lower` method can serve as a key function for case insensitive - sorts. Alternatively, an ad-hoc key function can be built from a + sorts. Alternatively, a key function can be built from a :keyword:`lambda` expression such as ``lambda r: (r[0], r[2])``. Also, the :mod:`operator` module provides three key function constructors: :func:`~operator.attrgetter`, :func:`~operator.itemgetter`, and diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 750ddbe074..ca8e1cb87b 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -886,7 +886,7 @@ Argument Clinic generates code that does it for you (in the parsing function). Advanced converters ------------------- -Remeber those format units you skipped for your first +Remember those format units you skipped for your first time because they were advanced? Here's how to handle those too. The trick is, all those format units take arguments--either @@ -1020,12 +1020,12 @@ any of the default arguments you can omit the parentheses. the ``"as"`` should come before the return converter.) There's one additional complication when using return converters: how do you -indicate an error has occured? Normally, a function returns a valid (non-``NULL``) +indicate an error has occurred? Normally, a function returns a valid (non-``NULL``) pointer for success, and ``NULL`` for failure. But if you use an integer return converter, all integers are valid. How can Argument Clinic detect an error? Its solution: each return converter implicitly looks for a special value that indicates an error. If you return that value, and an error has been set (``PyErr_Occurred()`` returns a true -value), then the generated code will propogate the error. Otherwise it will +value), then the generated code will propagate the error. Otherwise it will encode the value you return like normal. Currently Argument Clinic supports only a few return converters:: @@ -1573,7 +1573,7 @@ The fourth new directive is ``set``:: ``line_prefix`` is a string that will be prepended to every line of Clinic's output; ``line_suffix`` is a string that will be appended to every line of Clinic's output. -Both of these suport two format strings: +Both of these support two format strings: ``{block comment start}`` Turns into the string ``/*``, the start-comment text sequence for C files. diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst index 114ec09b74..e31b6c2bf4 100644 --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -325,6 +325,15 @@ which, when run, will produce:: MainThread: Look out! +.. versionchanged:: 3.5 + Prior to Python 3.5, the :class:`QueueListener` always passed every message + received from the queue to every handler it was initialized with. (This was + because it was assumed that level filtering was all done on the other side, + where the queue is filled.) From 3.5 onwards, this behaviour can be changed + by passing a keyword argument ``respect_handler_level=True`` to the + listener's constructor. When this is done, the listener compares the level + of each message with the handler's level, and only passes a message to a + handler if it's appropriate to do so. .. _network-logging: @@ -1680,7 +1689,7 @@ as in the following complete example:: def main(): logging.basicConfig(level=logging.INFO, format='%(message)s') - logging.info(_('message 1', set_value=set([1, 2, 3]), snowman='\u2603')) + logging.info(_('message 1', set_value={1, 2, 3}, snowman='\u2603')) if __name__ == '__main__': main() diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst index 5e875cdee2..a2aaf367c7 100644 --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -207,13 +207,12 @@ that's ``str``/``bytes`` in Python 2 and ``bytes`` in Python 3). The following table lists the **unique** methods of each data type across Python 2 & 3 (e.g., the ``decode()`` method is usable on the equivalent binary data type in either Python 2 or 3, but it can't be used by the text data type consistently -between Python 2 and 3 because ``str`` in Python 3 doesn't have the method). +between Python 2 and 3 because ``str`` in Python 3 doesn't have the method). Do +note that as of Python 3.5 the ``__mod__`` method was added to the bytes type. ======================== ===================== **Text data** **Binary data** ------------------------ --------------------- -__mod__ (``%`` operator) ------------------------- --------------------- \ decode ------------------------ --------------------- encode @@ -348,10 +347,12 @@ tox with your continuous integration system so that you never accidentally break Python 2 or 3 support. You may also want to use use the ``-bb`` flag with the Python 3 interpreter to -trigger an exception when you are comparing bytes to strings. Usually it's -simply ``False``, but if you made a mistake in your separation of text/binary -data handling you may be accidentally comparing text and binary data. This flag -will raise an exception when that occurs to help track down such cases. +trigger an exception when you are comparing bytes to strings or bytes to an int +(the latter is available starting in Python 3.5). By default type-differing +comparisons simply return ``False``, but if you made a mistake in your +separation of text/binary data handling or indexing on bytes you wouldn't easily +find the mistake. This flag will raise an exception when these kinds of +comparisons occur, making the mistake much easier to track down. And that's mostly it! At this point your code base is compatible with both Python 2 and 3 simultaneously. Your testing will also be set up so that you diff --git a/Doc/howto/regex.rst b/Doc/howto/regex.rst index fbe763b3f2..ad2c6ab7d5 100644 --- a/Doc/howto/regex.rst +++ b/Doc/howto/regex.rst @@ -852,7 +852,7 @@ keep track of the group numbers. There are two features which help with this problem. Both of them use a common syntax for regular expression extensions, so we'll look at that first. -Perl 5 is well-known for its powerful additions to standard regular expressions. +Perl 5 is well known for its powerful additions to standard regular expressions. For these new features the Perl developers couldn't choose new single-keystroke metacharacters or new special sequences beginning with ``\`` without making Perl's regular expressions confusingly different from standard REs. If they chose ``&`` as a @@ -1138,7 +1138,7 @@ Empty matches are replaced only when they're not adjacent to a previous match. If *replacement* is a string, any backslash escapes in it are processed. That is, ``\n`` is converted to a single newline character, ``\r`` is converted to a -carriage return, and so forth. Unknown escapes such as ``\j`` are left alone. +carriage return, and so forth. Unknown escapes such as ``\&`` are left alone. Backreferences, such as ``\6``, are replaced with the substring matched by the corresponding group in the RE. This lets you incorporate portions of the original text in the resulting replacement string. diff --git a/Doc/howto/sockets.rst b/Doc/howto/sockets.rst index d5aff9081d..04394d49f0 100644 --- a/Doc/howto/sockets.rst +++ b/Doc/howto/sockets.rst @@ -234,7 +234,7 @@ messages to be sent back to back (without some kind of reply), and you pass following message. You'll need to put that aside and hold onto it, until it's needed. -Prefixing the message with it's length (say, as 5 numeric characters) gets more +Prefixing the message with its length (say, as 5 numeric characters) gets more complex, because (believe it or not), you may not get all 5 characters in one ``recv``. In playing around, you'll get away with it; but in high network loads, your code will very quickly break unless you use two ``recv`` loops - the first diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst index b49ac391ce..ee31a9c981 100644 --- a/Doc/howto/unicode.rst +++ b/Doc/howto/unicode.rst @@ -280,8 +280,9 @@ and optionally an *errors* argument. The *errors* argument specifies the response when the input string can't be converted according to the encoding's rules. Legal values for this argument are ``'strict'`` (raise a :exc:`UnicodeDecodeError` exception), ``'replace'`` (use -``U+FFFD``, ``REPLACEMENT CHARACTER``), or ``'ignore'`` (just leave the -character out of the Unicode result). +``U+FFFD``, ``REPLACEMENT CHARACTER``), ``'ignore'`` (just leave the +character out of the Unicode result), or ``'backslashreplace'`` (inserts a +``\xNN`` escape sequence). The following examples show the differences:: >>> b'\x80abc'.decode("utf-8", "strict") #doctest: +NORMALIZE_WHITESPACE @@ -291,6 +292,8 @@ The following examples show the differences:: invalid start byte >>> b'\x80abc'.decode("utf-8", "replace") '\ufffdabc' + >>> b'\x80abc'.decode("utf-8", "backslashreplace") + '\\x80abc' >>> b'\x80abc'.decode("utf-8", "ignore") 'abc' @@ -325,8 +328,9 @@ The *errors* parameter is the same as the parameter of the :meth:`~bytes.decode` method but supports a few more possible handlers. As well as ``'strict'``, ``'ignore'``, and ``'replace'`` (which in this case inserts a question mark instead of the unencodable character), there is -also ``'xmlcharrefreplace'`` (inserts an XML character reference) and -``backslashreplace`` (inserts a ``\uNNNN`` escape sequence). +also ``'xmlcharrefreplace'`` (inserts an XML character reference), +``backslashreplace`` (inserts a ``\uNNNN`` escape sequence) and +``namereplace`` (inserts a ``\N{...}`` escape sequence). The following example shows the different results:: @@ -346,6 +350,8 @@ The following example shows the different results:: b'ꀀabcd޴' >>> u.encode('ascii', 'backslashreplace') b'\\ua000abcd\\u07b4' + >>> u.encode('ascii', 'namereplace') + b'\\N{YI SYLLABLE IT}abcd\\u07b4' The low-level routines for registering and accessing the available encodings are found in the :mod:`codecs` module. Implementing new diff --git a/Doc/includes/noddy.c b/Doc/includes/noddy.c index 8f79fcf6bf..19a27a89e8 100644 --- a/Doc/includes/noddy.c +++ b/Doc/includes/noddy.c @@ -38,7 +38,7 @@ static PyModuleDef noddymodule = { }; PyMODINIT_FUNC -PyInit_noddy(void) +PyInit_noddy(void) { PyObject* m; diff --git a/Doc/includes/run-func.c b/Doc/includes/run-func.c index 1c9860d7a8..986d670319 100644 --- a/Doc/includes/run-func.c +++ b/Doc/includes/run-func.c @@ -13,7 +13,7 @@ main(int argc, char *argv[]) } Py_Initialize(); - pName = PyUnicode_FromString(argv[1]); + pName = PyUnicode_DecodeFSDefault(argv[1]); /* Error checking of pName left out */ pModule = PyImport_Import(pName); diff --git a/Doc/install/index.rst b/Doc/install/index.rst index 8f3ad7241a..876f350f90 100644 --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -361,7 +361,7 @@ And here are the values used on Windows: Type of file Installation directory =============== =========================================================== modules :file:`{userbase}\\Python{XY}\\site-packages` -scripts :file:`{userbase}\\Scripts` +scripts :file:`{userbase}\\Python{XY}\\Scripts` data :file:`{userbase}` C headers :file:`{userbase}\\Python{XY}\\Include\\{distname}` =============== =========================================================== diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 73f94808b6..24060f06bc 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -135,7 +135,7 @@ ArgumentParser objects formatter_class=argparse.HelpFormatter, \ prefix_chars='-', fromfile_prefix_chars=None, \ argument_default=None, conflict_handler='error', \ - add_help=True) + add_help=True, allow_abbrev=True) Create a new :class:`ArgumentParser` object. All parameters should be passed as keyword arguments. Each parameter has its own more detailed description @@ -169,6 +169,12 @@ ArgumentParser objects * add_help_ - Add a -h/--help option to the parser (default: ``True``) + * allow_abbrev_ - Allows long options to be abbreviated if the + abbreviation is unambiguous. (default: ``True``) + + .. versionchanged:: 3.5 + *allow_abbrev* parameter was added. + The following sections describe how each of these are used. @@ -518,6 +524,26 @@ calls, we supply ``argument_default=SUPPRESS``:: >>> parser.parse_args([]) Namespace() +.. _allow_abbrev: + +allow_abbrev +^^^^^^^^^^^^ + +Normally, when you pass an argument list to the +:meth:`~ArgumentParser.parse_args` method of a :class:`ArgumentParser`, +it :ref:`recognizes abbreviations <prefix-matching>` of long options. + +This feature can be disabled by setting ``allow_abbrev`` to ``False``:: + + >>> parser = argparse.ArgumentParser(prog='PROG', allow_abbrev=False) + >>> parser.add_argument('--foobar', action='store_true') + >>> parser.add_argument('--foonley', action='store_false') + >>> parser.parse_args(['--foon']) + usage: PROG [-h] [--foobar] [--foonley] + PROG: error: unrecognized arguments: --foon + +.. versionadded:: 3.5 + conflict_handler ^^^^^^^^^^^^^^^^ @@ -1410,9 +1436,9 @@ argument:: Argument abbreviations (prefix matching) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The :meth:`~ArgumentParser.parse_args` method allows long options to be -abbreviated to a prefix, if the abbreviation is unambiguous (the prefix matches -a unique option):: +The :meth:`~ArgumentParser.parse_args` method :ref:`by default <allow_abbrev>` +allows long options to be abbreviated to a prefix, if the abbreviation is +unambiguous (the prefix matches a unique option):: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-bacon') @@ -1426,6 +1452,7 @@ a unique option):: PROG: error: ambiguous option: -ba could match -badger, -bacon An error is produced for arguments that could produce more than one options. +This feature can be disabled by setting :ref:`allow_abbrev` to ``False``. Beyond ``sys.argv`` @@ -1910,7 +1937,7 @@ Customizing file parsing Arguments that are read from a file (see the *fromfile_prefix_chars* keyword argument to the :class:`ArgumentParser` constructor) are read one - argument per line. :meth:`convert_arg_line_to_args` can be overriden for + argument per line. :meth:`convert_arg_line_to_args` can be overridden for fancier reading. This method takes a single argument *arg_line* which is a string read from diff --git a/Doc/library/asynchat.rst b/Doc/library/asynchat.rst index c6fa06162e..794da8cced 100644 --- a/Doc/library/asynchat.rst +++ b/Doc/library/asynchat.rst @@ -147,40 +147,6 @@ connection requests. by the channel after :meth:`found_terminator` is called. -asynchat - Auxiliary Classes ------------------------------------------- - -.. class:: fifo(list=None) - - A :class:`fifo` holding data which has been pushed by the application but - not yet popped for writing to the channel. A :class:`fifo` is a list used - to hold data and/or producers until they are required. If the *list* - argument is provided then it should contain producers or data items to be - written to the channel. - - - .. method:: is_empty() - - Returns ``True`` if and only if the fifo is empty. - - - .. method:: first() - - Returns the least-recently :meth:`push`\ ed item from the fifo. - - - .. method:: push(data) - - Adds the given data (which may be a string or a producer object) to the - producer fifo. - - - .. method:: pop() - - If the fifo is not empty, returns ``True, first()``, deleting the popped - item. Returns ``False, None`` for an empty fifo. - - .. _asynchat-example: asynchat Example diff --git a/Doc/library/asyncio-eventloops.rst b/Doc/library/asyncio-eventloops.rst index afb8b9f35b..ae3bf90298 100644 --- a/Doc/library/asyncio-eventloops.rst +++ b/Doc/library/asyncio-eventloops.rst @@ -100,8 +100,6 @@ Common limits of Windows event loops: :class:`ProactorEventLoop` specific limits: -- SSL is not supported: :meth:`~BaseEventLoop.create_connection` and - :meth:`~BaseEventLoop.create_server` cannot be used with SSL for example - :meth:`~BaseEventLoop.create_datagram_endpoint` (UDP) is not supported - :meth:`~BaseEventLoop.add_reader` and :meth:`~BaseEventLoop.add_writer` are not supported @@ -112,6 +110,10 @@ The best resolution is 0.5 msec. The resolution depends on the hardware <http://fr.wikipedia.org/wiki/High_Precision_Event_Timer>`_) and on the Windows configuration. See :ref:`asyncio delayed calls <asyncio-delayed-calls>`. +.. versionchanged:: 3.5 + + :class:`ProactorEventLoop` now supports SSL. + Mac OS X ^^^^^^^^ diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index 33706726a3..f11c09ac29 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -8,7 +8,6 @@ Queues: * :class:`Queue` * :class:`PriorityQueue` * :class:`LifoQueue` -* :class:`JoinableQueue` asyncio queue API was designed to be close to classes of the :mod:`queue` module (:class:`~queue.Queue`, :class:`~queue.PriorityQueue`, @@ -144,16 +143,6 @@ LifoQueue first. -JoinableQueue -^^^^^^^^^^^^^ - -.. class:: JoinableQueue - - Deprecated alias for :class:`Queue`. - - .. deprecated:: 3.4.4 - - Exceptions ^^^^^^^^^^ diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index 488cda5968..1b8d9cffc6 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -120,6 +120,10 @@ All of the classes in this module may safely be accessed from multiple threads. .. versionchanged:: 3.4 The ``'x'`` (exclusive creation) mode was added. + .. versionchanged:: 3.5 + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + Incremental (de)compression --------------------------- @@ -162,15 +166,32 @@ Incremental (de)compression you need to decompress a multi-stream input with :class:`BZ2Decompressor`, you must use a new decompressor for each stream. - .. method:: decompress(data) + .. method:: decompress(data, max_length=-1) + + Decompress *data* (a :term:`bytes-like object`), returning + uncompressed data as bytes. Some of *data* may be buffered + internally, for use in later calls to :meth:`decompress`. The + returned data should be concatenated with the output of any + previous calls to :meth:`decompress`. + + If *max_length* is nonnegative, returns at most *max_length* + bytes of decompressed data. If this limit is reached and further + output can be produced, the :attr:`~.needs_input` attribute will + be set to ``False``. In this case, the next call to + :meth:`~.decompress` may provide *data* as ``b''`` to obtain + more of the output. - Provide data to the decompressor object. Returns a chunk of decompressed - data if possible, or an empty byte string otherwise. + If all of the input data was decompressed and returned (either + because this was less than *max_length* bytes, or because + *max_length* was negative), the :attr:`~.needs_input` attribute + will be set to ``True``. - Attempting to decompress data after the end of the current stream is - reached raises an :exc:`EOFError`. If any data is found after the end of - the stream, it is ignored and saved in the :attr:`unused_data` attribute. + Attempting to decompress data after the end of stream is reached + raises an `EOFError`. Any data found after the end of the + stream is ignored and saved in the :attr:`~.unused_data` attribute. + .. versionchanged:: 3.5 + Added the *max_length* parameter. .. attribute:: eof @@ -186,6 +207,13 @@ Incremental (de)compression If this attribute is accessed before the end of the stream has been reached, its value will be ``b''``. + .. attribute:: needs_input + + ``False`` if the :meth:`.decompress` method can provide more + decompressed data before requiring new uncompressed input. + + .. versionadded:: 3.5 + One-shot (de)compression ------------------------ diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst index fa131453c7..0a9e884280 100644 --- a/Doc/library/cgi.rst +++ b/Doc/library/cgi.rst @@ -157,6 +157,9 @@ return bytes):: if not line: break linecount = linecount + 1 +:class:`FieldStorage` objects also support being used in a :keyword:`with` +statement, which will automatically close them when done. + If an error is encountered when obtaining the contents of an uploaded file (for example, when the user interrupts the form submission by clicking on a Back or Cancel button) the :attr:`~FieldStorage.done` attribute of the @@ -182,6 +185,10 @@ A form submitted via POST that also has a query string will contain both The :attr:`~FieldStorage.file` attribute is automatically closed upon the garbage collection of the creating :class:`FieldStorage` instance. +.. versionchanged:: 3.5 + Added support for the context management protocol to the + :class:`FieldStorage` class. + Higher Level Interface ---------------------- diff --git a/Doc/library/cmd.rst b/Doc/library/cmd.rst index 80d2a5d2c2..1ab2d7423f 100644 --- a/Doc/library/cmd.rst +++ b/Doc/library/cmd.rst @@ -259,7 +259,7 @@ immediate playback:: 'Move turtle to an absolute position with changing orientation. GOTO 100 200' goto(*parse(arg)) def do_home(self, arg): - 'Return turtle to the home postion: HOME' + 'Return turtle to the home position: HOME' home() def do_circle(self, arg): 'Draw circle with given radius an options extent and steps: CIRCLE 50' diff --git a/Doc/library/code.rst b/Doc/library/code.rst index 5b5d7cc8c1..275201c69b 100644 --- a/Doc/library/code.rst +++ b/Doc/library/code.rst @@ -4,6 +4,7 @@ .. module:: code :synopsis: Facilities to implement read-eval-print loops. +**Source code:** :source:`Lib/code.py` The ``code`` module provides facilities to implement read-eval-print loops in Python. Two classes and convenience functions are included which can be used to @@ -113,6 +114,9 @@ Interactive Interpreter Objects because it is within the interpreter object implementation. The output is written by the :meth:`write` method. + .. versionchanged:: 3.5 The full chained traceback is displayed instead + of just the primary traceback. + .. method:: InteractiveInterpreter.write(data) @@ -165,4 +169,3 @@ interpreter objects as well as the following additions. newline. When the user enters the EOF key sequence, :exc:`EOFError` is raised. The base implementation reads from ``sys.stdin``; a subclass may replace this with a different implementation. - diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 19d71921d2..0430cb92a4 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -7,6 +7,7 @@ .. sectionauthor:: Marc-André Lemburg <mal@lemburg.com> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/codecs.py` .. index:: single: Unicode @@ -29,10 +30,9 @@ module features are restricted to use specifically with The module defines the following functions for encoding and decoding with any codec: -.. function:: encode(obj, [encoding[, errors]]) +.. function:: encode(obj, encoding='utf-8', errors='strict') - Encodes *obj* using the codec registered for *encoding*. The default - encoding is ``utf-8``. + Encodes *obj* using the codec registered for *encoding*. *Errors* may be given to set the desired error handling scheme. The default error handler is ``'strict'`` meaning that encoding errors raise @@ -40,10 +40,9 @@ any codec: :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more information on codec error handling. -.. function:: decode(obj, [encoding[, errors]]) +.. function:: decode(obj, encoding='utf-8', errors='strict') - Decodes *obj* using the codec registered for *encoding*. The default - encoding is ``utf-8``. + Decodes *obj* using the codec registered for *encoding*. *Errors* may be given to set the desired error handling scheme. The default error handler is ``'strict'`` meaning that decoding errors raise @@ -104,7 +103,6 @@ The full details for each codec can also be looked up directly: To simplify access to the various codec components, the module provides these additional functions which use :func:`lookup` for the codec lookup: - .. function:: getencoder(encoding) Look up the codec for the given encoding and return its encoder function. @@ -274,6 +272,7 @@ implement the file protocols. Codec authors also need to define how the codec will handle encoding and decoding errors. +.. _surrogateescape: .. _error-handlers: Error Handlers @@ -315,10 +314,14 @@ The following error handlers are only applicable to | | reference (only for encoding). Implemented | | | in :func:`xmlcharrefreplace_errors`. | +-------------------------+-----------------------------------------------+ -| ``'backslashreplace'`` | Replace with backslashed escape sequences | -| | (only for encoding). Implemented in | +| ``'backslashreplace'`` | Replace with backslashed escape sequences. | +| | Implemented in | | | :func:`backslashreplace_errors`. | +-------------------------+-----------------------------------------------+ +| ``'namereplace'`` | Replace with ``\N{...}`` escape sequences | +| | (only for encoding). Implemented in | +| | :func:`namereplace_errors`. | ++-------------------------+-----------------------------------------------+ | ``'surrogateescape'`` | On decoding, replace byte with individual | | | surrogate code ranging from ``U+DC80`` to | | | ``U+DCFF``. This code will then be turned | @@ -344,6 +347,13 @@ In addition, the following error handler is specific to the given codecs: .. versionchanged:: 3.4 The ``'surrogatepass'`` error handlers now works with utf-16\* and utf-32\* codecs. +.. versionadded:: 3.5 + The ``'namereplace'`` error handler. + +.. versionchanged:: 3.5 + The ``'backslashreplace'`` error handlers now works with decoding and + translating. + The set of allowed values can be extended by registering a new named error handler: @@ -411,9 +421,17 @@ functions: .. function:: backslashreplace_errors(exception) - Implements the ``'backslashreplace'`` error handling (for encoding with + Implements the ``'backslashreplace'`` error handling (for + :term:`text encodings <text encoding>` only): malformed data is + replaced by a backslashed escape sequence. + +.. function:: namereplace_errors(exception) + + Implements the ``'namereplace'`` error handling (for encoding with :term:`text encodings <text encoding>` only): the - unencodable character is replaced by a backslashed escape sequence. + unencodable character is replaced by a ``\N{...}`` escape sequence. + + .. versionadded:: 3.5 .. _codec-objects: @@ -1444,4 +1462,3 @@ This module implements a variant of the UTF-8 codec: On encoding a UTF-8 encoded BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this is only done once (on the first write to the byte stream). For decoding an optional UTF-8 encoded BOM at the start of the data will be skipped. - diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index 599e9fa03a..99c43113e3 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -179,7 +179,7 @@ Notes on using :class:`Set` and :class:`MutableSet` as a mixin: (3) The :class:`Set` mixin provides a :meth:`_hash` method to compute a hash value for the set; however, :meth:`__hash__` is not defined because not all sets - are hashable or immutable. To add set hashabilty using mixins, + are hashable or immutable. To add set hashability using mixins, inherit from both :meth:`Set` and :meth:`Hashable`, then define ``__hash__ = Set._hash``. diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 0acde1292a..dd6395d7b9 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -437,6 +437,13 @@ or subtracting from an empty counter. Remove all elements from the deque leaving it with length 0. + .. method:: copy() + + Create a shallow copy of the deque. + + .. versionadded:: 3.5 + + .. method:: count(x) Count the number of deque elements equal to *x*. @@ -457,6 +464,21 @@ or subtracting from an empty counter. elements in the iterable argument. + .. method:: index(x[, start[, end]]) + + Return the position of *x* in the deque. Returns the first match + or raises :exc:`ValueError` if not found. + + .. versionadded:: 3.5 + + + .. method:: insert(i, x) + + Insert *x* into the deque at position *i*. + + .. versionadded:: 3.5 + + .. method:: pop() Remove and return an element from the right side of the deque. If no @@ -504,6 +526,9 @@ the :keyword:`in` operator, and subscript references such as ``d[-1]``. Indexed access is O(1) at both ends but slows to O(n) in the middle. For fast random access, use lists instead. +Starting in version 3.5, deques support ``__add__()``, ``__mul__()``, +and ``__imul__()``. + Example: .. doctest:: @@ -982,12 +1007,15 @@ The :class:`OrderedDict` constructor and :meth:`update` method both accept keyword arguments, but their order is lost because Python's function call semantics pass-in keyword arguments using a regular unordered dictionary. +.. versionchanged:: 3.5 + The items, keys, and values :term:`views <view>` of :class:`OrderedDict` now + support reverse iteration using :func:`reversed`. :class:`OrderedDict` Examples and Recipes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Since an ordered dictionary remembers its insertion order, it can be used -in conjuction with sorting to make a sorted dictionary:: +in conjunction with sorting to make a sorted dictionary:: >>> # regular unsorted dictionary >>> d = {'banana': 3, 'apple':4, 'pear': 1, 'orange': 2} diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 9712de2fef..57f4804979 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -42,7 +42,8 @@ compile Python sources. .. cmdoption:: -q - Do not print the list of files compiled, print only error messages. + Do not print the list of files compiled. If passed once, error messages will + still be printed. If passed twice (``-qq``), all output is suppressed. .. cmdoption:: -d destdir @@ -70,9 +71,33 @@ compile Python sources. is to write files to their :pep:`3147` locations and names, which allows byte-code files from multiple versions of Python to coexist. +.. cmdoption:: -r + + Control the maximum recursion level for subdirectories. + If this is given, then ``-l`` option will not be taken into account. + :program:`python -m compileall <directory> -r 0` is equivalent to + :program:`python -m compileall <directory> -l`. + +.. cmdoption:: -j N + + Use *N* workers to compile the files within the given directory. + If ``0`` is used, then the result of :func:`os.cpu_count()` + will be used. + .. versionchanged:: 3.2 Added the ``-i``, ``-b`` and ``-h`` options. +.. versionchanged:: 3.5 + Added the ``-j`` and ``-r`` options. + +.. versionchanged:: 3.5 + ``-q`` option was changed to a multilevel value. + +.. versionchanged:: 3.5 + ``-b`` will always produce a byte-code file ending in ``.pyc``, never + ``.pyo``. + + There is no command-line option to control the optimization level used by the :func:`compile` function, because the Python interpreter itself already provides the option: :program:`python -O -m compileall`. @@ -80,7 +105,7 @@ provides the option: :program:`python -O -m compileall`. Public functions ---------------- -.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) +.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1) Recursively descend the directory tree named by *dir*, compiling all :file:`.py` files along the way. @@ -101,8 +126,9 @@ Public functions file considered for compilation, and if it returns a true value, the file is skipped. - If *quiet* is true, nothing is printed to the standard output unless errors - occur. + If *quiet* is ``False`` or ``0`` (the default), the filenames and other + information are printed to standard out. Set to ``1``, only errors are + printed. Set to ``2``, all output is suppressed. If *legacy* is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of @@ -113,11 +139,26 @@ Public functions *optimize* specifies the optimization level for the compiler. It is passed to the built-in :func:`compile` function. + The argument *workers* specifies how many workers are used to + compile files in parallel. The default is to not use multiple workers. + If the platform can't use multiple workers and *workers* argument is given, + then a :exc:`NotImplementedError` will be raised. + If *workers* is lower than ``0``, a :exc:`ValueError` will be raised. + .. versionchanged:: 3.2 Added the *legacy* and *optimize* parameter. + .. versionchanged:: 3.5 + Added the *workers* parameter. + + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. -.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. + +.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1) Compile the file with path *fullname*. @@ -131,8 +172,9 @@ Public functions file being compiled, and if it returns a true value, the file is not compiled and ``True`` is returned. - If *quiet* is true, nothing is printed to the standard output unless errors - occur. + If *quiet* is ``False`` or ``0`` (the default), the filenames and other + information are printed to standard out. Set to ``1``, only errors are + printed. Set to ``2``, all output is suppressed. If *legacy* is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of @@ -145,8 +187,14 @@ Public functions .. versionadded:: 3.2 + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. + + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. -.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, legacy=False, optimize=-1) +.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1) Byte-compile all the :file:`.py` files found along ``sys.path``. If *skip_curdir* is true (the default), the current directory is not included @@ -157,6 +205,12 @@ Public functions .. versionchanged:: 3.2 Added the *legacy* and *optimize* parameter. + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. + + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. To force a recompile of all the :file:`.py` files in the :file:`Lib/` subdirectory and all its subdirectories:: diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index 48b43628b5..11b3916883 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -38,7 +38,7 @@ Executor Objects future = executor.submit(pow, 323, 1235) print(future.result()) - .. method:: map(func, *iterables, timeout=None) + .. method:: map(func, *iterables, timeout=None, chunksize=1) Equivalent to :func:`map(func, *iterables) <map>` except *func* is executed asynchronously and several calls to *func* may be made concurrently. The @@ -48,7 +48,16 @@ Executor Objects *timeout* can be an int or a float. If *timeout* is not specified or ``None``, there is no limit to the wait time. If a call raises an exception, then that exception will be raised when its value is - retrieved from the iterator. + retrieved from the iterator. When using :class:`ProcessPoolExecutor`, this + method chops *iterables* into a number of chunks which it submits to the + pool as separate tasks. The (approximate) size of these chunks can be + specified by setting *chunksize* to a positive integer. For very long + iterables, using a large value for *chunksize* can significantly improve + performance compared to the default size of 1. With :class:`ThreadPoolExecutor`, + *chunksize* has no effect. + + .. versionchanged:: 3.5 + Added the *chunksize* argument. .. method:: shutdown(wait=True) @@ -115,11 +124,19 @@ And:: executor.submit(wait_on_future) -.. class:: ThreadPoolExecutor(max_workers) +.. class:: ThreadPoolExecutor(max_workers=None) An :class:`Executor` subclass that uses a pool of at most *max_workers* threads to execute calls asynchronously. + .. versionchanged:: 3.5 + If *max_workers* is ``None`` or + not given, it will default to the number of processors on the machine, + multiplied by ``5``, assuming that :class:`ThreadPoolExecutor` is often + used to overlap I/O instead of CPU work and the number of workers + should be higher than the number of workers + for :class:`ProcessPoolExecutor`. + .. _threadpoolexecutor-example: @@ -175,6 +192,8 @@ to a :class:`ProcessPoolExecutor` will result in deadlock. An :class:`Executor` subclass that executes calls asynchronously using a pool of at most *max_workers* processes. If *max_workers* is ``None`` or not given, it will default to the number of processors on the machine. + If *max_workers* is lower or equal to ``0``, then a :exc:`ValueError` + will be raised. .. versionchanged:: 3.3 When one of the worker processes terminates abruptly, a diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst index 024d27cb84..c9187a3441 100644 --- a/Doc/library/configparser.rst +++ b/Doc/library/configparser.rst @@ -11,6 +11,8 @@ .. sectionauthor:: Christopher G. Petrilli <petrilli@amber.org> .. sectionauthor:: Łukasz Langa <lukasz@langa.pl> +**Source code:** :source:`Lib/configparser.py` + .. index:: pair: .ini; file pair: configuration; file @@ -142,12 +144,13 @@ datatypes, you should convert on your own: >>> float(topsecret['CompressionLevel']) 9.0 -Extracting Boolean values is not that simple, though. Passing the value -to ``bool()`` would do no good since ``bool('False')`` is still -``True``. This is why config parsers also provide :meth:`getboolean`. -This method is case-insensitive and recognizes Boolean values from -``'yes'``/``'no'``, ``'on'``/``'off'`` and ``'1'``/``'0'`` [1]_. -For example: +Since this task is so common, config parsers provide a range of handy getter +methods to handle integers, floats and booleans. The last one is the most +interesting because simply passing the value to ``bool()`` would do no good +since ``bool('False')`` is still ``True``. This is why config parsers also +provide :meth:`getboolean`. This method is case-insensitive and recognizes +Boolean values from ``'yes'``/``'no'``, ``'on'``/``'off'``, +``'true'``/``'false'`` and ``'1'``/``'0'`` [1]_. For example: .. doctest:: @@ -159,10 +162,8 @@ For example: True Apart from :meth:`getboolean`, config parsers also provide equivalent -:meth:`getint` and :meth:`getfloat` methods, but these are far less -useful since conversion using :func:`int` and :func:`float` is -sufficient for these types. - +:meth:`getint` and :meth:`getfloat` methods. You can register your own +converters and customize the provided ones. [1]_ Fallback Values --------------- @@ -317,11 +318,11 @@ from ``get()`` calls. .. class:: ExtendedInterpolation() An alternative handler for interpolation which implements a more advanced - syntax, used for instance in ``zc.buildout``. Extended interpolation is + syntax, used for instance in ``zc.buildout``. Extended interpolation is using ``${section:option}`` to denote a value from a foreign section. - Interpolation can span multiple levels. For convenience, if the ``section:`` - part is omitted, interpolation defaults to the current section (and possibly - the default values from the special section). + Interpolation can span multiple levels. For convenience, if the + ``section:`` part is omitted, interpolation defaults to the current section + (and possibly the default values from the special section). For example, the configuration specified above with basic interpolation, would look like this with extended interpolation: @@ -386,7 +387,7 @@ However, there are a few differences that should be taken into account: * All sections include ``DEFAULTSECT`` values as well which means that ``.clear()`` on a section may not leave the section visibly empty. This is because default values cannot be deleted from the section (because technically - they are not there). If they are overriden in the section, deleting causes + they are not there). If they are overridden in the section, deleting causes the default value to be visible again. Trying to delete a default value causes a ``KeyError``. @@ -399,13 +400,13 @@ However, there are a few differences that should be taken into account: * ``parser.popitem()`` never returns it. * ``parser.get(section, option, **kwargs)`` - the second argument is **not** - a fallback value. Note however that the section-level ``get()`` methods are + a fallback value. Note however that the section-level ``get()`` methods are compatible both with the mapping protocol and the classic configparser API. * ``parser.items()`` is compatible with the mapping protocol (returns a list of *section_name*, *section_proxy* pairs including the DEFAULTSECT). However, this method can also be invoked with arguments: ``parser.items(section, raw, - vars)``. The latter call returns a list of *option*, *value* pairs for + vars)``. The latter call returns a list of *option*, *value* pairs for a specified ``section``, with all interpolations expanded (unless ``raw=True`` is provided). @@ -539,9 +540,9 @@ the :meth:`__init__` options: * *delimiters*, default value: ``('=', ':')`` - Delimiters are substrings that delimit keys from values within a section. The - first occurrence of a delimiting substring on a line is considered a delimiter. - This means values (but not keys) can contain the delimiters. + Delimiters are substrings that delimit keys from values within a section. + The first occurrence of a delimiting substring on a line is considered + a delimiter. This means values (but not keys) can contain the delimiters. See also the *space_around_delimiters* argument to :meth:`ConfigParser.write`. @@ -553,7 +554,7 @@ the :meth:`__init__` options: Comment prefixes are strings that indicate the start of a valid comment within a config file. *comment_prefixes* are used only on otherwise empty lines (optionally indented) whereas *inline_comment_prefixes* can be used after - every valid value (e.g. section names, options and empty lines as well). By + every valid value (e.g. section names, options and empty lines as well). By default inline comments are disabled and ``'#'`` and ``';'`` are used as prefixes for whole line comments. @@ -563,10 +564,10 @@ the :meth:`__init__` options: Please note that config parsers don't support escaping of comment prefixes so using *inline_comment_prefixes* may prevent users from specifying option - values with characters used as comment prefixes. When in doubt, avoid setting - *inline_comment_prefixes*. In any circumstances, the only way of storing - comment prefix characters at the beginning of a line in multiline values is to - interpolate the prefix, for example:: + values with characters used as comment prefixes. When in doubt, avoid + setting *inline_comment_prefixes*. In any circumstances, the only way of + storing comment prefix characters at the beginning of a line in multiline + values is to interpolate the prefix, for example:: >>> from configparser import ConfigParser, ExtendedInterpolation >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) @@ -611,7 +612,7 @@ the :meth:`__init__` options: When set to ``True``, the parser will not allow for any section or option duplicates while reading from a single source (using :meth:`read_file`, - :meth:`read_string` or :meth:`read_dict`). It is recommended to use strict + :meth:`read_string` or :meth:`read_dict`). It is recommended to use strict parsers in new applications. .. versionchanged:: 3.2 @@ -646,12 +647,12 @@ the :meth:`__init__` options: The convention of allowing a special section of default values for other sections or interpolation purposes is a powerful concept of this library, - letting users create complex declarative configurations. This section is + letting users create complex declarative configurations. This section is normally called ``"DEFAULT"`` but this can be customized to point to any - other valid section name. Some typical values include: ``"general"`` or - ``"common"``. The name provided is used for recognizing default sections when - reading from any source and is used when writing configuration back to - a file. Its current value can be retrieved using the + other valid section name. Some typical values include: ``"general"`` or + ``"common"``. The name provided is used for recognizing default sections + when reading from any source and is used when writing configuration back to + a file. Its current value can be retrieved using the ``parser_instance.default_section`` attribute and may be modified at runtime (i.e. to convert files from one format to another). @@ -660,14 +661,30 @@ the :meth:`__init__` options: Interpolation behaviour may be customized by providing a custom handler through the *interpolation* argument. ``None`` can be used to turn off interpolation completely, ``ExtendedInterpolation()`` provides a more - advanced variant inspired by ``zc.buildout``. More on the subject in the + advanced variant inspired by ``zc.buildout``. More on the subject in the `dedicated documentation section <#interpolation-of-values>`_. :class:`RawConfigParser` has a default value of ``None``. +* *converters*, default value: not set + + Config parsers provide option value getters that perform type conversion. By + default :meth:`getint`, :meth:`getfloat`, and :meth:`getboolean` are + implemented. Should other getters be desirable, users may define them in + a subclass or pass a dictionary where each key is a name of the converter and + each value is a callable implementing said conversion. For instance, passing + ``{'decimal': decimal.Decimal}`` would add :meth:`getdecimal` on both the + parser object and all section proxies. In other words, it will be possible + to write both ``parser_instance.getdecimal('section', 'key', fallback=0)`` + and ``parser_instance['section'].getdecimal('key', 0)``. + + If the converter needs to access the state of the parser, it can be + implemented as a method on a config parser subclass. If the name of this + method starts with ``get``, it will be available on all section proxies, in + the dict-compatible form (see the ``getdecimal()`` example above). More advanced customization may be achieved by overriding default values of -these parser attributes. The defaults are defined on the classes, so they -may be overriden by subclasses or by attribute assignment. +these parser attributes. The defaults are defined on the classes, so they may +be overridden by subclasses or by attribute assignment. .. attribute:: BOOLEAN_STATES @@ -725,10 +742,11 @@ may be overriden by subclasses or by attribute assignment. .. attribute:: SECTCRE - A compiled regular expression used to parse section headers. The default - matches ``[section]`` to the name ``"section"``. Whitespace is considered part - of the section name, thus ``[ larch ]`` will be read as a section of name - ``" larch "``. Override this attribute if that's unsuitable. For example: + A compiled regular expression used to parse section headers. The default + matches ``[section]`` to the name ``"section"``. Whitespace is considered + part of the section name, thus ``[ larch ]`` will be read as a section of + name ``" larch "``. Override this attribute if that's unsuitable. For + example: .. doctest:: @@ -859,7 +877,7 @@ interpolation if an option used is not defined elsewhere. :: ConfigParser Objects -------------------- -.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation()) +.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation(), converters={}) The main configuration parser. When *defaults* is given, it is initialized into the dictionary of intrinsic defaults. When *dict_type* is given, it @@ -869,8 +887,8 @@ ConfigParser Objects When *delimiters* is given, it is used as the set of substrings that divide keys from values. When *comment_prefixes* is given, it will be used as the set of substrings that prefix comments in otherwise empty lines. - Comments can be indented. When *inline_comment_prefixes* is given, it will be - used as the set of substrings that prefix comments in non-empty lines. + Comments can be indented. When *inline_comment_prefixes* is given, it will + be used as the set of substrings that prefix comments in non-empty lines. When *strict* is ``True`` (the default), the parser won't allow for any section or option duplicates while reading from a single source (file, @@ -884,13 +902,13 @@ ConfigParser Objects When *default_section* is given, it specifies the name for the special section holding default values for other sections and interpolation purposes - (normally named ``"DEFAULT"``). This value can be retrieved and changed on + (normally named ``"DEFAULT"``). This value can be retrieved and changed on runtime using the ``default_section`` instance attribute. Interpolation behaviour may be customized by providing a custom handler through the *interpolation* argument. ``None`` can be used to turn off interpolation completely, ``ExtendedInterpolation()`` provides a more - advanced variant inspired by ``zc.buildout``. More on the subject in the + advanced variant inspired by ``zc.buildout``. More on the subject in the `dedicated documentation section <#interpolation-of-values>`_. All option names used in interpolation will be passed through the @@ -899,6 +917,12 @@ ConfigParser Objects converts option names to lower case), the values ``foo %(bar)s`` and ``foo %(BAR)s`` are equivalent. + When *converters* is given, it should be a dictionary where each key + represents the name of a type converter and each value is a callable + implementing the conversion from string to the desired datatype. Every + converter gets its own corresponding :meth:`get*()` method on the parser + object and section proxies. + .. versionchanged:: 3.1 The default *dict_type* is :class:`collections.OrderedDict`. @@ -907,6 +931,9 @@ ConfigParser Objects *empty_lines_in_values*, *default_section* and *interpolation* were added. + .. versionchanged:: 3.5 + The *converters* argument was added. + .. method:: defaults() @@ -944,7 +971,7 @@ ConfigParser Objects .. method:: has_option(section, option) If the given *section* exists, and contains the given *option*, return - :const:`True`; otherwise return :const:`False`. If the specified + :const:`True`; otherwise return :const:`False`. If the specified *section* is :const:`None` or an empty string, DEFAULT is assumed. @@ -1069,7 +1096,7 @@ ConfigParser Objects :meth:`get` method. .. versionchanged:: 3.2 - Items present in *vars* no longer appear in the result. The previous + Items present in *vars* no longer appear in the result. The previous behaviour mixed actual parser options with variables provided for interpolation. @@ -1170,7 +1197,7 @@ RawConfigParser Objects .. note:: Consider using :class:`ConfigParser` instead which checks types of - the values to be stored internally. If you don't want interpolation, you + the values to be stored internally. If you don't want interpolation, you can use ``ConfigParser(interpolation=None)``. @@ -1181,7 +1208,7 @@ RawConfigParser Objects *default section* name is passed, :exc:`ValueError` is raised. Type of *section* is not checked which lets users create non-string named - sections. This behaviour is unsupported and may cause internal errors. + sections. This behaviour is unsupported and may cause internal errors. .. method:: set(section, option, value) @@ -1282,3 +1309,4 @@ Exceptions .. [1] Config parsers allow for heavy customization. If you are interested in changing the behaviour outlined by the footnote reference, consult the `Customizing Parser Behaviour`_ section. + diff --git a/Doc/library/constants.rst b/Doc/library/constants.rst index 42b5af23a5..d5a0f09173 100644 --- a/Doc/library/constants.rst +++ b/Doc/library/constants.rst @@ -45,7 +45,6 @@ A small number of constants live in the built-in namespace. They are: for more details. - .. data:: Ellipsis The same as ``...``. Special value used mostly in conjunction with extended diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index 6f36864ac2..550b3476bd 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -172,6 +172,16 @@ Functions and classes provided: .. versionadded:: 3.4 +.. function:: redirect_stderr(new_target) + + Similar to :func:`~contextlib.redirect_stdout` but redirecting + :data:`sys.stderr` to another file or file-like object. + + This context manager is :ref:`reentrant <reentrant-cms>`. + + .. versionadded:: 3.5 + + .. class:: ContextDecorator() A base class that enables a context manager to also be used as a decorator. diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index 9f7b58afa7..325a421993 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -5,6 +5,7 @@ :synopsis: Write and read tabular data to and from delimited files. .. sectionauthor:: Skip Montanaro <skip@pobox.com> +**Source code:** :source:`Lib/csv.py` .. index:: single: csv @@ -418,7 +419,7 @@ Writer Objects :class:`Writer` objects (:class:`DictWriter` instances and objects returned by the :func:`writer` function) have the following public methods. A *row* must be -a sequence of strings or numbers for :class:`Writer` objects and a dictionary +an iterable of strings or numbers for :class:`Writer` objects and a dictionary mapping fieldnames to strings or numbers (by passing them through :func:`str` first) for :class:`DictWriter` objects. Note that complex numbers are written out surrounded by parens. This may cause some problems for other programs which @@ -430,6 +431,8 @@ read CSV files (assuming they support complex numbers at all). Write the *row* parameter to the writer's file object, formatted according to the current dialect. + .. versionchanged:: 3.5 + Added support of arbitrary iterables. .. method:: csvwriter.writerows(rows) diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 7dd86138c6..f82f425ac4 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -7,6 +7,8 @@ .. sectionauthor:: Tim Peters <tim@zope.com> .. sectionauthor:: A.M. Kuchling <amk@amk.ca> +**Source code:** :source:`Lib/datetime.py` + .. XXX what order should the types be discussed in? The :mod:`datetime` module supplies classes for manipulating dates and times in @@ -757,13 +759,19 @@ Other constructors, all class methods: :attr:`tzinfo` ``None``. This may raise :exc:`OverflowError`, if the timestamp is out of the range of values supported by the platform C :c:func:`gmtime` function, and :exc:`OSError` on :c:func:`gmtime` failure. - It's common for this to be restricted to years in 1970 through 2038. See also - :meth:`fromtimestamp`. + It's common for this to be restricted to years in 1970 through 2038. + + To get an aware :class:`.datetime` object, call :meth:`fromtimestamp`:: + + datetime.fromtimestamp(timestamp, timezone.utc) + + On the POSIX compliant platforms, it is equivalent to the following + expression:: - On the POSIX compliant platforms, ``utcfromtimestamp(timestamp)`` - is equivalent to the following expression:: + datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(seconds=timestamp) - datetime(1970, 1, 1) + timedelta(seconds=timestamp) + except the latter formula always supports the full years range: between + :const:`MINYEAR` and :const:`MAXYEAR` inclusive. .. versionchanged:: 3.3 Raise :exc:`OverflowError` instead of :exc:`ValueError` if the timestamp @@ -1376,10 +1384,13 @@ Supported operations: * efficient pickling -* in Boolean contexts, a :class:`.time` object is considered to be true if and - only if, after converting it to minutes and subtracting :meth:`utcoffset` (or - ``0`` if that's ``None``), the result is non-zero. +In boolean contexts, a :class:`.time` object is always considered to be true. +.. versionchanged:: 3.5 + Before Python 3.5, a :class:`.time` object was considered to be false if it + represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. Instance methods: diff --git a/Doc/library/dbm.rst b/Doc/library/dbm.rst index e6a82d67ce..3f3c43d438 100644 --- a/Doc/library/dbm.rst +++ b/Doc/library/dbm.rst @@ -325,13 +325,18 @@ The module defines the following: dumbdbm database is created, files with :file:`.dat` and :file:`.dir` extensions are created. - The optional *flag* argument is currently ignored; the database is always opened - for update, and will be created if it does not exist. + The optional *flag* argument supports only the semantics of ``'c'`` + and ``'n'`` values. Other values will default to database being always + opened for update, and will be created if it does not exist. The optional *mode* argument is the Unix mode of the file, used only when the database has to be created. It defaults to octal ``0o666`` (and will be modified by the prevailing umask). + .. versionchanged:: 3.5 + :func:`.open` always creates a new database when the flag has the value + ``'n'``. + In addition to the methods provided by the :class:`collections.abc.MutableMapping` class, :class:`dumbdbm` objects provide the following methods: diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index 759be70c9f..5d7ffffbe9 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -12,6 +12,8 @@ .. moduleauthor:: Stefan Krah <skrah at bytereef.org> .. sectionauthor:: Raymond D. Hettinger <python at rcn.com> +**Source code:** :source:`Lib/decimal.py` + .. import modules for testing inline doctests with the Sphinx doctest builder .. testsetup:: * @@ -742,7 +744,7 @@ Decimal objects * ``"NaN"``, indicating that the operand is a quiet NaN (Not a Number). * ``"sNaN"``, indicating that the operand is a signaling NaN. - .. method:: quantize(exp, rounding=None, context=None, watchexp=True) + .. method:: quantize(exp, rounding=None, context=None) Return a value equal to the first operand after rounding and having the exponent of the second operand. @@ -765,14 +767,8 @@ Decimal objects ``context`` argument; if neither argument is given the rounding mode of the current thread's context is used. - If *watchexp* is set (default), then an error is returned whenever the - resulting exponent is greater than :attr:`Emax` or less than - :attr:`Etiny`. - - .. deprecated:: 3.3 - *watchexp* is an implementation detail from the pure Python version - and is not present in the C version. It will be removed in version - 3.4, where it defaults to ``True``. + An error is returned whenever the resulting exponent is greater than + :attr:`Emax` or less than :attr:`Etiny`. .. method:: radix() @@ -2092,4 +2088,3 @@ Alternatively, inputs can be rounded upon creation using the >>> Context(prec=5, rounding=ROUND_DOWN).create_decimal('1.2345678') Decimal('1.2345') - diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index 5f72ea6233..442706556e 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -7,6 +7,8 @@ .. sectionauthor:: Tim Peters <tim_one@users.sourceforge.net> .. Markup by Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/difflib.py` + .. testsetup:: import sys @@ -25,7 +27,9 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. little fancier than, an algorithm published in the late 1980's by Ratcliff and Obershelp under the hyperbolic name "gestalt pattern matching." The idea is to find the longest contiguous matching subsequence that contains no "junk" - elements (the Ratcliff and Obershelp algorithm doesn't address junk). The same + elements; these "junk" elements are ones that are uninteresting in some + sense, such as blank lines or whitespace. (Handling junk is an + extension to the Ratcliff and Obershelp algorithm.) The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence. This does not yield minimal edit sequences, but does tend to yield matches that "look right" to people. @@ -100,7 +104,8 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. The following methods are public: - .. method:: make_file(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5) + .. method:: make_file(fromlines, tolines, fromdesc='', todesc='', context=False, \ + numlines=5, *, charset='utf-8') Compares *fromlines* and *tolines* (lists of strings) and returns a string which is a complete HTML file containing a table showing line by line differences with @@ -119,6 +124,10 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. the next difference highlight at the top of the browser without any leading context). + .. versionchanged:: 3.5 + *charset* keyword-only argument was added. The default charset of + HTML document changed from ``'ISO-8859-1'`` to ``'utf-8'``. + .. method:: make_table(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5) Compares *fromlines* and *tolines* (lists of strings) and returns a string which @@ -208,7 +217,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. Compare *a* and *b* (lists of strings); return a :class:`Differ`\ -style delta (a :term:`generator` generating the delta lines). - Optional keyword parameters *linejunk* and *charjunk* are for filter functions + Optional keyword parameters *linejunk* and *charjunk* are filtering functions (or ``None``): *linejunk*: A function that accepts a single string argument, and returns @@ -222,7 +231,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. *charjunk*: A function that accepts a character (a string of length 1), and returns if the character is junk, or false if not. The default is module-level function :func:`IS_CHARACTER_JUNK`, which filters out whitespace characters (a - blank or tab; note: bad idea to include newline in this!). + blank or tab; it's a bad idea to include newline in this!). :file:`Tools/scripts/ndiff.py` is a command-line front-end to this function. @@ -622,6 +631,12 @@ The :class:`Differ` class has this constructor: length 1), and returns true if the character is junk. The default is ``None``, meaning that no character is considered junk. + These junk-filtering functions speed up matching to find + differences and do not cause any differing lines or characters to + be ignored. Read the description of the + :meth:`~SequenceMatcher.find_longest_match` method's *isjunk* + parameter for an explanation. + :class:`Differ` objects are used (deltas generated) via a single method: diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index b816dcc4d3..b00d619d83 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -48,8 +48,9 @@ code. .. class:: Bytecode(x, *, first_line=None, current_offset=None) - Analyse the bytecode corresponding to a function, method, string of source - code, or a code object (as returned by :func:`compile`). + + Analyse the bytecode corresponding to a function, generator, method, string + of source code, or a code object (as returned by :func:`compile`). This is a convenience wrapper around many of the functions listed below, most notably :func:`get_instructions`, as iterating over a :class:`Bytecode` @@ -109,7 +110,7 @@ operation is being performed, so the intermediate analysis object isn't useful: .. function:: code_info(x) Return a formatted multi-line string with detailed code object information - for the supplied function, method, source code string or code object. + for the supplied function, generator, method, source code string or code object. Note that the exact contents of code info strings are highly implementation dependent and they may change arbitrarily across Python VMs or Python @@ -136,11 +137,11 @@ operation is being performed, so the intermediate analysis object isn't useful: .. function:: dis(x=None, *, file=None) Disassemble the *x* object. *x* can denote either a module, a class, a - method, a function, a code object, a string of source code or a byte sequence - of raw bytecode. For a module, it disassembles all functions. For a class, - it disassembles all methods. For a code object or sequence of raw bytecode, - it prints one line per bytecode instruction. Strings are first compiled to - code objects with the :func:`compile` built-in function before being + method, a function, a generator, a code object, a string of source code or + a byte sequence of raw bytecode. For a module, it disassembles all functions. + For a class, it disassembles all methods. For a code object or sequence of + raw bytecode, it prints one line per bytecode instruction. Strings are first + compiled to code objects with the :func:`compile` built-in function before being disassembled. If no object is provided, this function disassembles the last traceback. @@ -361,6 +362,13 @@ result back on the stack. Implements ``TOS = TOS1 * TOS``. +.. opcode:: BINARY_MATRIX_MULTIPLY + + Implements ``TOS = TOS1 @ TOS``. + + .. versionadded:: 3.5 + + .. opcode:: BINARY_FLOOR_DIVIDE Implements ``TOS = TOS1 // TOS``. @@ -433,6 +441,13 @@ the original TOS1. Implements in-place ``TOS = TOS1 * TOS``. +.. opcode:: INPLACE_MATRIX_MULTIPLY + + Implements in-place ``TOS = TOS1 @ TOS``. + + .. versionadded:: 3.5 + + .. opcode:: INPLACE_FLOOR_DIVIDE Implements in-place ``TOS = TOS1 // TOS``. diff --git a/Doc/library/distribution.rst b/Doc/library/distribution.rst index c4954d1b4a..3e6e84b42a 100644 --- a/Doc/library/distribution.rst +++ b/Doc/library/distribution.rst @@ -12,3 +12,4 @@ with a local index server, or without any index server at all. distutils.rst ensurepip.rst venv.rst + zipapp.rst diff --git a/Doc/library/doctest.rst b/Doc/library/doctest.rst index 2e372de34a..9aa9ea6614 100644 --- a/Doc/library/doctest.rst +++ b/Doc/library/doctest.rst @@ -1058,15 +1058,9 @@ from text files and modules with doctests: This function uses the same search technique as :func:`testmod`. - .. note:: - Unlike :func:`testmod` and :class:`DocTestFinder`, this function raises - a :exc:`ValueError` if *module* contains no docstrings. You can prevent - this error by passing a :class:`DocTestFinder` instance as the - *test_finder* argument with its *exclude_empty* keyword argument set - to ``False``:: - - >>> finder = doctest.DocTestFinder(exclude_empty=False) - >>> suite = doctest.DocTestSuite(test_finder=finder) + .. versionchanged:: 3.5 + :func:`DocTestSuite` returns an empty :class:`unittest.TestSuite` if *module* + contains no docstrings instead of raising :exc:`ValueError`. Under the covers, :func:`DocTestSuite` creates a :class:`unittest.TestSuite` out diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst index 1d70225fc8..67d0a67954 100644 --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -195,7 +195,8 @@ Here are the classes: set of the text and is passed as an argument to the :class:`~email.mime.nonmultipart.MIMENonMultipart` constructor; it defaults to ``us-ascii`` if the string contains only ``ascii`` code points, and - ``utf-8`` otherwise. + ``utf-8`` otherwise. The *_charset* parameter accepts either a string or a + :class:`~email.charset.Charset` instance. Unless the *_charset* argument is explicitly set to ``None``, the MIMEText object created will have both a :mailheader:`Content-Type` header @@ -206,3 +207,6 @@ Here are the classes: ``Content-Transfer-Encoding`` header, after which a ``set_payload`` call will automatically encode the new payload (and add a new :mailheader:`Content-Transfer-Encoding` header). + + .. versionchanged:: 3.5 + *_charset* also accepts :class:`~email.charset.Charset` instances. diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index cf095592f6..d3b838c01c 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -314,11 +314,11 @@ Then:: >>> str(Mood.funky) 'my custom str! 1' -The rules for what is allowed are as follows: _sunder_ names (starting and -ending with a single underscore) are reserved by enum and cannot be used; -all other attributes defined within an enumeration will become members of this -enumeration, with the exception of *__dunder__* names and descriptors (methods -are also descriptors). +The rules for what is allowed are as follows: names that start and end with a +with a single underscore are reserved by enum and cannot be used; all other +attributes defined within an enumeration will become members of this +enumeration, with the exception of special methods (:meth:`__str__`, +:meth:`__add__`, etc.) and descriptors (methods are also descriptors). Note: if your enumeration defines :meth:`__new__` and/or :meth:`__init__` then whatever value(s) were given to the enum member will be passed into those @@ -400,7 +400,8 @@ The second argument is the *source* of enumeration member names. It can be a whitespace-separated string of names, a sequence of names, a sequence of 2-tuples with key/value pairs, or a mapping (e.g. dictionary) of names to values. The last two options enable assigning arbitrary values to -enumerations; the others auto-assign increasing integers starting with 1. A +enumerations; the others auto-assign increasing integers starting with 1 (use +the ``start`` parameter to specify a different starting value). A new class derived from :class:`Enum` is returned. In other words, the above assignment to :class:`Animal` is equivalent to:: @@ -438,12 +439,12 @@ SomeData in the global scope:: The complete signature is:: - Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=<mixed-in class>) + Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=<mixed-in class>, start=1) :value: What the new Enum class will record as its name. :names: The Enum members. This can be a whitespace or comma separated string - (values will start at 1):: + (values will start at 1 unless otherwise specified):: 'red green blue' | 'red,green,blue' | 'red, green, blue' @@ -465,6 +466,11 @@ The complete signature is:: :type: type to mix in to new Enum class. +:start: number to start counting at if only names are passed in + +.. versionchanged:: 3.5 + The *start* parameter was added. + Derived Enumerations -------------------- diff --git a/Doc/library/errno.rst b/Doc/library/errno.rst index d2163b6258..22a5cbc450 100644 --- a/Doc/library/errno.rst +++ b/Doc/library/errno.rst @@ -41,7 +41,10 @@ defined by the module. The specific list of defined symbols is available as .. data:: EINTR - Interrupted system call + Interrupted system call. + + .. seealso:: + This error is mapped to the exception :exc:`InterruptedError`. .. data:: EIO diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 271a5c8219..bddd0edc6c 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -536,7 +536,12 @@ depending on the system error code. .. exception:: InterruptedError Raised when a system call is interrupted by an incoming signal. - Corresponds to :c:data:`errno` ``EINTR``. + Corresponds to :c:data:`errno` :py:data:`~errno.EINTR`. + + .. versionchanged:: 3.5 + Python now retries system calls when a syscall is interrupted by a + signal, except if the signal handler raises an exception (see :pep:`475` + for the rationale), instead of raising :exc:`InterruptedError`. .. exception:: IsADirectoryError diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst index eb2016a7b9..3a5badd0ff 100644 --- a/Doc/library/faulthandler.rst +++ b/Doc/library/faulthandler.rst @@ -47,6 +47,9 @@ Dumping the traceback Dump the tracebacks of all threads into *file*. If *all_threads* is ``False``, dump only the current thread. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + Fault handler state ------------------- @@ -59,6 +62,12 @@ Fault handler state produce tracebacks for every running thread. Otherwise, dump only the current thread. + The *file* must be kept open until the fault handler is disabled: see + :ref:`issue with file descriptors <faulthandler-fd>`. + + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: disable() Disable the fault handler: uninstall the signal handlers installed by @@ -82,9 +91,16 @@ Dumping the tracebacks after a timeout call replaces previous parameters and resets the timeout. The timer has a sub-second resolution. + The *file* must be kept open until the traceback is dumped or + :func:`cancel_dump_traceback_later` is called: see :ref:`issue with file + descriptors <faulthandler-fd>`. + This function is implemented using a watchdog thread and therefore is not available if Python is compiled with threads disabled. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: cancel_dump_traceback_later() Cancel the last call to :func:`dump_traceback_later`. @@ -99,8 +115,14 @@ Dumping the traceback on a user signal the traceback of all threads, or of the current thread if *all_threads* is ``False``, into *file*. Call the previous handler if chain is ``True``. + The *file* must be kept open until the signal is unregistered by + :func:`unregister`: see :ref:`issue with file descriptors <faulthandler-fd>`. + Not available on Windows. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: unregister(signum) Unregister a user signal: uninstall the handler of the *signum* signal @@ -110,6 +132,8 @@ Dumping the traceback on a user signal Not available on Windows. +.. _faulthandler-fd: + Issue with file descriptors --------------------------- diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst index 8e932fb954..432140f279 100644 --- a/Doc/library/fcntl.rst +++ b/Doc/library/fcntl.rst @@ -28,41 +28,41 @@ descriptor. The module defines the following functions: -.. function:: fcntl(fd, op[, arg]) +.. function:: fcntl(fd, cmd, arg=0) - Perform the operation *op* on file descriptor *fd* (file objects providing + Perform the operation *cmd* on file descriptor *fd* (file objects providing a :meth:`~io.IOBase.fileno` method are accepted as well). The values used - for *op* are operating system dependent, and are available as constants + for *cmd* are operating system dependent, and are available as constants in the :mod:`fcntl` module, using the same names as used in the relevant C - header files. The argument *arg* is optional, and defaults to the integer - value ``0``. When present, it can either be an integer value, or a string. - With the argument missing or an integer value, the return value of this function - is the integer return value of the C :c:func:`fcntl` call. When the argument is - a string it represents a binary structure, e.g. created by :func:`struct.pack`. - The binary data is copied to a buffer whose address is passed to the C - :c:func:`fcntl` call. The return value after a successful call is the contents - of the buffer, converted to a string object. The length of the returned string - will be the same as the length of the *arg* argument. This is limited to 1024 - bytes. If the information returned in the buffer by the operating system is - larger than 1024 bytes, this is most likely to result in a segmentation - violation or a more subtle data corruption. + header files. The argument *arg* can either be an integer value, or a + :class:`bytes` object. With an integer value, the return value of this + function is the integer return value of the C :c:func:`fcntl` call. When + the argument is bytes it represents a binary structure, e.g. created by + :func:`struct.pack`. The binary data is copied to a buffer whose address is + passed to the C :c:func:`fcntl` call. The return value after a successful + call is the contents of the buffer, converted to a :class:`bytes` object. + The length of the returned object will be the same as the length of the + *arg* argument. This is limited to 1024 bytes. If the information returned + in the buffer by the operating system is larger than 1024 bytes, this is + most likely to result in a segmentation violation or a more subtle data + corruption. If the :c:func:`fcntl` fails, an :exc:`OSError` is raised. -.. function:: ioctl(fd, op[, arg[, mutate_flag]]) +.. function:: ioctl(fd, request, arg=0, mutate_flag=True) This function is identical to the :func:`~fcntl.fcntl` function, except that the argument handling is even more complicated. - The op parameter is limited to values that can fit in 32-bits. - Additional constants of interest for use as the *op* argument can be + The *request* parameter is limited to values that can fit in 32-bits. + Additional constants of interest for use as the *request* argument can be found in the :mod:`termios` module, under the same names as used in the relevant C header files. - The parameter *arg* can be one of an integer, absent (treated identically to the - integer ``0``), an object supporting the read-only buffer interface (most likely - a plain Python string) or an object supporting the read-write buffer interface. + The parameter *arg* can be one of an integer, an object supporting the + read-only buffer interface (like :class:`bytes`) or an object supporting + the read-write buffer interface (like :class:`bytearray`). In all but the last case, behaviour is as for the :func:`~fcntl.fcntl` function. @@ -72,7 +72,7 @@ The module defines the following functions: If it is false, the buffer's mutability is ignored and behaviour is as for a read-only buffer, except that the 1024 byte limit mentioned above is avoided -- - so long as the buffer you pass is as least as long as what the operating system + so long as the buffer you pass is at least as long as what the operating system wants to put there, things should work. If *mutate_flag* is true (the default), then the buffer is (in effect) passed @@ -97,25 +97,25 @@ The module defines the following functions: array('h', [13341]) -.. function:: flock(fd, op) +.. function:: flock(fd, operation) - Perform the lock operation *op* on file descriptor *fd* (file objects providing + Perform the lock operation *operation* on file descriptor *fd* (file objects providing a :meth:`~io.IOBase.fileno` method are accepted as well). See the Unix manual :manpage:`flock(2)` for details. (On some systems, this function is emulated using :c:func:`fcntl`.) -.. function:: lockf(fd, operation, [length, [start, [whence]]]) +.. function:: lockf(fd, cmd, len=0, start=0, whence=0) This is essentially a wrapper around the :func:`~fcntl.fcntl` locking calls. - *fd* is the file descriptor of the file to lock or unlock, and *operation* + *fd* is the file descriptor of the file to lock or unlock, and *cmd* is one of the following values: * :const:`LOCK_UN` -- unlock * :const:`LOCK_SH` -- acquire a shared lock * :const:`LOCK_EX` -- acquire an exclusive lock - When *operation* is :const:`LOCK_SH` or :const:`LOCK_EX`, it can also be + When *cmd* is :const:`LOCK_SH` or :const:`LOCK_EX`, it can also be bitwise ORed with :const:`LOCK_NB` to avoid blocking on lock acquisition. If :const:`LOCK_NB` is used and the lock cannot be acquired, an :exc:`OSError` will be raised and the exception will have an *errno* @@ -124,7 +124,7 @@ The module defines the following functions: systems, :const:`LOCK_EX` can only be used if the file descriptor refers to a file opened for writing. - *length* is the number of bytes to lock, *start* is the byte offset at + *len* is the number of bytes to lock, *start* is the byte offset at which the lock starts, relative to *whence*, and *whence* is as with :func:`io.IOBase.seek`, specifically: @@ -133,7 +133,7 @@ The module defines the following functions: * :const:`2` -- relative to the end of the file (:data:`os.SEEK_END`) The default for *start* is 0, which means to start at the beginning of the file. - The default for *length* is 0 which means to lock to the end of the file. The + The default for *len* is 0 which means to lock to the end of the file. The default for *whence* is also 0. Examples (all on a SVR4 compliant system):: @@ -147,9 +147,9 @@ Examples (all on a SVR4 compliant system):: rv = fcntl.fcntl(f, fcntl.F_SETLKW, lockdata) Note that in the first example the return value variable *rv* will hold an -integer value; in the second example it will hold a string value. The structure -lay-out for the *lockdata* variable is system dependent --- therefore using the -:func:`flock` call may be better. +integer value; in the second example it will hold a :class:`bytes` object. The +structure lay-out for the *lockdata* variable is system dependent --- therefore +using the :func:`flock` call may be better. .. seealso:: diff --git a/Doc/library/formatter.rst b/Doc/library/formatter.rst index 1847a8094b..a515f74f64 100644 --- a/Doc/library/formatter.rst +++ b/Doc/library/formatter.rst @@ -5,7 +5,7 @@ :synopsis: Generic output formatter and device interface. :deprecated: -.. deprecated:: 3.4 +.. deprecated-removed:: 3.4 3.6 Due to lack of usage, the formatter module has been deprecated and is slated for removal in Python 3.6. diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index d9e5cfb5fc..456dd09821 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -156,11 +156,12 @@ are always available. They are listed here in alphabetical order. .. function:: chr(i) - Return the string representing a character whose Unicode code point is the integer - *i*. For example, ``chr(97)`` returns the string ``'a'``. This is the - inverse of :func:`ord`. The valid range for the argument is from 0 through - 1,114,111 (0x10FFFF in base 16). :exc:`ValueError` will be raised if *i* is - outside that range. + Return the string representing a character whose Unicode code point is the + integer *i*. For example, ``chr(97)`` returns the string ``'a'``, while + ``chr(957)`` returns the string ``'ν'``. This is the inverse of :func:`ord`. + + The valid range for the argument is from 0 through 1,114,111 (0x10FFFF in + base 16). :exc:`ValueError` will be raised if *i* is outside that range. .. function:: classmethod(function) @@ -972,9 +973,11 @@ are always available. They are listed here in alphabetical order. Characters not supported by the encoding are replaced with the appropriate XML character reference ``&#nnn;``. - * ``'backslashreplace'`` (also only supported when writing) - replaces unsupported characters with Python's backslashed escape - sequences. + * ``'backslashreplace'`` replaces malformed data by Python's backslashed + escape sequences. + + * ``'namereplace'`` (also only supported when writing) + replaces unsupported characters with ``\N{...}`` escape sequences. .. index:: single: universal newlines; open() built-in function @@ -999,8 +1002,8 @@ are always available. They are listed here in alphabetical order. If *closefd* is ``False`` and a file descriptor rather than a filename was given, the underlying file descriptor will be kept open when the file is - closed. If a filename is given *closefd* has no effect and must be ``True`` - (the default). + closed. If a filename is given *closefd* must be ``True`` (the default) + otherwise an error will be raised. A custom opener can be used by passing a callable as *opener*. The underlying file descriptor for the file object is then obtained by calling *opener* with @@ -1062,14 +1065,18 @@ are always available. They are listed here in alphabetical order. The ``'U'`` mode. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + -.. XXX works for bytes too, but should it? .. function:: ord(c) Given a string representing one Unicode character, return an integer - representing the Unicode code - point of that character. For example, ``ord('a')`` returns the integer ``97`` - and ``ord('\u2020')`` returns ``8224``. This is the inverse of :func:`chr`. + representing the Unicode code point of that character. For example, + ``ord('a')`` returns the integer ``97`` and ``ord('ν')`` returns ``957``. + This is the inverse of :func:`chr`. .. function:: pow(x, y[, z]) diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index 813554206a..d11c2e1282 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -186,7 +186,7 @@ values but should not rebind them): added to this list rather than freed. .. versionchanged:: 3.2 - If this list is non-empty at interpreter shutdown, a + If this list is non-empty at :term:`interpreter shutdown`, a :exc:`ResourceWarning` is emitted, which is silent by default. If :const:`DEBUG_UNCOLLECTABLE` is set, in addition all uncollectable objects are printed. @@ -252,8 +252,8 @@ The following constants are provided for use with :func:`set_debug`: to the ``garbage`` list. .. versionchanged:: 3.2 - Also print the contents of the :data:`garbage` list at interpreter - shutdown, if it isn't empty. + Also print the contents of the :data:`garbage` list at + :term:`interpreter shutdown`, if it isn't empty. .. data:: DEBUG_SAVEALL diff --git a/Doc/library/gettext.rst b/Doc/library/gettext.rst index ff23b59156..514cc5a989 100644 --- a/Doc/library/gettext.rst +++ b/Doc/library/gettext.rst @@ -344,9 +344,9 @@ will assume message ids as Unicode strings, not byte strings. The entire set of key/value pairs are placed into a dictionary and set as the "protected" :attr:`_info` instance variable. -If the :file:`.mo` file's magic number is invalid, or if other problems occur -while reading the file, instantiating a :class:`GNUTranslations` class can raise -:exc:`OSError`. +If the :file:`.mo` file's magic number is invalid, the major version number is +unexpected, or if other problems occur while reading the file, instantiating a +:class:`GNUTranslations` class can raise :exc:`OSError`. The following methods are overridden from the base class implementation: diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst index abcbf380d9..50f38a4d36 100644 --- a/Doc/library/glob.rst +++ b/Doc/library/glob.rst @@ -29,7 +29,7 @@ For example, ``'[?]'`` matches the character ``'?'``. The :mod:`pathlib` module offers high-level path objects. -.. function:: glob(pathname) +.. function:: glob(pathname, *, recursive=False) Return a possibly-empty list of path names that match *pathname*, which must be a string containing a path specification. *pathname* can be either absolute @@ -37,8 +37,19 @@ For example, ``'[?]'`` matches the character ``'?'``. :file:`../../Tools/\*/\*.gif`), and can contain shell-style wildcards. Broken symlinks are included in the results (as in the shell). + If *recursive* is true, the pattern "``**``" will match any files and zero or + more directories and subdirectories. If the pattern is followed by a + ``os.sep``, only directories and subdirectories match. -.. function:: iglob(pathname) + .. note:: + Using the "``**``" pattern in large directory trees may consume + an inordinate amount of time. + + .. versionchanged:: 3.5 + Support for recursive globs using "``**``". + + +.. function:: iglob(pathname, recursive=False) Return an :term:`iterator` which yields the same values as :func:`glob` without actually storing them all simultaneously. @@ -55,8 +66,9 @@ For example, ``'[?]'`` matches the character ``'?'``. .. versionadded:: 3.4 -For example, consider a directory containing only the following files: -:file:`1.gif`, :file:`2.txt`, and :file:`card.gif`. :func:`glob` will produce +For example, consider a directory containing the following files: +:file:`1.gif`, :file:`2.txt`, :file:`card.gif` and a subdirectory :file:`sub` +which contains only the file :file:`3.txt`. :func:`glob` will produce the following results. Notice how any leading components of the path are preserved. :: @@ -67,6 +79,10 @@ preserved. :: ['1.gif', 'card.gif'] >>> glob.glob('?.gif') ['1.gif'] + >>> glob.glob('**/*.txt', recursive=True) + ['2.txt', 'sub/3.txt'] + >>> glob.glob('./**/', recursive=True) + ['./', './sub/'] If the directory contains files starting with ``.`` they won't be matched by default. For example, consider a directory containing :file:`card.gif` and diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst index 78536fab51..a8e77049d3 100644 --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -90,13 +90,9 @@ The module defines the following items: is no compression. The default is ``9``. The *mtime* argument is an optional numeric timestamp to be written to - the stream when compressing. All :program:`gzip` compressed streams are - required to contain a timestamp. If omitted or ``None``, the current - time is used. This module ignores the timestamp when decompressing; - however, some programs, such as :program:`gunzip`\ , make use of it. - The format of the timestamp is the same as that of the return value of - ``time.time()`` and of the ``st_mtime`` attribute of the object returned - by ``os.stat()``. + the last modification time field in the stream when compressing. It + should only be provided in compression mode. If omitted or ``None``, the + current time is used. See the :attr:`mtime` attribute for more details. Calling a :class:`GzipFile` object's :meth:`close` method does not close *fileobj*, since you might wish to append more material after the compressed @@ -108,9 +104,9 @@ The module defines the following items: including iteration and the :keyword:`with` statement. Only the :meth:`truncate` method isn't implemented. - :class:`GzipFile` also provides the following method: + :class:`GzipFile` also provides the following method and attribute: - .. method:: peek([n]) + .. method:: peek(n) Read *n* uncompressed bytes without advancing the file position. At most one single read on the compressed stream is done to satisfy @@ -124,9 +120,21 @@ The module defines the following items: .. versionadded:: 3.2 + .. attribute:: mtime + + When decompressing, the value of the last modification time field in + the most recently read header may be read from this attribute, as an + integer. The initial value before reading any headers is ``None``. + + All :program:`gzip` compressed streams are required to contain this + timestamp field. Some programs, such as :program:`gunzip`\ , make use + of the timestamp. The format is the same as the return value of + :func:`time.time` and the :attr:`~os.stat_result.st_mtime` attribute of + the object returned by :func:`os.stat`. + .. versionchanged:: 3.1 Support for the :keyword:`with` statement was added, along with the - *mtime* argument. + *mtime* constructor argument and :attr:`mtime` attribute. .. versionchanged:: 3.2 Support for zero-padded and unseekable files was added. @@ -137,6 +145,12 @@ The module defines the following items: .. versionchanged:: 3.4 Added support for the ``'x'`` and ``'xb'`` modes. + .. versionchanged:: 3.5 + Added support for writing arbitrary + :term:`bytes-like objects <bytes-like object>`. + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + .. function:: compress(data, compresslevel=9) diff --git a/Doc/library/heapq.rst b/Doc/library/heapq.rst index f8970bed73..9fbbcc6904 100644 --- a/Doc/library/heapq.rst +++ b/Doc/library/heapq.rst @@ -82,7 +82,7 @@ The following functions are provided: The module also offers three general purpose functions based on heaps. -.. function:: merge(*iterables) +.. function:: merge(*iterables, key=None, reverse=False) Merge multiple sorted inputs into a single sorted output (for example, merge timestamped entries from multiple log files). Returns an :term:`iterator` @@ -92,6 +92,18 @@ The module also offers three general purpose functions based on heaps. not pull the data into memory all at once, and assumes that each of the input streams is already sorted (smallest to largest). + Has two optional arguments which must be specified as keyword arguments. + + *key* specifies a :term:`key function` of one argument that is used to + extract a comparison key from each input element. The default value is + ``None`` (compare the elements directly). + + *reverse* is a boolean value. If set to ``True``, then the input elements + are merged as if each comparison were reversed. + + .. versionchanged:: 3.5 + Added the optional *key* and *reverse* parameters. + .. function:: nlargest(n, iterable, key=None) diff --git a/Doc/library/html.parser.rst b/Doc/library/html.parser.rst index 44b7d6ea6d..b84c60b708 100644 --- a/Doc/library/html.parser.rst +++ b/Doc/library/html.parser.rst @@ -16,21 +16,13 @@ This module defines a class :class:`HTMLParser` which serves as the basis for parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. -.. class:: HTMLParser(strict=False, *, convert_charrefs=False) +.. class:: HTMLParser(*, convert_charrefs=True) - Create a parser instance. + Create a parser instance able to parse invalid markup. - If *convert_charrefs* is ``True`` (default: ``False``), all character + If *convert_charrefs* is ``True`` (the default), all character references (except the ones in ``script``/``style`` elements) are automatically converted to the corresponding Unicode characters. - The use of ``convert_charrefs=True`` is encouraged and will become - the default in Python 3.5. - - If *strict* is ``False`` (the default), the parser will accept and parse - invalid markup. If *strict* is ``True`` the parser will raise an - :exc:`~html.parser.HTMLParseError` exception instead [#]_ when it's not - able to parse the markup. The use of ``strict=True`` is discouraged and - the *strict* argument is deprecated. An :class:`.HTMLParser` instance is fed HTML data and calls handler methods when start tags, end tags, text, comments, and other markup elements are @@ -40,31 +32,11 @@ parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. This parser does not check that end tags match start tags or call the end-tag handler for elements which are closed implicitly by closing an outer element. - .. versionchanged:: 3.2 - *strict* argument added. - - .. deprecated-removed:: 3.3 3.5 - The *strict* argument and the strict mode have been deprecated. - The parser is now able to accept and parse invalid markup too. - .. versionchanged:: 3.4 *convert_charrefs* keyword argument added. -An exception is defined as well: - - -.. exception:: HTMLParseError - - Exception raised by the :class:`HTMLParser` class when it encounters an error - while parsing and *strict* is ``True``. This exception provides three - attributes: :attr:`msg` is a brief message explaining the error, - :attr:`lineno` is the number of the line on which the broken construct was - detected, and :attr:`offset` is the number of characters into the line at - which the construct starts. - - .. deprecated-removed:: 3.3 3.5 - This exception has been deprecated because it's never raised by the parser - (when the default non-strict mode is used). + .. versionchanged:: 3.5 + The default value for argument *convert_charrefs* is now ``True``. Example HTML Parser Application @@ -246,8 +218,7 @@ implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`): The *data* parameter will be the entire contents of the declaration inside the ``<![...]>`` markup. It is sometimes useful to be overridden by a - derived class. The base class implementation raises an :exc:`HTMLParseError` - when *strict* is ``True``. + derived class. The base class implementation does nothing. .. _htmlparser-examples: @@ -358,9 +329,3 @@ Parsing invalid HTML (e.g. unquoted attributes) also works:: Data : tag soup End tag : p End tag : a - -.. rubric:: Footnotes - -.. [#] For backward compatibility reasons *strict* mode does not raise - exceptions for all non-compliant HTML. That is, some invalid HTML - is tolerated even in *strict* mode. diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index fc45103474..5e2d68ce09 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -175,6 +175,17 @@ The following exceptions are raised as appropriate: is received in the HTTP protocol from the server. +.. exception:: RemoteDisconnected + + A subclass of :exc:`ConnectionResetError` and :exc:`BadStatusLine`. Raised + by :meth:`HTTPConnection.getresponse` when the attempt to read the response + results in no data read from the connection, indicating that the remote end + has closed the connection. + + .. versionadded:: 3.5 + Previously, :exc:`BadStatusLine`\ ``('')`` was raised. + + The constants defined in this module are: .. data:: HTTP_PORT @@ -186,221 +197,15 @@ The constants defined in this module are: The default port for the HTTPS protocol (always ``443``). -and also the following constants for integer status codes: - -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| Constant | Value | Definition | -+==========================================+=========+=======================================================================+ -| :const:`CONTINUE` | ``100`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.1.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.1.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SWITCHING_PROTOCOLS` | ``101`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.1.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.1.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PROCESSING` | ``102`` | WEBDAV, `RFC 2518, Section 10.1 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_102>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`OK` | ``200`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`CREATED` | ``201`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`ACCEPTED` | ``202`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NON_AUTHORITATIVE_INFORMATION` | ``203`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NO_CONTENT` | ``204`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`RESET_CONTENT` | ``205`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PARTIAL_CONTENT` | ``206`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.7 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.7>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MULTI_STATUS` | ``207`` | WEBDAV `RFC 2518, Section 10.2 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_207>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`IM_USED` | ``226`` | Delta encoding in HTTP, | -| | | :rfc:`3229`, Section 10.4.1 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MULTIPLE_CHOICES` | ``300`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MOVED_PERMANENTLY` | ``301`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FOUND` | ``302`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SEE_OTHER` | ``303`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_MODIFIED` | ``304`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`USE_PROXY` | ``305`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`TEMPORARY_REDIRECT` | ``307`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.8 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.8>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`BAD_REQUEST` | ``400`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNAUTHORIZED` | ``401`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PAYMENT_REQUIRED` | ``402`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FORBIDDEN` | ``403`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_FOUND` | ``404`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`METHOD_NOT_ALLOWED` | ``405`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_ACCEPTABLE` | ``406`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.7 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.7>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PROXY_AUTHENTICATION_REQUIRED` | ``407`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.8 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.8>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_TIMEOUT` | ``408`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.9 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.9>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`CONFLICT` | ``409`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.10 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.10>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`GONE` | ``410`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.11 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.11>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`LENGTH_REQUIRED` | ``411`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.12 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.12>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PRECONDITION_FAILED` | ``412`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.13 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.13>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_ENTITY_TOO_LARGE` | ``413`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.14 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.14>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_URI_TOO_LONG` | ``414`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.15 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.15>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNSUPPORTED_MEDIA_TYPE` | ``415`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.16 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.16>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUESTED_RANGE_NOT_SATISFIABLE` | ``416`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.17 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.17>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`EXPECTATION_FAILED` | ``417`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.18 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.18>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNPROCESSABLE_ENTITY` | ``422`` | WEBDAV, `RFC 2518, Section 10.3 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_422>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`LOCKED` | ``423`` | WEBDAV `RFC 2518, Section 10.4 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_423>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FAILED_DEPENDENCY` | ``424`` | WEBDAV, `RFC 2518, Section 10.5 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_424>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UPGRADE_REQUIRED` | ``426`` | HTTP Upgrade to TLS, | -| | | :rfc:`2817`, Section 6 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PRECONDITION_REQUIRED` | ``428`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 3 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`TOO_MANY_REQUESTS` | ``429`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 4 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_HEADER_FIELDS_TOO_LARGE` | ``431`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 5 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`INTERNAL_SERVER_ERROR` | ``500`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_IMPLEMENTED` | ``501`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`BAD_GATEWAY` | ``502`` | HTTP/1.1 `RFC 2616, Section | -| | | 10.5.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SERVICE_UNAVAILABLE` | ``503`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`GATEWAY_TIMEOUT` | ``504`` | HTTP/1.1 `RFC 2616, Section | -| | | 10.5.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`HTTP_VERSION_NOT_SUPPORTED` | ``505`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`INSUFFICIENT_STORAGE` | ``507`` | WEBDAV, `RFC 2518, Section 10.6 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_507>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_EXTENDED` | ``510`` | An HTTP Extension Framework, | -| | | :rfc:`2774`, Section 7 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NETWORK_AUTHENTICATION_REQUIRED` | ``511`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 6 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ - -.. versionchanged:: 3.3 - Added codes ``428``, ``429``, ``431`` and ``511`` from :rfc:`6585`. - - .. data:: responses This dictionary maps the HTTP 1.1 status codes to the W3C names. Example: ``http.client.responses[http.client.NOT_FOUND]`` is ``'Not Found'``. +See :ref:`http-status-codes` for a list of HTTP status codes that are +available in this module as constants. + .. _httpconnection-objects: @@ -453,6 +258,11 @@ HTTPConnection Objects Note that you must have read the whole response before you can send a new request to the server. + .. versionchanged:: 3.5 + If a :exc:`ConnectionError` or subclass is raised, the + :class:`HTTPConnection` object will be ready to reconnect when + a new request is sent. + .. method:: HTTPConnection.set_debuglevel(level) @@ -491,7 +301,9 @@ HTTPConnection Objects .. method:: HTTPConnection.connect() - Connect to the server specified when the object was created. + Connect to the server specified when the object was created. By default, + this is called automatically when making a request if the client does not + already have a connection. .. method:: HTTPConnection.close() diff --git a/Doc/library/http.cookies.rst b/Doc/library/http.cookies.rst index 646f2e8860..7c85d0954d 100644 --- a/Doc/library/http.cookies.rst +++ b/Doc/library/http.cookies.rst @@ -143,26 +143,43 @@ Morsel Objects The keys are case-insensitive. + .. versionchanged:: 3.5 + :meth:`~Morsel.__eq__` now takes :attr:`~Morsel.key` and :attr:`~Morsel.value` + into account. + .. attribute:: Morsel.value The value of the cookie. + .. deprecated:: 3.5 + assigning to ``value``; use :meth:`~Morsel.set` instead. + .. attribute:: Morsel.coded_value The encoded value of the cookie --- this is what should be sent. + .. deprecated:: 3.5 + assigning to ``coded_value``; use :meth:`~Morsel.set` instead. + .. attribute:: Morsel.key The name of the cookie. + .. deprecated:: 3.5 + assigning to ``key``; use :meth:`~Morsel.set` instead. + .. method:: Morsel.set(key, value, coded_value) Set the *key*, *value* and *coded_value* attributes. + .. deprecated:: 3.5 + The undocumented *LegalChars* parameter is ignored and will be removed in + a future version. + .. method:: Morsel.isReservedKey(K) @@ -193,6 +210,30 @@ Morsel Objects The meaning for *attrs* is the same as in :meth:`output`. +.. method:: Morsel.update(values) + + Update the values in the Morsel dictionary with the values in the dictionary + *values*. Raise an error if any of the keys in the *values* dict is not a + valid :rfc:`2109` attribute. + + .. versionchanged:: 3.5 + an error is raised for invalid keys. + + +.. method:: Morsel.copy(value) + + Return a shallow copy of the Morsel object. + + .. versionchanged:: 3.5 + return a Morsel object instead of a dict. + + +.. method:: Morsel.setdefault(key, value=None) + + Raise an error if key is not a valid :rfc:`2109` attribute, otherwise + behave the same as :meth:`dict.setdefault`. + + .. _cookie-example: Example diff --git a/Doc/library/http.rst b/Doc/library/http.rst index a387a37ddd..b6f2c58218 100644 --- a/Doc/library/http.rst +++ b/Doc/library/http.rst @@ -1,7 +1,16 @@ :mod:`http` --- HTTP modules ============================ -``http`` is a package that collects several modules for working with the +.. module:: http + :synopsis: HTTP status codes and messages + +.. index:: + pair: HTTP; protocol + single: HTTP; http (standard module) + +**Source code:** :source:`Lib/http/__init__.py` + +:mod:`http` is a package that collects several modules for working with the HyperText Transfer Protocol: * :mod:`http.client` is a low-level HTTP protocol client; for high-level URL @@ -9,3 +18,105 @@ HyperText Transfer Protocol: * :mod:`http.server` contains basic HTTP server classes based on :mod:`socketserver` * :mod:`http.cookies` has utilities for implementing state management with cookies * :mod:`http.cookiejar` provides persistence of cookies + +:mod:`http` is also a module that defines a number of HTTP status codes and +associated messages through the :class:`http.HTTPStatus` enum: + +.. class:: HTTPStatus + + .. versionadded:: 3.5 + + A subclass of :class:`enum.IntEnum` that defines a set of HTTP status codes, + reason phrases and long descriptions written in English. + + Usage:: + + >>> from http import HTTPStatus + >>> HTTPStatus.OK + <HTTPStatus.OK: 200> + >>> HTTPStatus.OK == 200 + True + >>> http.HTTPStatus.OK.value + 200 + >>> HTTPStatus.OK.phrase + 'OK' + >>> HTTPStatus.OK.description + 'Request fulfilled, document follows' + >>> list(HTTPStatus) + [<HTTPStatus.CONTINUE: 100>, <HTTPStatus.SWITCHING_PROTOCOLS: 101>, ...] + +.. _http-status-codes: + +HTTP status codes +----------------- + +Supported, +`IANA-registered <http://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml>`_ +status codes available in :class:`http.HTTPStatus` are: + +======= =================================== ================================================================== +Code Enum Name Details +======= =================================== ================================================================== +``100`` ``CONTINUE`` HTTP/1.1 :rfc:`7231`, Section 6.2.1 +``101`` ``SWITCHING_PROTOCOLS`` HTTP/1.1 :rfc:`7231`, Section 6.2.2 +``102`` ``PROCESSING`` WebDAV :rfc:`2518`, Section 10.1 +``200`` ``OK`` HTTP/1.1 :rfc:`7231`, Section 6.3.1 +``201`` ``CREATED`` HTTP/1.1 :rfc:`7231`, Section 6.3.2 +``202`` ``ACCEPTED`` HTTP/1.1 :rfc:`7231`, Section 6.3.3 +``203`` ``NON_AUTHORITATIVE_INFORMATION`` HTTP/1.1 :rfc:`7231`, Section 6.3.4 +``204`` ``NO_CONTENT`` HTTP/1.1 :rfc:`7231`, Section 6.3.5 +``205`` ``RESET_CONTENT`` HTTP/1.1 :rfc:`7231`, Section 6.3.6 +``206`` ``PARTIAL_CONTENT`` HTTP/1.1 :rfc:`7233`, Section 4.1 +``207`` ``MULTI_STATUS`` WebDAV :rfc:`4918`, Section 11.1 +``208`` ``ALREADY_REPORTED`` WebDAV Binding Extensions :rfc:`5842`, Section 7.1 (Experimental) +``226`` ``IM_USED`` Delta Encoding in HTTP :rfc:`3229`, Section 10.4.1 +``300`` ``MULTIPLE_CHOICES`` HTTP/1.1 :rfc:`7231`, Section 6.4.1 +``301`` ``MOVED_PERMANENTLY`` HTTP/1.1 :rfc:`7231`, Section 6.4.2 +``302`` ``FOUND`` HTTP/1.1 :rfc:`7231`, Section 6.4.3 +``303`` ``SEE_OTHER`` HTTP/1.1 :rfc:`7231`, Section 6.4.4 +``304`` ``NOT_MODIFIED`` HTTP/1.1 :rfc:`7232`, Section 4.1 +``305`` ``USE_PROXY`` HTTP/1.1 :rfc:`7231`, Section 6.4.5 +``307`` ``TEMPORARY_REDIRECT`` HTTP/1.1 :rfc:`7231`, Section 6.4.7 +``308`` ``PERMANENT_REDIRECT`` Permanent Redirect :rfc:`7238`, Section 3 (Experimental) +``400`` ``BAD_REQUEST`` HTTP/1.1 :rfc:`7231`, Section 6.5.1 +``401`` ``UNAUTHORIZED`` HTTP/1.1 Authentication :rfc:`7235`, Section 3.1 +``402`` ``PAYMENT_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.2 +``403`` ``FORBIDDEN`` HTTP/1.1 :rfc:`7231`, Section 6.5.3 +``404`` ``NOT_FOUND`` HTTP/1.1 :rfc:`7231`, Section 6.5.4 +``405`` ``METHOD_NOT_ALLOWED`` HTTP/1.1 :rfc:`7231`, Section 6.5.5 +``406`` ``NOT_ACCEPTABLE`` HTTP/1.1 :rfc:`7231`, Section 6.5.6 +``407`` ``PROXY_AUTHENTICATION_REQUIRED`` HTTP/1.1 Authentication :rfc:`7235`, Section 3.2 +``408`` ``REQUEST_TIMEOUT`` HTTP/1.1 :rfc:`7231`, Section 6.5.7 +``409`` ``CONFLICT`` HTTP/1.1 :rfc:`7231`, Section 6.5.8 +``410`` ``GONE`` HTTP/1.1 :rfc:`7231`, Section 6.5.9 +``411`` ``LENGTH_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.10 +``412`` ``PRECONDITION_FAILED`` HTTP/1.1 :rfc:`7232`, Section 4.2 +``413`` ``REQUEST_ENTITY_TOO_LARGE`` HTTP/1.1 :rfc:`7231`, Section 6.5.11 +``414`` ``REQUEST_URI_TOO_LONG`` HTTP/1.1 :rfc:`7231`, Section 6.5.12 +``415`` ``UNSUPPORTED_MEDIA_TYPE`` HTTP/1.1 :rfc:`7231`, Section 6.5.13 +``416`` ``REQUEST_RANGE_NOT_SATISFIABLE`` HTTP/1.1 Range Requests :rfc:`7233`, Section 4.4 +``417`` ``EXPECTATION_FAILED`` HTTP/1.1 :rfc:`7231`, Section 6.5.14 +``422`` ``UNPROCESSABLE_ENTITY`` WebDAV :rfc:`4918`, Section 11.2 +``423`` ``LOCKED`` WebDAV :rfc:`4918`, Section 11.3 +``424`` ``FAILED_DEPENDENCY`` WebDAV :rfc:`4918`, Section 11.4 +``426`` ``UPGRADE_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.15 +``428`` ``PRECONDITION_REQUIRED`` Additional HTTP Status Codes :rfc:`6585` +``429`` ``TOO_MANY_REQUESTS`` Additional HTTP Status Codes :rfc:`6585` +``431`` ``REQUEST_HEADER_FIELDS_TOO_LARGE`` Additional HTTP Status Codes :rfc:`6585` +``500`` ``INTERNAL_SERVER_ERROR`` HTTP/1.1 :rfc:`7231`, Section 6.6.1 +``501`` ``NOT_IMPLEMENTED`` HTTP/1.1 :rfc:`7231`, Section 6.6.2 +``502`` ``BAD_GATEWAY`` HTTP/1.1 :rfc:`7231`, Section 6.6.3 +``503`` ``SERVICE_UNAVAILABLE`` HTTP/1.1 :rfc:`7231`, Section 6.6.4 +``504`` ``GATEWAY_TIMEOUT`` HTTP/1.1 :rfc:`7231`, Section 6.6.5 +``505`` ``HTTP_VERSION_NOT_SUPPORTED`` HTTP/1.1 :rfc:`7231`, Section 6.6.6 +``506`` ``VARIANT_ALSO_NEGOTIATES`` Transparent Content Negotiation in HTTP :rfc:`2295`, Section 8.1 (Experimental) +``507`` ``INSUFFICIENT_STORAGE`` WebDAV :rfc:`4918`, Section 11.5 +``508`` ``LOOP_DETECTED`` WebDAV Binding Extensions :rfc:`5842`, Section 7.2 (Experimental) +``510`` ``NOT_EXTENDED`` An HTTP Extension Framework :rfc:`2774`, Section 7 (Experimental) +``511`` ``NETWORK_AUTHENTICATION_REQUIRED`` Additional HTTP Status Codes :rfc:`6585`, Section 6 +======= =================================== ================================================================== + +In order to preserve backwards compatibility, enum values are also present +in the :mod:`http.client` module in the form of constants. The enum name is +equal to the constant name (i.e. ``http.HTTPStatus.OK`` is also available as +``http.client.OK``). diff --git a/Doc/library/http.server.rst b/Doc/library/http.server.rst index a75015555d..1c3e202609 100644 --- a/Doc/library/http.server.rst +++ b/Doc/library/http.server.rst @@ -232,7 +232,7 @@ of which this module provides three different variants: .. method:: send_response_only(code, message=None) - Sends the reponse header only, used for the purposes when ``100 + Sends the response header only, used for the purposes when ``100 Continue`` response is sent by the server to the client. The headers not buffered and sent directly the output stream.If the *message* is not specified, the HTTP message corresponding the response *code* is sent. diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index fa736fe3af..f263bab9f0 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -37,6 +37,19 @@ base class: initialized. If *host* is not specified, ``''`` (the local host) is used. If *port* is omitted, the standard IMAP4 port (143) is used. + The :class:`IMAP4` class supports the :keyword:`with` statement. When used + like this, the IMAP4 ``LOGOUT`` command is issued automatically when the + :keyword:`with` statement exits. E.g.:: + + >>> from imaplib import IMAP4 + >>> with IMAP4("domain.org") as M: + ... M.noop() + ... + ('OK', [b'Nothing Accomplished. d25if65hy903weo.87']) + + .. versionchanged:: 3.5 + Support for the :keyword:`with` statement was added. + Three exceptions are defined as attributes of the :class:`IMAP4` class: diff --git a/Doc/library/imghdr.rst b/Doc/library/imghdr.rst index 9e8952339c..c60df24c5a 100644 --- a/Doc/library/imghdr.rst +++ b/Doc/library/imghdr.rst @@ -48,6 +48,16 @@ from :func:`what`: +------------+-----------------------------------+ | ``'png'`` | Portable Network Graphics | +------------+-----------------------------------+ +| ``'webp'`` | WebP files | ++------------+-----------------------------------+ +| ``'exr'`` | OpenEXR Files | ++------------+-----------------------------------+ + +.. versionadded:: 3.5 + The *exr* format was added. + +.. versionchanged:: 3.5 + The *webp* type was added. You can extend the list of file types :mod:`imghdr` can recognize by appending to this variable: diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst index c2dbdc5cf9..55573460e1 100644 --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -203,11 +203,9 @@ file paths. value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. The ``cpython-32`` string comes from the current magic tag (see :func:`get_tag`; if :attr:`sys.implementation.cache_tag` is not defined then - :exc:`NotImplementedError` will be raised). The returned path will end in - ``.pyc`` when ``__debug__`` is ``True`` or ``.pyo`` for an optimized Python - (i.e. ``__debug__`` is ``False``). By passing in ``True`` or ``False`` for - *debug_override* you can override the system's value for ``__debug__`` for - extension selection. + :exc:`NotImplementedError` will be raised). By passing in ``True`` or + ``False`` for *debug_override* you can override the system's value for + ``__debug__``, leading to optimized bytecode. *path* need not exist. @@ -218,6 +216,9 @@ file paths. .. deprecated:: 3.4 Use :func:`importlib.util.cache_from_source` instead. + .. versionchanged:: 3.5 + The *debug_override* parameter no longer creates a ``.pyo`` file. + .. function:: source_from_cache(path) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index 91328af701..bd9c5ebc79 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -69,6 +69,10 @@ Functions An implementation of the built-in :func:`__import__` function. + .. note:: + Programmatic importing of modules should use :func:`import_module` + instead of this function. + .. function:: import_module(name, package=None) Import a module. The *name* argument specifies what module to @@ -81,12 +85,15 @@ Functions The :func:`import_module` function acts as a simplifying wrapper around :func:`importlib.__import__`. This means all semantics of the function are - derived from :func:`importlib.__import__`, including requiring the package - from which an import is occurring to have been previously imported - (i.e., *package* must already be imported). The most important difference - is that :func:`import_module` returns the specified package or module - (e.g. ``pkg.mod``), while :func:`__import__` returns the - top-level package or module (e.g. ``pkg``). + derived from :func:`importlib.__import__`. The most important difference + between these two functions is that :func:`import_module` returns the + specified package or module (e.g. ``pkg.mod``), while :func:`__import__` + returns the top-level package or module (e.g. ``pkg``). + + If you are dynamically importing a module that was created since the + interpreter began execution (e.g., created a Python source file), you may + need to call :func:`invalidate_caches` in order for the new module to be + noticed by the import system. .. versionchanged:: 3.3 Parent packages are automatically imported. @@ -347,13 +354,16 @@ ABC hierarchy:: .. method:: create_module(spec) - An optional method that returns the module object to use when - importing a module. create_module() may also return ``None``, - indicating that the default module creation should take place - instead. + A method that returns the module object to use when + importing a module. This method may return ``None``, + indicating that default module creation semantics should take place. .. versionadded:: 3.4 + .. versionchanged:: 3.5 + Starting in Python 3.6, this method will not be optional when + :meth:`exec_module` is defined. + .. method:: exec_module(module) An abstract method that executes the module in its own namespace @@ -417,7 +427,7 @@ ABC hierarchy:: .. deprecated:: 3.4 The recommended API for loading a module is :meth:`exec_module` - (and optionally :meth:`create_module`). Loaders should implement + (and :meth:`create_module`). Loaders should implement it instead of load_module(). The import machinery takes care of all the other responsibilities of load_module() when exec_module() is implemented. @@ -499,7 +509,7 @@ ABC hierarchy:: .. versionchanged:: 3.4 Raises :exc:`ImportError` instead of :exc:`NotImplementedError`. - .. method:: source_to_code(data, path='<string>') + .. staticmethod:: source_to_code(data, path='<string>') Create a code object from Python source. @@ -508,8 +518,14 @@ ABC hierarchy:: the "path" to where the source code originated from, which can be an abstract concept (e.g. location in a zip file). + With the subsequent code object one can execute it in a module by + running ``exec(code, module.__dict__)``. + .. versionadded:: 3.4 + .. versionchanged:: 3.5 + Made the method static. + .. method:: exec_module(module) Implementation of :meth:`Loader.exec_module`. @@ -695,6 +711,9 @@ find and load modules. .. versionadded:: 3.3 + .. deprecated:: 3.5 + Use :attr:`BYTECODE_SUFFIXES` instead. + .. attribute:: OPTIMIZED_BYTECODE_SUFFIXES A list of strings representing the file suffixes for optimized bytecode @@ -702,14 +721,19 @@ find and load modules. .. versionadded:: 3.3 + .. deprecated:: 3.5 + Use :attr:`BYTECODE_SUFFIXES` instead. + .. attribute:: BYTECODE_SUFFIXES A list of strings representing the recognized file suffixes for bytecode - modules. Set to either :attr:`DEBUG_BYTECODE_SUFFIXES` or - :attr:`OPTIMIZED_BYTECODE_SUFFIXES` based on whether ``__debug__`` is true. + modules (including the leading dot). .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The value is no longer dependent on ``__debug__``. + .. attribute:: EXTENSION_SUFFIXES A list of strings representing the recognized file suffixes for @@ -788,6 +812,11 @@ find and load modules. .. versionadded:: 3.4 + .. versionchanged:: 3.5 + If the current working directory -- represented by an empty string -- + is no longer valid then ``None`` is returned but no value is cached + in :data:`sys.path_importer_cache`. + .. classmethod:: find_module(fullname, path=None) A legacy wrapper around :meth:`find_spec`. @@ -1053,23 +1082,37 @@ an :term:`importer`. .. versionadded:: 3.4 -.. function:: cache_from_source(path, debug_override=None) +.. function:: cache_from_source(path, debug_override=None, *, optimization=None) - Return the :pep:`3147` path to the byte-compiled file associated with the - source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return + Return the :pep:`3147`/:pep:`488` path to the byte-compiled file associated + with the source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. The ``cpython-32`` string comes from the current magic tag (see :func:`get_tag`; if :attr:`sys.implementation.cache_tag` is not defined then - :exc:`NotImplementedError` will be raised). The returned path will end in - ``.pyc`` when ``__debug__`` is ``True`` or ``.pyo`` for an optimized Python - (i.e. ``__debug__`` is ``False``). By passing in ``True`` or ``False`` for - *debug_override* you can override the system's value for ``__debug__`` for - extension selection. - - *path* need not exist. + :exc:`NotImplementedError` will be raised). + + The *optimization* parameter is used to specify the optimization level of the + bytecode file. An empty string represents no optimization, so + ``/foo/bar/baz.py`` with an *optimization* of ``''`` will result in a + bytecode path of ``/foo/bar/__pycache__/baz.cpython-32.pyc``. ``None`` causes + the interpter's optimization level to be used. Any other value's string + representation being used, so ``/foo/bar/baz.py`` with an *optimization* of + ``2`` will lead to the bytecode path of + ``/foo/bar/__pycache__/baz.cpython-32.opt-2.pyc``. The string representation + of *optimization* can only be alphanumeric, else :exc:`ValueError` is raised. + + The *debug_override* parameter is deprecated and can be used to override + the system's value for ``__debug__``. A ``True`` value is the equivalent of + setting *optimization* to the empty string. A ``False`` value is the same as + setting *optimization* to ``1``. If both *debug_override* an *optimization* + are not ``None`` then :exc:`TypeError` is raised. .. versionadded:: 3.4 + .. versionchanged ::3.5 + The *optimization* parameter was added and the *debug_override* parameter + was deprecated. + .. function:: source_from_cache(path) @@ -1077,7 +1120,7 @@ an :term:`importer`. file path. For example, if *path* is ``/foo/bar/__pycache__/baz.cpython-32.pyc`` the returned path would be ``/foo/bar/baz.py``. *path* need not exist, however if it does not conform - to :pep:`3147` format, a ``ValueError`` is raised. If + to :pep:`3147` or :pep`488` format, a ``ValueError`` is raised. If :attr:`sys.implementation.cache_tag` is not defined, :exc:`NotImplementedError` is raised. @@ -1123,6 +1166,21 @@ an :term:`importer`. .. versionadded:: 3.4 +.. function:: module_from_spec(spec) + + Create a new module based on **spec** and ``spec.loader.create_module()``. + + If ``spec.loader.create_module()`` does not return ``None``, then any + pre-existing attributes will not be reset. Also, no :exc:`AttributeError` + will be raised if triggered while accessing **spec** or setting an attribute + on the module. + + This function is preferred over using :class:`types.ModuleType` to create a + new module as **spec** is used to set as many import-controlled attributes on + the module as possible. + + .. versionadded:: 3.5 + .. decorator:: module_for_loader A :term:`decorator` for :meth:`importlib.abc.Loader.load_module` @@ -1201,3 +1259,39 @@ an :term:`importer`. module will be file-based. .. versionadded:: 3.4 + +.. class:: LazyLoader(loader) + + A class which postpones the execution of the loader of a module until the + module has an attribute accessed. + + This class **only** works with loaders that define + :meth:`~importlib.abc.Loader.exec_module` as control over what module type + is used for the module is required. For those same reasons, the loader's + :meth:`~importlib.abc.Loader.create_module` method will be ignored (i.e., the + loader's method should only return ``None``). Finally, + modules which substitute the object placed into :attr:`sys.modules` will + not work as there is no way to properly replace the module references + throughout the interpreter safely; :exc:`ValueError` is raised if such a + substitution is detected. + + .. note:: + For projects where startup time is critical, this class allows for + potentially minimizing the cost of loading a module if it is never used. + For projects where startup time is not essential then use of this class is + **heavily** discouraged due to error messages created during loading being + postponed and thus occurring out of context. + + .. versionadded:: 3.5 + + .. classmethod:: factory(loader) + + A static method which returns a callable that creates a lazy loader. This + is meant to be used in situations where the loader is passed by class + instead of by instance. + :: + + suffixes = importlib.machinery.SOURCE_SUFFIXES + loader = importlib.machinery.SourceFileLoader + lazy_loader = importlib.util.LazyLoader.factory(loader) + finder = importlib.machinery.FileFinder(path, [(lazy_loader, suffixes)]) diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index c540b11645..471200fa78 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -159,6 +159,16 @@ attributes: | | | arguments and local | | | | variables | +-----------+-----------------+---------------------------+ +| generator | __name__ | name | ++-----------+-----------------+---------------------------+ +| | __qualname__ | qualified name | ++-----------+-----------------+---------------------------+ +| | gi_frame | frame | ++-----------+-----------------+---------------------------+ +| | gi_running | is the generator running? | ++-----------+-----------------+---------------------------+ +| | gi_code | code | ++-----------+-----------------+---------------------------+ | builtin | __doc__ | documentation string | +-----------+-----------------+---------------------------+ | | __name__ | original name of this | @@ -169,6 +179,12 @@ attributes: | | | ``None`` | +-----------+-----------------+---------------------------+ +.. versionchanged:: 3.5 + + Add ``__qualname__`` attribute to generators. The ``__name__`` attribute of + generators is now set from the function name, instead of the code name, and + it can now be modified. + .. function:: getmembers(object[, predicate]) @@ -340,6 +356,9 @@ Retrieving source code .. function:: getdoc(object) Get the documentation string for an object, cleaned up with :func:`cleandoc`. + If the documentation string for an object is not provided and the object is + a class, a method, a property or a descriptor, retrieve the documentation + string from the inheritance hierarchy. .. function:: getcomments(object) @@ -462,6 +481,9 @@ function. Signature objects are *immutable*. Use :meth:`Signature.replace` to make a modified copy. + .. versionchanged:: 3.5 + Signature objects are picklable and hashable. + .. attribute:: Signature.empty A special class-level marker to specify absence of a return annotation. @@ -506,12 +528,29 @@ function. >>> str(new_sig) "(a, b) -> 'new return anno'" + .. classmethod:: Signature.from_callable(obj) + + Return a :class:`Signature` (or its subclass) object for a given callable + ``obj``. This method simplifies subclassing of :class:`Signature`: + + :: + + class MySignature(Signature): + pass + sig = MySignature.from_callable(min) + assert isinstance(sig, MySignature) + + .. versionadded:: 3.5 + .. class:: Parameter(name, kind, \*, default=Parameter.empty, annotation=Parameter.empty) Parameter objects are *immutable*. Instead of modifying a Parameter object, you can use :meth:`Parameter.replace` to create a modified copy. + .. versionchanged:: 3.5 + Parameter objects are picklable and hashable. + .. attribute:: Parameter.empty A special class-level marker to specify absence of default values and @@ -849,11 +888,17 @@ Classes and functions The interpreter stack --------------------- -When the following functions return "frame records," each record is a tuple of -six items: the frame object, the filename, the line number of the current line, +When the following functions return "frame records," each record is a +:term:`named tuple` +``FrameInfo(frame, filename, lineno, function, code_context, index)``. +The tuple contains the frame object, the filename, the line number of the +current line, the function name, a list of lines of context from the source code, and the index of the current line within that list. +.. versionchanged:: 3.5 + Return a named tuple instead of a tuple. + .. note:: Keeping references to frame objects, as found in the first element of the frame diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 3adf6e9c33..f009c65cf0 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -339,8 +339,11 @@ I/O Base Classes if *size* is not specified). The current stream position isn't changed. This resizing can extend or reduce the current file size. In case of extension, the contents of the new file area depend on the platform - (on most systems, additional bytes are zero-filled, on Windows they're - undetermined). The new file size is returned. + (on most systems, additional bytes are zero-filled). The new file size + is returned. + + .. versionchanged:: 3.5 + Windows will now zero-fill files when extending. .. method:: writable() @@ -391,8 +394,8 @@ I/O Base Classes .. method:: readinto(b) Read up to ``len(b)`` bytes into :class:`bytearray` *b* and return the - number of bytes read. If the object is in non-blocking mode and no - bytes are available, ``None`` is returned. + number of bytes read. If the object is in non-blocking mode and no bytes + are available, ``None`` is returned. .. method:: write(b) @@ -465,9 +468,10 @@ I/O Base Classes .. method:: read1(size=-1) - Read and return up to *size* bytes, with at most one call to the underlying - raw stream's :meth:`~RawIOBase.read` method. This can be useful if you - are implementing your own buffering on top of a :class:`BufferedIOBase` + Read and return up to *size* bytes, with at most one call to the + underlying raw stream's :meth:`~RawIOBase.read` (or + :meth:`~RawIOBase.readinto`) method. This can be useful if you are + implementing your own buffering on top of a :class:`BufferedIOBase` object. .. method:: readinto(b) @@ -478,8 +482,19 @@ I/O Base Classes Like :meth:`read`, multiple reads may be issued to the underlying raw stream, unless the latter is interactive. - A :exc:`BlockingIOError` is raised if the underlying raw stream is in - non blocking-mode, and has no data available at the moment. + A :exc:`BlockingIOError` is raised if the underlying raw stream is in non + blocking-mode, and has no data available at the moment. + + .. method:: readinto1(b) + + Read up to ``len(b)`` bytes into bytearray *b*, ,using at most one call to + the underlying raw stream's :meth:`~RawIOBase.read` (or + :meth:`~RawIOBase.readinto`) method. Return the number of bytes read. + + A :exc:`BlockingIOError` is raised if the underlying raw stream is in non + blocking-mode, and has no data available at the moment. + + .. versionadded:: 3.5 .. method:: write(b) @@ -507,9 +522,12 @@ Raw File I/O The *name* can be one of two things: * a character string or :class:`bytes` object representing the path to the - file which will be opened; + file which will be opened. In this case closefd must be True (the default) + otherwise an error will be raised. * an integer representing the number of an existing OS-level file descriptor - to which the resulting :class:`FileIO` object will give access. + to which the resulting :class:`FileIO` object will give access. When the + FileIO object is closed this fd will be closed as well, unless *closefd* + is set to ``False``. The *mode* can be ``'r'``, ``'w'``, ``'x'`` or ``'a'`` for reading (default), writing, exclusive creation or appending. The file will be @@ -598,6 +616,11 @@ than raw I/O does. In :class:`BytesIO`, this is the same as :meth:`read`. + .. method:: readinto1() + + In :class:`BytesIO`, this is the same as :meth:`readinto`. + + .. versionadded:: 3.5 .. class:: BufferedReader(raw, buffer_size=DEFAULT_BUFFER_SIZE) @@ -807,11 +830,13 @@ Text I/O exception if there is an encoding error (the default of ``None`` has the same effect), or pass ``'ignore'`` to ignore errors. (Note that ignoring encoding errors can lead to data loss.) ``'replace'`` causes a replacement marker - (such as ``'?'``) to be inserted where there is malformed data. When - writing, ``'xmlcharrefreplace'`` (replace with the appropriate XML character - reference) or ``'backslashreplace'`` (replace with backslashed escape - sequences) can be used. Any other error handling name that has been - registered with :func:`codecs.register_error` is also valid. + (such as ``'?'``) to be inserted where there is malformed data. + ``'backslashreplace'`` causes malformed data to be replaced by a + backslashed escape sequence. When writing, ``'xmlcharrefreplace'`` + (replace with the appropriate XML character reference) or ``'namereplace'`` + (replace with ``\N{...}`` escape sequences) can be used. Any other error + handling name that has been registered with + :func:`codecs.register_error` is also valid. .. index:: single: universal newlines; io.TextIOWrapper class diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst index ca8798058e..7b594408c8 100644 --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -103,7 +103,7 @@ write code that handles both IP versions correctly. 1. A string in decimal-dot notation, consisting of four decimal integers in the inclusive range 0-255, separated by dots (e.g. ``192.168.0.1``). Each integer represents an octet (byte) in the address. Leading zeroes are - tolerated only for values less then 8 (as there is no ambiguity + tolerated only for values less than 8 (as there is no ambiguity between the decimal and octal interpretations of such strings). 2. An integer that fits into 32 bits. 3. An integer packed into a :class:`bytes` object of length 4 (most @@ -146,6 +146,20 @@ write code that handles both IP versions correctly. the appropriate length (most significant octet first). This is 4 bytes for IPv4 and 16 bytes for IPv6. + .. attribute:: reverse_pointer + + The name of the reverse DNS PTR record for the IP address, e.g.:: + + >>> ipaddress.ip_address("127.0.0.1").reverse_pointer + '1.0.0.127.in-addr.arpa' + >>> ipaddress.ip_address("2001:db8::1").reverse_pointer + '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa' + + This is the name that could be used for performing a PTR lookup, not the + resolved hostname itself. + + .. versionadded:: 3.5 + .. attribute:: is_multicast ``True`` if the address is reserved for multicast use. See @@ -226,6 +240,7 @@ write code that handles both IP versions correctly. :class:`IPv4Address` class: .. attribute:: packed + .. attribute:: reverse_pointer .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast @@ -377,6 +392,12 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. 3. An integer packed into a :class:`bytes` object of length 4, big-endian. The interpretation is similar to an integer *address*. + 4. A two-tuple of an address description and a netmask, where the address + description is either a string, a 32-bits integer, a 4-bytes packed + integer, or an existing IPv4Address object; and the netmask is either + an integer representing the prefix length (e.g. ``24``) or a string + representing the prefix mask (e.g. ``255.255.255.0``). + An :exc:`AddressValueError` is raised if *address* is not a valid IPv4 address. A :exc:`NetmaskValueError` is raised if the mask is not valid for an IPv4 address. @@ -389,6 +410,10 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. objects will raise :exc:`TypeError` if the argument's IP version is incompatible to ``self`` + .. versionchanged:: 3.5 + + Added the two-tuple form for the *address* constructor parameter. + .. attribute:: version .. attribute:: max_prefixlen @@ -553,6 +578,11 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. 3. An integer packed into a :class:`bytes` object of length 16, bit-endian. The interpretation is similar to an integer *address*. + 4. A two-tuple of an address description and a netmask, where the address + description is either a string, a 128-bits integer, a 16-bytes packed + integer, or an existing IPv4Address object; and the netmask is an + integer representing the prefix length. + An :exc:`AddressValueError` is raised if *address* is not a valid IPv6 address. A :exc:`NetmaskValueError` is raised if the mask is not valid for an IPv6 address. @@ -561,6 +591,10 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. then :exc:`ValueError` is raised. Otherwise, the host bits are masked out to determine the appropriate network address. + .. versionchanged:: 3.5 + + Added the two-tuple form for the *address* constructor parameter. + .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index f489535c53..8c7592d17e 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -87,10 +87,15 @@ loops that truncate the stream. .. function:: accumulate(iterable[, func]) - Make an iterator that returns accumulated sums. Elements may be any addable - type including :class:`~decimal.Decimal` or :class:`~fractions.Fraction`. - If the optional *func* argument is supplied, it should be a function of two - arguments and it will be used instead of addition. + Make an iterator that returns accumulated sums, or accumulated + results of other binary functions (specified via the optional + *func* argument). If *func* is supplied, it should be a function + of two arguments. Elements of the input *iterable* may be any type + that can be accepted as arguments to *func*. (For example, with + the default operation of addition, elements may be any addable + type including :class:`~decimal.Decimal` or + :class:`~fractions.Fraction`.) If the input iterable is empty, the + output iterable will also be empty. Equivalent to:: @@ -99,7 +104,10 @@ loops that truncate the stream. # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 it = iter(iterable) - total = next(it) + try: + total = next(it) + except StopIteration: + return yield total for element in it: total = func(total, element) @@ -400,7 +408,10 @@ loops that truncate the stream. def _grouper(self, tgtkey): while self.currkey == tgtkey: yield self.currvalue - self.currvalue = next(self.it) # Exit on StopIteration + try: + self.currvalue = next(self.it) + except StopIteration: + return self.currkey = self.keyfunc(self.currvalue) @@ -424,7 +435,10 @@ loops that truncate the stream. # islice('ABCDEFG', 0, None, 2) --> A C E G s = slice(*args) it = iter(range(s.start or 0, s.stop or sys.maxsize, s.step or 1)) - nexti = next(it) + try: + nexti = next(it) + except StopIteration: + return for i, element in enumerate(iterable): if i == nexti: yield element @@ -582,7 +596,10 @@ loops that truncate the stream. def gen(mydeque): while True: if not mydeque: # when the local deque is empty - newval = next(it) # fetch a new value and + try: + newval = next(it) # fetch a new value and + except StopIteration: + return for d in deques: # load it to all the deques d.append(newval) yield mydeque.popleft() @@ -657,6 +674,11 @@ which incur interpreter overhead. "Return function(0), function(1), ..." return map(function, count(start)) + def tail(n, iterable): + "Return an iterator over the last n items" + # tail(3, 'ABCDEFG') --> E F G + return iter(collections.deque(iterable, maxlen=n)) + def consume(iterator, n): "Advance the iterator n-steps ahead. If n is none, consume entirely." # Use functions that consume iterators at C speed. diff --git a/Doc/library/json.rst b/Doc/library/json.rst index 6f5f8b15a8..49bb0905e0 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -106,6 +106,8 @@ Using json.tool from the shell to validate and pretty-print:: $ echo '{1.2:3.4}' | python -m json.tool Expecting property name enclosed in double quotes: line 1 column 2 (char 1) +See :ref:`json-commandline` for detailed documentation. + .. highlight:: python3 .. note:: @@ -248,7 +250,7 @@ Basic Usage will be passed to the constructor of the class. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. .. function:: loads(s, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) @@ -259,7 +261,7 @@ Basic Usage *encoding* which is ignored and deprecated. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. Encoders and Decoders --------------------- @@ -332,13 +334,16 @@ Encoders and Decoders ``'\n'``, ``'\r'`` and ``'\0'``. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. .. method:: decode(s) Return the Python representation of *s* (a :class:`str` instance containing a JSON document) + :exc:`JSONDecodeError` will be raised if the given JSON document is not + valid. + .. method:: raw_decode(s) Decode a JSON document from *s* (a :class:`str` beginning with a @@ -467,6 +472,36 @@ Encoders and Decoders mysocket.write(chunk) +Exceptions +---------- + +.. exception:: JSONDecodeError(msg, doc, pos, end=None) + + Subclass of :exc:`ValueError` with the following additional attributes: + + .. attribute:: msg + + The unformatted error message. + + .. attribute:: doc + + The JSON document being parsed. + + .. attribute:: pos + + The start index of *doc* where parsing failed. + + .. attribute:: lineno + + The line corresponding to *pos*. + + .. attribute:: colno + + The column corresponding to *pos*. + + .. versionadded:: 3.5 + + Standard Compliance and Interoperability ---------------------------------------- @@ -588,6 +623,68 @@ when serializing Python :class:`int` values of extremely large magnitude, or when serializing instances of "exotic" numerical types such as :class:`decimal.Decimal`. +.. highlight:: bash +.. module:: json.tool + +.. _json-commandline: + +Command Line Interface +---------------------- + +The :mod:`json.tool` module provides a simple command line interface to validate +and pretty-print JSON objects. + +If the optional ``infile`` and ``outfile`` arguments are not +specified, :attr:`sys.stdin` and :attr:`sys.stdout` will be used respectively:: + + $ echo '{"json": "obj"}' | python -m json.tool + { + "json": "obj" + } + $ echo '{1.2:3.4}' | python -m json.tool + Expecting property name enclosed in double quotes: line 1 column 2 (char 1) + +.. versionchanged:: 3.5 + The output is now in the same order as the input. Use the + :option:`--sort-keys` option to sort the output of dictionaries + alphabetically by key. + +Command line options +^^^^^^^^^^^^^^^^^^^^ + +.. cmdoption:: infile + + The JSON file to be validated or pretty-printed:: + + $ python -m json.tool mp_films.json + [ + { + "title": "And Now for Something Completely Different", + "year": 1971 + }, + { + "title": "Monty Python and the Holy Grail", + "year": 1975 + } + ] + + If *infile* is not specified, read from :attr:`sys.stdin`. + +.. cmdoption:: outfile + + Write the output of the *infile* to the given *outfile*. Otherwise, write it + to :attr:`sys.stdout`. + +.. cmdoption:: --sort-keys + + Sort the output of dictionaries alphabetically by key. + + .. versionadded:: 3.5 + +.. cmdoption:: -h, --help + + Show the help message. + .. rubric:: Footnotes diff --git a/Doc/library/linecache.rst b/Doc/library/linecache.rst index f18b1cdac0..6c92cc58d2 100644 --- a/Doc/library/linecache.rst +++ b/Doc/library/linecache.rst @@ -47,6 +47,14 @@ The :mod:`linecache` module defines the following functions: changed on disk, and you require the updated version. If *filename* is omitted, it will check all the entries in the cache. +.. function:: lazycache(filename, module_globals) + + Capture enough detail about a non-file based module to permit getting its + lines later via :func:`getline` even if *module_globals* is None in the later + call. This avoids doing I/O until a line is actually needed, without having + to carry the module globals around indefinitely. + + .. versionadded:: 3.5 Example:: diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst index 9600193547..bc7f5f932b 100644 --- a/Doc/library/locale.rst +++ b/Doc/library/locale.rst @@ -387,6 +387,14 @@ The :mod:`locale` module defines the following exception and functions: ``str(float)``, but takes the decimal point into account. +.. function:: delocalize(string) + + Converts a string into a normalized number string, following the + :const:`LC_NUMERIC` settings. + + .. versionadded:: 3.5 + + .. function:: atof(string) Converts a string to a floating point number, following the :const:`LC_NUMERIC` diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst index 63aba1e194..67403a9bc6 100644 --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -436,7 +436,7 @@ sends logging output to a network socket. The base class uses a TCP socket. .. method:: createSocket() Tries to create a socket; on failure, uses an exponential back-off - algorithm. On intial failure, the handler will drop the message it was + algorithm. On initial failure, the handler will drop the message it was trying to send. When subsequent messages are handled by the same instance, it will not try connecting until some time has passed. The default parameters are such that the initial delay is one second, and if @@ -853,7 +853,7 @@ supports sending logging messages to a Web server, using either ``GET`` or credentials, you should also specify secure=True so that your userid and password are not passed in cleartext across the wire. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 The *context* parameter was added. .. method:: mapLogRecord(record) @@ -953,13 +953,20 @@ applications where threads servicing clients need to respond as quickly as possible, while any potentially slow operations (such as sending an email via :class:`SMTPHandler`) are done on a separate thread. -.. class:: QueueListener(queue, *handlers) +.. class:: QueueListener(queue, *handlers, respect_handler_level=False) Returns a new instance of the :class:`QueueListener` class. The instance is initialized with the queue to send messages to and a list of handlers which will handle entries placed on the queue. The queue can be any queue- like object; it's passed as-is to the :meth:`dequeue` method, which needs - to know how to get messages from it. + to know how to get messages from it. If ``respect_handler_level`` is ``True``, + a handler's level is respected (compared with the level for the message) when + deciding whether to pass messages to that handler; otherwise, the behaviour + is as in previous Python versions - to always pass each message to each + handler. + + .. versionchanged:: 3.5 + The ``respect_handler_levels`` argument was added. .. method:: dequeue(block) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 96861a7b9e..7915c277df 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -159,11 +159,13 @@ is the module's name in the Python package namespace. *msg* using the string formatting operator. (Note that this means that you can use keywords in the format string, together with a single dictionary argument.) - There are three keyword arguments in *kwargs* which are inspected: *exc_info* - which, if it does not evaluate as false, causes exception information to be + There are three keyword arguments in *kwargs* which are inspected: + *exc_info*, *stack_info*, and *extra*. + + If *exc_info* does not evaluate as false, it causes exception information to be added to the logging message. If an exception tuple (in the format returned by - :func:`sys.exc_info`) is provided, it is used; otherwise, :func:`sys.exc_info` - is called to get the exception information. + :func:`sys.exc_info`) or an exception instance is provided, it is used; + otherwise, :func:`sys.exc_info` is called to get the exception information. The second optional keyword argument is *stack_info*, which defaults to ``False``. If true, stack information is added to the logging @@ -220,6 +222,9 @@ is the module's name in the Python package namespace. .. versionadded:: 3.2 The *stack_info* parameter was added. + .. versionchanged:: 3.5 + The *exc_info* parameter can now accept exception instances. + .. method:: Logger.info(msg, *args, **kwargs) diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst index b71051da85..054600530b 100644 --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -110,6 +110,10 @@ Reading and writing compressed files .. versionchanged:: 3.4 Added support for the ``"x"`` and ``"xb"`` modes. + .. versionchanged:: 3.5 + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + Compressing and decompressing data in memory -------------------------------------------- @@ -221,13 +225,32 @@ Compressing and decompressing data in memory decompress a multi-stream input with :class:`LZMADecompressor`, you must create a new decompressor for each stream. - .. method:: decompress(data) + .. method:: decompress(data, max_length=-1) + + Decompress *data* (a :term:`bytes-like object`), returning + uncompressed data as bytes. Some of *data* may be buffered + internally, for use in later calls to :meth:`decompress`. The + returned data should be concatenated with the output of any + previous calls to :meth:`decompress`. + + If *max_length* is nonnegative, returns at most *max_length* + bytes of decompressed data. If this limit is reached and further + output can be produced, the :attr:`~.needs_input` attribute will + be set to ``False``. In this case, the next call to + :meth:`~.decompress` may provide *data* as ``b''`` to obtain + more of the output. + + If all of the input data was decompressed and returned (either + because this was less than *max_length* bytes, or because + *max_length* was negative), the :attr:`~.needs_input` attribute + will be set to ``True``. - Decompress *data* (a :class:`bytes` object), returning a :class:`bytes` - object containing the decompressed data for at least part of the input. - Some of *data* may be buffered internally, for use in later calls to - :meth:`decompress`. The returned data should be concatenated with the - output of any previous calls to :meth:`decompress`. + Attempting to decompress data after the end of stream is reached + raises an `EOFError`. Any data found after the end of the + stream is ignored and saved in the :attr:`~.unused_data` attribute. + + .. versionchanged:: 3.5 + Added the *max_length* parameter. .. attribute:: check @@ -245,6 +268,12 @@ Compressing and decompressing data in memory Before the end of the stream is reached, this will be ``b""``. + .. attribute:: needs_input + + ``False`` if the :meth:`.decompress` method can provide more + decompressed data before requiring new uncompressed input. + + .. versionadded:: 3.5 .. function:: compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None) diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 3c41672095..eda0056fc9 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -383,6 +383,22 @@ Constants The mathematical constant e = 2.718281..., to available precision. +.. data:: inf + + A floating-point positive infinity. (For negative infinity, use + ``-math.inf``.) Equivalent to the output of ``float('inf')``. + + .. versionadded:: 3.5 + + +.. data:: nan + + A floating-point "not a number" (NaN) value. Equivalent to the output of + ``float('nan')``. + + .. versionadded:: 3.5 + + .. impl-detail:: The :mod:`math` module consists mostly of thin wrappers around the platform C diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index 18e05e31f7..b74a8231ea 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -174,6 +174,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length Optional arguments *start* and *end* are interpreted as in slice notation. Returns ``-1`` on failure. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: flush([offset[, size]]) @@ -234,6 +237,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length Optional arguments *start* and *end* are interpreted as in slice notation. Returns ``-1`` on failure. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: seek(pos[, whence]) @@ -261,6 +267,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length were written. If the mmap was created with :const:`ACCESS_READ`, then writing to it will raise a :exc:`TypeError` exception. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: write_byte(byte) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 4d5f30841e..8703f8f5ec 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -1352,6 +1352,9 @@ processes. Note that accessing the ctypes object through the wrapper can be a lot slower than accessing the raw ctypes object. + .. versionchanged:: 3.5 + Synchronized objects support the :term:`context manager` protocol. + The table below compares the syntax for creating shared ctypes objects from shared memory with the normal ctypes syntax. (In the table ``MyStruct`` is some @@ -1839,9 +1842,9 @@ itself. This means, for example, that one shared object can contain a second: >>> l = manager.list(range(10)) >>> l._callmethod('__len__') 10 - >>> l._callmethod('__getitem__', (slice(2, 7),)) # equiv to `l[2:7]` + >>> l._callmethod('__getitem__', (slice(2, 7),)) # equivalent to l[2:7] [2, 3, 4, 5, 6] - >>> l._callmethod('__getitem__', (20,)) # equiv to `l[20]` + >>> l._callmethod('__getitem__', (20,)) # equivalent to l[20] Traceback (most recent call last): ... IndexError: list index out of range diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst index f9e2a3d003..c01e63b77a 100644 --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -138,6 +138,14 @@ The mathematical and bitwise operations are the most numerous: Return ``a * b``, for *a* and *b* numbers. +.. function:: matmul(a, b) + __matmul__(a, b) + + Return ``a @ b``. + + .. versionadded:: 3.5 + + .. function:: neg(obj) __neg__(obj) @@ -391,6 +399,8 @@ Python syntax and the functions in the :mod:`operator` module. +-----------------------+-------------------------+---------------------------------------+ | Multiplication | ``a * b`` | ``mul(a, b)`` | +-----------------------+-------------------------+---------------------------------------+ +| Matrix Multiplication | ``a @ b`` | ``matmul(a, b)`` | ++-----------------------+-------------------------+---------------------------------------+ | Negation (Arithmetic) | ``- a`` | ``neg(a)`` | +-----------------------+-------------------------+---------------------------------------+ | Negation (Logical) | ``not a`` | ``not_(a)`` | @@ -499,6 +509,14 @@ will perform the update, so no subsequent assignment is necessary: ``a = imul(a, b)`` is equivalent to ``a *= b``. +.. function:: imatmul(a, b) + __imatmul__(a, b) + + ``a = imatmul(a, b)`` is equivalent to ``a @= b``. + + .. versionadded:: 3.5 + + .. function:: ior(a, b) __ior__(a, b) diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index 92631b2e41..e4fe44ed94 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -66,11 +66,24 @@ the :mod:`glob` module.) empty string (``''``). +.. function:: commonpath(paths) + + Return the longest common sub-path of each pathname in the sequence + *paths*. Raise ValueError if *paths* contains both absolute and relative + pathnames, or if *paths* is empty. Unlike :func:`commonprefix`, this + returns a valid path. + + Availability: Unix, Windows + + .. versionadded:: 3.5 + + .. function:: commonprefix(list) - Return the longest path prefix (taken character-by-character) that is a prefix - of all paths in *list*. If *list* is empty, return the empty string (``''``). - Note that this may return invalid paths because it works a character at a time. + Return the longest path prefix (taken character-by-character) that is a + prefix of all paths in *list*. If *list* is empty, return the empty string + (``''``). Note that this may return invalid paths because it works a + character at a time. To obtain a valid path, see :func:`commonpath`. .. function:: dirname(path) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 41efccd980..bd09937aeb 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -78,9 +78,10 @@ uses the file system encoding to perform this conversion (see .. versionchanged:: 3.1 On some systems, conversion using the file system encoding may fail. In this - case, Python uses the ``surrogateescape`` encoding error handler, which means - that undecodable bytes are replaced by a Unicode character U+DCxx on - decoding, and these are again translated to the original byte on encoding. + case, Python uses the :ref:`surrogateescape encoding error handler + <surrogateescape>`, which means that undecodable bytes are replaced by a + Unicode character U+DCxx on decoding, and these are again translated to the + original byte on encoding. The file system encoding must guarantee to successfully decode all bytes @@ -804,8 +805,21 @@ as internal buffering of data. most *length* bytes in size. As of Python 3.3, this is equivalent to ``os.truncate(fd, length)``. + Availability: Unix, Windows. + + .. versionchanged:: 3.5 + Added support for Windows + +.. function:: get_blocking(fd) + + Get the blocking mode of the file descriptor: ``False`` if the + :data:`O_NONBLOCK` flag is set, ``True`` if the flag is cleared. + + See also :func:`set_blocking` and :meth:`socket.socket.setblocking`. + Availability: Unix. + .. versionadded:: 3.5 .. function:: isatty(fd) @@ -893,6 +907,11 @@ as internal buffering of data. .. versionadded:: 3.3 The *dir_fd* argument. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + The following constants are options for the *flags* parameter to the :func:`~os.open` function. They can be combined using the bitwise OR operator ``|``. Some of them are not available on all platforms. For descriptions of @@ -1070,6 +1089,11 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window :func:`popen` or :func:`fdopen`, or :data:`sys.stdin`, use its :meth:`~file.read` or :meth:`~file.readline` methods. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: sendfile(out, in, offset, nbytes) sendfile(out, in, offset, nbytes, headers=None, trailers=None, flags=0) @@ -1107,6 +1131,18 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window .. versionadded:: 3.3 +.. function:: set_blocking(fd, blocking) + + Set the blocking mode of the specified file descriptor. Set the + :data:`O_NONBLOCK` flag if blocking is ``False``, clear the flag otherwise. + + See also :func:`get_blocking` and :meth:`socket.socket.setblocking`. + + Availability: Unix. + + .. versionadded:: 3.5 + + .. data:: SF_NODISKIO SF_MNOWAIT SF_SYNC @@ -1173,6 +1209,11 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window :func:`fdopen`, or :data:`sys.stdout` or :data:`sys.stderr`, use its :meth:`~file.write` method. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: writev(fd, buffers) @@ -1577,6 +1618,11 @@ features: Availability: Unix, Windows. + .. seealso:: + + The :func:`scandir` function returns the directory entries with more + information than just the name. + .. versionchanged:: 3.2 The *path* parameter became optional. @@ -1869,6 +1915,178 @@ features: The *dir_fd* parameter. +.. function:: scandir(path='.') + + Return an iterator of :class:`DirEntry` objects corresponding to the entries + in the directory given by *path*. The entries are yielded in arbitrary + order, and the special entries ``'.'`` and ``'..'`` are not included. + + On Windows, *path* must of type :class:`str`. On POSIX, *path* can be of + type :class:`str` or :class:`bytes`. If *path* is of type :class:`bytes`, + the :attr:`~DirEntry.name` and :attr:`~DirEntry.path` attributes of + :class:`DirEntry` are also of type ``bytes``. Use :func:`~os.fsencode` and + :func:`~os.fsdecode` to encode and decode paths. + + The :func:`scandir` function is recommended, instead of :func:`listdir`, + when the file type of entries is used. In most cases, the file type of a + :class:`DirEntry` is retrieved directly by :func:`scandir`, no system call + is required. If only the name of entries is used, :func:`listdir` can + be more efficient than :func:`scandir`. + + The following example shows a simple use of :func:`scandir` to display all + the files excluding directories in the given *path* that don't start with + ``'.'``:: + + for entry in os.scandir(path): + if not entry.name.startswith('.') and entry.is_file(): + print(entry.name) + + .. note:: + + On Unix-based systems, :func:`scandir` uses the system's + `opendir() <http://pubs.opengroup.org/onlinepubs/009695399/functions/opendir.html>`_ + and + `readdir() <http://pubs.opengroup.org/onlinepubs/009695399/functions/readdir_r.html>`_ + functions. On Windows, it uses the Win32 + `FindFirstFileW <http://msdn.microsoft.com/en-us/library/windows/desktop/aa364418(v=vs.85).aspx>`_ + and + `FindNextFileW <http://msdn.microsoft.com/en-us/library/windows/desktop/aa364428(v=vs.85).aspx>`_ + functions. + + .. seealso:: + + The :func:`listdir` function returns the names of the directory entries. + + .. versionadded:: 3.5 + + +.. class:: DirEntry + + Object yielded by :func:`scandir` to expose the file path and other file + attributes of a directory entry. + + :func:`scandir` will provide as much of this information as possible without + making additional system calls. When a ``stat()`` or ``lstat()`` system call + is made, the ``DirEntry`` object cache the result . + + ``DirEntry`` instances are not intended to be stored in long-lived data + structures; if you know the file metadata has changed or if a long time has + elapsed since calling :func:`scandir`, call ``os.stat(entry.path)`` to fetch + up-to-date information. + + Because the ``DirEntry`` methods can make operating system calls, they may + also raise :exc:`OSError`. For example, if a file is deleted between calling + :func:`scandir` and calling :func:`DirEntry.stat`, a + :exc:`FileNotFoundError` exception can be raised. Unfortunately, the + behaviour on errors depends on the platform. If you need very fine-grained + control over errors, you can catch :exc:`OSError` when calling one of the + ``DirEntry`` methods and handle as appropriate. + + Attributes and methods on a ``DirEntry`` instance are as follows: + + .. attribute:: name + + The entry's base filename, relative to the :func:`scandir` *path* + argument. + + The :attr:`name` type is :class:`str`. On POSIX, it can be of type + :class:`bytes` if the type of the :func:`scandir` *path* argument is also + :class:`bytes`. Use :func:`~os.fsdecode` to decode the name. + + .. attribute:: path + + The entry's full path name: equivalent to ``os.path.join(scandir_path, + entry.name)`` where *scandir_path* is the :func:`scandir` *path* + argument. The path is only absolute if the :func:`scandir` *path* + argument is absolute. + + The :attr:`name` type is :class:`str`. On POSIX, it can be of type + :class:`bytes` if the type of the :func:`scandir` *path* argument is also + :class:`bytes`. Use :func:`~os.fsdecode` to decode the path. + + .. method:: inode() + + Return the inode number of the entry. + + The result is cached in the object, use ``os.stat(entry.path, + follow_symlinks=False).st_ino`` to fetch up-to-date information. + + On POSIX, no system call is required. + + .. method:: is_dir(\*, follow_symlinks=True) + + If *follow_symlinks* is ``True`` (the default), return ``True`` if the + entry is a directory or a symbolic link pointing to a directory, + return ``False`` if it points to another kind of file, if it doesn't + exist anymore or if it is a broken symbolic link. + + If *follow_symlinks* is ``False``, return ``True`` only if this entry + is a directory, return ``False`` if it points to a symbolic link or + another kind of file, if the entry doesn't exist anymore or if it is a + broken symbolic link + + The result is cached in the object. Call :func:`stat.S_ISDIR` with + :func:`os.stat` to fetch up-to-date information. + + The method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is catched. + + In most cases, no system call is required. + + .. method:: is_file(\*, follow_symlinks=True) + + If *follow_symlinks* is ``True`` (the default), return ``True`` if the + entry is a regular file or a symbolic link pointing to a regular file, + return ``False`` if it points to another kind of file, if it doesn't + exist anymore or if it is a broken symbolic link. + + If *follow_symlinks* is ``False``, return ``True`` only if this entry + is a regular file, return ``False`` if it points to a symbolic link or + another kind of file, if it doesn't exist anymore or if it is a broken + symbolic link. + + The result is cached in the object. Call :func:`stat.S_ISREG` with + :func:`os.stat` to fetch up-to-date information. + + The method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is catched. + + In most cases, no system call is required. + + .. method:: is_symlink() + + Return ``True`` if this entry is a symbolic link or a broken symbolic + link, return ``False`` if it points to a another kind of file or if the + entry doesn't exist anymore. + + The result is cached in the object. Call :func:`os.path.islink` to fetch + up-to-date information. + + The method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is catched. + + In most cases, no system call is required. + + .. method:: stat(\*, follow_symlinks=True) + + Return a :class:`stat_result` object for this entry. This function + normally follows symbolic links; to stat a symbolic link add the + argument ``follow_symlinks=False``. + + On Windows, the ``st_ino``, ``st_dev`` and ``st_nlink`` attributes of the + :class:`stat_result` are always set to zero. Call :func:`os.stat` to + get these attributes. + + The result is cached in the object. Call :func:`os.stat` to fetch + up-to-date information. + + On Windows, ``DirEntry.stat(follow_symlinks=False)`` doesn't require a + system call. ``DirEntry.stat()`` requires a system call if the entry is a + symbolic link. + + .. versionadded:: 3.5 + + .. function:: stat(path, \*, dir_fd=None, follow_symlinks=True) Get the status of a file or a file descriptor. Perform the equivalent of a @@ -2044,6 +2262,15 @@ features: File type. + On Windows systems, the following attribute is also available: + + .. attribute:: st_file_attributes + + Windows file attributes: ``dwFileAttributes`` member of the + ``BY_HANDLE_FILE_INFORMATION`` structure returned by + :c:func:`GetFileInformationByHandle`. See the ``FILE_ATTRIBUTE_*`` + constants in the :mod:`stat` module. + The standard module :mod:`stat` defines functions and constants that are useful for extracting information from a :c:type:`stat` structure. (On Windows, some items are filled with dummy values.) @@ -2061,6 +2288,9 @@ features: Added the :attr:`st_atime_ns`, :attr:`st_mtime_ns`, and :attr:`st_ctime_ns` members. + .. versionadded:: 3.5 + Added the :attr:`st_file_attributes` member on Windows. + .. function:: stat_float_times([newvalue]) @@ -2264,10 +2494,12 @@ features: This function can support :ref:`specifying a file descriptor <path_fd>`. - Availability: Unix. + Availability: Unix, Windows. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + Added support for Windows .. function:: unlink(path, *, dir_fd=None) @@ -2406,6 +2638,11 @@ features: for name in dirs: os.rmdir(os.path.join(root, name)) + .. versionchanged:: 3.5 + The function now calls :func:`os.scandir` instead of :func:`os.listdir`. + The usage of :func:`os.scandir` reduces the number of calls to + :func:`os.stat`. + .. function:: fwalk(top='.', topdown=True, onerror=None, *, follow_symlinks=False, dir_fd=None) @@ -2990,6 +3227,10 @@ written in Python, such as a mail server's external command delivery program. doesn't work if it is. Use the :func:`os.path.normpath` function to ensure that the path is properly encoded for Win32. + To reduce interpreter startup overhead, the Win32 :c:func:`ShellExecute` + function is not resolved until this function is first called. If the function + cannot be resolved, :exc:`NotImplementedError` will be raised. + Availability: Windows. @@ -3137,6 +3378,11 @@ written in Python, such as a mail server's external command delivery program. id is known, not necessarily a child process. The :func:`spawn\* <spawnl>` functions called with :const:`P_NOWAIT` return suitable process handles. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: wait3(options) diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst index bb5081afde..c60d596506 100644 --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -148,21 +148,30 @@ and (read-only) attributes: .. method:: oss_audio_device.write(data) - Write the Python string *data* to the audio device and return the number of - bytes written. If the audio device is in blocking mode (the default), the - entire string is always written (again, this is different from usual Unix device - semantics). If the device is in non-blocking mode, some data may not be written + Write a :term:`bytes-like object` *data* to the audio device and return the + number of bytes written. If the audio device is in blocking mode (the + default), the entire data is always written (again, this is different from + usual Unix device semantics). If the device is in non-blocking mode, some + data may not be written ---see :meth:`writeall`. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: oss_audio_device.writeall(data) - Write the entire Python string *data* to the audio device: waits until the audio - device is able to accept data, writes as much data as it will accept, and - repeats until *data* has been completely written. If the device is in blocking - mode (the default), this has the same effect as :meth:`write`; :meth:`writeall` - is only useful in non-blocking mode. Has no return value, since the amount of - data written is always equal to the amount of data supplied. + Write a :term:`bytes-like object` *data* to the audio device: waits until + the audio device is able to accept data, writes as much data as it will + accept, and repeats until *data* has been completely written. If the device + is in blocking mode (the default), this has the same effect as + :meth:`write`; :meth:`writeall` is only useful in non-blocking mode. Has + no return value, since the amount of data written is always equal to the + amount of data supplied. + + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. versionchanged:: 3.2 Audio device objects also support the context management protocol, i.e. they can diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 24e2a308df..2f0654440d 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -628,6 +628,17 @@ call fails (for example because the path doesn't exist): PosixPath('/home/antoine/pathlib') +.. classmethod:: Path.home() + + Return a new path object representing the user's home directory (as + returned by :func:`os.path.expanduser` with ``~`` construct):: + + >>> Path.home() + PosixPath('/home/antoine') + + .. versionadded:: 3.5 + + .. method:: Path.stat() Return information about this path (similarly to :func:`os.stat`). @@ -670,6 +681,18 @@ call fails (for example because the path doesn't exist): symlink *points to* an existing file or directory. +.. method:: Path.expanduser() + + Return a new path with expanded ``~`` and ``~user`` constructs, + as returned by :meth:`os.path.expanduser`:: + + >>> p = PosixPath('~/films/Monty Python') + >>> p.expanduser() + PosixPath('/home/eric/films/Monty Python') + + .. versionadded:: 3.5 + + .. method:: Path.glob(pattern) Glob the given *pattern* in the directory represented by this path, @@ -791,7 +814,7 @@ call fails (for example because the path doesn't exist): the symbolic link's information rather than its target's. -.. method:: Path.mkdir(mode=0o777, parents=False) +.. method:: Path.mkdir(mode=0o777, parents=False, exist_ok=False) Create a new directory at this given path. If *mode* is given, it is combined with the process' ``umask`` value to determine the file mode @@ -805,6 +828,16 @@ call fails (for example because the path doesn't exist): If *parents* is false (the default), a missing parent raises :exc:`FileNotFoundError`. + If *exist_ok* is false (the default), an :exc:`FileExistsError` is + raised if the target directory already exists. + + If *exist_ok* is true, :exc:`FileExistsError` exceptions will be + ignored (same behavior as the POSIX ``mkdir -p`` command), but only if the + last path component is not an existing non-directory file. + + .. versionchanged:: 3.5 + The *exist_ok* parameter was added. + .. method:: Path.open(mode='r', buffering=-1, encoding=None, errors=None, newline=None) @@ -824,6 +857,34 @@ call fails (for example because the path doesn't exist): if the file's uid isn't found in the system database. +.. method:: Path.read_bytes() + + Return the binary contents of the pointed-to file as a bytes object:: + + >>> p = Path('my_binary_file') + >>> p.write_bytes(b'Binary file contents') + 20 + >>> p.read_bytes() + b'Binary file contents' + + .. versionadded:: 3.5 + + +.. method:: Path.read_text(encoding=None, errors=None) + + Return the decoded contents of the pointed-to file as a string:: + + >>> p = Path('my_text_file') + >>> p.write_text('Text file contents') + 18 + >>> p.read_text() + 'Text file contents' + + The optional parameters have the same meaning as in :func:`open`. + + .. versionadded:: 3.5 + + .. method:: Path.rename(target) Rename this file or directory to the given *target*. *target* can be @@ -884,6 +945,25 @@ call fails (for example because the path doesn't exist): Remove this directory. The directory must be empty. +.. method:: Path.samefile(other_path) + + Return whether this path points to the same file as *other_path*, which + can be either a Path object, or a string. The semantics are similar + to :func:`os.path.samefile` and :func:`os.path.samestat`. + + An :exc:`OSError` can be raised if either file cannot be accessed for some + reason. + + >>> p = Path('spam') + >>> q = Path('eggs') + >>> p.samefile(q) + False + >>> p.samefile('spam') + True + + .. versionadded:: 3.5 + + .. method:: Path.symlink_to(target, target_is_directory=False) Make this path a symbolic link to *target*. Under Windows, @@ -917,3 +997,33 @@ call fails (for example because the path doesn't exist): Remove this file or symbolic link. If the path points to a directory, use :func:`Path.rmdir` instead. + + +.. method:: Path.write_bytes(data) + + Open the file pointed to in bytes mode, write *data* to it, and close the + file:: + + >>> p = Path('my_binary_file') + >>> p.write_bytes(b'Binary file contents') + 20 + >>> p.read_bytes() + b'Binary file contents' + + An existing file of the same name is overwritten. + + .. versionadded:: 3.5 + + +.. method:: Path.write_text(data, encoding=None, errors=None) + + Open the file pointed to in text mode, write *data* to it, and close the + file:: + + >>> p = Path('my_text_file') + >>> p.write_text('Text file contents') + 18 + >>> p.read_text() + 'Text file contents' + + .. versionadded:: 3.5 diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index 47356f9f0c..4ce4d345af 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -859,7 +859,7 @@ For the simplest code, use the :func:`dump` and :func:`load` functions. :: data = { 'a': [1, 2.0, 3, 4+6j], 'b': ("character string", b"byte string"), - 'c': set([None, True, False]) + 'c': {None, True, False} } with open('data.pickle', 'wb') as f: diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 13ea7b9b33..5d3295db7e 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -58,7 +58,7 @@ support. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism - is now fully PEP 302 compliant and available in :mod:`importlib` + is now fully PEP 302 compliant and available in :mod:`importlib`. .. class:: ImpLoader(fullname, file, filename, etc) @@ -67,7 +67,7 @@ support. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism - is now fully PEP 302 compliant and available in :mod:`importlib` + is now fully PEP 302 compliant and available in :mod:`importlib`. .. function:: find_loader(fullname) diff --git a/Doc/library/plistlib.rst b/Doc/library/plistlib.rst index 2bef120f1d..416559114b 100644 --- a/Doc/library/plistlib.rst +++ b/Doc/library/plistlib.rst @@ -129,7 +129,7 @@ The following functions are deprecated: and binary) file object. Returns the unpacked root object (which usually is a dictionary). - This function calls :func:`load` to do the actual work, the the documentation + This function calls :func:`load` to do the actual work, see the documentation of :func:`that function <load>` for an explanation of the keyword arguments. .. note:: diff --git a/Doc/library/pprint.rst b/Doc/library/pprint.rst index c0589a31a9..0b44dc865e 100644 --- a/Doc/library/pprint.rst +++ b/Doc/library/pprint.rst @@ -235,10 +235,10 @@ In its basic form, :func:`pprint` shows the whole object:: 'classifiers': ['Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 2 :: Only'], - 'description': 'An extensible framework for Python programming, ' - 'with special focus\r\n' - 'on event-based network programming and ' - 'multiprotocol integration.', + 'description': 'An extensible framework for Python programming, with ' + 'special focus\r\n' + 'on event-based network programming and multiprotocol ' + 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', 'home_page': 'http://twistedmatrix.com/', @@ -288,10 +288,10 @@ contents):: 'cheesecake_documentation_id': None, 'cheesecake_installability_id': None, 'classifiers': [...], - 'description': 'An extensible framework for Python programming, ' - 'with special focus\r\n' - 'on event-based network programming and ' - 'multiprotocol integration.', + 'description': 'An extensible framework for Python programming, with ' + 'special focus\r\n' + 'on event-based network programming and multiprotocol ' + 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', 'home_page': 'http://twistedmatrix.com/', @@ -323,13 +323,12 @@ cannot be split, the specified width will be exceeded:: 'cheesecake_installability_id': None, 'classifiers': [...], 'description': 'An extensible ' - 'framework for ' - 'Python programming, ' - 'with special ' - 'focus\r\n' - 'on event-based ' - 'network programming ' - 'and multiprotocol ' + 'framework for Python ' + 'programming, with ' + 'special focus\r\n' + 'on event-based network ' + 'programming and ' + 'multiprotocol ' 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', @@ -344,8 +343,8 @@ cannot be split, the specified width will be exceeded:: 'release_url': 'http://pypi.python.org/pypi/Twisted/12.3.0', 'requires_python': None, 'stable_version': None, - 'summary': 'An asynchronous ' - 'networking framework ' - 'written in Python', + 'summary': 'An asynchronous networking ' + 'framework written in ' + 'Python', 'version': '12.3.0'}, 'urls': [{...}, {...}]} diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst index bae8450b7c..3647571090 100644 --- a/Doc/library/py_compile.rst +++ b/Doc/library/py_compile.rst @@ -29,9 +29,9 @@ byte-code cache files in the directory containing the source code. .. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1) Compile a source file to byte-code and write out the byte-code cache file. - The source code is loaded from the file name *file*. The byte-code is - written to *cfile*, which defaults to the :PEP:`3147` path, ending in - ``.pyc`` (``.pyo`` if optimization is enabled in the current interpreter). + The source code is loaded from the file name *file*. The byte-code is + written to *cfile*, which defaults to the :pep:`3147`/:pep`488` path, ending + in ``.pyc``. For example, if *file* is ``/foo/bar/baz.py`` *cfile* will default to ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. If *dfile* is specified, it is used as the name of the source file in error messages when @@ -68,7 +68,7 @@ byte-code cache files in the directory containing the source code. .. function:: main(args=None) Compile several source files. The files named in *args* (or on the command - line, if *args* is ``None``) are compiled and the resulting bytecode is + line, if *args* is ``None``) are compiled and the resulting byte-code is cached in the normal manner. This function does not search a directory structure to locate source files; it only compiles files named explicitly. If ``'-'`` is the only parameter in args, the list of files is taken from @@ -86,4 +86,3 @@ could not be compiled. Module :mod:`compileall` Utilities to compile all Python source files in a directory tree. - diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 680d690052..1cb0935377 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -158,22 +158,32 @@ fully processed by daemon consumer threads. Example of how to wait for enqueued tasks to be completed:: - def worker(): - while True: - item = q.get() - do_work(item) - q.task_done() - - q = Queue() - for i in range(num_worker_threads): - t = Thread(target=worker) - t.daemon = True + def worker(): + while True: + item = q.get() + if item is None: + break + do_work(item) + q.task_done() + + q = queue.Queue() + threads = [] + for i in range(num_worker_threads): + t = threading.Thread(target=worker) t.start() + threads.append(t) - for item in source(): - q.put(item) + for item in source(): + q.put(item) - q.join() # block until all tasks are done + # block until all tasks are done + q.join() + + # stop workers + for i in range(num_worker_threads): + q.put(None) + for t in threads: + t.join() .. seealso:: diff --git a/Doc/library/random.rst b/Doc/library/random.rst index 11dd367f8a..f8b772749f 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -46,8 +46,7 @@ from sources provided by the operating system. .. warning:: The pseudo-random generators of this module should not be used for - security purposes. Use :func:`os.urandom` or :class:`SystemRandom` if - you require a cryptographically secure pseudo-random number generator. + security purposes. Bookkeeping functions: diff --git a/Doc/library/re.rst b/Doc/library/re.rst index c3c8b65d8d..888458449a 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -281,9 +281,7 @@ The special characters are: assertion`. ``(?<=abc)def`` will find a match in ``abcdef``, since the lookbehind will back up 3 characters and check if the contained pattern matches. The contained pattern must only match strings of some fixed length, meaning that - ``abc`` or ``a|b`` are allowed, but ``a*`` and ``a{3,4}`` are not. Group - references are not supported even if they match strings of some fixed length. - Note that + ``abc`` or ``a|b`` are allowed, but ``a*`` and ``a{3,4}`` are not. Note that patterns which start with positive lookbehind assertions will not match at the beginning of the string being searched; you will most likely want to use the :func:`search` function rather than the :func:`match` function: @@ -299,12 +297,14 @@ The special characters are: >>> m.group(0) 'egg' + .. versionchanged: 3.5 + Added support for group references of fixed length. + ``(?<!...)`` Matches if the current position in the string is not preceded by a match for ``...``. This is called a :dfn:`negative lookbehind assertion`. Similar to positive lookbehind assertions, the contained pattern must only match strings of - some fixed length and shouldn't contain group references. - Patterns which start with negative lookbehind assertions may + some fixed length. Patterns which start with negative lookbehind assertions may match at the beginning of the string being searched. ``(?(id/name)yes-pattern|no-pattern)`` @@ -438,6 +438,10 @@ three digits in length. .. versionchanged:: 3.3 The ``'\u'`` and ``'\U'`` escape sequences have been added. +.. deprecated-removed:: 3.5 3.6 + Unknown escapes consist of ``'\'`` and ASCII letter now raise a + deprecation warning and will be forbidden in Python 3.6. + .. seealso:: @@ -524,7 +528,11 @@ form. current locale. The use of this flag is discouraged as the locale mechanism is very unreliable, and it only handles one "culture" at a time anyway; you should use Unicode matching instead, which is the default in Python 3 - for Unicode (str) patterns. + for Unicode (str) patterns. This flag makes sense only with bytes patterns. + + .. deprecated-removed:: 3.5 3.6 + Deprecated the use of :const:`re.LOCALE` with string patterns or + :const:`re.ASCII`. .. data:: M @@ -625,17 +633,37 @@ form. That way, separator components are always found at the same relative indices within the result list. - Note that *split* will never split a string on an empty pattern match. - For example: + .. note:: + + :func:`split` doesn't currently split a string on an empty pattern match. + For example: - >>> re.split('x*', 'foo') - ['foo'] - >>> re.split("(?m)^$", "foo\n\nbar\n") - ['foo\n\nbar\n'] + >>> re.split('x*', 'axbc') + ['a', 'bc'] + + Even though ``'x*'`` also matches 0 'x' before 'a', between 'b' and 'c', + and after 'c', currently these matches are ignored. The correct behavior + (i.e. splitting on empty matches too and returning ``['', 'a', 'b', 'c', + '']``) will be implemented in future versions of Python, but since this + is a backward incompatible change, a :exc:`FutureWarning` will be raised + in the meanwhile. + + Patterns that can only match empty strings currently never split the + string. Since this doesn't match the expected behavior, a + :exc:`ValueError` will be raised starting from Python 3.5:: + + >>> re.split("^$", "foo\n\nbar\n", flags=re.M) + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + ... + ValueError: split() requires a non-empty pattern match. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Splitting on a pattern that could match an empty string now raises + a warning. Patterns that can only match empty strings are now rejected. .. function:: findall(pattern, string, flags=0) @@ -663,7 +691,7 @@ form. *string* is returned unchanged. *repl* can be a string or a function; if it is a string, any backslash escapes in it are processed. That is, ``\n`` is converted to a single newline character, ``\r`` is converted to a carriage return, and - so forth. Unknown escapes such as ``\j`` are left alone. Backreferences, such + so forth. Unknown escapes such as ``\&`` are left alone. Backreferences, such as ``\6``, are replaced with the substring matched by group 6 in the pattern. For example: @@ -705,6 +733,13 @@ form. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. + + .. deprecated-removed:: 3.5 3.6 + Unknown escapes consist of ``'\'`` and ASCII letter now raise a + deprecation warning and will be forbidden in Python 3.6. + .. function:: subn(pattern, repl, string, count=0, flags=0) @@ -714,6 +749,9 @@ form. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. + .. function:: escape(string) @@ -730,13 +768,36 @@ form. Clear the regular expression cache. -.. exception:: error +.. exception:: error(msg, pattern=None, pos=None) Exception raised when a string passed to one of the functions here is not a valid regular expression (for example, it might contain unmatched parentheses) or when some other error occurs during compilation or matching. It is never an - error if a string contains no match for a pattern. + error if a string contains no match for a pattern. The error instance has + the following additional attributes: + + .. attribute:: msg + + The unformatted error message. + + .. attribute:: pattern + + The regular expression pattern. + + .. attribute:: pos + + The index of *pattern* where compilation failed. + + .. attribute:: lineno + + The line corresponding to *pos*. + + .. attribute:: colno + + The column corresponding to *pos*. + .. versionchanged:: 3.5 + Added additional attributes. .. _re-objects: @@ -889,6 +950,8 @@ Match objects support the following methods and attributes: (``\g<1>``, ``\g<name>``) are replaced by the contents of the corresponding group. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. .. method:: match.group([group1, ...]) diff --git a/Doc/library/readline.rst b/Doc/library/readline.rst index 692310b941..3864f0d225 100644 --- a/Doc/library/readline.rst +++ b/Doc/library/readline.rst @@ -59,6 +59,14 @@ The :mod:`readline` module defines the following functions: Save a readline history file. The default filename is :file:`~/.history`. +.. function:: append_history_file(nelements[, filename]) + + Append the last *nelements* of history to a file. The default filename is + :file:`~/.history`. The file must already exist. + + .. versionadded:: 3.5 + + .. function:: clear_history() Clear the current history. (Note: this function is not available if the @@ -209,6 +217,26 @@ from the user's :envvar:`PYTHONSTARTUP` file. :: This code is actually automatically run when Python is run in :ref:`interactive mode <tut-interactive>` (see :ref:`rlcompleter-config`). +The following example achieves the same goal but supports concurrent interactive +sessions, by only appending the new history. :: + + import atexit + import os + import realine + histfile = os.path.join(os.path.expanduser("~"), ".python_history") + + try: + readline.read_history_file(histfile) + h_len = readline.get_history_length() + except FileNotFoundError: + open(histfile, 'wb').close() + h_len = 0 + + def save(prev_h_len, histfile): + new_h_len = readline.get_history_length() + readline.append_history_file(new_h_len - prev_h_len, histfile) + atexit.register(save, h_len, histfile) + The following example extends the :class:`code.InteractiveConsole` class to support history save/restore. :: @@ -234,4 +262,3 @@ support history save/restore. :: def save_history(self, histfile): readline.write_history_file(histfile) - diff --git a/Doc/library/resource.rst b/Doc/library/resource.rst index f8112cc0e4..7c0e4caf75 100644 --- a/Doc/library/resource.rst +++ b/Doc/library/resource.rst @@ -45,7 +45,7 @@ this module for those platforms. .. data:: RLIM_INFINITY - Constant used to represent the the limit for an unlimited resource. + Constant used to represent the limit for an unlimited resource. .. function:: getrlimit(resource) diff --git a/Doc/library/select.rst b/Doc/library/select.rst index 734e426acd..a62dc844e4 100644 --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -145,6 +145,13 @@ The module defines the following: library, and does not handle file descriptors that don't originate from WinSock. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + + .. attribute:: PIPE_BUF The minimum number of bytes which can be written without blocking to a pipe @@ -207,7 +214,7 @@ object. .. warning:: Registering a file descriptor that's already registered is not an - error, but the result is undefined. The appropiate action is to + error, but the result is undefined. The appropriate action is to unregister or modify it first. This is an important difference compared with :c:func:`poll`. @@ -242,6 +249,12 @@ object. returning. If *timeout* is omitted, -1, or :const:`None`, the call will block until there is an event for this poll object. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _epoll-objects: @@ -322,6 +335,12 @@ Edge and Level Trigger Polling (epoll) Objects Wait for events. timeout in seconds (float) + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _poll-objects: @@ -401,6 +420,12 @@ linearly scanned again. :c:func:`select` is O(highest file descriptor), while returning. If *timeout* is omitted, negative, or :const:`None`, the call will block until there is an event for this poll object. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _kqueue-objects: @@ -435,6 +460,12 @@ Kqueue Objects - max_events must be 0 or a positive integer - timeout in seconds (floats possible) + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _kevent-objects: diff --git a/Doc/library/selectors.rst b/Doc/library/selectors.rst index 98377c890f..f6ef24b6ae 100644 --- a/Doc/library/selectors.rst +++ b/Doc/library/selectors.rst @@ -45,6 +45,7 @@ Classes hierarchy:: +-- SelectSelector +-- PollSelector +-- EpollSelector + +-- DevpollSelector +-- KqueueSelector @@ -158,6 +159,12 @@ below: timeout has elapsed if the current process receives a signal: in this case, an empty list will be returned. + .. versionchanged:: 3.5 + The selector is now retried with a recomputed timeout when interrupted + by a signal if the signal handler did not raise an exception (see + :pep:`475` for the rationale), instead of returning an empty list + of events before the timeout. + .. method:: close() Close the selector. @@ -207,6 +214,16 @@ below: This returns the file descriptor used by the underlying :func:`select.epoll` object. +.. class:: DevpollSelector() + + :func:`select.devpoll`-based selector. + + .. method:: fileno() + + This returns the file descriptor used by the underlying + :func:`select.devpoll` object. + + .. versionadded:: 3.5 .. class:: KqueueSelector() diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index cd86e92944..3b467e01c2 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -191,7 +191,8 @@ Directory and files operations match one of the glob-style *patterns* provided. See the example below. -.. function:: copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, ignore_dangling_symlinks=False) +.. function:: copytree(src, dst, symlinks=False, ignore=None, \ + copy_function=copy2, ignore_dangling_symlinks=False) Recursively copy an entire directory tree rooted at *src*, returning the destination directory. The destination @@ -282,7 +283,7 @@ Directory and files operations .. versionadded:: 3.3 -.. function:: move(src, dst) +.. function:: move(src, dst, copy_function=copy2) Recursively move a file or directory (*src*) to another location (*dst*) and return the destination. @@ -292,15 +293,26 @@ Directory and files operations be overwritten depending on :func:`os.rename` semantics. If the destination is on the current filesystem, then :func:`os.rename` is - used. Otherwise, *src* is copied (using :func:`shutil.copy2`) to *dst* and - then removed. In case of symlinks, a new symlink pointing to the target of - *src* will be created in or as *dst* and *src* will be removed. + used. Otherwise, *src* is copied to *dst* using *copy_function* and then + removed. In case of symlinks, a new symlink pointing to the target of *src* + will be created in or as *dst* and *src* will be removed. + + If *copy_function* is given, it must be a callable that takes two arguments + *src* and *dst*, and will be used to copy *src* to *dest* if + :func:`os.rename` cannot be used. If the source is a directory, + :func:`copytree` is called, passing it the :func:`copy_function`. The + default *copy_function* is :func:`copy2`. Using :func:`copy` as the + *copy_function* allows the move to succeed when it is not possible to also + copy the metadata, at the expense of not copying any of the metadata. .. versionchanged:: 3.3 Added explicit symlink handling for foreign filesystems, thus adapting it to the behavior of GNU's :program:`mv`. Now returns *dst*. + .. versionchanged:: 3.5 + Added the *copy_function* keyword argument. + .. function:: disk_usage(path) Return disk usage statistics about the given path as a :term:`named tuple` @@ -338,7 +350,7 @@ Directory and files operations On Windows, the current directory is always prepended to the *path* whether or not you use the default or provide your own, which is the behavior the - command shell uses when finding executables. Additionaly, when finding the + command shell uses when finding executables. Additionally, when finding the *cmd* in the *path*, the ``PATHEXT`` environment variable is checked. For example, if you call ``shutil.which("python")``, :func:`which` will search ``PATHEXT`` to know that it should look for ``python.exe`` within the *path* @@ -418,6 +430,26 @@ Another example that uses the *ignore* argument to add a logging call:: copytree(source, destination, ignore=_logpath) +.. _shutil-rmtree-example: + +rmtree example +~~~~~~~~~~~~~~ + +This example shows how to remove a directory tree on Windows where some +of the files have their read-only bit set. It uses the onerror callback +to clear the readonly bit and reattempt the remove. Any subsequent failure +will propagate. :: + + import os, stat + import shutil + + def remove_readonly(func, path, _): + "Clear the readonly bit and reattempt the removal" + os.chmod(path, stat.S_IWRITE) + func(path) + + shutil.rmtree(directory, onerror=remove_readonly) + .. _archiving-operations: Archiving operations @@ -434,7 +466,8 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. *base_name* is the name of the file to create, including the path, minus any format-specific extension. *format* is the archive format: one of - "zip", "tar", "bztar" (if the :mod:`bz2` module is available) or "gztar". + "zip", "tar", "bztar" (if the :mod:`bz2` module is available), "xztar" + (if the :mod:`lzma` module is available) or "gztar". *root_dir* is a directory that will be the root directory of the archive; for example, we typically chdir into *root_dir* before creating the @@ -457,6 +490,9 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. The *verbose* argument is unused and deprecated. + .. versionchanged:: 3.5 + Added support for the *xztar* format. + .. function:: get_archive_formats() @@ -467,6 +503,7 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. - *gztar*: gzip'ed tar-file - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available.) - *tar*: uncompressed tar file - *zip*: ZIP file @@ -542,6 +579,7 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. - *gztar*: gzip'ed tar-file - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available.) - *tar*: uncompressed tar file - *zip*: ZIP file diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 84e283683d..553920d885 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -65,6 +65,16 @@ Besides, only the main thread is allowed to set a new signal handler. Module contents --------------- +.. versionchanged:: 3.5 + signal (SIG*), handler (:const:`SIG_DFL`, :const:`SIG_IGN`) and sigmask + (:const:`SIG_BLOCK`, :const:`SIG_UNBLOCK`, :const:`SIG_SETMASK`) + related constants listed below were turned into + :class:`enums <enum.IntEnum>`. + :func:`getsignal`, :func:`pthread_sigmask`, :func:`sigpending` and + :func:`sigwait` functions return human-readable + :class:`enums <enum.IntEnum>`. + + The variables defined in the :mod:`signal` module are: @@ -308,6 +318,9 @@ The :mod:`signal` module defines the following functions: attempting to call it from other threads will cause a :exc:`ValueError` exception to be raised. + .. versionchanged:: 3.5 + On Windows, the function now also supports socket handles. + .. function:: siginterrupt(signalnum, flag) @@ -395,6 +408,11 @@ The :mod:`signal` module defines the following functions: .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now retried if interrupted by a signal not in *sigset* + and the signal handler does not raise an exception (see :pep:`475` for + the rationale). + .. function:: sigtimedwait(sigset, timeout) @@ -409,6 +427,11 @@ The :mod:`signal` module defines the following functions: .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now retried with the recomputed *timeout* if interrupted + by a signal not in *sigset* and the signal handler does not raise an + exception (see :pep:`475` for the rationale). + .. _signal-example: diff --git a/Doc/library/site.rst b/Doc/library/site.rst index e57b8ccc2d..43daf790b7 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -26,24 +26,23 @@ additions, call the :func:`site.main` function. :option:`-S`. .. index:: - pair: site-python; directory pair: site-packages; directory It starts by constructing up to four directories from a head and a tail part. For the head part, it uses ``sys.prefix`` and ``sys.exec_prefix``; empty heads are skipped. For the tail part, it uses the empty string and then :file:`lib/site-packages` (on Windows) or -:file:`lib/python{X.Y}/site-packages` and then :file:`lib/site-python` (on -Unix and Macintosh). For each of the distinct head-tail combinations, it sees -if it refers to an existing directory, and if so, adds it to ``sys.path`` and -also inspects the newly added path for configuration files. +:file:`lib/python{X.Y}/site-packages` (on Unix and Macintosh). For each +of the distinct head-tail combinations, it sees if it refers to an existing +directory, and if so, adds it to ``sys.path`` and also inspects the newly +added path for configuration files. -.. deprecated:: 3.4 - Support for the "site-python" directory will be removed in 3.5. +.. versionchanged:: 3.5 + Support for the "site-python" directory has been removed. If a file named "pyvenv.cfg" exists one directory above sys.executable, sys.prefix and sys.exec_prefix are set to that directory and -it is also checked for site-packages and site-python (sys.base_prefix and +it is also checked for site-packages (sys.base_prefix and sys.base_exec_prefix will always be the "real" prefixes of the Python installation). If "pyvenv.cfg" (a bootstrap configuration file) contains the key "include-system-site-packages" set to anything other than "false" @@ -184,7 +183,7 @@ Module contents unless the Python interpreter was started with the :option:`-S` flag. .. versionchanged:: 3.3 - This function used to be called unconditionnally. + This function used to be called unconditionally. .. function:: addsitedir(sitedir, known_paths=None) @@ -195,8 +194,7 @@ Module contents .. function:: getsitepackages() - Return a list containing all global site-packages directories (and possibly - site-python). + Return a list containing all global site-packages directories. .. versionadded:: 3.2 diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst index 3ebed06260..3e0c6fbff5 100644 --- a/Doc/library/smtpd.rst +++ b/Doc/library/smtpd.rst @@ -20,7 +20,8 @@ specific mail-sending strategies. Additionally the SMTPChannel may be extended to implement very specific interaction behaviour with SMTP clients. -The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE extension. +The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE and :rfc:`6531` +SMTPUTF8 extensions. SMTPServer Objects @@ -28,7 +29,7 @@ SMTPServer Objects .. class:: SMTPServer(localaddr, remoteaddr, data_size_limit=33554432,\ - map=None) + map=None, enable_SMTPUTF8=False, decode_data=True) Create a new :class:`SMTPServer` object, which binds to local address *localaddr*. It will treat *remoteaddr* as an upstream SMTP relayer. It @@ -39,18 +40,47 @@ SMTPServer Objects accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no limit. + *enable_SMTPUTF8* determins whether the ``SMTPUTF8`` extension (as defined + in :RFC:`6531`) should be enabled. The default is ``False``. If + *enable_SMTPUTF* is set to ``True``, the :meth:`process_smtputf8_message` + method must be defined. A :exc:`ValueError` is raised if both + *enable_SMTPUTF8* and *decode_data* are set to ``True`` at the same time. + A dictionary can be specified in *map* to avoid using a global socket map. + *decode_data* specifies whether the data portion of the SMTP transaction + should be decoded using UTF-8. The default is ``True`` for backward + compatibility reasons, but will change to ``False`` in Python 3.6. Specify + the keyword value explicitly to avoid the :exc:`DeprecationWarning`. + .. method:: process_message(peer, mailfrom, rcpttos, data) - Raise :exc:`NotImplementedError` exception. Override this in subclasses to + Raise a :exc:`NotImplementedError` exception. Override this in subclasses to do something useful with this message. Whatever was passed in the constructor as *remoteaddr* will be available as the :attr:`_remoteaddr` attribute. *peer* is the remote host's address, *mailfrom* is the envelope originator, *rcpttos* are the envelope recipients and *data* is a string - containing the contents of the e-mail (which should be in :rfc:`2822` + containing the contents of the e-mail (which should be in :rfc:`5321` format). + If the *decode_data* constructor keyword is set to ``True``, the *data* + argument will be a unicode string. If it is set to ``False``, it + will be a bytes object. + + Return ``None`` to request a normal ``250 Ok`` response; otherwise + return the desired response string in :RFC:`5321` format. + + .. method:: process_smtputf8_message(peer, mailfrom, rcpttos, data) + + Raise a :exc:`NotImplementedError` exception. Override this in + subclasses to do something useful with messages when *enable_SMTPUTF8* + has been set to ``True`` and the SMTP client requested ``SMTPUTF8``, + since this method is called rather than :meth:`process_message` when the + client actively requests ``SMTPUTF8``. The *data* argument will always + be a bytes object, and any non-``None`` return value should conform to + :rfc:`6531`; otherwise, the API is the same as for + :meth:`process_message`. + .. attribute:: channel_class Override this in subclasses to use a custom :class:`SMTPChannel` for @@ -59,6 +89,13 @@ SMTPServer Objects .. versionchanged:: 3.4 The *map* argument was added. + .. versionchanged:: 3.5 + *localaddr* and *remoteaddr* may now contain IPv6 addresses. + + .. versionadded:: 3.5 + the *decode_data* and *enable_SMTPUTF8* constructor arguments, and the + :meth:`process_smtputf8_message` method. + DebuggingServer Objects ----------------------- @@ -97,7 +134,7 @@ SMTPChannel Objects ------------------- .. class:: SMTPChannel(server, conn, addr, data_size_limit=33554432,\ - map=None)) + map=None, enable_SMTPUTF8=False, decode_data=True) Create a new :class:`SMTPChannel` object which manages the communication between the server and a single SMTP client. @@ -108,11 +145,24 @@ SMTPChannel Objects accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no limit. + *enable_SMTPUTF8* determins whether the ``SMTPUTF8`` extension (as defined + in :RFC:`6531`) should be enabled. The default is ``False``. A + :exc:`ValueError` is raised if both *enable_SMTPUTF8* and *decode_data* are + set to ``True`` at the same time. + A dictionary can be specified in *map* to avoid using a global socket map. + *decode_data* specifies whether the data portion of the SMTP transaction + should be decoded using UTF-8. The default is ``True`` for backward + compatibility reasons, but will change to ``False`` in Python 3.6. Specify + the keyword value explicitly to avoid the :exc:`DeprecationWarning`. + To use a custom SMTPChannel implementation you need to override the :attr:`SMTPServer.channel_class` of your :class:`SMTPServer`. + .. versionchanged:: 3.5 + the *decode_data* and *enable_SMTPUTF8* arguments were added. + The :class:`SMTPChannel` has the following instance variables: .. attribute:: smtp_server diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index 8e1bfb57f9..74de77b94e 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -240,8 +240,7 @@ An :class:`SMTP` instance has the following methods: the server is stored as the :attr:`ehlo_resp` attribute, :attr:`does_esmtp` is set to true or false depending on whether the server supports ESMTP, and :attr:`esmtp_features` will be a dictionary containing the names of the - SMTP service extensions this server supports, and their - parameters (if any). + SMTP service extensions this server supports, and their parameters (if any). Unless you wish to use :meth:`has_extn` before sending mail, it should not be necessary to call this method explicitly. It will be implicitly called by @@ -291,6 +290,42 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPException` No suitable authentication method was found. + Each of the authentication methods supported by :mod:`smtplib` are tried in + turn if they are advertised as supported by the server (see :meth:`auth` + for a list of supported authentication methods). + + +.. method:: SMTP.auth(mechanism, authobject) + + Issue an ``SMTP`` ``AUTH`` command for the specified authentication + *mechanism*, and handle the challenge response via *authobject*. + + *mechanism* specifies which authentication mechanism is to + be used as argument to the ``AUTH`` command; the valid values are + those listed in the ``auth`` element of :attr:`esmtp_features`. + + *authobject* must be a callable object taking a single argument: + + data = authobject(challenge) + + It will be called to process the server's challenge response; the + *challenge* argument it is passed will be a ``bytes``. It should return + ``bytes`` *data* that will be base64 encoded and sent to the server. + + The ``SMTP`` class provides ``authobjects`` for the ``CRAM-MD5``, ``PLAIN``, + and ``LOGIN`` mechanisms; they are named ``SMTP.auth_cram_md5``, + ``SMTP.auth_plain``, and ``SMTP.auth_login`` respectively. They all require + that the ``user`` and ``password`` properties of the ``SMTP`` instance are + set to appropriate values. + + User code does not normally need to call ``auth`` directly, but can instead + call the :meth:`login` method, which will try each of the above mechanisms in + turn, in the order listed. ``auth`` is exposed to facilitate the + implementation of authentication methods not (or not yet) supported directly + by :mod:`smtplib`. + + .. versionadded:: 3.5 + .. method:: SMTP.starttls(keyfile=None, certfile=None, context=None) diff --git a/Doc/library/sndhdr.rst b/Doc/library/sndhdr.rst index f36df68703..f8b5d8b246 100644 --- a/Doc/library/sndhdr.rst +++ b/Doc/library/sndhdr.rst @@ -16,8 +16,9 @@ The :mod:`sndhdr` provides utility functions which attempt to determine the type of sound data which is in a file. When these functions are able to determine -what type of sound data is stored in a file, they return a tuple ``(type, -sampling_rate, channels, frames, bits_per_sample)``. The value for *type* +what type of sound data is stored in a file, they return a +:func:`~collections.namedtuple`, containing five attributes: (``filetype``, +``framerate``, ``nchannels``, ``nframes``, ``sampwidth``). The value for *type* indicates the data type and will be one of the strings ``'aifc'``, ``'aiff'``, ``'au'``, ``'hcom'``, ``'sndr'``, ``'sndt'``, ``'voc'``, ``'wav'``, ``'8svx'``, ``'sb'``, ``'ub'``, or ``'ul'``. The *sampling_rate* will be either the actual @@ -31,13 +32,19 @@ be the sample size in bits or ``'A'`` for A-LAW or ``'U'`` for u-LAW. .. function:: what(filename) Determines the type of sound data stored in the file *filename* using - :func:`whathdr`. If it succeeds, returns a tuple as described above, otherwise + :func:`whathdr`. If it succeeds, returns a namedtuple as described above, otherwise ``None`` is returned. + .. versionchanged:: 3.5 + Result changed from a tuple to a namedtuple. + .. function:: whathdr(filename) Determines the type of sound data stored in a file based on the file header. - The name of the file is given by *filename*. This function returns a tuple as + The name of the file is given by *filename*. This function returns a namedtuple as described above on success, or ``None``. + .. versionchanged:: 3.5 + Result changed from a tuple to a namedtuple. + diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index b703e9e9a6..84fa10d774 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -46,17 +46,20 @@ created. Socket addresses are represented as follows: - The address of an :const:`AF_UNIX` socket bound to a file system node is represented as a string, using the file system encoding and the ``'surrogateescape'`` error handler (see :pep:`383`). An address in - Linux's abstract namespace is returned as a :class:`bytes` object with + Linux's abstract namespace is returned as a :term:`bytes-like object` with an initial null byte; note that sockets in this namespace can communicate with normal file system sockets, so programs intended to run on Linux may need to deal with both types of address. A string or - :class:`bytes` object can be used for either type of address when + bytes-like object can be used for either type of address when passing it as an argument. .. versionchanged:: 3.3 Previously, :const:`AF_UNIX` socket paths were assumed to use UTF-8 encoding. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + - A pair ``(host, port)`` is used for the :const:`AF_INET` address family, where *host* is a string representing either a hostname in Internet domain notation like ``'daring.cwi.nl'`` or an IPv4 address like ``'100.50.200.5'``, @@ -276,6 +279,18 @@ Constants .. versionadded:: 3.4 +.. data:: CAN_RAW_FD_FRAMES + + Enables CAN FD support in a CAN_RAW socket. This is disabled by default. + This allows your application to send both CAN and CAN FD frames; however, + you one must accept both CAN and CAN FD frames when reading from the socket. + + This constant is documented in the Linux documentation. + + Availability: Linux >= 3.6. + + .. versionadded:: 3.5 + .. data:: AF_RDS PF_RDS SOL_RDS @@ -352,7 +367,6 @@ The following functions all create :ref:`socket objects <socket-objects>`. type, and protocol number. Address family, socket type, and protocol number are as for the :func:`.socket` function above. The default family is :const:`AF_UNIX` if defined on the platform; otherwise, the default is :const:`AF_INET`. - Availability: Unix. The newly created sockets are :ref:`non-inheritable <fd_inheritance>`. @@ -363,6 +377,9 @@ The following functions all create :ref:`socket objects <socket-objects>`. .. versionchanged:: 3.4 The returned sockets are now non-inheritable. + .. versionchanged:: 3.5 + Windows support added. + .. function:: create_connection(address[, timeout[, source_address]]) @@ -607,8 +624,8 @@ The :mod:`socket` module also offers various network-related services: .. function:: inet_ntoa(packed_ip) - Convert a 32-bit packed IPv4 address (a bytes object four characters in - length) to its standard dotted-quad string representation (for example, + Convert a 32-bit packed IPv4 address (a :term:`bytes-like object` four + bytes in length) to its standard dotted-quad string representation (for example, '123.45.67.89'). This is useful when conversing with a program that uses the standard C library and needs objects of type :c:type:`struct in_addr`, which is the C type for the 32-bit packed binary data this function takes as an @@ -619,6 +636,9 @@ The :mod:`socket` module also offers various network-related services: support IPv6, and :func:`inet_ntop` should be used instead for IPv4/v6 dual stack support. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. function:: inet_pton(address_family, ip_string) @@ -641,22 +661,26 @@ The :mod:`socket` module also offers various network-related services: .. function:: inet_ntop(address_family, packed_ip) - Convert a packed IP address (a bytes object of some number of characters) to its - standard, family-specific string representation (for example, ``'7.10.0.5'`` or - ``'5aef:2b::8'``). :func:`inet_ntop` is useful when a library or network protocol - returns an object of type :c:type:`struct in_addr` (similar to :func:`inet_ntoa`) - or :c:type:`struct in6_addr`. + Convert a packed IP address (a :term:`bytes-like object` of some number of + bytes) to its standard, family-specific string representation (for + example, ``'7.10.0.5'`` or ``'5aef:2b::8'``). + :func:`inet_ntop` is useful when a library or network protocol returns an + object of type :c:type:`struct in_addr` (similar to :func:`inet_ntoa`) or + :c:type:`struct in6_addr`. Supported values for *address_family* are currently :const:`AF_INET` and - :const:`AF_INET6`. If the string *packed_ip* is not the correct length for the - specified address family, :exc:`ValueError` will be raised. A - :exc:`OSError` is raised for errors from the call to :func:`inet_ntop`. + :const:`AF_INET6`. If the bytes object *packed_ip* is not the correct + length for the specified address family, :exc:`ValueError` will be raised. + A :exc:`OSError` is raised for errors from the call to :func:`inet_ntop`. Availability: Unix (maybe not all platforms), Windows. .. versionchanged:: 3.4 Windows support added + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. XXX: Are sendmsg(), recvmsg() and CMSG_*() available on any @@ -781,6 +805,11 @@ to sockets. .. versionchanged:: 3.4 The socket is now non-inheritable. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.bind(address) @@ -813,6 +842,19 @@ to sockets. Connect to a remote socket at *address*. (The format of *address* depends on the address family --- see above.) + If the connection is interrupted by a signal, the method waits until the + connection completes, or raise a :exc:`socket.timeout` on timeout, if the + signal handler doesn't raise an exception and the socket is blocking or has + a timeout. For non-blocking sockets, the method raises an + :exc:`InterruptedError` exception if the connection is interrupted by a + signal (or the exception raised by the signal handler). + + .. versionchanged:: 3.5 + The method now waits until the connection completes instead of raising an + :exc:`InterruptedError` exception if the connection is interrupted by a + signal, the signal handler doesn't raise an exception and the socket is + blocking or has a timeout (see the :pep:`475` for the rationale). + .. method:: socket.connect_ex(address) @@ -908,12 +950,15 @@ to sockets. On other platforms, the generic :func:`fcntl.fcntl` and :func:`fcntl.ioctl` functions may be used; they accept a socket object as their first argument. -.. method:: socket.listen(backlog) +.. method:: socket.listen([backlog]) - Listen for connections made to the socket. The *backlog* argument specifies the - maximum number of queued connections and should be at least 0; the maximum value - is system-dependent (usually 5), the minimum value is forced to 0. + Enable a server to accept connections. If *backlog* is specified, it must + be at least 0 (if it is lower, it is set to 0); it specifies the number of + unaccepted connections that the system will allow before refusing new + connections. If not specified, a default reasonable value is chosen. + .. versionchanged:: 3.5 + The *backlog* parameter is now optional. .. method:: socket.makefile(mode='r', buffering=None, *, encoding=None, \ errors=None, newline=None) @@ -951,6 +996,11 @@ to sockets. For best match with hardware and network realities, the value of *bufsize* should be a relatively small power of 2, for example, 4096. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvfrom(bufsize[, flags]) @@ -960,6 +1010,11 @@ to sockets. :manpage:`recv(2)` for the meaning of the optional argument *flags*; it defaults to zero. (The format of *address* depends on the address family --- see above.) + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvmsg(bufsize[, ancbufsize[, flags]]) @@ -1026,6 +1081,11 @@ to sockets. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvmsg_into(buffers[, ancbufsize[, flags]]) @@ -1092,6 +1152,11 @@ to sockets. application needs to attempt delivery of the remaining data. For further information on this topic, consult the :ref:`socket-howto`. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendall(bytes[, flags]) @@ -1102,6 +1167,15 @@ to sockets. success. On error, an exception is raised, and there is no way to determine how much data, if any, was successfully sent. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time data is sent successfuly. + The socket timeout is now the maximum total duration to send all data. + + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendto(bytes, address) socket.sendto(bytes, flags, address) @@ -1112,6 +1186,11 @@ to sockets. bytes sent. (The format of *address* depends on the address family --- see above.) + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendmsg(buffers[, ancdata[, flags[, address]]]) @@ -1148,6 +1227,26 @@ to sockets. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + +.. method:: socket.sendfile(file, offset=0, count=None) + + Send a file until EOF is reached by using high-performance + :mod:`os.sendfile` and return the total number of bytes which were sent. + *file* must be a regular file object opened in binary mode. If + :mod:`os.sendfile` is not available (e.g. Windows) or *file* is not a + regular file :meth:`send` will be used instead. *offset* tells from where to + start reading the file. If specified, *count* is the total number of bytes + to transmit as opposed to sending the file until EOF is reached. File + position is updated on return or also in case of error in which case + :meth:`file.tell() <io.IOBase.tell>` can be used to figure out the number of + bytes which were sent. The socket must be of :const:`SOCK_STREAM` type. Non- + blocking sockets are not supported. + + .. versionadded:: 3.5 .. method:: socket.set_inheritable(inheritable) @@ -1187,11 +1286,15 @@ to sockets. Set the value of the given socket option (see the Unix manual page :manpage:`setsockopt(2)`). The needed symbolic constants are defined in the - :mod:`socket` module (:const:`SO_\*` etc.). The value can be an integer or a - bytes object representing a buffer. In the latter case it is up to the caller to + :mod:`socket` module (:const:`SO_\*` etc.). The value can be an integer or + a :term:`bytes-like object` representing a buffer. In the latter case it is + up to the caller to ensure that the bytestring contains the proper bits (see the optional built-in module :mod:`struct` for a way to encode C structures as bytestrings). + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: socket.shutdown(how) @@ -1447,7 +1550,7 @@ After binding (:const:`CAN_RAW`) or connecting (:const:`CAN_BCM`) the socket, yo can use the :meth:`socket.send`, and the :meth:`socket.recv` operations (and their counterparts) on the socket object as usual. -This example might require special priviledge:: +This example might require special privileges:: import socket import struct diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst index 1ec44389bf..9db36d5510 100644 --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -113,7 +113,7 @@ the request handler class :meth:`handle` method. Another approach to handling multiple simultaneous requests in an environment that supports neither threads nor :func:`~os.fork` (or where these are too expensive or inappropriate for the service) is to maintain an explicit table of -partially finished requests and to use :func:`~select.select` to decide which +partially finished requests and to use :mod:`selectors` to decide which request to work on next (or whether to handle a new incoming request). This is particularly important for stream services where each client can potentially be connected for a long time (if threads or subprocesses cannot be used). See @@ -136,7 +136,7 @@ Server Objects .. method:: BaseServer.fileno() Return an integer file descriptor for the socket on which the server is - listening. This function is most commonly passed to :func:`select.select`, to + listening. This function is most commonly passed to :mod:`selectors`, to allow monitoring multiple servers in the same process. diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 6097e7a30d..fc69a804d5 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -649,6 +649,9 @@ Row Objects This method returns a list of column names. Immediately after a query, it is the first member of each tuple in :attr:`Cursor.description`. + .. versionchanged:: 3.5 + Added support of slicing. + Let's assume we initialize a table as in the example given above:: conn = sqlite3.connect(":memory:") diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index bc28e97bab..31b4658008 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -315,6 +315,8 @@ Random generation For almost all applications :func:`os.urandom` is preferable. + For almost all applications :func:`os.urandom` is preferable. + .. versionadded:: 3.3 .. function:: RAND_status() @@ -335,6 +337,8 @@ Random generation See http://egd.sourceforge.net/ or http://prngd.sourceforge.net/ for sources of entropy-gathering daemons. + Availability: not available with LibreSSL. + .. function:: RAND_add(bytes, entropy) Mix the given *bytes* into the SSL pseudo-random number generator. The @@ -342,6 +346,9 @@ Random generation string (so you can always use :const:`0.0`). See :rfc:`1750` for more information on sources of entropy. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + Certificate handling ^^^^^^^^^^^^^^^^^^^^ @@ -350,10 +357,9 @@ Certificate handling Verify that *cert* (in decoded format as returned by :meth:`SSLSocket.getpeercert`) matches the given *hostname*. The rules applied are those for checking the identity of HTTPS servers as outlined - in :rfc:`2818` and :rfc:`6125`, except that IP addresses are not currently - supported. In addition to HTTPS, this function should be suitable for - checking the identity of servers in various SSL-based protocols such as - FTPS, IMAPS, POPS and others. + in :rfc:`2818` and :rfc:`6125`. In addition to HTTPS, this function + should be suitable for checking the identity of servers in various + SSL-based protocols such as FTPS, IMAPS, POPS and others. :exc:`CertificateError` is raised on failure. On success, the function returns nothing:: @@ -375,22 +381,38 @@ Certificate handling IDN A-labels such as ``www*.xn--pthon-kva.org`` are still supported, but ``x*.python.org`` no longer matches ``xn--tda.python.org``. -.. function:: cert_time_to_seconds(timestring) + .. versionchanged:: 3.5 + Matching of IP addresses, when present in the subjectAltName field + of the certificate, is now supported. + +.. function:: cert_time_to_seconds(cert_time) - Returns a floating-point value containing a normal seconds-after-the-epoch - time value, given the time-string representing the "notBefore" or "notAfter" - date from a certificate. + Return the time in seconds since the Epoch, given the ``cert_time`` + string representing the "notBefore" or "notAfter" date from a + certificate in ``"%b %d %H:%M:%S %Y %Z"`` strptime format (C + locale). - Here's an example:: + Here's an example: - >>> import ssl - >>> ssl.cert_time_to_seconds("May 9 00:00:00 2007 GMT") - 1178694000.0 - >>> import time - >>> time.ctime(ssl.cert_time_to_seconds("May 9 00:00:00 2007 GMT")) - 'Wed May 9 00:00:00 2007' + .. doctest:: newcontext -.. function:: get_server_certificate(addr, ssl_version=PROTOCOL_SSLv3, ca_certs=None) + >>> import ssl + >>> timestamp = ssl.cert_time_to_seconds("Jan 5 09:34:43 2018 GMT") + >>> timestamp + 1515144883 + >>> from datetime import datetime + >>> print(datetime.utcfromtimestamp(timestamp)) + 2018-01-05 09:34:43 + + "notBefore" or "notAfter" dates must use GMT (:rfc:`5280`). + + .. versionchanged:: 3.5 + Interpret the input time as a time in UTC as specified by 'GMT' + timezone in the input string. Local timezone was used + previously. Return an integer (no fractions of a second in the + input format) + +.. function:: get_server_certificate(addr, ssl_version=PROTOCOL_SSLv23, ca_certs=None) Given the address ``addr`` of an SSL-protected server, as a (*hostname*, *port-number*) pair, fetches the server's certificate, and returns it as a @@ -404,6 +426,10 @@ Certificate handling .. versionchanged:: 3.3 This function is now IPv6-compatible. + .. versionchanged:: 3.5 + The default *ssl_version* is changed from :data:`PROTOCOL_SSLv3` to + :data:`PROTOCOL_SSLv23` for maximum compatibility with modern servers. + .. function:: DER_cert_to_PEM_cert(DER_cert_bytes) Given a certificate as a DER-encoded blob of bytes, returns a PEM-encoded @@ -671,6 +697,13 @@ Constants .. versionadded:: 3.3 +.. data:: HAS_ALPN + + Whether the OpenSSL library has built-in support for the *Application-Layer + Protocol Negotiation* TLS extension as described in :rfc:`7301`. + + .. versionadded:: 3.5 + .. data:: HAS_ECDH Whether the OpenSSL library has built-in support for Elliptic Curve-based @@ -788,6 +821,8 @@ SSL Sockets (but passing a non-zero ``flags`` argument is not allowed) - :meth:`~socket.socket.send()`, :meth:`~socket.socket.sendall()` (with the same limitation) + - :meth:`~socket.socket.sendfile()` (but :mod:`os.sendfile` will be used + for plain-text sockets only, else :meth:`~socket.socket.send()` will be used) - :meth:`~socket.socket.shutdown()` However, since the SSL (and TLS) protocol has its own framing atop @@ -798,6 +833,15 @@ SSL Sockets Usually, :class:`SSLSocket` are not created directly, but using the :func:`wrap_socket` function or the :meth:`SSLContext.wrap_socket` method. + .. versionchanged:: 3.5 + The :meth:`sendfile` method was added. + + .. versionchanged:: 3.5 + The :meth:`shutdown` does not reset the socket timeout each time bytes + are received or sent. The socket timeout is now to maximum total duration + of the shutdown. + + SSL sockets also have the following additional methods and attributes: .. method:: SSLSocket.read(len=0, buffer=None) @@ -812,6 +856,11 @@ SSL sockets also have the following additional methods and attributes: As at any time a re-negotiation is possible, a call to :meth:`read` can also cause write operations. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration to read up to *len* + bytes. + .. method:: SSLSocket.write(buf) Write *buf* to the SSL socket and return the number of bytes written. The @@ -823,6 +872,10 @@ SSL sockets also have the following additional methods and attributes: As at any time a re-negotiation is possible, a call to :meth:`write` can also cause read operations. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration to write *buf*. + .. note:: The :meth:`~SSLSocket.read` and :meth:`~SSLSocket.write` methods are the @@ -844,6 +897,10 @@ SSL sockets also have the following additional methods and attributes: :attr:`~SSLContext.check_hostname` attribute of the socket's :attr:`~SSLSocket.context` is true. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration of the handshake. + .. method:: SSLSocket.getpeercert(binary_form=False) If there is no certificate for the peer on the other end of the connection, @@ -917,6 +974,17 @@ SSL sockets also have the following additional methods and attributes: version of the SSL protocol that defines its use, and the number of secret bits being used. If no connection has been established, returns ``None``. +.. method:: SSLSocket.shared_ciphers() + + Return the list of ciphers shared by the client during the handshake. Each + entry of the returned list is a three-value tuple containing the name of the + cipher, the version of the SSL protocol that defines its use, and the number + of secret bits the cipher uses. :meth:`~SSLSocket.shared_ciphers` returns + ``None`` if no connection has been established or the socket is a client + socket. + + .. versionadded:: 3.5 + .. method:: SSLSocket.compression() Return the compression algorithm being used as a string, or ``None`` @@ -940,12 +1008,22 @@ SSL sockets also have the following additional methods and attributes: .. versionadded:: 3.3 +.. method:: SSLSocket.selected_alpn_protocol() + + Return the protocol that was selected during the TLS handshake. If + :meth:`SSLContext.set_alpn_protocols` was not called, if the other party does + not support ALPN, if this socket does not support any of the client's + proposed protocols, or if the handshake has not happened yet, ``None`` is + returned. + + .. versionadded:: 3.5 + .. method:: SSLSocket.selected_npn_protocol() - Returns the protocol that was selected during the TLS/SSL handshake. If - :meth:`SSLContext.set_npn_protocols` was not called, or if the other party - does not support NPN, or if the handshake has not yet happened, this will - return ``None``. + Return the higher-level protocol that was selected during the TLS/SSL + handshake. If :meth:`SSLContext.set_npn_protocols` was not called, or + if the other party does not support NPN, or if the handshake has not yet + happened, this will return ``None``. .. versionadded:: 3.3 @@ -957,6 +1035,16 @@ SSL sockets also have the following additional methods and attributes: returned socket should always be used for further communication with the other side of the connection, rather than the original socket. +.. method:: SSLSocket.version() + + Return the actual SSL protocol version negotiated by the connection + as a string, or ``None`` is no secure connection is established. + As of this writing, possible return values include ``"SSLv2"``, + ``"SSLv3"``, ``"TLSv1"``, ``"TLSv1.1"`` and ``"TLSv1.2"``. + Recent OpenSSL versions may define more return values. + + .. versionadded:: 3.5 + .. method:: SSLSocket.pending() Returns the number of already decrypted bytes available for read, pending on @@ -1135,6 +1223,20 @@ to speed up repeated connections from the same clients. when connected, the :meth:`SSLSocket.cipher` method of SSL sockets will give the currently selected cipher. +.. method:: SSLContext.set_alpn_protocols(protocols) + + Specify which protocols the socket should advertise during the SSL/TLS + handshake. It should be a list of ASCII strings, like ``['http/1.1', + 'spdy/2']``, ordered by preference. The selection of a protocol will happen + during the handshake, and will play out according to :rfc:`7301`. After a + successful handshake, the :meth:`SSLSocket.selected_alpn_protocol` method will + return the agreed-upon protocol. + + This method will raise :exc:`NotImplementedError` if :data:`HAS_ALPN` is + False. + + .. versionadded:: 3.5 + .. method:: SSLContext.set_npn_protocols(protocols) Specify which protocols the socket should advertise during the SSL/TLS @@ -1175,7 +1277,7 @@ to speed up repeated connections from the same clients. Due to the early negotiation phase of the TLS connection, only limited methods and attributes are usable like - :meth:`SSLSocket.selected_npn_protocol` and :attr:`SSLSocket.context`. + :meth:`SSLSocket.selected_alpn_protocol` and :attr:`SSLSocket.context`. :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.cipher` and :meth:`SSLSocket.compress` methods require that the TLS connection has progressed beyond the TLS Client Hello and therefore @@ -1251,10 +1353,20 @@ to speed up repeated connections from the same clients. quite similarly to HTTP virtual hosts. Specifying *server_hostname* will raise a :exc:`ValueError` if *server_side* is true. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 Always allow a server_hostname to be passed, even if OpenSSL does not have SNI. +.. method:: SSLContext.wrap_bio(incoming, outgoing, server_side=False, \ + server_hostname=None) + + Create a new :class:`SSLObject` instance by wrapping the BIO objects + *incoming* and *outgoing*. The SSL routines will read input data from the + incoming BIO and write data to the outgoing BIO. + + The *server_side* and *server_hostname* parameters have the same meaning as + in :meth:`SSLContext.wrap_socket`. + .. method:: SSLContext.session_stats() Get statistics about the SSL sessions created or managed by this context. @@ -1627,7 +1739,7 @@ are finished with the client (or the client is finished with you):: And go back to listening for new client connections (of course, a real server would probably handle each client connection in a separate thread, or put -the sockets in non-blocking mode and use an event loop). +the sockets in :ref:`non-blocking mode <ssl-nonblocking>` and use an event loop). .. _ssl-nonblocking: @@ -1649,6 +1761,12 @@ thus several things you need to be aware of: socket first, and attempts to *read* from the SSL socket may require a prior *write* to the underlying socket. + .. versionchanged:: 3.5 + + In earlier Python versions, the :meth:`!SSLSocket.send` method + returned zero instead of raising :exc:`SSLWantWriteError` or + :exc:`SSLWantReadError`. + - Calling :func:`~select.select` tells you that the OS-level socket can be read from (or written to), but it does not imply that there is sufficient data at the upper SSL layer. For example, only part of an SSL frame might @@ -1681,13 +1799,143 @@ thus several things you need to be aware of: .. seealso:: - The :mod:`asyncio` module supports non-blocking SSL sockets and provides a + The :mod:`asyncio` module supports :ref:`non-blocking SSL sockets + <ssl-nonblocking>` and provides a higher level API. It polls for events using the :mod:`selectors` module and handles :exc:`SSLWantWriteError`, :exc:`SSLWantReadError` and :exc:`BlockingIOError` exceptions. It runs the SSL handshake asynchronously as well. +Memory BIO Support +------------------ + +.. versionadded:: 3.5 + +Ever since the SSL module was introduced in Python 2.6, the :class:`SSLSocket` +class has provided two related but distinct areas of functionality: + +- SSL protocol handling +- Network IO + +The network IO API is identical to that provided by :class:`socket.socket`, +from which :class:`SSLSocket` also inherits. This allows an SSL socket to be +used as a drop-in replacement for a regular socket, making it very easy to add +SSL support to an existing application. + +Combining SSL protocol handling and network IO usually works well, but there +are some cases where it doesn't. An example is async IO frameworks that want to +use a different IO multiplexing model than the "select/poll on a file +descriptor" (readiness based) model that is assumed by :class:`socket.socket` +and by the internal OpenSSL socket IO routines. This is mostly relevant for +platforms like Windows where this model is not efficient. For this purpose, a +reduced scope variant of :class:`SSLSocket` called :class:`SSLObject` is +provided. + +.. class:: SSLObject + + A reduced-scope variant of :class:`SSLSocket` representing an SSL protocol + instance that does not contain any network IO methods. This class is + typically used by framework authors that want to implement asynchronous IO + for SSL through memory buffers. + + This class implements an interface on top of a low-level SSL object as + implemented by OpenSSL. This object captures the state of an SSL connection + but does not provide any network IO itself. IO needs to be performed through + separate "BIO" objects which are OpenSSL's IO abstraction layer. + + An :class:`SSLObject` instance can be created using the + :meth:`~SSLContext.wrap_bio` method. This method will create the + :class:`SSLObject` instance and bind it to a pair of BIOs. The *incoming* + BIO is used to pass data from Python to the SSL protocol instance, while the + *outgoing* BIO is used to pass data the other way around. + + The following methods are available: + + - :attr:`~SSLSocket.context` + - :attr:`~SSLSocket.server_side` + - :attr:`~SSLSocket.server_hostname` + - :meth:`~SSLSocket.read` + - :meth:`~SSLSocket.write` + - :meth:`~SSLSocket.getpeercert` + - :meth:`~SSLSocket.selected_npn_protocol` + - :meth:`~SSLSocket.cipher` + - :meth:`~SSLSocket.shared_ciphers` + - :meth:`~SSLSocket.compression` + - :meth:`~SSLSocket.pending` + - :meth:`~SSLSocket.do_handshake` + - :meth:`~SSLSocket.unwrap` + - :meth:`~SSLSocket.get_channel_binding` + + When compared to :class:`SSLSocket`, this object lacks the following + features: + + - Any form of network IO incluging methods such as ``recv()`` and + ``send()``. + + - There is no *do_handshake_on_connect* machinery. You must always manually + call :meth:`~SSLSocket.do_handshake` to start the handshake. + + - There is no handling of *suppress_ragged_eofs*. All end-of-file conditions + that are in violation of the protocol are reported via the + :exc:`SSLEOFError` exception. + + - The method :meth:`~SSLSocket.unwrap` call does not return anything, + unlike for an SSL socket where it returns the underlying socket. + + - The *server_name_callback* callback passed to + :meth:`SSLContext.set_servername_callback` will get an :class:`SSLObject` + instance instead of a :class:`SSLSocket` instance as its first parameter. + + Some notes related to the use of :class:`SSLObject`: + + - All IO on an :class:`SSLObject` is :ref:`non-blocking <ssl-nonblocking>`. + This means that for example :meth:`~SSLSocket.read` will raise an + :exc:`SSLWantReadError` if it needs more data than the incoming BIO has + available. + + - There is no module-level ``wrap_bio()`` call like there is for + :meth:`~SSLContext.wrap_socket`. An :class:`SSLObject` is always created + via an :class:`SSLContext`. + +An SSLObject communicates with the outside world using memory buffers. The +class :class:`MemoryBIO` provides a memory buffer that can be used for this +purpose. It wraps an OpenSSL memory BIO (Basic IO) object: + +.. class:: MemoryBIO + + A memory buffer that can be used to pass data between Python and an SSL + protocol instance. + + .. attribute:: MemoryBIO.pending + + Return the number of bytes currently in the memory buffer. + + .. attribute:: MemoryBIO.eof + + A boolean indicating whether the memory BIO is current at the end-of-file + position. + + .. method:: MemoryBIO.read(n=-1) + + Read up to *n* bytes from the memory buffer. If *n* is not specified or + negative, all bytes are returned. + + .. method:: MemoryBIO.write(buf) + + Write the bytes from *buf* to the memory BIO. The *buf* argument must be an + object supporting the buffer protocol. + + The return value is the number of bytes written, which is always equal to + the length of *buf*. + + .. method:: MemoryBIO.write_eof() + + Write an EOF marker to the memory BIO. After this method has been called, it + is illegal to call :meth:`~MemoryBIO.write`. The attribute :attr:`eof` will + become true after all data currently in the buffer has been read. + + .. _ssl-security: Security considerations diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index 24769f6893..845b2ef7da 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -126,7 +126,7 @@ Example:: if __name__ == '__main__': walktree(sys.argv[1], visitfile) -An additional utility function is provided to covert a file's mode in a human +An additional utility function is provided to convert a file's mode in a human readable string: .. function:: filemode(mode) @@ -399,3 +399,29 @@ The following flags can be used in the *flags* argument of :func:`os.chflags`: The file is a snapshot file. See the \*BSD or Mac OS systems man page :manpage:`chflags(2)` for more information. + +On Windows, the following file attribute constants are available for use when +testing bits in the ``st_file_attributes`` member returned by :func:`os.stat`. +See the `Windows API documentation +<http://msdn.microsoft.com/en-us/library/windows/desktop/gg258117.aspx>`_ +for more detail on the meaning of these constants. + +.. data:: FILE_ATTRIBUTE_ARCHIVE + FILE_ATTRIBUTE_COMPRESSED + FILE_ATTRIBUTE_DEVICE + FILE_ATTRIBUTE_DIRECTORY + FILE_ATTRIBUTE_ENCRYPTED + FILE_ATTRIBUTE_HIDDEN + FILE_ATTRIBUTE_INTEGRITY_STREAM + FILE_ATTRIBUTE_NORMAL + FILE_ATTRIBUTE_NOT_CONTENT_INDEXED + FILE_ATTRIBUTE_NO_SCRUB_DATA + FILE_ATTRIBUTE_OFFLINE + FILE_ATTRIBUTE_READONLY + FILE_ATTRIBUTE_REPARSE_POINT + FILE_ATTRIBUTE_SPARSE_FILE + FILE_ATTRIBUTE_SYSTEM + FILE_ATTRIBUTE_TEMPORARY + FILE_ATTRIBUTE_VIRTUAL + + .. versionadded:: 3.5 diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index a4324b550b..26ff30988e 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -354,7 +354,7 @@ Notes: The numeric literals accepted include the digits ``0`` to ``9`` or any Unicode equivalent (code points with the ``Nd`` property). - See http://www.unicode.org/Public/6.3.0/ucd/extracted/DerivedNumericType.txt + See http://www.unicode.org/Public/7.0.0/ucd/extracted/DerivedNumericType.txt for a complete list of code points with the ``Nd`` property. @@ -3089,6 +3089,203 @@ place, and instead produce new objects. always produces a new object, even if no changes were made. +.. _bytes-formatting: + +``printf``-style Bytes Formatting +---------------------------------- + +.. index:: + single: formatting, bytes (%) + single: formatting, bytearray (%) + single: interpolation, bytes (%) + single: interpolation, bytearray (%) + single: bytes; formatting + single: bytearray; formatting + single: bytes; interpolation + single: bytearray; interpolation + single: printf-style formatting + single: sprintf-style formatting + single: % formatting + single: % interpolation + +.. note:: + + The formatting operations described here exhibit a variety of quirks that + lead to a number of common errors (such as failing to display tuples and + dictionaries correctly). If the value being printed may be a tuple or + dictionary, wrap it in a tuple. + +Bytes objects (``bytes``/``bytearray``) have one unique built-in operation: +the ``%`` operator (modulo). +This is also known as the bytes *formatting* or *interpolation* operator. +Given ``format % values`` (where *format* is a bytes object), ``%`` conversion +specifications in *format* are replaced with zero or more elements of *values*. +The effect is similar to using the :c:func:`sprintf` in the C language. + +If *format* requires a single argument, *values* may be a single non-tuple +object. [5]_ Otherwise, *values* must be a tuple with exactly the number of +items specified by the format bytes object, or a single mapping object (for +example, a dictionary). + +A conversion specifier contains two or more characters and has the following +components, which must occur in this order: + +#. The ``'%'`` character, which marks the start of the specifier. + +#. Mapping key (optional), consisting of a parenthesised sequence of characters + (for example, ``(somename)``). + +#. Conversion flags (optional), which affect the result of some conversion + types. + +#. Minimum field width (optional). If specified as an ``'*'`` (asterisk), the + actual width is read from the next element of the tuple in *values*, and the + object to convert comes after the minimum field width and optional precision. + +#. Precision (optional), given as a ``'.'`` (dot) followed by the precision. If + specified as ``'*'`` (an asterisk), the actual precision is read from the next + element of the tuple in *values*, and the value to convert comes after the + precision. + +#. Length modifier (optional). + +#. Conversion type. + +When the right argument is a dictionary (or other mapping type), then the +formats in the bytes object *must* include a parenthesised mapping key into that +dictionary inserted immediately after the ``'%'`` character. The mapping key +selects the value to be formatted from the mapping. For example: + + >>> print(b'%(language)s has %(number)03d quote types.' % + ... {b'language': b"Python", b"number": 2}) + b'Python has 002 quote types.' + +In this case no ``*`` specifiers may occur in a format (since they require a +sequential parameter list). + +The conversion flag characters are: + ++---------+---------------------------------------------------------------------+ +| Flag | Meaning | ++=========+=====================================================================+ +| ``'#'`` | The value conversion will use the "alternate form" (where defined | +| | below). | ++---------+---------------------------------------------------------------------+ +| ``'0'`` | The conversion will be zero padded for numeric values. | ++---------+---------------------------------------------------------------------+ +| ``'-'`` | The converted value is left adjusted (overrides the ``'0'`` | +| | conversion if both are given). | ++---------+---------------------------------------------------------------------+ +| ``' '`` | (a space) A blank should be left before a positive number (or empty | +| | string) produced by a signed conversion. | ++---------+---------------------------------------------------------------------+ +| ``'+'`` | A sign character (``'+'`` or ``'-'``) will precede the conversion | +| | (overrides a "space" flag). | ++---------+---------------------------------------------------------------------+ + +A length modifier (``h``, ``l``, or ``L``) may be present, but is ignored as it +is not necessary for Python -- so e.g. ``%ld`` is identical to ``%d``. + +The conversion types are: + ++------------+-----------------------------------------------------+-------+ +| Conversion | Meaning | Notes | ++============+=====================================================+=======+ +| ``'d'`` | Signed integer decimal. | | ++------------+-----------------------------------------------------+-------+ +| ``'i'`` | Signed integer decimal. | | ++------------+-----------------------------------------------------+-------+ +| ``'o'`` | Signed octal value. | \(1) | ++------------+-----------------------------------------------------+-------+ +| ``'u'`` | Obsolete type -- it is identical to ``'d'``. | \(8) | ++------------+-----------------------------------------------------+-------+ +| ``'x'`` | Signed hexadecimal (lowercase). | \(2) | ++------------+-----------------------------------------------------+-------+ +| ``'X'`` | Signed hexadecimal (uppercase). | \(2) | ++------------+-----------------------------------------------------+-------+ +| ``'e'`` | Floating point exponential format (lowercase). | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'E'`` | Floating point exponential format (uppercase). | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'f'`` | Floating point decimal format. | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'F'`` | Floating point decimal format. | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'g'`` | Floating point format. Uses lowercase exponential | \(4) | +| | format if exponent is less than -4 or not less than | | +| | precision, decimal format otherwise. | | ++------------+-----------------------------------------------------+-------+ +| ``'G'`` | Floating point format. Uses uppercase exponential | \(4) | +| | format if exponent is less than -4 or not less than | | +| | precision, decimal format otherwise. | | ++------------+-----------------------------------------------------+-------+ +| ``'c'`` | Single byte (accepts integer or single | | +| | byte objects). | | ++------------+-----------------------------------------------------+-------+ +| ``'b'`` | Bytes (any object that follows the | \(5) | +| | :ref:`buffer protocol <bufferobjects>` or has | | +| | :meth:`__bytes__`). | | ++------------+-----------------------------------------------------+-------+ +| ``'s'`` | ``'s'`` is an alias for ``'b'`` and should only | \(6) | +| | be used for Python2/3 code bases. | | ++------------+-----------------------------------------------------+-------+ +| ``'a'`` | Bytes (converts any Python object using | \(5) | +| | ``repr(obj).encode('ascii','backslashreplace)``). | | ++------------+-----------------------------------------------------+-------+ +| ``'r'`` | ``'r'`` is an alias for ``'a'`` and should only | \(7) | +| | be used for Python2/3 code bases. | | ++------------+-----------------------------------------------------+-------+ +| ``'%'`` | No argument is converted, results in a ``'%'`` | | +| | character in the result. | | ++------------+-----------------------------------------------------+-------+ + +Notes: + +(1) + The alternate form causes a leading zero (``'0'``) to be inserted between + left-hand padding and the formatting of the number if the leading character + of the result is not already a zero. + +(2) + The alternate form causes a leading ``'0x'`` or ``'0X'`` (depending on whether + the ``'x'`` or ``'X'`` format was used) to be inserted between left-hand padding + and the formatting of the number if the leading character of the result is not + already a zero. + +(3) + The alternate form causes the result to always contain a decimal point, even if + no digits follow it. + + The precision determines the number of digits after the decimal point and + defaults to 6. + +(4) + The alternate form causes the result to always contain a decimal point, and + trailing zeroes are not removed as they would otherwise be. + + The precision determines the number of significant digits before and after the + decimal point and defaults to 6. + +(5) + If precision is ``N``, the output is truncated to ``N`` characters. + +(6) + ``b'%s'`` is deprecated, but will not be removed during the 3.x series. + +(7) + ``b'%r'`` is deprecated, but will not be removed during the 3.x series. + +(8) + See :pep:`237`. + +.. note:: + + The bytearray version of this method does *not* operate in place - it + always produces a new object, even if no changes were made. + +.. seealso:: :pep:`461`. +.. versionadded:: 3.5 + .. _typememoryview: Memory Views @@ -3117,10 +3314,8 @@ copying. the view. The :class:`~memoryview.itemsize` attribute will give you the number of bytes in a single element. - A :class:`memoryview` supports slicing to expose its data. If - :class:`~memoryview.format` is one of the native format specifiers - from the :mod:`struct` module, indexing will return a single element - with the correct type. Full slicing will result in a subview:: + A :class:`memoryview` supports slicing and indexing to expose its data. + One-dimensional slicing will result in a subview:: >>> v = memoryview(b'abcefg') >>> v[1] @@ -3132,25 +3327,29 @@ copying. >>> bytes(v[1:4]) b'bce' - Other native formats:: + If :class:`~memoryview.format` is one of the native format specifiers + from the :mod:`struct` module, indexing with an integer or a tuple of + integers is also supported and returns a single *element* with + the correct type. One-dimensional memoryviews can be indexed + with an integer or a one-integer tuple. Multi-dimensional memoryviews + can be indexed with tuples of exactly *ndim* integers where *ndim* is + the number of dimensions. Zero-dimensional memoryviews can be indexed + with the empty tuple. + + Here is an example with a non-byte format:: >>> import array >>> a = array.array('l', [-11111111, 22222222, -33333333, 44444444]) - >>> a[0] + >>> m = memoryview(a) + >>> m[0] -11111111 - >>> a[-1] + >>> m[-1] 44444444 - >>> a[2:3].tolist() - [-33333333] - >>> a[::2].tolist() + >>> m[::2].tolist() [-11111111, -33333333] - >>> a[::-1].tolist() - [44444444, -33333333, 22222222, -11111111] - .. versionadded:: 3.3 - - If the underlying object is writable, the memoryview supports slice - assignment. Resizing is not allowed:: + If the underlying object is writable, the memoryview supports + one-dimensional slice assignment. Resizing is not allowed:: >>> data = bytearray(b'abcefg') >>> v = memoryview(data) @@ -3183,12 +3382,16 @@ copying. True .. versionchanged:: 3.3 + One-dimensional memoryviews can now be sliced. One-dimensional memoryviews with formats 'B', 'b' or 'c' are now hashable. .. versionchanged:: 3.4 memoryview is now registered automatically with :class:`collections.abc.Sequence` + .. versionchanged:: 3.5 + memoryviews can now be indexed with tuple of integers. + :class:`memoryview` has several methods: .. method:: __eq__(exporter) diff --git a/Doc/library/string.rst b/Doc/library/string.rst index f9da5fa7ad..eab9f370f6 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -95,6 +95,10 @@ implementation as the built-in :meth:`format` method. an arbitrary set of positional and keyword arguments. :meth:`format` is just a wrapper that calls :meth:`vformat`. + .. deprecated:: 3.5 + Passing a format string as keyword argument *format_string* has been + deprecated. + .. method:: vformat(format_string, args, kwargs) This function does the actual work of formatting. It is exposed as a diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 36cbf3c5f3..49a82bd92a 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -635,6 +635,7 @@ Instances of the :class:`Popen` class have the following methods: must be bytes or, if *universal_newlines* was ``True``, a string. :meth:`communicate` returns a tuple ``(stdout_data, stderr_data)``. + The data will be bytes or, if *universal_newlines* was ``True``, strings. Note that if you want to send data to the process's stdin, you need to create the Popen object with ``stdin=PIPE``. Similarly, to get anything other than diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 3024086a22..3080b69445 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -167,7 +167,7 @@ always available. .. data:: dont_write_bytecode - If this is true, Python won't try to write ``.pyc`` or ``.pyo`` files on the + If this is true, Python won't try to write ``.pyc`` files on the import of source modules. This value is initially set to ``True`` or ``False`` depending on the :option:`-B` command line option and the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable, but you can set it @@ -718,6 +718,14 @@ always available. value of :func:`intern` around to benefit from it. +.. function:: is_finalizing() + + Return :const:`True` if the Python interpreter is + :term:`shutting down <interpreter shutdown>`, :const:`False` otherwise. + + .. versionadded:: 3.5 + + .. data:: last_type last_value last_traceback @@ -1223,4 +1231,3 @@ always available. .. rubric:: Citations .. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf\ . - diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index 4536ecaab0..4fd94fd90d 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -62,6 +62,23 @@ Some facts and figures: +------------------+---------------------------------------------+ | ``'r:xz'`` | Open for reading with lzma compression. | +------------------+---------------------------------------------+ + | ``'x'`` or | Create a tarfile exclusively without | + | ``'x:'`` | compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ + | ``'x:gz'`` | Create a tarfile with gzip compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ + | ``'x:bz2'`` | Create a tarfile with bzip2 compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ + | ``'x:xz'`` | Create a tarfile with lzma compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ | ``'a' or 'a:'`` | Open for appending with no compression. The | | | file is created if it does not exist. | +------------------+---------------------------------------------+ @@ -82,9 +99,9 @@ Some facts and figures: If *fileobj* is specified, it is used as an alternative to a :term:`file object` opened in binary mode for *name*. It is supposed to be at position 0. - For modes ``'w:gz'``, ``'r:gz'``, ``'w:bz2'``, ``'r:bz2'``, :func:`tarfile.open` - accepts the keyword argument *compresslevel* to specify the compression level of - the file. + For modes ``'w:gz'``, ``'r:gz'``, ``'w:bz2'``, ``'r:bz2'``, ``'x:gz'``, + ``'x:bz2'``, :func:`tarfile.open` accepts the keyword argument + *compresslevel* to specify the compression level of the file. For special purposes, there is a second format for *mode*: ``'filemode|[compression]'``. :func:`tarfile.open` will return a :class:`TarFile` @@ -127,6 +144,8 @@ Some facts and figures: | | writing. | +-------------+--------------------------------------------+ + .. versionchanged:: 3.5 + The ``'x'`` (exclusive creation) mode was added. .. class:: TarFile @@ -252,8 +271,8 @@ be finalized; only the internally used file object will be closed. See the In this case, the file object's :attr:`name` attribute is used if it exists. *mode* is either ``'r'`` to read from an existing archive, ``'a'`` to append - data to an existing file or ``'w'`` to create a new file overwriting an existing - one. + data to an existing file, ``'w'`` to create a new file overwriting an existing + one or ``'x'`` to create a new file only if it's not exists. If *fileobj* is given, it is used for reading or writing data. If it can be determined, *mode* is overridden by *fileobj*'s mode. *fileobj* will be used @@ -292,12 +311,14 @@ be finalized; only the internally used file object will be closed. See the to be handled. The default settings will work for most users. See section :ref:`tar-unicode` for in-depth information. - .. versionchanged:: 3.2 - Use ``'surrogateescape'`` as the default for the *errors* argument. - The *pax_headers* argument is an optional dictionary of strings which will be added as a pax global header if *format* is :const:`PAX_FORMAT`. + .. versionchanged:: 3.2 + Use ``'surrogateescape'`` as the default for the *errors* argument. + + .. versionchanged:: 3.5 + The ``'x'`` (exclusive creation) mode was added. .. classmethod:: TarFile.open(...) @@ -328,11 +349,15 @@ be finalized; only the internally used file object will be closed. See the returned by :meth:`getmembers`. -.. method:: TarFile.list(verbose=True) +.. method:: TarFile.list(verbose=True, *, members=None) Print a table of contents to ``sys.stdout``. If *verbose* is :const:`False`, only the names of the members are printed. If it is :const:`True`, output - similar to that of :program:`ls -l` is produced. + similar to that of :program:`ls -l` is produced. If optional *members* is + given, it must be a subset of the list returned by :meth:`getmembers`. + + .. versionchanged:: 3.5 + Added the *members* parameter. .. method:: TarFile.next() diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 44d025defe..2c6837744b 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -54,6 +54,13 @@ The module defines the following user-callable items: underlying true file object. This file-like object can be used in a :keyword:`with` statement, just like a normal file. + The :py:data:`os.O_TMPFILE` flag is used if it is available and works + (Linux-specific, require Linux kernel 3.11 or later). + + .. versionchanged:: 3.5 + + The :py:data:`os.O_TMPFILE` flag is now used if available. + .. function:: NamedTemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None, delete=True) diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index 0133a8f5e7..5269994781 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -630,7 +630,7 @@ item to the buffer only needs to wake up one consumer thread. cv.wait() Therefore, the same rules apply as with :meth:`wait`: The lock must be - held when called and is re-aquired on return. The predicate is evaluated + held when called and is re-acquired on return. The predicate is evaluated with the lock held. .. versionadded:: 3.2 diff --git a/Doc/library/time.rst b/Doc/library/time.rst index 8d8b7d4c55..3d335c8867 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -314,9 +314,9 @@ The module defines the following functions and data items: processes running for more than 49 days. On more recent versions of Windows and on other operating systems, :func:`monotonic` is system-wide. - Availability: Windows, Mac OS X, Linux, FreeBSD, OpenBSD, Solaris. - .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now always available. .. function:: perf_counter() @@ -350,6 +350,11 @@ The module defines the following functions and data items: requested by an arbitrary amount because of the scheduling of other activity in the system. + .. versionchanged:: 3.5 + The function now sleeps at least *secs* even if the sleep is interrupted + by a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale). + .. function:: strftime(format[, t]) diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst index 503a705fbc..3ef9f3e396 100644 --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -59,18 +59,26 @@ Python Interface The module defines three convenience functions and a public class: -.. function:: timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000) +.. function:: timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000, globals=None) Create a :class:`Timer` instance with the given statement, *setup* code and *timer* function and run its :meth:`.timeit` method with *number* executions. + The optional *globals* argument specifies a namespace in which to execute the + code. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. -.. function:: repeat(stmt='pass', setup='pass', timer=<default timer>, repeat=3, number=1000000) + +.. function:: repeat(stmt='pass', setup='pass', timer=<default timer>, repeat=3, number=1000000, globals=None) Create a :class:`Timer` instance with the given statement, *setup* code and *timer* function and run its :meth:`.repeat` method with the given *repeat* - count and *number* executions. + count and *number* executions. The optional *globals* argument specifies a + namespace in which to execute the code. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. .. function:: default_timer() @@ -80,7 +88,7 @@ The module defines three convenience functions and a public class: :func:`time.perf_counter` is now the default timer. -.. class:: Timer(stmt='pass', setup='pass', timer=<timer function>) +.. class:: Timer(stmt='pass', setup='pass', timer=<timer function>, globals=None) Class for timing execution speed of small code snippets. @@ -88,7 +96,9 @@ The module defines three convenience functions and a public class: for setup, and a timer function. Both statements default to ``'pass'``; the timer function is platform-dependent (see the module doc string). *stmt* and *setup* may also contain multiple statements separated by ``;`` - or newlines, as long as they don't contain multi-line string literals. + or newlines, as long as they don't contain multi-line string literals. The + statement will by default be executed within timeit's namespace; this behavior + can be controlled by passing a namespace to *globals*. To measure the execution time of the first statement, use the :meth:`.timeit` method. The :meth:`.repeat` method is a convenience to call :meth:`.timeit` @@ -99,6 +109,8 @@ The module defines three convenience functions and a public class: will then be executed by :meth:`.timeit`. Note that the timing overhead is a little larger in this case because of the extra function calls. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. .. method:: Timer.timeit(number=1000000) @@ -167,7 +179,7 @@ Command-Line Interface When called as a program from the command line, the following form is used:: - python -m timeit [-n N] [-r N] [-s S] [-t] [-c] [-h] [statement ...] + python -m timeit [-n N] [-r N] [-u U] [-s S] [-t] [-c] [-h] [statement ...] Where the following options are understood: @@ -196,6 +208,12 @@ Where the following options are understood: use :func:`time.time` (deprecated) +.. cmdoption:: -u, --unit=U + + specify a time unit for timer output; can select usec, msec, or sec + + .. versionadded:: 3.5 + .. cmdoption:: -c, --clock use :func:`time.clock` (deprecated) @@ -318,3 +336,17 @@ To give the :mod:`timeit` module access to functions you define, you can pass a if __name__ == '__main__': import timeit print(timeit.timeit("test()", setup="from __main__ import test")) + +Another option is to pass :func:`globals` to the *globals* parameter, which will cause the code +to be executed within your current global namespace. This can be more convenient +than individually specifying imports:: + + def f(x): + return x**2 + def g(x): + return x**4 + def h(x): + return x**8 + + import timeit + print(timeit.timeit('[func(42) for func in (f,g,h)]', globals=globals())) diff --git a/Doc/library/tkinter.ttk.rst b/Doc/library/tkinter.ttk.rst index 6f8bf1c315..b0eefcb702 100644 --- a/Doc/library/tkinter.ttk.rst +++ b/Doc/library/tkinter.ttk.rst @@ -1167,7 +1167,7 @@ Ttk Styling Each widget in :mod:`ttk` is assigned a style, which specifies the set of elements making up the widget and how they are arranged, along with dynamic and default settings for element options. By default the style name is the -same as the widget's class name, but it may be overriden by the widget's style +same as the widget's class name, but it may be overridden by the widget's style option. If you don't know the class name of a widget, use the method :meth:`Misc.winfo_class` (somewidget.winfo_class()). diff --git a/Doc/library/token.rst b/Doc/library/token.rst index 4cd709814c..88fb38bc1d 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -93,6 +93,7 @@ The token constants are: DOUBLESLASH DOUBLESLASHEQUAL AT + ATEQUAL RARROW ELLIPSIS OP diff --git a/Doc/library/traceback.rst b/Doc/library/traceback.rst index 15fbedcf33..c6b5d0c956 100644 --- a/Doc/library/traceback.rst +++ b/Doc/library/traceback.rst @@ -136,6 +136,162 @@ The module defines the following functions: .. versionadded:: 3.4 +.. function:: walk_stack(f) + + Walk a stack following ``f.f_back`` from the given frame, yielding the frame + and line number for each frame. If *f* is ``None``, the current stack is + used. This helper is used with :meth:`StackSummary.extract`. + + .. versionadded:: 3.5 + +.. function:: walk_tb(tb) + + Walk a traceback following ``tb_next`` yielding the frame and line number + for each frame. This helper is used with :meth:`StackSummary.extract`. + + .. versionadded:: 3.5 + +The module also defines the following classes: + +:class:`TracebackException` Objects +----------------------------------- + +.. versionadded:: 3.5 + +:class:`TracebackException` objects are created from actual exceptions to +capture data for later printing in a lightweight fashion. + +.. class:: TracebackException(exc_type, exc_value, exc_traceback, *, limit=None, lookup_lines=True, capture_locals=False) + + Capture an exception for later rendering. *limit*, *lookup_lines* and + *capture_locals* are as for the :class:`StackSummary` class. + + Note that when locals are captured, they are also shown in the traceback. + + .. attribute:: __cause__ + + A :class:`TracebackException` of the original ``__cause__``. + + .. attribute:: __context__ + + A :class:`TracebackException` of the original ``__context__``. + + .. attribute:: __suppress_context__ + + The ``__suppress_context__`` value from the original exception. + + .. attribute:: stack + + A :class:`StackSummary` representing the traceback. + + .. attribute:: exc_type + + The class of the original traceback. + + .. attribute:: filename + + For syntax errors - the file name where the error occurred. + + .. attribute:: lineno + + For syntax errors - the line number where the error occurred. + + .. attribute:: text + + For syntax errors - the text where the error occurred. + + .. attribute:: offset + + For syntax errors - the offset into the text where the error occurred. + + .. attribute:: msg + + For syntax errors - the compiler error message. + + .. classmethod:: from_exception(exc, *, limit=None, lookup_lines=True, capture_locals=False) + + Capture an exception for later rendering. *limit*, *lookup_lines* and + *capture_locals* are as for the :class:`StackSummary` class. + + Note that when locals are captured, they are also shown in the traceback. + + .. method:: format(*, chain=True) + + Format the exception. + + If *chain* is not ``True``, ``__cause__`` and ``__context__`` will not + be formatted. + + The return value is a generator of strings, each ending in a newline and + some containing internal newlines. :func:`~traceback.print_exception` + is a wrapper around this method which just prints the lines to a file. + + The message indicating which exception occurred is always the last + string in the output. + + .. method:: format_exception_only() + + Format the exception part of the traceback. + + The return value is a generator of strings, each ending in a newline. + + Normally, the generator emits a single string; however, for + :exc:`SyntaxError` exceptions, it emits several lines that (when + printed) display detailed information about where the syntax + error occurred. + + The message indicating which exception occurred is always the last + string in the output. + + +:class:`StackSummary` Objects +----------------------------- + +.. versionadded:: 3.5 + +:class:`StackSummary` objects represent a call stack ready for formatting. + +.. class:: StackSummary + + .. classmethod:: extract(frame_gen, *, limit=None, lookup_lines=True, capture_locals=False) + + Construct a :class:`StackSummary` object from a frame generator (such as + is returned by :func:`~traceback.walk_stack` or + :func:`~traceback.walk_tb`). + + If *limit* is supplied, only this many frames are taken from *frame_gen*. + If *lookup_lines* is ``False``, the returned :class:`FrameSummary` + objects will not have read their lines in yet, making the cost of + creating the :class:`StackSummary` cheaper (which may be valuable if it + may not actually get formatted). If *capture_locals* is ``True`` the + local variables in each :class:`FrameSummary` are captured as object + representations. + + .. classmethod:: from_list(a_list) + + Construct a :class:`StackSummary` object from a supplied old-style list + of tuples. Each tuple should be a 4-tuple with filename, lineno, name, + line as the elements. + + +:class:`FrameSummary` Objects +----------------------------- + +.. versionadded:: 3.5 + +:class:`FrameSummary` objects represent a single frame in a traceback. + +.. class:: FrameSummary(filename, lineno, name, lookup_line=True, locals=None, line=None) + + Represent a single frame in the traceback or stack that is being formatted + or printed. It may optionally have a stringified version of the frames + locals included in it. If *lookup_line* is ``False``, the source code is not + looked up until the :class:`FrameSummary` has the :attr:`~FrameSummary.line` + attribute accessed (which also happens when casting it to a tuple). + :attr:`~FrameSummary.line` may be directly provided, and will prevent line + lookups happening at all. *locals* is an optional local variable + dictionary, and if supplied the variable representations are stored in the + summary for later display. .. _traceback-example: diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst index 552e84b4e0..13c81a7fed 100644 --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -350,7 +350,7 @@ Functions the *nframe* parameter of the :func:`start` function to store more frames. The :mod:`tracemalloc` module must be tracing memory allocations to take a - snapshot, see the the :func:`start` function. + snapshot, see the :func:`start` function. See also the :func:`get_object_traceback` function. @@ -363,7 +363,7 @@ Filter Filter on traces of memory blocks. See the :func:`fnmatch.fnmatch` function for the syntax of - *filename_pattern*. The ``'.pyc'`` and ``'.pyo'`` file extensions are + *filename_pattern*. The ``'.pyc'`` file extension is replaced with ``'.py'``. Examples: @@ -374,6 +374,10 @@ Filter :mod:`tracemalloc` module * ``Filter(False, "<unknown>")`` excludes empty tracebacks + + .. versionchanged:: 3.5 + The ``'.pyo'`` file extension is no longer replaced with ``'.py'``. + .. attribute:: inclusive If *inclusive* is ``True`` (include), only trace memory blocks allocated @@ -631,4 +635,3 @@ Traceback obj = Object() File "test.py", line 12 tb = tracemalloc.get_object_traceback(f()) - diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index 5899f3dddc..30dd6eff2d 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -1809,7 +1809,7 @@ Input methods Pop up a dialog window for input of a number. title is the title of the dialog window, prompt is a text mostly describing what numerical information - to input. default: default value, minval: minimum value for imput, + to input. default: default value, minval: minimum value for input, maxval: maximum value for input The number input must be in the range minval .. maxval if these are given. If not, a hint is issued and the dialog remains open for @@ -1879,7 +1879,7 @@ Settings and special methods >>> cv = screen.getcanvas() >>> cv - <turtle.ScrolledCanvas object at ...> + <turtle.ScrolledCanvas object ...> .. function:: getshapes() @@ -2351,6 +2351,9 @@ The demo scripts are: | | pairwise in opposite | shapesize, tilt, | | | direction | get_shapepoly, update | +----------------+------------------------------+-----------------------+ +| sorting_animate| visual demonstration of | simple alignment, | +| | different sorting methods | randomization | ++----------------+------------------------------+-----------------------+ | tree | a (graphical) breadth | :func:`clone` | | | first tree (using generators)| | +----------------+------------------------------+-----------------------+ @@ -2402,7 +2405,7 @@ Changes since Python 3.0 Accordingly the latter has got an alias: :meth:`Screen.onkeyrelease`. - The method :meth:`Screen.mainloop` has been added. So when working only - with Screen and Turtle objects one must not additonally import + with Screen and Turtle objects one must not additionally import :func:`mainloop` anymore. - Two input methods has been added :meth:`Screen.textinput` and diff --git a/Doc/library/types.rst b/Doc/library/types.rst index abdb939d1a..34fffe6863 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -115,6 +115,10 @@ Standard names are defined for the following types: The type of :term:`modules <module>`. Constructor takes the name of the module to be created and optionally its :term:`docstring`. + .. note:: + Use :func:`importlib.util.module_from_spec` to create a new module if you + wish to set the various import-controlled attributes. + .. attribute:: __doc__ The :term:`docstring` of the module. Defaults to ``None``. diff --git a/Doc/library/unicodedata.rst b/Doc/library/unicodedata.rst index 3b3d3a0a8f..24ddef8a32 100644 --- a/Doc/library/unicodedata.rst +++ b/Doc/library/unicodedata.rst @@ -15,8 +15,8 @@ This module provides access to the Unicode Character Database (UCD) which defines character properties for all Unicode characters. The data contained in -this database is compiled from the `UCD version 6.3.0 -<http://www.unicode.org/Public/6.3.0/ucd>`_. +this database is compiled from the `UCD version 7.0.0 +<http://www.unicode.org/Public/7.0.0/ucd>`_. The module uses the same names and symbols as defined by Unicode Standard Annex #44, `"Unicode Character Database" @@ -166,6 +166,6 @@ Examples: .. rubric:: Footnotes -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NameAliases.txt +.. [#] http://www.unicode.org/Public/7.0.0/ucd/NameAliases.txt -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NamedSequences.txt +.. [#] http://www.unicode.org/Public/7.0.0/ucd/NamedSequences.txt diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index 91d5a27827..f76c301724 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -181,71 +181,77 @@ The Mock Class -------------- -`Mock` is a flexible mock object intended to replace the use of stubs and +:class:`Mock` is a flexible mock object intended to replace the use of stubs and test doubles throughout your code. Mocks are callable and create attributes as new mocks when you access them [#]_. Accessing the same attribute will always return the same mock. Mocks record how you use them, allowing you to make assertions about what your code has done to them. -:class:`MagicMock` is a subclass of `Mock` with all the magic methods +:class:`MagicMock` is a subclass of :class:`Mock` with all the magic methods pre-created and ready to use. There are also non-callable variants, useful when you are mocking out objects that aren't callable: :class:`NonCallableMock` and :class:`NonCallableMagicMock` The :func:`patch` decorators makes it easy to temporarily replace classes -in a particular module with a `Mock` object. By default `patch` will create -a `MagicMock` for you. You can specify an alternative class of `Mock` using -the `new_callable` argument to `patch`. +in a particular module with a :class:`Mock` object. By default :func:`patch` will create +a :class:`MagicMock` for you. You can specify an alternative class of :class:`Mock` using +the *new_callable* argument to :func:`patch`. -.. class:: Mock(spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, **kwargs) +.. class:: Mock(spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, unsafe=False, **kwargs) - Create a new `Mock` object. `Mock` takes several optional arguments + Create a new :class:`Mock` object. :class:`Mock` takes several optional arguments that specify the behaviour of the Mock object: - * `spec`: This can be either a list of strings or an existing object (a + * *spec*: This can be either a list of strings or an existing object (a class or instance) that acts as the specification for the mock object. If you pass in an object then a list of strings is formed by calling dir on the object (excluding unsupported magic attributes and methods). - Accessing any attribute not in this list will raise an `AttributeError`. + Accessing any attribute not in this list will raise an :exc:`AttributeError`. - If `spec` is an object (rather than a list of strings) then + If *spec* is an object (rather than a list of strings) then :attr:`~instance.__class__` returns the class of the spec object. This - allows mocks to pass `isinstance` tests. + allows mocks to pass :func:`isinstance` tests. - * `spec_set`: A stricter variant of `spec`. If used, attempting to *set* + * *spec_set*: A stricter variant of *spec*. If used, attempting to *set* or get an attribute on the mock that isn't on the object passed as - `spec_set` will raise an `AttributeError`. + *spec_set* will raise an :exc:`AttributeError`. - * `side_effect`: A function to be called whenever the Mock is called. See + * *side_effect*: A function to be called whenever the Mock is called. See the :attr:`~Mock.side_effect` attribute. Useful for raising exceptions or dynamically changing return values. The function is called with the same arguments as the mock, and unless it returns :data:`DEFAULT`, the return value of this function is used as the return value. - Alternatively `side_effect` can be an exception class or instance. In + Alternatively *side_effect* can be an exception class or instance. In this case the exception will be raised when the mock is called. - If `side_effect` is an iterable then each call to the mock will return + If *side_effect* is an iterable then each call to the mock will return the next value from the iterable. - A `side_effect` can be cleared by setting it to `None`. + A *side_effect* can be cleared by setting it to ``None``. - * `return_value`: The value returned when the mock is called. By default + * *return_value*: The value returned when the mock is called. By default this is a new Mock (created on first access). See the :attr:`return_value` attribute. - * `wraps`: Item for the mock object to wrap. If `wraps` is not None then + * *unsafe*: By default if any attribute starts with *assert* or + *assret* will raise an :exc:`AttributeError`. Passing ``unsafe=True`` + will allow access to these attributes. + + .. versionadded:: 3.5 + + * *wraps*: Item for the mock object to wrap. If *wraps* is not None then calling the Mock will pass the call through to the wrapped object (returning the real result). Attribute access on the mock will return a Mock object that wraps the corresponding attribute of the wrapped object (so attempting to access an attribute that doesn't exist will - raise an `AttributeError`). + raise an :exc:`AttributeError`). - If the mock has an explicit `return_value` set then calls are not passed - to the wrapped object and the `return_value` is returned instead. + If the mock has an explicit *return_value* set then calls are not passed + to the wrapped object and the *return_value* is returned instead. - * `name`: If the mock has a name then it will be used in the repr of the + * *name*: If the mock has a name then it will be used in the repr of the mock. This can be useful for debugging. The name is propagated to child mocks. @@ -315,6 +321,20 @@ the `new_callable` argument to `patch`. >>> calls = [call(4), call(2), call(3)] >>> mock.assert_has_calls(calls, any_order=True) + .. method:: assert_not_called(*args, **kwargs) + + Assert the mock was never called. + + >>> m = Mock() + >>> m.hello.assert_not_called() + >>> obj = m.hello() + >>> m.hello.assert_not_called() + Traceback (most recent call last): + ... + AssertionError: Expected 'hello' to not have been called. Called 1 times. + + .. versionadded:: 3.5 + .. method:: reset_mock() @@ -1032,25 +1052,31 @@ patch default because it can be dangerous. With it switched on you can write passing tests against APIs that don't actually exist! - Patch can be used as a `TestCase` class decorator. It works by + .. note:: + + .. versionchanged:: 3.5 + If you are patching builtins in a module then you don't + need to pass ``create=True``, it will be added by default. + + Patch can be used as a :class:`TestCase` class decorator. It works by decorating each test method in the class. This reduces the boilerplate - code when your test methods share a common patchings set. `patch` finds - tests by looking for method names that start with `patch.TEST_PREFIX`. - By default this is `test`, which matches the way `unittest` finds tests. - You can specify an alternative prefix by setting `patch.TEST_PREFIX`. + code when your test methods share a common patchings set. :func:`patch` finds + tests by looking for method names that start with ``patch.TEST_PREFIX``. + By default this is ``'test'``, which matches the way :mod:`unittest` finds tests. + You can specify an alternative prefix by setting ``patch.TEST_PREFIX``. Patch can be used as a context manager, with the with statement. Here the patching applies to the indented block after the with statement. If you use "as" then the patched object will be bound to the name after the - "as"; very useful if `patch` is creating a mock object for you. + "as"; very useful if :func:`patch` is creating a mock object for you. - `patch` takes arbitrary keyword arguments. These will be passed to - the `Mock` (or `new_callable`) on construction. + :func:`patch` takes arbitrary keyword arguments. These will be passed to + the :class:`Mock` (or *new_callable*) on construction. - `patch.dict(...)`, `patch.multiple(...)` and `patch.object(...)` are + ``patch.dict(...)``, ``patch.multiple(...)`` and ``patch.object(...)`` are available for alternate use-cases. -`patch` as function decorator, creating the mock for you and passing it into +:func:`patch` as function decorator, creating the mock for you and passing it into the decorated function: >>> @patch('__main__.SomeClass') @@ -1392,15 +1418,31 @@ method of a :class:`TestCase`: ... assert package.module.Class is self.MockClass ... - As an added bonus you no longer need to keep a reference to the `patcher` + As an added bonus you no longer need to keep a reference to the ``patcher`` object. It is also possible to stop all patches which have been started by using -`patch.stopall`. +:func:`patch.stopall`. .. function:: patch.stopall - Stop all active patches. Only stops patches started with `start`. + Stop all active patches. Only stops patches started with ``start``. + + +.. _patch-builtins: + +patch builtins +~~~~~~~~~~~~~~ +You can patch any builtins within a module. The following example patches +builtin :func:`ord`: + + >>> @patch('__main__.ord') + ... def test(mock_ord): + ... mock_ord.return_value = 101 + ... print(ord('c')) + ... + >>> test() + 101 TEST_PREFIX @@ -1587,7 +1629,7 @@ The full list of supported magic methods is: * Context manager: ``__enter__`` and ``__exit__`` * Unary numeric methods: ``__neg__``, ``__pos__`` and ``__invert__`` * The numeric methods (including right hand and in-place variants): - ``__add__``, ``__sub__``, ``__mul__``, ``__div__``,``__truediv__``, + ``__add__``, ``__sub__``, ``__mul__``, ``__matmul__``, ``__div__``, ``__truediv__``, ``__floordiv__``, ``__mod__``, ``__divmod__``, ``__lshift__``, ``__rshift__``, ``__and__``, ``__xor__``, ``__or__``, and ``__pow__`` * Numeric conversion methods: ``__complex__``, ``__int__``, ``__float__`` @@ -2013,7 +2055,7 @@ Mocking context managers with a :class:`MagicMock` is common enough and fiddly enough that a helper function is useful. >>> m = mock_open() - >>> with patch('__main__.open', m, create=True): + >>> with patch('__main__.open', m): ... with open('foo', 'w') as h: ... h.write('some stuff') ... @@ -2028,7 +2070,7 @@ enough that a helper function is useful. And for reading files: - >>> with patch('__main__.open', mock_open(read_data='bibble'), create=True) as m: + >>> with patch('__main__.open', mock_open(read_data='bibble')) as m: ... with open('foo') as h: ... result = h.read() ... diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index bdb18bc41f..70f99a0772 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -214,9 +214,16 @@ Command-line options Stop the test run on the first error or failure. +.. cmdoption:: --locals + + Show local variables in tracebacks. + .. versionadded:: 3.2 The command-line options ``-b``, ``-c`` and ``-f`` were added. +.. versionadded:: 3.5 + The command-line option ``--locals``. + The command line can also be used for test discovery, for running all of the tests in a project or just a subset. @@ -1543,6 +1550,20 @@ Loading and running tests :data:`unittest.defaultTestLoader`. Using a subclass or instance, however, allows customization of some configurable properties. + :class:`TestLoader` objects have the following attributes: + + + .. attribute:: errors + + A list of the non-fatal errors encountered while loading tests. Not reset + by the loader at any point. Fatal errors are signalled by the relevant + a method raising an exception to the caller. Non-fatal errors are also + indicated by a synthetic test that will raise the original error when + run. + + .. versionadded:: 3.5 + + :class:`TestLoader` objects have the following methods: @@ -1552,7 +1573,7 @@ Loading and running tests :class:`testCaseClass`. - .. method:: loadTestsFromModule(module) + .. method:: loadTestsFromModule(module, pattern=None) Return a suite of all tests cases contained in the given module. This method searches *module* for classes derived from :class:`TestCase` and @@ -1569,11 +1590,18 @@ Loading and running tests If a module provides a ``load_tests`` function it will be called to load the tests. This allows modules to customize test loading. - This is the `load_tests protocol`_. + This is the `load_tests protocol`_. The *pattern* argument is passed as + the third argument to ``load_tests``. .. versionchanged:: 3.2 Support for ``load_tests`` added. + .. versionchanged:: 3.5 + The undocumented and unofficial *use_load_tests* default argument is + deprecated and ignored, although it is still accepted for backward + compatibility. The method also now accepts a keyword-only argument + *pattern* which is passed to ``load_tests`` as the third argument. + .. method:: loadTestsFromName(name, module=None) @@ -1599,6 +1627,12 @@ Loading and running tests The method optionally resolves *name* relative to the given *module*. + .. versionchanged:: 3.5 + If an :exc:`ImportError` or :exc:`AttributeError` occurs while traversing + *name* then a synthetic test that raises that error when run will be + returned. These errors are included in the errors accumulated by + self.errors. + .. method:: loadTestsFromNames(names, module=None) @@ -1625,18 +1659,22 @@ Loading and running tests the start directory is not the top level directory then the top level directory must be specified separately. - If importing a module fails, for example due to a syntax error, then this - will be recorded as a single error and discovery will continue. If the - import failure is due to :exc:`SkipTest` being raised, it will be recorded - as a skip instead of an error. + If importing a module fails, for example due to a syntax error, then + this will be recorded as a single error and discovery will continue. If + the import failure is due to :exc:`SkipTest` being raised, it will be + recorded as a skip instead of an error. - If a test package name (directory with :file:`__init__.py`) matches the - pattern then the package will be checked for a ``load_tests`` - function. If this exists then it will be called with *loader*, *tests*, - *pattern*. + If a package (a directory containing a file named :file:`__init__.py`) is + found, the package will be checked for a ``load_tests`` function. If this + exists then it will be called + ``package.load_tests(loader, tests, pattern)``. Test discovery takes care + to ensure that a package is only checked for tests once during an + invocation, even if the load_tests function itself calls + ``loader.discover``. - If ``load_tests`` exists then discovery does *not* recurse into the package, - ``load_tests`` is responsible for loading all tests in the package. + If ``load_tests`` exists then discovery does *not* recurse into the + package, ``load_tests`` is responsible for loading all tests in the + package. The pattern is deliberately not stored as a loader attribute so that packages can continue discovery themselves. *top_level_dir* is stored so @@ -1655,6 +1693,11 @@ Loading and running tests the same even if the underlying file system's ordering is not dependent on file name. + .. versionchanged:: 3.5 + Found packages are now checked for ``load_tests`` regardless of + whether their path matches *pattern*, because it is impossible for + a package name to match the default pattern. + The following attributes of a :class:`TestLoader` can be configured either by subclassing or assignment on an instance: @@ -1737,12 +1780,10 @@ Loading and running tests Set to ``True`` when the execution of tests should stop by :meth:`stop`. - .. attribute:: testsRun The total number of tests run so far. - .. attribute:: buffer If set to true, ``sys.stdout`` and ``sys.stderr`` will be buffered in between @@ -1752,7 +1793,6 @@ Loading and running tests .. versionadded:: 3.2 - .. attribute:: failfast If set to true :meth:`stop` will be called on the first failure or error, @@ -1760,6 +1800,11 @@ Loading and running tests .. versionadded:: 3.2 + .. attribute:: tb_locals + + If set to true then local variables will be shown in tracebacks. + + .. versionadded:: 3.5 .. method:: wasSuccessful() @@ -1770,7 +1815,6 @@ Loading and running tests Returns ``False`` if there were any :attr:`unexpectedSuccesses` from tests marked with the :func:`expectedFailure` decorator. - .. method:: stop() This method can be called to signal that the set of tests being run should @@ -1902,12 +1946,14 @@ Loading and running tests .. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, failfast=False, \ - buffer=False, resultclass=None, warnings=None) + buffer=False, resultclass=None, warnings=None, *, tb_locals=False) A basic test runner implementation that outputs results to a stream. If *stream* is ``None``, the default, :data:`sys.stderr` is used as the output stream. This class has a few configurable parameters, but is essentially very simple. Graphical - applications which run test suites should provide alternate implementations. + applications which run test suites should provide alternate implementations. Such + implementations should accept ``**kwargs`` as the interface to construct runners + changes when features are added to unittest. By default this runner shows :exc:`DeprecationWarning`, :exc:`PendingDeprecationWarning`, :exc:`ResourceWarning` and @@ -1926,6 +1972,9 @@ Loading and running tests The default stream is set to :data:`sys.stderr` at instantiation time rather than import time. + .. versionchanged:: 3.5 + Added the tb_locals parameter. + .. method:: _makeResult() This method returns the instance of ``TestResult`` used by :meth:`run`. @@ -2023,7 +2072,10 @@ test runs or test discovery by implementing a function called ``load_tests``. If a test module defines ``load_tests`` it will be called by :meth:`TestLoader.loadTestsFromModule` with the following arguments:: - load_tests(loader, standard_tests, None) + load_tests(loader, standard_tests, pattern) + +where *pattern* is passed straight through from ``loadTestsFromModule``. It +defaults to ``None``. It should return a :class:`TestSuite`. @@ -2045,21 +2097,12 @@ A typical ``load_tests`` function that loads tests from a specific set of suite.addTests(tests) return suite -If discovery is started, either from the command line or by calling -:meth:`TestLoader.discover`, with a pattern that matches a package -name then the package :file:`__init__.py` will be checked for ``load_tests``. - -.. note:: - - The default pattern is ``'test*.py'``. This matches all Python files - that start with ``'test'`` but *won't* match any test directories. - - A pattern like ``'test*'`` will match test packages as well as - modules. - -If the package :file:`__init__.py` defines ``load_tests`` then it will be -called and discovery not continued into the package. ``load_tests`` -is called with the following arguments:: +If discovery is started in a directory containing a package, either from the +command line or by calling :meth:`TestLoader.discover`, then the package +:file:`__init__.py` will be checked for ``load_tests``. If that function does +not exist, discovery will recurse into the package as though it were just +another directory. Otherwise, discovery of the package's tests will be left up +to ``load_tests`` which is called with the following arguments:: load_tests(loader, standard_tests, pattern) @@ -2078,6 +2121,11 @@ continue (and potentially modify) test discovery. A 'do nothing' standard_tests.addTests(package_tests) return standard_tests +.. versionchanged:: 3.5 + Discovery no longer checks package names for matching *pattern* due to the + impossibility of package names matching the default pattern. + + Class and Module Fixtures ------------------------- diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index 154a521fa1..3ecdda18d7 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -268,6 +268,11 @@ or on combining URL components into a URL string. :func:`urlunsplit`, removing possible *scheme* and *netloc* parts. + .. versionchanged:: 3.5 + + Behaviour updated to match the semantics defined in :rfc:`3986`. + + .. function:: urldefrag(url) If *url* contains a fragment identifier, return a modified version of *url* diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index d878aac557..82fc1b2812 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -63,7 +63,7 @@ The :mod:`urllib.request` module defines the following functions: :class:`http.client.HTTPResponse` object which has the following :ref:`httpresponse-objects` methods. - For ftp, file, and data urls and requests explicity handled by legacy + For ftp, file, and data urls and requests explicitly handled by legacy :class:`URLopener` and :class:`FancyURLopener` classes, this function returns a :class:`urllib.response.addinfourl` object which can work as :term:`context manager` and has methods such as @@ -110,6 +110,7 @@ The :mod:`urllib.request` module defines the following functions: .. versionchanged:: 3.4.3 *context* was added. + .. function:: install_opener(opener) Install an :class:`OpenerDirector` instance as the default global opener. @@ -300,6 +301,17 @@ The following classes are provided: presented with a wrong Authentication scheme. +.. class:: HTTPBasicPriorAuthHandler(password_mgr=None) + + A variant of :class:`HTTPBasicAuthHandler` which automatically sends + authorization credentials with the first request, rather than waiting to + first receive a HTTP 401 "Unauthorised" error response. This allows + authentication to sites that don't provide a 401 response when receiving + a request without an Authorization header. Aside from this difference, + this behaves exactly as :class:`HTTPBasicAuthHandler`. + + .. versionadded:: 3.5 + .. class:: ProxyBasicAuthHandler(password_mgr=None) Handle authentication with the proxy. *password_mgr*, if given, should be @@ -1067,7 +1079,7 @@ The following W3C document, http://www.w3.org/International/O-charset\ , lists the various ways in which a (X)HTML or a XML document could have specified its encoding information. -As the python.org website uses *utf-8* encoding as specified in it's meta tag, we +As the python.org website uses *utf-8* encoding as specified in its meta tag, we will use the same for decoding the bytes object. :: >>> with urllib.request.urlopen('http://www.python.org/') as f: diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index 9ca60a903f..2e16077b8a 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -258,7 +258,7 @@ These method have the same issues as the and :meth:`keyrefs` method of are called in reverse order of creation. A finalizer will never invoke its callback during the later part of - the interpreter shutdown when module globals are liable to have + the :term:`interpreter shutdown` when module globals are liable to have been replaced by :const:`None`. .. method:: __call__() @@ -527,8 +527,8 @@ follows:: Starting with Python 3.4, :meth:`__del__` methods no longer prevent reference cycles from being garbage collected, and module globals are -no longer forced to :const:`None` during interpreter shutdown. So this -code should work without any issues on CPython. +no longer forced to :const:`None` during :term:`interpreter shutdown`. +So this code should work without any issues on CPython. However, handling of :meth:`__del__` methods is notoriously implementation specific, since it depends on internal details of the interpreter's garbage @@ -566,8 +566,8 @@ third party, such as running code when a module is unloaded:: .. note:: - If you create a finalizer object in a daemonic thread just as the - the program exits then there is the possibility that the finalizer + If you create a finalizer object in a daemonic thread just as the program + exits then there is the possibility that the finalizer does not get called at exit. However, in a daemonic thread :func:`atexit.register`, ``try: ... finally: ...`` and ``with: ...`` do not guarantee that cleanup occurs either. diff --git a/Doc/library/webbrowser.rst b/Doc/library/webbrowser.rst index ef63769999..aa5e4ad15d 100644 --- a/Doc/library/webbrowser.rst +++ b/Doc/library/webbrowser.rst @@ -20,7 +20,7 @@ available. If text-mode browsers are used, the calling process will block until the user exits the browser. If the environment variable :envvar:`BROWSER` exists, it is interpreted as the -:data:`os.pathsep`-separated list of browsers to try ahead of the the platform +:data:`os.pathsep`-separated list of browsers to try ahead of the platform defaults. When the value of a list part contains the string ``%s``, then it is interpreted as a literal browser command line to be used with the argument URL substituted for ``%s``; if the part does not contain ``%s``, it is simply diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst index a9e19dad98..de74c30250 100644 --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -184,10 +184,11 @@ This module provides a single class, :class:`Headers`, for convenient manipulation of WSGI response headers using a mapping-like interface. -.. class:: Headers(headers) +.. class:: Headers([headers]) Create a mapping-like object wrapping *headers*, which must be a list of header - name/value tuples as described in :pep:`3333`. + name/value tuples as described in :pep:`3333`. The default value of *headers* is + an empty list. :class:`Headers` objects support typical mapping operations including :meth:`__getitem__`, :meth:`get`, :meth:`__setitem__`, :meth:`setdefault`, @@ -251,6 +252,10 @@ manipulation of WSGI response headers using a mapping-like interface. Content-Disposition: attachment; filename="bud.gif" + .. versionchanged:: 3.5 + *headers* parameter is optional. + + :mod:`wsgiref.simple_server` -- a simple WSGI HTTP server --------------------------------------------------------- diff --git a/Doc/library/xml.dom.rst b/Doc/library/xml.dom.rst index 19512ed3f7..4914738dfc 100644 --- a/Doc/library/xml.dom.rst +++ b/Doc/library/xml.dom.rst @@ -412,7 +412,7 @@ objects: .. method:: NodeList.item(i) Return the *i*'th item from the sequence, if there is one, or ``None``. The - index *i* is not allowed to be less then zero or greater than or equal to the + index *i* is not allowed to be less than zero or greater than or equal to the length of the sequence. diff --git a/Doc/library/xml.sax.reader.rst b/Doc/library/xml.sax.reader.rst index 3ab60636c5..31ca260f7c 100644 --- a/Doc/library/xml.sax.reader.rst +++ b/Doc/library/xml.sax.reader.rst @@ -100,8 +100,10 @@ The :class:`XMLReader` interface supports the following methods: system identifier (a string identifying the input source -- typically a file name or an URL), a file-like object, or an :class:`InputSource` object. When :meth:`parse` returns, the input is completely processed, and the parser object - can be discarded or reset. As a limitation, the current implementation only - accepts byte streams; processing of character streams is for further study. + can be discarded or reset. + + .. versionchanged:: 3.5 + Added support of character streams. .. method:: XMLReader.getContentHandler() @@ -288,8 +290,7 @@ InputSource Objects .. method:: InputSource.setByteStream(bytefile) - Set the byte stream (a Python file-like object which does not perform - byte-to-character conversion) for this input source. + Set the byte stream (a :term:`binary file`) for this input source. The SAX parser will ignore this if there is also a character stream specified, but it will use a byte stream in preference to opening a URI connection itself. @@ -308,8 +309,7 @@ InputSource Objects .. method:: InputSource.setCharacterStream(charfile) - Set the character stream for this input source. (The stream must be a Python 1.6 - Unicode-wrapped file-like that performs conversion to strings.) + Set the character stream (a :term:`text file`) for this input source. If there is a character stream specified, the SAX parser will ignore any byte stream and will not attempt to open a URI connection to the system identifier. diff --git a/Doc/library/xml.sax.rst b/Doc/library/xml.sax.rst index e95d6b0f85..55f97999de 100644 --- a/Doc/library/xml.sax.rst +++ b/Doc/library/xml.sax.rst @@ -47,7 +47,11 @@ The convenience functions are: .. function:: parseString(string, handler, error_handler=handler.ErrorHandler()) Similar to :func:`parse`, but parses from a buffer *string* received as a - parameter. + parameter. *string* must be a :class:`str` instance or a + :term:`bytes-like object`. + + .. versionchanged:: 3.5 + Added support of :class:`str` instances. A typical SAX application uses three kinds of objects: readers, handlers and input sources. "Reader" in this context is another term for parser, i.e. some diff --git a/Doc/library/xmlrpc.client.rst b/Doc/library/xmlrpc.client.rst index cc5e83a706..e199931b55 100644 --- a/Doc/library/xmlrpc.client.rst +++ b/Doc/library/xmlrpc.client.rst @@ -27,7 +27,7 @@ between conformable Python objects and XML on the wire. constructed data. If you need to parse untrusted or unauthenticated data see :ref:`xml-vulnerabilities`. -.. versionchanged:: 3.4.3 +.. versionchanged:: 3.5 For https URIs, :mod:`xmlrpc.client` now performs all the necessary certificate and hostname checks by default @@ -129,7 +129,7 @@ between conformable Python objects and XML on the wire. :class:`Server` is retained as an alias for :class:`ServerProxy` for backwards compatibility. New code should use :class:`ServerProxy`. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 Added the *context* argument. @@ -200,6 +200,11 @@ grouped under the reserved :attr:`system` attribute: no such string is available, an empty string is returned. The documentation string may contain HTML markup. +.. versionchanged:: 3.5 + + Instances of :class:`ServerProxy` support the :term:`context manager` protocol + for closing the underlying transport. + A working example follows. The server code:: @@ -217,9 +222,9 @@ The client code for the preceding server:: import xmlrpc.client - proxy = xmlrpc.client.ServerProxy("http://localhost:8000/") - print("3 is even: %s" % str(proxy.is_even(3))) - print("100 is even: %s" % str(proxy.is_even(100))) + with xmlrpc.client.ServerProxy("http://localhost:8000/") as proxy: + print("3 is even: %s" % str(proxy.is_even(3))) + print("100 is even: %s" % str(proxy.is_even(100))) .. _datetime-objects: @@ -527,14 +532,14 @@ Example of Client Usage from xmlrpc.client import ServerProxy, Error # server = ServerProxy("http://localhost:8000") # local server - server = ServerProxy("http://betty.userland.com") + with ServerProxy("http://betty.userland.com") as proxy: - print(server) + print(proxy) - try: - print(server.examples.getStateName(41)) - except Error as v: - print("ERROR", v) + try: + print(proxy.examples.getStateName(41)) + except Error as v: + print("ERROR", v) To access an XML-RPC server through a proxy, you need to define a custom transport. The following example shows how: diff --git a/Doc/library/zipapp.rst b/Doc/library/zipapp.rst new file mode 100644 index 0000000000..b4484c87d0 --- /dev/null +++ b/Doc/library/zipapp.rst @@ -0,0 +1,259 @@ +:mod:`zipapp` --- Manage executable python zip archives +======================================================= + +.. module:: zipapp + :synopsis: Manage executable python zip archives + + +.. index:: + single: Executable Zip Files + +.. versionadded:: 3.5 + +**Source code:** :source:`Lib/zipapp.py` + +-------------- + +This module provides tools to manage the creation of zip files containing +Python code, which can be :ref:`executed directly by the Python interpreter +<using-on-interface-options>`. The module provides both a +:ref:`zipapp-command-line-interface` and a :ref:`zipapp-python-api`. + + +Basic Example +------------- + +The following example shows how the :ref:`command-line-interface` +can be used to create an executable archive from a directory containing +Python code. When run, the archive will execute the ``main`` function from +the module ``myapp`` in the archive. + +.. code-block:: sh + + $ python -m zipapp myapp -m "myapp:main" + $ python myapp.pyz + <output from myapp> + + +.. _zipapp-command-line-interface: + +Command-Line Interface +---------------------- + +When called as a program from the command line, the following form is used: + +.. code-block:: sh + + $ python -m zipapp source [options] + +If *source* is a directory, this will create an archive from the contents of +*source*. If *source* is a file, it should be an archive, and it will be +copied to the target archive (or the contents of its shebang line will be +displayed if the --info option is specified). + +The following options are understood: + +.. program:: zipapp + +.. cmdoption:: -o <output>, --output=<output> + + Write the output to a file named *output*. If this option is not specified, + the output filename will be the same as the input *source*, with the + extension ``.pyz`` added. If an explicit filename is given, it is used as + is (so a ``.pyz`` extension should be included if required). + + An output filename must be specified if the *source* is an archive (and in + that case, *output* must not be the same as *source*). + +.. cmdoption:: -p <interpreter>, --python=<interpreter> + + Add a ``#!`` line to the archive specifying *interpreter* as the command + to run. Also, on POSIX, make the archive executable. The default is to + write no ``#!`` line, and not make the file executable. + +.. cmdoption:: -m <mainfn>, --main=<mainfn> + + Write a ``__main__.py`` file to the archive that executes *mainfn*. The + *mainfn* argument should have the form "pkg.mod:fn", where "pkg.mod" is a + package/module in the archive, and "fn" is a callable in the given module. + The ``__main__.py`` file will execute that callable. + + :option:`--main` cannot be specified when copying an archive. + +.. cmdoption:: --info + + Display the interpreter embedded in the archive, for diagnostic purposes. In + this case, any other options are ignored and SOURCE must be an archive, not a + directory. + +.. cmdoption:: -h, --help + + Print a short usage message and exit. + + +.. _zipapp-python-api: + +Python API +---------- + +The module defines two convenience functions: + + +.. function:: create_archive(source, target=None, interpreter=None, main=None) + + Create an application archive from *source*. The source can be any + of the following: + + * The name of a directory, or a :class:`pathlib.Path` object referring + to a directory, in which case a new application archive will be + created from the content of that directory. + * The name of an existing application archive file, or a :class:`pathlib.Path` + object referring to such a file, in which case the file is copied to + the target (modifying it to reflect the value given for the *interpreter* + argument). The file name should include the ``.pyz`` extension, if required. + * A file object open for reading in bytes mode. The content of the + file should be an application archive, and the file object is + assumed to be positioned at the start of the archive. + + The *target* argument determines where the resulting archive will be + written: + + * If it is the name of a file, or a :class:`pathlb.Path` object, + the archive will be written to that file. + * If it is an open file object, the archive will be written to that + file object, which must be open for writing in bytes mode. + * If the target is omitted (or None), the source must be a directory + and the target will be a file with the same name as the source, with + a ``.pyz`` extension added. + + The *interpreter* argument specifies the name of the Python + interpreter with which the archive will be executed. It is written as + a "shebang" line at the start of the archive. On POSIX, this will be + interpreted by the OS, and on Windows it will be handled by the Python + launcher. Omitting the *interpreter* results in no shebang line being + written. If an interpreter is specified, and the target is a + filename, the executable bit of the target file will be set. + + The *main* argument specifies the name of a callable which will be + used as the main program for the archive. It can only be specified if + the source is a directory, and the source does not already contain a + ``__main__.py`` file. The *main* argument should take the form + "pkg.module:callable" and the archive will be run by importing + "pkg.module" and executing the given callable with no arguments. It + is an error to omit *main* if the source is a directory and does not + contain a ``__main__.py`` file, as otherwise the resulting archive + would not be executable. + + If a file object is specified for *source* or *target*, it is the + caller's responsibility to close it after calling create_archive. + + When copying an existing archive, file objects supplied only need + ``read`` and ``readline``, or ``write`` methods. When creating an + archive from a directory, if the target is a file object it will be + passed to the ``zipfile.ZipFile`` class, and must supply the methods + needed by that class. + +.. function:: get_interpreter(archive) + + Return the interpreter specified in the ``#!`` line at the start of the + archive. If there is no ``#!`` line, return :const:`None`. + The *archive* argument can be a filename or a file-like object open + for reading in bytes mode. It is assumed to be at the start of the archive. + + +.. _zipapp-examples: + +Examples +-------- + +Pack up a directory into an archive, and run it. + +.. code-block:: sh + + $ python -m zipapp myapp + $ python myapp.pyz + <output from myapp> + +The same can be done using the :func:`create_archive` functon:: + + >>> import zipapp + >>> zipapp.create_archive('myapp.pyz', 'myapp') + +To make the application directly executable on POSIX, specify an interpreter +to use. + +.. code-block:: sh + + $ python -m zipapp myapp -p "/usr/bin/env python" + $ ./myapp.pyz + <output from myapp> + +To replace the shebang line on an existing archive, create a modified archive +using the :func:`create_archive` function:: + + >>> import zipapp + >>> zipapp.create_archive('old_archive.pyz', 'new_archive.pyz', '/usr/bin/python3') + +To update the file in place, do the replacement in memory using a :class:`BytesIO` +object, and then overwrite the source afterwards. Note that there is a risk +when overwriting a file in place that an error will result in the loss of +the original file. This code does not protect against such errors, but +production code should do so. Also, this method will only work if the archive +fits in memory:: + + >>> import zipapp + >>> import io + >>> temp = io.BytesIO() + >>> zipapp.create_archive('myapp.pyz', temp, '/usr/bin/python2') + >>> with open('myapp.pyz', 'wb') as f: + >>> f.write(temp.getvalue()) + +Note that if you specify an interpreter and then distribute your application +archive, you need to ensure that the interpreter used is portable. The Python +launcher for Windows supports most common forms of POSIX ``#!`` line, but there +are other issues to consider: + +* If you use "/usr/bin/env python" (or other forms of the "python" command, + such as "/usr/bin/python"), you need to consider that your users may have + either Python 2 or Python 3 as their default, and write your code to work + under both versions. +* If you use an explicit version, for example "/usr/bin/env python3" your + application will not work for users who do not have that version. (This + may be what you want if you have not made your code Python 2 compatible). +* There is no way to say "python X.Y or later", so be careful of using an + exact version like "/usr/bin/env python3.4" as you will need to change your + shebang line for users of Python 3.5, for example. + +The Python Zip Application Archive Format +----------------------------------------- + +Python has been able to execute zip files which contain a ``__main__.py`` file +since version 2.6. In order to be executed by Python, an application archive +simply has to be a standard zip file containing a ``__main__.py`` file which +will be run as the entry point for the application. As usual for any Python +script, the parent of the script (in this case the zip file) will be placed on +:data:`sys.path` and thus further modules can be imported from the zip file. + +The zip file format allows arbitrary data to be prepended to a zip file. The +zip application format uses this ability to prepend a standard POSIX "shebang" +line to the file (``#!/path/to/interpreter``). + +Formally, the Python zip application format is therefore: + +1. An optional shebang line, containing the characters ``b'#!'`` followed by an + interpreter name, and then a newline (``b'\n'``) character. The interpreter + name can be anything acceptable to the OS "shebang" processing, or the Python + launcher on Windows. The interpreter should be encoded in UTF-8 on Windows, + and in :func:`sys.getfilesystemencoding()` on POSIX. +2. Standard zipfile data, as generated by the :mod:`zipfile` module. The + zipfile content *must* include a file called ``__main__.py`` (which must be + in the "root" of the zipfile - i.e., it cannot be in a subdirectory). The + zipfile data can be compressed or uncompressed. + +If an application archive has a shebang line, it may have the executable bit set +on POSIX systems, to allow it to be executed directly. + +There is no requirement that the tools in this module are used to create +application archives - the module is a convenience, but archives in the above +format created by any means are acceptable to Python. + diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index f3e54ca922..4e1a87fde1 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -134,12 +134,16 @@ ZipFile Objects Open a ZIP file, where *file* can be either a path to a file (a string) or a file-like object. The *mode* parameter should be ``'r'`` to read an existing - file, ``'w'`` to truncate and write a new file, or ``'a'`` to append to an - existing file. If *mode* is ``'a'`` and *file* refers to an existing ZIP + file, ``'w'`` to truncate and write a new file, ``'x'`` to exclusive create + and write a new file, or ``'a'`` to append to an existing file. + If *mode* is ``'x'`` and *file* refers to an existing file, + a :exc:`FileExistsError` will be raised. + If *mode* is ``'a'`` and *file* refers to an existing ZIP file, then additional files are added to it. If *file* does not refer to a ZIP file, then a new ZIP archive is appended to the file. This is meant for adding a ZIP archive to another file (such as :file:`python.exe`). If *mode* is ``a`` and the file does not exist at all, it is created. + If *mode* is ``r`` or ``a``, the file should be seekable. *compression* is the ZIP compression method to use when writing the archive, and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized @@ -151,7 +155,7 @@ ZipFile Objects extensions when the zipfile is larger than 2 GiB. If it is false :mod:`zipfile` will raise an exception when the ZIP file would require ZIP64 extensions. - If the file is created with mode ``'a'`` or ``'w'`` and then + If the file is created with mode ``'w'``, ``'x'`` or ``'a'`` and then :meth:`closed <close>` without adding any files to the archive, the appropriate ZIP structures for an empty archive will be written to the file. @@ -171,6 +175,10 @@ ZipFile Objects .. versionchanged:: 3.4 ZIP64 extensions are enabled by default. + .. versionchanged:: 3.5 + Added support for writing to unseekable streams. + Added support for the ``'x'`` mode. + .. method:: ZipFile.close() @@ -226,14 +234,8 @@ ZipFile Objects .. note:: - If the ZipFile was created by passing in a file-like object as the first - argument to the constructor, then the object returned by :meth:`.open` shares the - ZipFile's file pointer. Under these circumstances, the object returned by - :meth:`.open` should not be used after any additional operations are performed - on the ZipFile object. If the ZipFile was created by passing in a string (the - filename) as the first argument to the constructor, then :meth:`.open` will - create a new file object that will be held by the ZipExtFile, allowing it to - operate independently of the ZipFile. + Objects returned by :meth:`.open` can operate independently of the + ZipFile. .. note:: @@ -314,7 +316,8 @@ ZipFile Objects *arcname* (by default, this will be the same as *filename*, but without a drive letter and with leading path separators removed). If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for - the new entry. The archive must be open with mode ``'w'`` or ``'a'`` -- calling + the new entry. + The archive must be open with mode ``'w'``, ``'x'`` or ``'a'`` -- calling :meth:`write` on a ZipFile created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling :meth:`write` on a closed ZipFile will raise a :exc:`RuntimeError`. @@ -336,16 +339,16 @@ ZipFile Objects If ``arcname`` (or ``filename``, if ``arcname`` is not given) contains a null byte, the name of the file in the archive will be truncated at the null byte. - .. method:: ZipFile.writestr(zinfo_or_arcname, bytes[, compress_type]) Write the string *bytes* to the archive; *zinfo_or_arcname* is either the file name it will be given in the archive, or a :class:`ZipInfo` instance. If it's an instance, at least the filename, date, and time must be given. If it's a - name, the date and time is set to the current date and time. The archive must be - opened with mode ``'w'`` or ``'a'`` -- calling :meth:`writestr` on a ZipFile - created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling - :meth:`writestr` on a closed ZipFile will raise a :exc:`RuntimeError`. + name, the date and time is set to the current date and time. + The archive must be opened with mode ``'w'``, ``'x'`` or ``'a'`` -- calling + :meth:`writestr` on a ZipFile created with mode ``'r'`` will raise a + :exc:`RuntimeError`. Calling :meth:`writestr` on a closed ZipFile will + raise a :exc:`RuntimeError`. If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for the new entry, or in the *zinfo_or_arcname* @@ -373,7 +376,8 @@ The following data attributes are also available: .. attribute:: ZipFile.comment The comment text associated with the ZIP file. If assigning a comment to a - :class:`ZipFile` instance created with mode 'a' or 'w', this should be a + :class:`ZipFile` instance created with mode ``'w'``, ``'x'`` or ``'a'``, + this should be a string no longer than 65535 bytes. Comments longer than this will be truncated in the written archive when :meth:`close` is called. @@ -403,8 +407,7 @@ The :class:`PyZipFile` constructor takes the same parameters as the archive. If the *optimize* parameter to :class:`PyZipFile` was not given or ``-1``, - the corresponding file is a :file:`\*.pyo` file if available, else a - :file:`\*.pyc` file, compiling if necessary. + the corresponding file is a :file:`\*.pyc` file, compiling if necessary. If the *optimize* parameter to :class:`PyZipFile` was ``0``, ``1`` or ``2``, only files with that optimization level (see :func:`compile`) are @@ -567,4 +570,3 @@ Instances have the following attributes: .. attribute:: ZipInfo.file_size Size of the uncompressed file. - diff --git a/Doc/library/zipimport.rst b/Doc/library/zipimport.rst index 2cf508b85c..8a5d5d19df 100644 --- a/Doc/library/zipimport.rst +++ b/Doc/library/zipimport.rst @@ -20,10 +20,10 @@ subdirectory. For example, the path :file:`example.zip/lib/` would only import from the :file:`lib/` subdirectory within the archive. Any files may be present in the ZIP archive, but only files :file:`.py` and -:file:`.py[co]` are available for import. ZIP import of dynamic modules +:file:`.pyc` are available for import. ZIP import of dynamic modules (:file:`.pyd`, :file:`.so`) is disallowed. Note that if an archive only contains :file:`.py` files, Python will not attempt to modify the archive by adding the -corresponding :file:`.pyc` or :file:`.pyo` file, meaning that if a ZIP archive +corresponding :file:`.pyc` file, meaning that if a ZIP archive doesn't contain :file:`.pyc` files, importing may be rather slow. ZIP archives with an archive comment are currently not supported. @@ -161,4 +161,3 @@ Here is an example that imports a module from a ZIP archive - note that the >>> import jwzthreading >>> jwzthreading.__file__ 'example.zip/jwzthreading.py' - diff --git a/Doc/make.bat b/Doc/make.bat index 251f822abf..3ff91f288d 100644 --- a/Doc/make.bat +++ b/Doc/make.bat @@ -8,9 +8,17 @@ set this=%~n0 if "%SPHINXBUILD%" EQU "" set SPHINXBUILD=sphinx-build
if "%PYTHON%" EQU "" set PYTHON=py
-if DEFINED ProgramFiles(x86) set _PRGMFLS=%ProgramFiles(x86)%
-if NOT DEFINED ProgramFiles(x86) set _PRGMFLS=%ProgramFiles%
-if "%HTMLHELP%" EQU "" set HTMLHELP=%_PRGMFLS%\HTML Help Workshop\hhc.exe
+if "%1" NEQ "htmlhelp" goto :skiphhcsearch
+if exist "%HTMLHELP%" goto :skiphhcsearch
+
+rem Search for HHC in likely places
+set HTMLHELP=
+where hhc /q && set HTMLHELP=hhc && goto :skiphhcsearch
+where /R ..\externals hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" where /R "%ProgramFiles(x86)%" hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" where /R "%ProgramFiles%" hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" echo Cannot find HHC on PATH or in externals & exit /B 1
+:skiphhcsearch
if "%DISTVERSION%" EQU "" for /f "usebackq" %%v in (`%PYTHON% tools/extensions/patchlevel.py`) do set DISTVERSION=%%v
@@ -36,7 +44,8 @@ if errorlevel 9009 ( echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
- goto end
+ popd
+ exit /B 1
)
rem Targets that do require sphinx-build and have their own label
diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index dda18ba6ac..df8b2454bb 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1986,6 +1986,7 @@ left undefined. .. method:: object.__add__(self, other) object.__sub__(self, other) object.__mul__(self, other) + object.__matmul__(self, other) object.__truediv__(self, other) object.__floordiv__(self, other) object.__mod__(self, other) @@ -2002,15 +2003,16 @@ left undefined. builtin: pow builtin: pow - These methods are called to implement the binary arithmetic operations (``+``, - ``-``, ``*``, ``/``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, ``<<``, - ``>>``, ``&``, ``^``, ``|``). For instance, to evaluate the expression - ``x + y``, where *x* is an instance of a class that has an :meth:`__add__` - method, ``x.__add__(y)`` is called. The :meth:`__divmod__` method should be the - equivalent to using :meth:`__floordiv__` and :meth:`__mod__`; it should not be - related to :meth:`__truediv__`. Note that :meth:`__pow__` should be defined - to accept an optional third argument if the ternary version of the built-in - :func:`pow` function is to be supported. + These methods are called to implement the binary arithmetic operations + (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, + :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``). For instance, to + evaluate the expression ``x + y``, where *x* is an instance of a class that + has an :meth:`__add__` method, ``x.__add__(y)`` is called. The + :meth:`__divmod__` method should be the equivalent to using + :meth:`__floordiv__` and :meth:`__mod__`; it should not be related to + :meth:`__truediv__`. Note that :meth:`__pow__` should be defined to accept + an optional third argument if the ternary version of the built-in :func:`pow` + function is to be supported. If one of those methods does not support the operation with the supplied arguments, it should return ``NotImplemented``. @@ -2019,6 +2021,7 @@ left undefined. .. method:: object.__radd__(self, other) object.__rsub__(self, other) object.__rmul__(self, other) + object.__rmatmul__(self, other) object.__rtruediv__(self, other) object.__rfloordiv__(self, other) object.__rmod__(self, other) @@ -2034,14 +2037,14 @@ left undefined. builtin: divmod builtin: pow - These methods are called to implement the binary arithmetic operations (``+``, - ``-``, ``*``, ``/``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, - ``<<``, ``>>``, ``&``, ``^``, ``|``) with reflected (swapped) operands. - These functions are only called if the left operand does not support the - corresponding operation and the operands are of different types. [#]_ For - instance, to evaluate the expression ``x - y``, where *y* is an instance of - a class that has an :meth:`__rsub__` method, ``y.__rsub__(x)`` is called if - ``x.__sub__(y)`` returns *NotImplemented*. + These methods are called to implement the binary arithmetic operations + (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, + :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``) with reflected + (swapped) operands. These functions are only called if the left operand does + not support the corresponding operation and the operands are of different + types. [#]_ For instance, to evaluate the expression ``x - y``, where *y* is + an instance of a class that has an :meth:`__rsub__` method, ``y.__rsub__(x)`` + is called if ``x.__sub__(y)`` returns *NotImplemented*. .. index:: builtin: pow @@ -2059,6 +2062,7 @@ left undefined. .. method:: object.__iadd__(self, other) object.__isub__(self, other) object.__imul__(self, other) + object.__imatmul__(self, other) object.__itruediv__(self, other) object.__ifloordiv__(self, other) object.__imod__(self, other) @@ -2070,17 +2074,17 @@ left undefined. object.__ior__(self, other) These methods are called to implement the augmented arithmetic assignments - (``+=``, ``-=``, ``*=``, ``/=``, ``//=``, ``%=``, ``**=``, ``<<=``, ``>>=``, - ``&=``, ``^=``, ``|=``). These methods should attempt to do the operation - in-place (modifying *self*) and return the result (which could be, but does - not have to be, *self*). If a specific method is not defined, the augmented - assignment falls back to the normal methods. For instance, if *x* is an - instance of a class with an :meth:`__iadd__` method, ``x += y`` is equivalent - to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and ``y.__radd__(x)`` - are considered, as with the evaluation of ``x + y``. In certain situations, - augmented assignment can result in unexpected errors (see - :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in - fact part of the data model. + (``+=``, ``-=``, ``*=``, ``@=``, ``/=``, ``//=``, ``%=``, ``**=``, ``<<=``, + ``>>=``, ``&=``, ``^=``, ``|=``). These methods should attempt to do the + operation in-place (modifying *self*) and return the result (which could be, + but does not have to be, *self*). If a specific method is not defined, the + augmented assignment falls back to the normal methods. For instance, if *x* + is an instance of a class with an :meth:`__iadd__` method, ``x += y`` is + equivalent to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and + ``y.__radd__(x)`` are considered, as with the evaluation of ``x + y``. In + certain situations, augmented assignment can result in unexpected errors (see + :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in fact + part of the data model. .. method:: object.__neg__(self) diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 1a5088abbc..12f9f2f485 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -891,8 +891,9 @@ from the power operator, there are only two levels, one for multiplicative operators and one for additive operators: .. productionlist:: - m_expr: `u_expr` | `m_expr` "*" `u_expr` | `m_expr` "//" `u_expr` | `m_expr` "/" `u_expr` - : | `m_expr` "%" `u_expr` + m_expr: `u_expr` | `m_expr` "*" `u_expr` | `m_expr` "@" `m_expr` | + : `m_expr` "//" `u_expr`| `m_expr` "/" `u_expr` | + : `m_expr` "%" `u_expr` a_expr: `m_expr` | `a_expr` "+" `m_expr` | `a_expr` "-" `m_expr` .. index:: single: multiplication @@ -903,6 +904,13 @@ the other must be a sequence. In the former case, the numbers are converted to a common type and then multiplied together. In the latter case, sequence repetition is performed; a negative repetition factor yields an empty sequence. +.. index:: single: matrix multiplication + +The ``@`` (at) operator is intended to be used for matrix multiplication. No +builtin Python types implement this operator. + +.. versionadded:: 3.5 + .. index:: exception: ZeroDivisionError single: division @@ -1346,8 +1354,9 @@ precedence and have a left-to-right chaining feature as described in the +-----------------------------------------------+-------------------------------------+ | ``+``, ``-`` | Addition and subtraction | +-----------------------------------------------+-------------------------------------+ -| ``*``, ``/``, ``//``, ``%`` | Multiplication, division, remainder | -| | [#]_ | +| ``*``, ``@``, ``/``, ``//``, ``%`` | Multiplication, matrix | +| | multiplication division, | +| | remainder [#]_ | +-----------------------------------------------+-------------------------------------+ | ``+x``, ``-x``, ``~x`` | Positive, negative, bitwise NOT | +-----------------------------------------------+-------------------------------------+ diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index e9b7e53247..dec5f8f14b 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -339,6 +339,7 @@ of what happens during the loading portion of import:: module = None if spec.loader is not None and hasattr(spec.loader, 'create_module'): + # It is assumed 'exec_module' will also be defined on the loader. module = spec.loader.create_module(spec) if module is None: module = ModuleType(spec.name) @@ -427,7 +428,7 @@ Module loaders may opt in to creating the module object during loading by implementing a :meth:`~importlib.abc.Loader.create_module` method. It takes one argument, the module spec, and returns the new module object to use during loading. ``create_module()`` does not need to set any attributes -on the module object. If the loader does not define ``create_module()``, the +on the module object. If the method returns ``None``, the import machinery will create the new module itself. .. versionadded:: 3.4 @@ -459,7 +460,13 @@ import machinery will create the new module itself. * If loading fails, the loader must remove any modules it has inserted into :data:`sys.modules`, but it must remove **only** the failing - module, and only if the loader itself has loaded it explicitly. + module(s), and only if the loader itself has loaded the module(s) + explicitly. + +.. versionchanged:: 3.5 + A :exc:`DeprecationWarning` is raised when ``exec_module()`` is defined but + ``create_module()`` is not. Starting in Python 3.6 it will be an error to not + define ``create_module()`` on a loader attached to a ModuleSpec. Module spec ----------- @@ -639,7 +646,7 @@ path entry finder that knows how to handle that particular kind of path. The default set of path entry finders implement all the semantics for finding modules on the file system, handling special file types such as Python source -code (``.py`` files), Python byte code (``.pyc`` and ``.pyo`` files) and +code (``.py`` files), Python byte code (``.pyc`` files) and shared libraries (e.g. ``.so`` files). When supported by the :mod:`zipimport` module in the standard library, the default path entry finders also handle loading all of these file types (other than shared libraries) from zipfiles. @@ -753,6 +760,15 @@ hook` callables on :data:`sys.path_hooks`, then the following protocol is used to ask the finder for a module spec, which is then used when loading the module. +The current working directory -- denoted by an empty string -- is handled +slightly differently from other entries on :data:`sys.path`. First, if the +current working directory is found to not exist, no value is stored in +:data:`sys.path_importer_cache`. Second, the value for the current working +directory is looked up fresh for each module lookup. Third, the path used for +:data:`sys.path_importer_cache` and returned by +:meth:`importlib.machinery.PathFinder.find_spec` will be the actual current +working directory and not the empty string. + Path entry finder protocol -------------------------- diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index 2bf66b1862..c673791895 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -311,7 +311,7 @@ The Unicode category codes mentioned above stand for: * *Nd* - decimal numbers * *Pc* - connector punctuations * *Other_ID_Start* - explicit list of characters in `PropList.txt - <http://www.unicode.org/Public/6.3.0/ucd/PropList.txt>`_ to support backwards + <http://www.unicode.org/Public/7.0.0/ucd/PropList.txt>`_ to support backwards compatibility * *Other_ID_Continue* - likewise @@ -691,7 +691,7 @@ Operators The following tokens are operators:: - + - * ** / // % + + - * ** / // % @ << >> & | ^ ~ < > <= >= == != @@ -707,7 +707,7 @@ The following tokens serve as delimiters in the grammar:: ( ) [ ] { } , : . ; @ = -> - += -= *= /= //= %= + += -= *= /= //= %= @= &= |= ^= >>= <<= **= The period can also occur in floating-point and imaginary literals. A sequence @@ -728,4 +728,4 @@ occurrence outside string literals and comments is an unconditional error:: .. rubric:: Footnotes -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NameAliases.txt +.. [#] http://www.unicode.org/Public/7.0.0/ucd/NameAliases.txt diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 8946b4f354..5f605408b7 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -281,7 +281,7 @@ operation and an assignment statement: .. productionlist:: augmented_assignment_stmt: `augtarget` `augop` (`expression_list` | `yield_expression`) augtarget: `identifier` | `attributeref` | `subscription` | `slicing` - augop: "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**=" + augop: "+=" | "-=" | "*=" | "@=" | "/=" | "//=" | "%=" | "**=" : | ">>=" | "<<=" | "&=" | "^=" | "|=" (See section :ref:`primaries` for the syntax definitions of the last three diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 64a56655f7..9b78184d67 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -34,7 +34,7 @@ import suspicious ISSUE_URI = 'https://bugs.python.org/issue%s' -SOURCE_URI = 'https://hg.python.org/cpython/file/3.4/%s' +SOURCE_URI = 'https://hg.python.org/cpython/file/default/%s' # monkey-patch reST parser to disable alphabetic and roman enumerated lists from docutils.parsers.rst.states import Body diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index 48653af8ba..4b538a5368 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -122,6 +122,8 @@ library/ipaddress,,:db8,>>> ipaddress.IPv6Address('2001:db8::1000') library/ipaddress,,::,>>> ipaddress.IPv6Address('2001:db8::1000') library/ipaddress,,:db8,IPv6Address('2001:db8::1000') library/ipaddress,,::,IPv6Address('2001:db8::1000') +library/ipaddress,,:db8,">>> ipaddress.ip_address(""2001:db8::1"").reverse_pointer" +library/ipaddress,,::,">>> ipaddress.ip_address(""2001:db8::1"").reverse_pointer" library/ipaddress,,::,"""::abc:7:def""" library/ipaddress,,:def,"""::abc:7:def""" library/ipaddress,,::,::FFFF/96 @@ -136,12 +138,9 @@ library/ipaddress,,:db00,2001:db00::0/ffff:ff00:: library/ipaddress,,::,2001:db00::0/ffff:ff00:: library/itertools,,:step,elements from seq[start:stop:step] library/itertools,,:stop,elements from seq[start:stop:step] -library/linecache,,:sys,"sys:x:3:3:sys:/dev:/bin/sh" library/logging.handlers,,:port,host:port library/mmap,,:i2,obj[i1:i2] library/multiprocessing,,`,# Add more tasks using `put()` -library/multiprocessing,,`,">>> l._callmethod('__getitem__', (20,)) # equiv to `l[20]`" -library/multiprocessing,,`,">>> l._callmethod('__getslice__', (2, 7)) # equiv to `l[2:7]`" library/multiprocessing,,:queue,">>> QueueManager.register('get_queue', callable=lambda:queue)" library/multiprocessing,,`,# register the Foo class; make `f()` and `g()` accessible via proxy library/multiprocessing,,`,# register the Foo class; make `g()` and `_h()` accessible via proxy @@ -179,8 +178,6 @@ library/ssl,,:MyState,State or Province Name (full name) [Some-State]:MyState library/ssl,,:ops,Email Address []:ops@myserver.mygroup.myorganization.com library/ssl,,:Some,"Locality Name (eg, city) []:Some City" library/ssl,,:US,Country Name (2 letter code) [AU]:US -library/stdtypes,,::,>>> a[::-1].tolist() -library/stdtypes,,::,>>> a[::2].tolist() library/stdtypes,,:end,s[start:end] library/stdtypes,,::,>>> hash(v[::-2]) == hash(b'abcefg'[::-2]) library/stdtypes,,:len,s[len(s):len(s)] @@ -256,6 +253,11 @@ whatsnew/2.4,,:System, whatsnew/2.5,,:memory,:memory: whatsnew/2.5,,:step,[start:stop:step] whatsnew/2.5,,:stop,[start:stop:step] +whatsnew/2.7,,::,"ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]'," +whatsnew/2.7,,::,>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') +whatsnew/2.7,,:Sunday,'2009:4:Sunday' +whatsnew/2.7,,:Cookie,"export PYTHONWARNINGS=all,error:::Cookie:0" +whatsnew/2.7,,::,"export PYTHONWARNINGS=all,error:::Cookie:0" whatsnew/3.2,,:affe,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," whatsnew/3.2,,:affe,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') whatsnew/3.2,,:beef,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," @@ -271,14 +273,20 @@ whatsnew/3.2,,:feed,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe: whatsnew/3.2,,:gz,">>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:" whatsnew/3.2,,:location,zope9-location = ${zope9:location} whatsnew/3.2,,:prefix,zope-conf = ${custom:prefix}/etc/zope.conf -whatsnew/changelog,,:platform,:platform: whatsnew/changelog,,:gz,": TarFile opened with external fileobj and ""w:gz"" mode didn't" -whatsnew/changelog,,:PythonCmd,"With Tk < 8.5 _tkinter.c:PythonCmd() raised UnicodeDecodeError, caused" -whatsnew/changelog,,::,": Fix FTP tests for IPv6, bind to ""::1"" instead of ""localhost""." whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" -whatsnew/changelog,,:password,user:password -whatsnew/2.7,780,:Sunday,'2009:4:Sunday' -whatsnew/2.7,907,::,"export PYTHONWARNINGS=all,error:::Cookie:0" -whatsnew/2.7,907,:Cookie,"export PYTHONWARNINGS=all,error:::Cookie:0" -whatsnew/2.7,1657,::,>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') -whatsnew/2.7,1657,::,"ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]'," +library/stdtypes,3688,::,>>> m[::2].tolist() +library/tarfile,149,:xz,'x:xz' +library/xml.etree.elementtree,290,:sometag,prefix:sometag +library/xml.etree.elementtree,301,:fictional,"<actors xmlns:fictional=""http://characters.example.com""" +library/xml.etree.elementtree,301,:character,<fictional:character>Lancelot</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Archie Leach</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Sir Robin</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Gunther</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Commander Clement</fictional:character> +library/xml.etree.elementtree,332,:actor,"for actor in root.findall('real_person:actor', ns):" +library/xml.etree.elementtree,332,:name,"name = actor.find('real_person:name', ns)" +library/xml.etree.elementtree,332,:character,"for char in actor.findall('role:character', ns):" +library/zipapp,31,:main,"$ python -m zipapp myapp -m ""myapp:main""" +library/zipapp,82,:fn,"argument should have the form ""pkg.mod:fn"", where ""pkg.mod"" is a" +library/zipapp,155,:callable,"""pkg.module:callable"" and the archive will be run by importing" diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html index abdf070772..78e9c4f99e 100644 --- a/Doc/tools/templates/indexsidebar.html +++ b/Doc/tools/templates/indexsidebar.html @@ -3,8 +3,7 @@ <h3>Docs for other versions</h3> <ul> <li><a href="https://docs.python.org/2.7/">Python 2.7 (stable)</a></li> - <li><a href="https://docs.python.org/3.3/">Python 3.3 (stable)</a></li> - <li><a href="https://docs.python.org/3.5/">Python 3.5 (in development)</a></li> + <li><a href="https://docs.python.org/3.4/">Python 3.4 (stable)</a></li> <li><a href="https://www.python.org/doc/versions/">Old versions</a></li> </ul> diff --git a/Doc/tutorial/appendix.rst b/Doc/tutorial/appendix.rst index 8670efc8e9..67262a1d80 100644 --- a/Doc/tutorial/appendix.rst +++ b/Doc/tutorial/appendix.rst @@ -40,7 +40,7 @@ Executable Python Scripts On BSD'ish Unix systems, Python scripts can be made directly executable, like shell scripts, by putting the line :: - #!/usr/bin/env python3.4 + #!/usr/bin/env python3.5 (assuming that the interpreter is on the user's :envvar:`PATH`) at the beginning of the script and giving the file an executable mode. The ``#!`` must be the @@ -107,7 +107,7 @@ of your user site-packages directory. Start Python and run this code:: >>> import site >>> site.getusersitepackages() - '/home/user/.local/lib/python3.4/site-packages' + '/home/user/.local/lib/python3.5/site-packages' Now you can create a file named :file:`usercustomize.py` in that directory and put anything you want in it. It will affect every invocation of Python, unless diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst index 1ea299fbed..a2031ed867 100644 --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -73,10 +73,11 @@ objects: Return the number of times *x* appears in the list. -.. method:: list.sort() +.. method:: list.sort(key=None, reverse=False) :noindex: - Sort the items of the list in place. + Sort the items of the list in place (the arguments can be used for sort + customization, see :func:`sorted` for their explanation). .. method:: list.reverse() diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst index 80516349c7..d5789a62c9 100644 --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -10,13 +10,13 @@ Using the Python Interpreter Invoking the Interpreter ======================== -The Python interpreter is usually installed as :file:`/usr/local/bin/python3.4` +The Python interpreter is usually installed as :file:`/usr/local/bin/python3.5` on those machines where it is available; putting :file:`/usr/local/bin` in your Unix shell's search path makes it possible to start it by typing the command: .. code-block:: text - python3.4 + python3.5 to the shell. [#]_ Since the choice of the directory where the interpreter lives is an installation option, other places are possible; check with your local @@ -24,11 +24,11 @@ Python guru or system administrator. (E.g., :file:`/usr/local/python` is a popular alternative location.) On Windows machines, the Python installation is usually placed in -:file:`C:\\Python34`, though you can change this when you're running the +:file:`C:\\Python35`, though you can change this when you're running the installer. To add this directory to your path, you can type the following command into the command prompt in a DOS box:: - set path=%path%;C:\python34 + set path=%path%;C:\python35 Typing an end-of-file character (:kbd:`Control-D` on Unix, :kbd:`Control-Z` on Windows) at the primary prompt causes the interpreter to exit with a zero exit @@ -96,8 +96,8 @@ with the *secondary prompt*, by default three dots (``...``). The interpreter prints a welcome message stating its version number and a copyright notice before printing the first prompt:: - $ python3.4 - Python 3.4 (default, Mar 16 2014, 09:25:04) + $ python3.5 + Python 3.5 (default, Sep 16 2015, 09:25:04) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index fd361aed69..9ae64b0cf7 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -216,15 +216,15 @@ Some tips for experts: statements, the ``-OO`` switch removes both assert statements and __doc__ strings. Since some programs may rely on having these available, you should only use this option if you know what you're doing. "Optimized" modules have - a .pyo rather than a .pyc suffix and are usually smaller. Future releases may + an ``opt-`` tag and are usually smaller. Future releases may change the effects of optimization. -* A program doesn't run any faster when it is read from a ``.pyc`` or ``.pyo`` +* A program doesn't run any faster when it is read from a ``.pyc`` file than when it is read from a ``.py`` file; the only thing that's faster - about ``.pyc`` or ``.pyo`` files is the speed with which they are loaded. + about ``.pyc`` files is the speed with which they are loaded. -* The module :mod:`compileall` can create .pyc files (or .pyo files when - :option:`-O` is used) for all modules in a directory. +* The module :mod:`compileall` can create .pyc files for all modules in a + directory. * There is more detail on this process, including a flow chart of the decisions, in PEP 3147. @@ -548,4 +548,3 @@ modules found in a package. .. [#] In fact function definitions are also 'statements' that are 'executed'; the execution of a module-level function definition enters the function name in the module's global symbol table. - diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst index d71598efdc..598859de6a 100644 --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -15,7 +15,7 @@ operating system:: >>> import os >>> os.getcwd() # Return the current working directory - 'C:\\Python34' + 'C:\\Python35' >>> os.chdir('/server/accesslogs') # Change current working directory >>> os.system('mkdir today') # Run the command mkdir in the system shell 0 diff --git a/Doc/tutorial/stdlib2.rst b/Doc/tutorial/stdlib2.rst index c0197ea5ef..f7d2a0ac2a 100644 --- a/Doc/tutorial/stdlib2.rst +++ b/Doc/tutorial/stdlib2.rst @@ -18,7 +18,7 @@ abbreviated displays of large or deeply nested containers:: >>> import reprlib >>> reprlib.repr(set('supercalifragilisticexpialidocious')) - "set(['a', 'c', 'd', 'e', 'f', 'g', ...])" + "{'a', 'c', 'd', 'e', 'f', 'g', ...}" The :mod:`pprint` module offers more sophisticated control over printing both built-in and user defined objects in a way that is readable by the interpreter. @@ -277,7 +277,7 @@ applications include caching objects that are expensive to create:: Traceback (most recent call last): File "<stdin>", line 1, in <module> d['primary'] # entry was automatically removed - File "C:/python34/lib/weakref.py", line 46, in __getitem__ + File "C:/python35/lib/weakref.py", line 46, in __getitem__ o = self.data[key]() KeyError: 'primary' diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 4017ce897c..81cd38a533 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -190,13 +190,16 @@ Miscellaneous options .. cmdoption:: -b - Issue a warning when comparing str and bytes. Issue an error when the + Issue a warning when comparing :class:`bytes` or :class:`bytearray` with + :class:`str` or :class:`bytes` with :class:`int`. Issue an error when the option is given twice (:option:`-bb`). + .. versionchanged: 3.5 + Affects comparisons of :class:`bytes` with :class:`int`. .. cmdoption:: -B - If given, Python won't try to write ``.pyc`` or ``.pyo`` files on the + If given, Python won't try to write ``.pyc``` files on the import of source modules. See also :envvar:`PYTHONDONTWRITEBYTECODE`. diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc index 45bdd5a46d..02bcbeeb7f 100644 --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -21,11 +21,11 @@ subdirectory (on Windows, this is ``Lib\site-packages``). On Windows, you may have to invoke the ``pyvenv`` script as follows, if you don't have the relevant PATH and PATHEXT settings:: - c:\Temp>c:\Python34\python c:\Python34\Tools\Scripts\pyvenv.py myenv + c:\Temp>c:\Python35\python c:\Python35\Tools\Scripts\pyvenv.py myenv or equivalently:: - c:\Temp>c:\Python34\python -m venv myenv + c:\Temp>c:\Python35\python -m venv myenv The command, if run with ``-h``, will show the available options:: diff --git a/Doc/using/win_installer.png b/Doc/using/win_installer.png Binary files differnew file mode 100644 index 0000000000..4696bb2a40 --- /dev/null +++ b/Doc/using/win_installer.png diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index c05f72a853..7113dfa118 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -7,6 +7,7 @@ ************************* .. sectionauthor:: Robert Lehmann <lehmannro@gmail.com> +.. sectionauthor:: Steve Dower <steve.dower@microsoft.com> This document aims to give an overview of Windows-specific behaviour you should know about when using Python on Microsoft Windows. @@ -19,10 +20,172 @@ know about when using Python on Microsoft Windows. Installing Python ================= -Unlike most Unix systems and services, Windows does not require Python natively -and thus does not pre-install a version of Python. However, the CPython team +Unlike most Unix systems and services, Windows does not include a system +supported installation of Python. To make Python available, the CPython team has compiled Windows installers (MSI packages) with every `release -<https://www.python.org/download/releases/>`_ for many years. +<https://www.python.org/download/releases/>`_ for many years. These installers +are primarily intended to add a system-wide installation of Python, with the +core interpreter and library being shared by all application. Non-shared +layouts of the Python interpreter may also be created with the same installer, +however, the released installer is not intended for embedding in other +installers. + +Installation Steps +------------------ + +Four Python 3.5 installers are available for download - two each for the 32-bit +and 64-bit versions of the interpreter. The *web installer* is a small initial +download, and it will automatically download the required components as +necessary. The *offline installer* includes the components necessary for a +default installation and only requires an internet connection for optional +features. See :ref:`install-layout-option` for other ways to avoid downloading +during installation. + +After starting the installer, one of three options may be selected: + +.. image:: win_installer.png + +If you select "Install for All Users": + +* You may be required to provide administrative credentials or approval +* Python will be installed into your Program Files directory +* The :ref:`launcher` will be installed into your Windows directory +* The standard library, test suite, launcher and pip will be installed +* After installation, the standard library will be pre-compiled to bytecode +* If selected, the install directory will be added to :envvar:`PATH` + +If you select "Install Just for Me": + +* You will *not* need to be an administrator +* Python will be installed into your user directory +* The :ref:`launcher` will *also* be installed into your user directory +* The standard library, test suite, launcher and pip will be installed +* If selected, the install directory will be added to :envvar:`PATH` + +Selecting "Customize installation" will allow you to select the features to +install, the installation location and other options or post-install actions. +To install debugging symbols or binaries, you will need to use this option. + +.. _install-quiet-option: + +Installing Without UI +--------------------- + +All of the options available in the installer UI can also be specified from the +command line, allowing scripted installers to replicate an installation on many +machines without user interaction. These options may also be set without +suppressing the UI in order to change some of the defaults. + +To completely hide the installer UI and install Python silently, pass the +``/quiet`` (``/q``) option. To skip past the user interaction but still display +progress and errors, pass the ``/passive`` (``/p``) option. The ``/uninstall`` +option may be passed to immediately begin removing Python - no prompt will be +displayed. + +All other options are passed as ``name=value``, where the value is usually +``0`` to disable a feature, ``1`` to enable a feature, or a path. The full list +of available options is shown below. + ++---------------------------+--------------------------------------+--------------------------+ +| Name | Description | Default | ++===========================+======================================+==========================+ +| InstallAllUsers | Perform a system-wide installation. | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| TargetDir | The installation directory | Selected based on | +| | | InstallAllUsers | ++---------------------------+--------------------------------------+--------------------------+ +| DefaultAllUsersTargetDir | The default installation directory | :file:`%ProgramFiles%\\\ | +| | for all-user installs | Python X.Y` or :file:`\ | +| | | %ProgramFiles(x86)%\\\ | +| | | Python X.Y` | ++---------------------------+--------------------------------------+--------------------------+ +| DefaultJustForMeTargetDir | The default install directory for | :file:`%LocalAppData%\\\ | +| | just-for-me installs | Programs\\PythonXY` or | +| | | :file:`%LocalAppData%\\\ | +| | | Programs\\PythonXY-32` | ++---------------------------+--------------------------------------+--------------------------+ +| DefaultCustomTargetDir | The default custom install directory | (empty) | +| | displayed in the UI | | ++---------------------------+--------------------------------------+--------------------------+ +| AssociateFiles | Create file associations if the | 1 | +| | launcher is also installed. | | ++---------------------------+--------------------------------------+--------------------------+ +| CompileAll | Compile all ``.py`` files to | 0 | +| | ``.pyc``. | | ++---------------------------+--------------------------------------+--------------------------+ +| PrependPath | Add install and Scripts directories | 0 | +| | tho :envvar:`PATH` and ``.PY`` to | | +| | :envvar:`PATHEXT` | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_doc | Install Python manual | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_debug | Install debug binaries | 0 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_dev | Install developer headers and | 1 | +| | libraries | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_exe | Install :file:`python.exe` and | 1 | +| | related files | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_launcher | Install :ref:`launcher`. | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_lib | Install standard library and | 1 | +| | extension modules | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_pip | Install bundled pip and setuptools | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_symbols | Install debugging symbols (`*`.pdb) | 0 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_tcltk | Install Tcl/Tk support and IDLE | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_test | Install standard library test suite | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_tools | Install utility scripts | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| SimpleInstall | Disable most install UI | 0 | ++---------------------------+--------------------------------------+--------------------------+ + +For example, to silently install a default, system-wide Python installation, +you could use the following command (from an elevated command prompt):: + + python-3.5.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0 + +To allow users to easily install a personal copy of Python without the test +suite, you could provide a shortcut with the following command:: + + python-3.5.0.exe /passive InstallAllUsers=0 Include_launcher=0 Include_test=0 SimpleInstall=1 + +(Note that omitting the launcher also omits file associations, and is only +recommended for per-user installs when there is also a system-wide installation +that included the launcher.) + +.. _install-layout-option: + +Installing Without Downloading +------------------------------ + +As some features of Python are not included in the initial installer download, +selecting those features may require an internet connection. To avoid this +need, all possible components may be downloaded on-demand to create a complete +*layout* that will no longer require an internet connection regardless of the +selected features. Note that this download may be bigger than required, but +where a large number of installations are going to be performed it is very +useful to have a locally cached copy. + +Execute the following command from Command Prompt to download all possible +required files. Remember to substitute ``python-3.5.0.exe`` for the actual +name of your installer, and to create layouts in their own directories to +avoid collisions between files with the same name. + +:: + + python-3.5.0.exe /layout [optional target directory] + +You may also specify the ``/quiet`` option to hide the progress display. + + +Other Platforms +--------------- With ongoing development of Python, some platforms that used to be supported earlier are no longer supported (due to the lack of users or developers). @@ -66,19 +229,31 @@ key features: `ActivePython <http://www.activestate.com/activepython/>`_ Installer with multi-platform compatibility, documentation, PyWin32 -`Enthought Python Distribution <https://www.enthought.com/products/epd/>`_ - Popular modules (such as PyWin32) with their respective documentation, tool - suite for building extensible Python applications +`Anaconda <http://www.continuum.io/downloads/>`_ + Popular scientific modules (such as numpy, scipy and pandas) and the + ``conda`` package manager. + +`Canopy <https://www.enthought.com/products/canopy/>`_ + A "comprehensive Python analysis environment" with editors and other + development tools. -Notice that these packages are likely to install *older* versions of Python. +`WinPython <https://winpython.github.io/>`_ + Windows-specific distribution with prebuilt scientific packages and + tools for building packages. + +Note that these packages may not include the latest versions of Python or +other libraries, and are not maintained or supported by the core Python team. Configuring Python ================== -In order to run Python flawlessly, you might have to change certain environment -settings in Windows. +To run Python conveniently from a command prompt, you might consider changing +some default environment variables in Windows. While the installer provides an +option to configure the PATH and PATHEXT variables for you, this is only +reliable for a single, system-wide installation. If you regularly use multiple +versions of Python, consider using the :ref:`launcher`. .. _setting-envvars: @@ -86,163 +261,86 @@ settings in Windows. Excursus: Setting environment variables --------------------------------------- -Windows has a built-in dialog for changing environment variables (following -guide applies to XP classical view): Right-click the icon for your machine -(usually located on your Desktop and called "My Computer") and choose -:menuselection:`Properties` there. Then, open the :guilabel:`Advanced` tab -and click the :guilabel:`Environment Variables` button. +Windows allows environment variables to be configured permanently at both the +User level and the System level, or temporarily in a command prompt. + +To temporarily set environment variables, open Command Prompt and use the +:command:`set` command:: -In short, your path is: + C:\>set PATH=C:\Program Files\Python 3.5;%PATH% + C:\>set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib + C:\>python - :menuselection:`My Computer - --> Properties - --> Advanced - --> Environment Variables` +These changes will apply to any further commands executed in that console, and +will be inherited by any applications started from the console. +Including the variable name within percent signs will expand to the existing +value, allowing you to add your new value at either the start or the end. +Modifying :envvar:`PATH` by adding the directory containing +:program:`python.exe` to the start is a common way to ensure the correct version +of Python is launched. + +To permanently modify the default environment variables, click Start and search +for 'edit environment variables', or open System properties, :guilabel:`Advanced +system settings` and click the :guilabel:`Environment Variables` button. In this dialog, you can add or modify User and System variables. To change System variables, you need non-restricted access to your machine (i.e. Administrator rights). -Another way of adding variables to your environment is using the :command:`set` -command:: - - set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib - -To make this setting permanent, you could add the corresponding command line to -your :file:`autoexec.bat`. :program:`msconfig` is a graphical interface to this -file. +.. note:: -Viewing environment variables can also be done more straight-forward: The -command prompt will expand strings wrapped into percent signs automatically:: + Windows will concatenate User variables *after* System variables, which may + cause unexpected results when modifying :envvar:`PATH`. - echo %PATH% - -Consult :command:`set /?` for details on this behaviour. + The :envvar:`PYTHONPATH` variable is used by all versions of Python 2 and + Python 3, so you should not permanently configure this variable unless it + only includes code that is compatible all of your installed Python + versions. .. seealso:: - http://support.microsoft.com/kb/100843 + http://support.microsoft.com/kb/100843 Environment variables in Windows NT - http://support.microsoft.com/kb/310519 + http://technet.microsoft.com/en-us/library/cc754250.aspx + The SET command, for temporarily modifying environment variables + + http://technet.microsoft.com/en-us/library/cc755104.aspx + The SETX command, for permanently modifying environment variables + + http://support.microsoft.com/kb/310519 How To Manage Environment Variables in Windows XP - http://www.chem.gla.ac.uk/~louis/software/faq/q1.html + http://www.chem.gla.ac.uk/~louis/software/faq/q1.html Setting Environment variables, Louis J. Farrugia - .. _windows-path-mod: Finding the Python executable ----------------------------- -.. versionchanged:: 3.3 +.. versionchanged:: 3.5 Besides using the automatically created start menu entry for the Python -interpreter, you might want to start Python in the command prompt. As of -Python 3.3, the installer has an option to set that up for you. - -At the "Customize Python 3.3" screen, an option called -"Add python.exe to search path" can be enabled to have the installer place -your installation into the :envvar:`%PATH%`. This allows you to type -:command:`python` to run the interpreter. Thus, you can also execute your +interpreter, you might want to start Python in the command prompt. The +installer for Python 3.5 and later has an option to set that up for you. + +On the first page of the installer, an option labelled "Add Python 3.5 to +PATH" can be selected to have the installer add the install location into the +:envvar:`PATH`. The location of the :file:`Scripts\\` folder is also added. +This allows you to type :command:`python` to run the interpreter, and +:command:`pip` or . Thus, you can also execute your scripts with command line options, see :ref:`using-on-cmdline` documentation. If you don't enable this option at install time, you can always re-run the -installer to choose it. - -The alternative is manually modifying the :envvar:`%PATH%` using the -directions in :ref:`setting-envvars`. You need to set your :envvar:`%PATH%` -environment variable to include the directory of your Python distribution, -delimited by a semicolon from other entries. An example variable could look -like this (assuming the first two entries are Windows' default):: - - C:\WINDOWS\system32;C:\WINDOWS;C:\Python33 - - -Finding modules ---------------- - -Python usually stores its library (and thereby your site-packages folder) in the -installation directory. So, if you had installed Python to -:file:`C:\\Python\\`, the default library would reside in -:file:`C:\\Python\\Lib\\` and third-party modules should be stored in -:file:`C:\\Python\\Lib\\site-packages\\`. - -This is how :data:`sys.path` is populated on Windows: - -* An empty entry is added at the start, which corresponds to the current - directory. - -* If the environment variable :envvar:`PYTHONPATH` exists, as described in - :ref:`using-on-envvars`, its entries are added next. Note that on Windows, - paths in this variable must be separated by semicolons, to distinguish them - from the colon used in drive identifiers (``C:\`` etc.). - -* Additional "application paths" can be added in the registry as subkeys of - :samp:`\\SOFTWARE\\Python\\PythonCore\\{version}\\PythonPath` under both the - ``HKEY_CURRENT_USER`` and ``HKEY_LOCAL_MACHINE`` hives. Subkeys which have - semicolon-delimited path strings as their default value will cause each path - to be added to :data:`sys.path`. (Note that all known installers only use - HKLM, so HKCU is typically empty.) - -* If the environment variable :envvar:`PYTHONHOME` is set, it is assumed as - "Python Home". Otherwise, the path of the main Python executable is used to - locate a "landmark file" (``Lib\os.py``) to deduce the "Python Home". If a - Python home is found, the relevant sub-directories added to :data:`sys.path` - (``Lib``, ``plat-win``, etc) are based on that folder. Otherwise, the core - Python path is constructed from the PythonPath stored in the registry. - -* If the Python Home cannot be located, no :envvar:`PYTHONPATH` is specified in - the environment, and no registry entries can be found, a default path with - relative entries is used (e.g. ``.\Lib;.\plat-win``, etc). - -The end result of all this is: - -* When running :file:`python.exe`, or any other .exe in the main Python - directory (either an installed version, or directly from the PCbuild - directory), the core path is deduced, and the core paths in the registry are - ignored. Other "application paths" in the registry are always read. - -* When Python is hosted in another .exe (different directory, embedded via COM, - etc), the "Python Home" will not be deduced, so the core path from the - registry is used. Other "application paths" in the registry are always read. - -* If Python can't find its home and there is no registry (eg, frozen .exe, some - very strange installation setup) you get a path with some default, but - relative, paths. - - -Executing scripts ------------------ - -As of Python 3.3, Python includes a launcher which facilitates running Python -scripts. See :ref:`launcher` for more information. - -Executing scripts without the Python launcher ---------------------------------------------- - -Without the Python launcher installed, Python scripts (files with the extension -``.py``) will be executed by :program:`python.exe` by default. This executable -opens a terminal, which stays open even if the program uses a GUI. If you do -not want this to happen, use the extension ``.pyw`` which will cause the script -to be executed by :program:`pythonw.exe` by default (both executables are -located in the top-level of your Python installation directory). This -suppresses the terminal window on startup. - -You can also make all ``.py`` scripts execute with :program:`pythonw.exe`, -setting this through the usual facilities, for example (might require -administrative rights): - -#. Launch a command prompt. -#. Associate the correct file group with ``.py`` scripts:: - - assoc .py=Python.File - -#. Redirect all Python files to the new executable:: - - ftype Python.File=C:\Path\to\pythonw.exe "%1" %* +installer, select Modify, and enable it. Alternatively, you can manually +modify the :envvar:`PATH` using the directions in :ref:`setting-envvars`. You +need to set your :envvar:`PATH` environment variable to include the directory +of your Python installation, delimited by a semicolon from other entries. An +example variable could look like this (assuming the first two entries already +existed):: + C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.5 .. _launcher: @@ -251,21 +349,26 @@ Python Launcher for Windows .. versionadded:: 3.3 -The Python launcher for Windows is a utility which aids in the location and -execution of different Python versions. It allows scripts (or the +The Python launcher for Windows is a utility which aids in locating and +executing of different Python versions. It allows scripts (or the command-line) to indicate a preference for a specific Python version, and will locate and execute that version. +Unlike the :envvar:`PATH` variable, the launcher will correctly select the most +appropriate version of Python. It will prefer per-user installations over +system-wide ones, and orders by language version rather than using the most +recently installed version. + Getting started --------------- From the command-line ^^^^^^^^^^^^^^^^^^^^^ -You should ensure the launcher is on your PATH - depending on how it was -installed it may already be there, but check just in case it is not. - -From a command-prompt, execute the following command: +System-wide installations of Python 3.3 and later will put the launcher on your +:envvar:`PATH`. The launcher is compatible with all available versions of +Python, so it does not matter which version is installed. To check that the +launcher is available, execute the following command in Command Prompt: :: @@ -291,6 +394,28 @@ If you have a Python 3.x installed, try the command: You should find the latest version of Python 3.x starts. +If you see the following error, you do not have the launcher installed: + +:: + + 'py' is not recognized as an internal or external command, + operable program or batch file. + +Per-user installations of Python do not add the launcher to :envvar:`PATH` +unless the option was selected on installation. + +Virtual environments +^^^^^^^^^^^^^^^^^^^^ + +.. versionadded:: 3.5 + +If the launcher is run with no explicit Python version specification, and a +virtual environment (created with the standard library :mod:`venv` module or +the external ``virtualenv`` tool) active, the launcher will run the virtual +environment's interpreter rather than the global one. To run the global +interpreter, either deactivate the virtual environment, or explicitly specify +the global Python version. + From a script ^^^^^^^^^^^^^ @@ -326,7 +451,7 @@ From file associations ^^^^^^^^^^^^^^^^^^^^^^ The launcher should have been associated with Python files (i.e. ``.py``, -``.pyw``, ``.pyc``, ``.pyo`` files) when it was installed. This means that +``.pyw``, ``.pyc`` files) when it was installed. This means that when you double-click on one of these files from Windows explorer the launcher will be used, and therefore you can use the same facilities described above to have the script specify the version which should be used. @@ -365,6 +490,16 @@ be used by the launcher without modification. If you are writing a new script on Windows which you hope will be useful on Unix, you should use one of the shebang lines starting with ``/usr``. +Any of the above virtual commands can be suffixed with an explicit version +(either just the major version, or the major and minor version) - for example +``/usr/bin/python2.7`` - which will cause that specific version to be located +and used. + +The ``/usr/bin/env`` form of shebang line has one further special property. +Before looking for installed Python interpreters, this form will search the +executable :envvar:`PATH` for a Python executable. This corresponds to the +behaviour of the Unix ``env`` program, which performs a :envvar:`PATH` search. + Arguments in shebang lines -------------------------- @@ -383,17 +518,16 @@ Customization Customization via INI files ^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Two .ini files will be searched by the launcher - ``py.ini`` in the - current user's "application data" directory (i.e. the directory returned - by calling the Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA) - and ``py.ini`` in the same directory as the launcher. The same .ini - files are used for both the 'console' version of the launcher (i.e. - py.exe) and for the 'windows' version (i.e. pyw.exe) +Two .ini files will be searched by the launcher - ``py.ini`` in the current +user's "application data" directory (i.e. the directory returned by calling the +Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA) and ``py.ini`` in the +same directory as the launcher. The same .ini files are used for both the +'console' version of the launcher (i.e. py.exe) and for the 'windows' version +(i.e. pyw.exe) - Customization specified in the "application directory" will have - precedence over the one next to the executable, so a user, who may not - have write access to the .ini file next to the launcher, can override - commands in that global .ini file) +Customization specified in the "application directory" will have precedence over +the one next to the executable, so a user, who may not have write access to the +.ini file next to the launcher, can override commands in that global .ini file) Customizing default Python versions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -488,6 +622,78 @@ particular version was chosen and the exact command-line used to execute the target Python. + +Finding modules +=============== + +Python usually stores its library (and thereby your site-packages folder) in the +installation directory. So, if you had installed Python to +:file:`C:\\Python\\`, the default library would reside in +:file:`C:\\Python\\Lib\\` and third-party modules should be stored in +:file:`C:\\Python\\Lib\\site-packages\\`. + +This is how :data:`sys.path` is populated on Windows: + +* An empty entry is added at the start, which corresponds to the current + directory. + +* If the environment variable :envvar:`PYTHONPATH` exists, as described in + :ref:`using-on-envvars`, its entries are added next. Note that on Windows, + paths in this variable must be separated by semicolons, to distinguish them + from the colon used in drive identifiers (``C:\`` etc.). + +* Additional "application paths" can be added in the registry as subkeys of + :samp:`\\SOFTWARE\\Python\\PythonCore\\{version}\\PythonPath` under both the + ``HKEY_CURRENT_USER`` and ``HKEY_LOCAL_MACHINE`` hives. Subkeys which have + semicolon-delimited path strings as their default value will cause each path + to be added to :data:`sys.path`. (Note that all known installers only use + HKLM, so HKCU is typically empty.) + +* If the environment variable :envvar:`PYTHONHOME` is set, it is assumed as + "Python Home". Otherwise, the path of the main Python executable is used to + locate a "landmark file" (``Lib\os.py``) to deduce the "Python Home". If a + Python home is found, the relevant sub-directories added to :data:`sys.path` + (``Lib``, ``plat-win``, etc) are based on that folder. Otherwise, the core + Python path is constructed from the PythonPath stored in the registry. + +* If the Python Home cannot be located, no :envvar:`PYTHONPATH` is specified in + the environment, and no registry entries can be found, a default path with + relative entries is used (e.g. ``.\Lib;.\plat-win``, etc). + +The end result of all this is: + +* When running :file:`python.exe`, or any other .exe in the main Python + directory (either an installed version, or directly from the PCbuild + directory), the core path is deduced, and the core paths in the registry are + ignored. Other "application paths" in the registry are always read. + +* When Python is hosted in another .exe (different directory, embedded via COM, + etc), the "Python Home" will not be deduced, so the core path from the + registry is used. Other "application paths" in the registry are always read. + +* If Python can't find its home and there is no registry (eg, frozen .exe, some + very strange installation setup) you get a path with some default, but + relative, paths. + +For those who want to bundle Python into their application or distribution, the +following advice will prevent conflicts with other installations: + +* If you are loading :file:`python3.dll` or :file:`python35.dll` in your own + executable, explicitly call :c:func:`Py_SetPath` or (at least) + :c:func:`Py_SetProgramName` before :c:func:`Py_Initialize`. + +* Clear and/or overwrite :envvar:`PYTHONPATH` and set :envvar:`PYTHONHOME` + before launching :file:`python.exe` from your application. + +* If you cannot use the previous suggestions (for example, you are a + distribution that allows people to run :file:`python.exe` directly), ensure + that the landmark file (:file:`Lib\\os.py`) exists in your bundled library. + (Note that it will not be detected inside a ZIP file.) + +These will ensure that the files in a system-wide installation will not take +precedence over the copy of the standard library bundled with your application. +Otherwise, your users may experience problems using your application. + Additional modules ================== @@ -498,7 +704,6 @@ and external, and snippets exist to use these features. The Windows-specific standard modules are documented in :ref:`mswin-specific-services`. - PyWin32 ------- @@ -557,20 +762,8 @@ latest release's source or just grab a fresh `checkout <https://docs.python.org/devguide/setup.html#getting-the-source-code>`_. The source tree contains a build solution and project files for Microsoft -Visual C++, which is the compiler used to build the official Python releases. -View the :file:`readme.txt` in their respective directories: - -+--------------------+--------------+-----------------------+ -| Directory | MSVC version | Visual Studio version | -+====================+==============+=======================+ -| :file:`PC/VS9.0/` | 9.0 | 2008 | -+--------------------+--------------+-----------------------+ -| :file:`PCbuild/` | 10.0 | 2010 | -+--------------------+--------------+-----------------------+ - -Note that any build directories within the :file:`PC` directory are not -necessarily fully supported. The :file:`PCbuild` directory contains the files -for the compiler used to build the official release. +Visual Studio 2015, which is the compiler used to build the official Python +releases. These files are in the :file:`PCbuild` directory. Check :file:`PCbuild/readme.txt` for general information on the build process. @@ -603,5 +796,3 @@ Other resources :pep:`397` - Python launcher for Windows The proposal for the launcher to be included in the Python distribution. - - diff --git a/Doc/whatsnew/2.1.rst b/Doc/whatsnew/2.1.rst index 5058bf1d5a..ff1566226d 100644 --- a/Doc/whatsnew/2.1.rst +++ b/Doc/whatsnew/2.1.rst @@ -219,7 +219,7 @@ comparison. I won't cover the C API here, but will refer you to PEP 207, or to .. seealso:: - :pep:`207` - Rich Comparisions + :pep:`207` - Rich Comparisons Written by Guido van Rossum, heavily based on earlier work by David Ascher, and implemented by Guido van Rossum. diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst index 1fdb365847..1d4ce7236f 100644 --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -1579,7 +1579,7 @@ os avoid race conditions in multi-threaded programs. * The :mod:`os` module has a new :func:`~os.sendfile` function which provides - an efficent "zero-copy" way for copying data from one file (or socket) + an efficient "zero-copy" way for copying data from one file (or socket) descriptor to another. The phrase "zero-copy" refers to the fact that all of the copying of data between the two descriptors is done entirely by the kernel, with no copying of data into userspace buffers. :func:`~os.sendfile` @@ -1908,7 +1908,7 @@ socketserver :meth:`~socketserver.BaseServer.service_actions` that is called by the :meth:`~socketserver.BaseServer.serve_forever` method in the service loop. :class:`~socketserver.ForkingMixIn` now uses this to clean up zombie -child proceses. (Contributed by Justin Warkentin in :issue:`11109`.) +child processes. (Contributed by Justin Warkentin in :issue:`11109`.) sqlite3 @@ -2360,7 +2360,7 @@ Porting Python code bytecode file, make sure to call :func:`importlib.invalidate_caches` to clear out the cache for the finders to notice the new file. -* :exc:`ImportError` now uses the full name of the module that was attemped to +* :exc:`ImportError` now uses the full name of the module that was attempted to be imported. Doctests that check ImportErrors' message will need to be updated to use the full name of the module instead of just the tail of the name. diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst new file mode 100644 index 0000000000..a1130df3c8 --- /dev/null +++ b/Doc/whatsnew/3.5.rst @@ -0,0 +1,817 @@ +**************************** + What's New In Python 3.5 +**************************** + +:Release: |release| +:Date: |today| + +.. Rules for maintenance: + + * Anyone can add text to this document. Do not spend very much time + on the wording of your changes, because your text will probably + get rewritten to some degree. + + * The maintainer will go through Misc/NEWS periodically and add + changes; it's therefore more important to add your changes to + Misc/NEWS than to this file. + + * This is not a complete list of every single change; completeness + is the purpose of Misc/NEWS. Some changes I consider too small + or esoteric to include. If such a change is added to the text, + I'll just remove it. (This is another reason you shouldn't spend + too much time on writing your addition.) + + * If you want to draw your new text to the attention of the + maintainer, add 'XXX' to the beginning of the paragraph or + section. + + * It's OK to just add a fragmentary note about a change. For + example: "XXX Describe the transmogrify() function added to the + socket module." The maintainer will research the change and + write the necessary text. + + * You can comment out your additions if you like, but it's not + necessary (especially when a final release is some months away). + + * Credit the author of a patch or bugfix. Just the name is + sufficient; the e-mail address isn't necessary. + + * It's helpful to add the bug/patch number as a comment: + + XXX Describe the transmogrify() function added to the socket + module. + (Contributed by P.Y. Developer in :issue:`12345`.) + + This saves the maintainer the effort of going through the Mercurial log + when researching a change. + +This article explains the new features in Python 3.5, compared to 3.4. + +For full details, see the :source:`Misc/NEWS` file. + +.. note:: + + Prerelease users should be aware that this document is currently in draft + form. It will be updated substantially as Python 3.5 moves towards release, + so it's worth checking back even after reading earlier versions. + + +.. seealso:: + + :pep:`478` - Python 3.5 Release Schedule + + +Summary -- Release highlights +============================= + +.. This section singles out the most important changes in Python 3.5. + Brevity is key. + +New syntax features: + +* :pep:`465`, a new matrix multiplication operator: ``a @ b``. + +New library modules: + +* :mod:`zipapp`: :ref:`Improving Python ZIP Application Support + <whatsnew-zipapp>` (:pep:`441`). + +New built-in features: + +* ``bytes % args``, ``bytearray % args``: :pep:`461` - Adding ``%`` formatting + to bytes and bytearray + +Implementation improvements: + +* When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale), + :py:data:`sys.stdin` and :py:data:`sys.stdout` are now using the + ``surrogateescape`` error handler, instead of the ``strict`` error handler + (:issue:`19977`). + +* :pep:`488`, the elimination of ``.pyo`` files. + +Significantly Improved Library Modules: + +* None yet. + +Security improvements: + +* None yet. + +Please read on for a comprehensive list of user-facing changes. + + +.. PEP-sized items next. + +.. _pep-4XX: + +.. PEP 4XX: Virtual Environments +.. ============================= + + +.. (Implemented by Foo Bar.) + +.. .. seealso:: + + :pep:`4XX` - Python Virtual Environments + PEP written by Carl Meyer + + +PEP 461 - Adding % formatting to bytes and bytearray +---------------------------------------------------- + +This PEP proposes adding % formatting operations similar to Python 2's ``str`` +type to :class:`bytes` and :class:`bytearray`. + +Examples:: + + >>> b'Hello %s!' % b'World' + b'Hello World!' + >>> b'x=%i y=%f' % (1, 2.5) + b'x=1 y=2.500000' + +Unicode is not allowed for ``%s``, but it is accepted by ``%a`` (equivalent of +``repr(obj).encode('ascii', 'backslashreplace')``):: + + >>> b'Hello %s!' % 'World' + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: %b requires bytes, or an object that implements __bytes__, not 'str' + >>> b'price: %a' % '10€' + b"price: '10\\u20ac'" + +.. seealso:: + + :pep:`461` -- Adding % formatting to bytes and bytearray + + +PEP 465 - A dedicated infix operator for matrix multiplication +-------------------------------------------------------------- + +This PEP proposes a new binary operator to be used for matrix multiplication, +called ``@``. (Mnemonic: ``@`` is ``*`` for mATrices.) + +.. seealso:: + + :pep:`465` -- A dedicated infix operator for matrix multiplication + + +PEP 471 - os.scandir() function -- a better and faster directory iterator +------------------------------------------------------------------------- + +:pep:`471` adds a new directory iteration function, :func:`os.scandir`, +to the standard library. Additionally, :func:`os.walk` is now +implemented using :func:`os.scandir`, which speeds it up by 3-5 times +on POSIX systems and by 7-20 times on Windows systems. + +PEP and implementation written by Ben Hoyt with the help of Victor Stinner. + +.. seealso:: + + :pep:`471` -- os.scandir() function -- a better and faster directory + iterator + + +PEP 475: Retry system calls failing with EINTR +---------------------------------------------- + +:pep:`475` adds support for automatic retry of system calls failing with +:py:data:`~errno.EINTR`: this means that user code doesn't have to deal with +EINTR or :exc:`InterruptedError` manually, and should make it more robust +against asynchronous signal reception. + +.. seealso:: + + :pep:`475` -- Retry system calls failing with EINTR + + +PEP 486: Make the Python Launcher aware of virtual environments +--------------------------------------------------------------- + +:pep:`486` makes the Windows launcher (see :pep:`397`) aware of an active +virtual environment. When the default interpreter would be used and the +``VIRTUAL_ENV`` environment variable is set, the interpreter in the virtual +environment will be used. + +.. seealso:: + + :pep:`486` -- Make the Python Launcher aware of virtual environments + + +PEP 488: Elimination of PYO files +--------------------------------- + +:pep:`488` does away with the concept of ``.pyo`` files. This means that +``.pyc`` files represent both unoptimized and optimized bytecode. To prevent +the need to constantly regenerate bytecode files, ``.pyc`` files now have an +optional ``opt-`` tag in their name when the bytecode is optimized. This has +the side-effect of no more bytecode file name clashes when running under either +``-O`` or ``-OO``, thus allowing unoptimized, ``-O``, and ``-OO`` bytecode files +to all exist simultaneously. :func:`importlib.util.cache_from_source` has an +updated API to help with this change. + +.. seealso:: + + :pep:`488` -- Elimination of PYO files + + +Other Language Changes +====================== + +Some smaller changes made to the core Python language are: + +* Added the ``'namereplace'`` error handlers. The ``'backslashreplace'`` + error handlers now works with decoding and translating. + (Contributed by Serhiy Storchaka in :issue:`19676` and :issue:`22286`.) + +* The :option:`-b` option now affects comparisons of :class:`bytes` with + :class:`int`. (Contributed by Serhiy Storchaka in :issue:`23681`) + + +New Modules +=========== + +.. _whatsnew-zipapp: + +zipapp +------ + +The new :mod:`zipapp` module (specified in :pep:`441`) provides an API and +command line tool for creating executable Python Zip Applications, which +were introduced in Python 2.6 in :issue:`1739468` but which were not well +publicised, either at the time or since. + +With the new module, bundling your application is as simple as putting all +the files, including a ``__main__.py`` file, into a directory ``myapp`` +and running:: + + $ python -m zipapp myapp + $ python myapp.pyz + + +Improved Modules +================ + +argparse +-------- + +* :class:`~argparse.ArgumentParser` now allows to disable + :ref:`abbreviated usage <prefix-matching>` of long options by setting + :ref:`allow_abbrev` to ``False``. + (Contributed by Jonathan Paugh, Steven Bethard, paul j3 and Daniel Eriksson.) + +cgi +--- + +* :class:`~cgi.FieldStorage` now supports the context management protocol. + (Contributed by Berker Peksag in :issue:`20289`.) + +code +---- + +* The :func:`code.InteractiveInterpreter.showtraceback` method now prints + the full chained traceback, just like the interactive interpreter. + (Contributed by Claudiu Popa in :issue:`17442`.) + +compileall +---------- + +* :func:`compileall.compile_dir` and :mod:`compileall`'s command-line interface + can now do parallel bytecode compilation. + (Contributed by Claudiu Popa in :issue:`16104`.) + +contextlib +---------- + +* The new :func:`contextlib.redirect_stderr` context manager(similar to + :func:`contextlib.redirect_stdout`) makes it easier for utility scripts to + handle inflexible APIs that write their output to :data:`sys.stderr` and + don't provide any options to redirect it. + (Contributed by Berker Peksag in :issue:`22389`.) + +difflib +------- + +* The charset of the HTML document generated by :meth:`difflib.HtmlDiff.make_file` + can now be customized by using *charset* keyword-only parameter. The default + charset of HTML document changed from ``'ISO-8859-1'`` to ``'utf-8'``. + (Contributed by Berker Peksag in :issue:`2052`.) + +distutils +--------- + +* The ``build`` and ``build_ext`` commands now accept a ``-j`` + option to enable parallel building of extension modules. + (Contributed by Antoine Pitrou in :issue:`5309`.) + +doctest +------- + +* :func:`doctest.DocTestSuite` returns an empty :class:`unittest.TestSuite` if + *module* contains no docstrings instead of raising :exc:`ValueError`. + (Contributed by Glenn Jones in :issue:`15916`.) + +glob +---- + +* :func:`~glob.iglob` and :func:`~glob.glob` now support recursive search in + subdirectories using the "``**``" pattern. + (Contributed by Serhiy Storchaka in :issue:`13968`.) + +imaplib +------- + +* :class:`IMAP4` now supports the context management protocol. When used in a + :keyword:`with` statement, the IMAP4 ``LOGOUT`` command will be called + automatically at the end of the block. (Contributed by Tarek Ziadé and + Serhiy Storchaka in :issue:`4972`.) + +imghdr +------ + +* :func:`~imghdr.what` now recognizes the `OpenEXR <http://www.openexr.com>`_ + format. (Contributed by Martin Vignali and Claudiu Popa in :issue:`20295`.) + +importlib +--------- + +* :class:`importlib.util.LazyLoader` allows for the lazy loading of modules in + applications where startup time is paramount. + (Contributed by Brett Cannon in :issue:`17621`.) + +* :func:`importlib.abc.InspectLoader.source_to_code` is now a + static method to make it easier to work with source code in a string. + With a module object that you want to initialize you can then use + ``exec(code, module.__dict__)`` to execute the code in the module. + +* :func:`importlib.util.module_from_spec` is now the preferred way to create a + new module. Compared to :class:`types.ModuleType`, this new function will set + the various import-controlled attributes based on the passed-in spec object. + +inspect +------- + +* :class:`inspect.Signature` and :class:`inspect.Parameter` are now + picklable and hashable. (Contributed by Yury Selivanov in :issue:`20726` + and :issue:`20334`.) + +* New class method :meth:`inspect.Signature.from_callable`, which makes + subclassing of :class:`~inspect.Signature` easier. (Contributed + by Yury Selivanov and Eric Snow in :issue:`17373`.) + +ipaddress +--------- + +* :class:`ipaddress.IPv4Network` and :class:`ipaddress.IPv6Network` now + accept an ``(address, netmask)`` tuple argument, so as to easily construct + network objects from existing addresses. (Contributed by Peter Moody + and Antoine Pitrou in :issue:`16531`.) + +json +---- + +* The output of :mod:`json.tool` command line interface is now in the same + order as the input. Use the :option:`--sort-keys` option to sort the output + of dictionaries alphabetically by key. (Contributed by Berker Peksag in + :issue:`21650`.) + +* JSON decoder now raises :exc:`json.JSONDecodeError` instead of + :exc:`ValueError`. (Contributed by Serhiy Storchaka in :issue:`19361`.) + +os +-- + +* New :func:`os.scandir` function that exposes file information from + the operating system when listing a directory. :func:`os.scandir` + returns an iterator of :class:`os.DirEntry` objects corresponding to + the entries in the directory given by *path*. (Contributed by Ben + Hoyt with the help of Victor Stinner in :issue:`22524`.) + +* :class:`os.stat_result` now has a :attr:`~os.stat_result.st_file_attributes` + attribute on Windows. (Contributed by Ben Hoyt in :issue:`21719`.) + +os.path +------- + +* New :func:`~os.path.commonpath` function that extracts common path prefix. + Unlike the :func:`~os.path.commonprefix` function, it always returns a valid + patch. (Contributed by Rafik Draoui and Serhiy Storchaka in :issue:`10395`.) + +pickle +------ + +* Serializing more "lookupable" objects (such as unbound methods or nested + classes) now are supported with pickle protocols < 4. + (Contributed by Serhiy Storchaka in :issue:`23611`.) + +re +-- + +* Number of capturing groups in regular expression is no longer limited by 100. + (Contributed by Serhiy Storchaka in :issue:`22437`.) + +* Now unmatched groups are replaced with empty strings in :func:`re.sub` + and :func:`re.subn`. (Contributed by Serhiy Storchaka in :issue:`1519638`.) + +math +---- + +* :data:`math.inf` and :data:`math.nan` constants added. (Contributed by Mark + Dickinson in :issue:`23185`.) + +shutil +------ + +* :func:`~shutil.move` now accepts a *copy_function* argument, allowing, + for example, :func:`~shutil.copy` to be used instead of the default + :func:`~shutil.copy2` if there is a need to ignore metadata. (Contributed by + Claudiu Popa in :issue:`19840`.) + +signal +------ + +* On Windows, :func:`signal.set_wakeup_fd` now also supports socket handles. + (Contributed by Victor Stinner in :issue:`22018`.) + +* Different constants of :mod:`signal` module are now enumeration values using + the :mod:`enum` module. This allows meaningful names to be printed during + debugging, instead of integer “magic numbers”. (Contributed by Giampaolo + Rodola' in :issue:`21076`.) + +smtpd +----- + +* Both :class:`~smtpd.SMTPServer` and :class:`smtpd.SMTPChannel` now accept a + *decode_data* keyword to determine if the DATA portion of the SMTP + transaction is decoded using the ``utf-8`` codec or is instead provided to + :meth:`~smtpd.SMTPServer.process_message` as a byte string. The default + is ``True`` for backward compatibility reasons, but will change to ``False`` + in Python 3.6. (Contributed by Maciej Szulik in :issue:`19662`.) + +* It is now possible to provide, directly or via name resolution, IPv6 + addresses in the :class:`~smtpd.SMTPServer` constructor, and have it + successfully connect. (Contributed by Milan Oberkirch in :issue:`14758`.) + +* :mod:`~smtpd.SMTPServer` now supports :rfc:`6531` via the *enable_SMTPUTF8* + constructor argument and a user-provided + :meth:`~smtpd.SMTPServer.process_smtputf8_message` method. + +smtplib +------- + +* A new :meth:`~smtplib.SMTP.auth` method provides a convenient way to + implement custom authentication mechanisms. + (Contributed by Milan Oberkirch in :issue:`15014`.) + +sndhdr +------ + +* :func:`~sndhdr.what` and :func:`~sndhdr.whathdr` now return + :func:`~collections.namedtuple`. + (Contributed by Claudiu Popa in :issue:`18615`.) + +socket +------ + +* New :meth:`socket.socket.sendfile` method allows to send a file over a socket + by using high-performance :func:`os.sendfile` function on UNIX resulting in + uploads being from 2x to 3x faster than when using plain + :meth:`socket.socket.send`. + (Contributed by Giampaolo Rodola' in :issue:`17552`.) + +sysconfig +--------- + +* The user scripts directory on Windows is now versioned. + (Contributed by Paul Moore in :issue:`23437`.) + + +tarfile +------- + +* The :func:`tarfile.open` function now supports ``'x'`` (exclusive creation) + mode. (Contributed by Berker Peksag in :issue:`21717`.) + +time +---- + +* The :func:`time.monotonic` function is now always available. (Contributed by + Victor Stinner in :issue:`22043`.) + +urllib +------ + +* A new :class:`urllib.request.HTTPBasicPriorAuthHandler` allows HTTP Basic + Authentication credentials to be sent unconditionally with the first HTTP + request, rather than waiting for a HTTP 401 Unauthorized response from the + server. + (Contributed by Matej Cepl in :issue:`19494`.) + +wsgiref +------- + +* *headers* parameter of :class:`wsgiref.headers.Headers` is now optional. + (Contributed by Pablo Torres Navarrete and SilentGhost in :issue:`5800`.) + +xmlrpc +------ + +* :class:`xmlrpc.client.ServerProxy` is now a :term:`context manager`. + (Contributed by Claudiu Popa in :issue:`20627`.) + +xml.sax +------- + +* SAX parsers now support a character stream of + :class:`~xml.sax.xmlreader.InputSource` object. + (Contributed by Serhiy Storchaka in :issue:`2175`.) + +faulthandler +------------ + +* :func:`~faulthandler.enable`, :func:`~faulthandler.register`, + :func:`~faulthandler.dump_traceback` and + :func:`~faulthandler.dump_traceback_later` functions now accept file + descriptors. (Contributed by Wei Wu in :issue:`23566`.) + +zipfile +------- + +* Added support for writing ZIP files to unseekable streams. + (Contributed by Serhiy Storchaka in :issue:`23252`.) + +* The :func:`zipfile.ZipFile.open` function now supports ``'x'`` (exclusive + creation) mode. (Contributed by Serhiy Storchaka in :issue:`21717`.) + + +Optimizations +============= + +The following performance enhancements have been added: + +* :func:`os.walk` has been sped up by 3-5x on POSIX systems and 7-20x + on Windows. This was done using the new :func:`os.scandir` function, + which exposes file information from the underlying ``readdir`` and + ``FindFirstFile``/``FindNextFile`` system calls. (Contributed by + Ben Hoyt with help from Victor Stinner in :issue:`23605`.) + +* Construction of ``bytes(int)`` (filled by zero bytes) is faster and uses less + memory for large objects. ``calloc()`` is used instead of ``malloc()`` to + allocate memory for these objects. + +* Some operations on :class:`~ipaddress.IPv4Network` and + :class:`~ipaddress.IPv6Network` have been massively sped up, such as + :meth:`~ipaddress.IPv4Network.subnets`, :meth:`~ipaddress.IPv4Network.supernet`, + :func:`~ipaddress.summarize_address_range`, :func:`~ipaddress.collapse_addresses`. + The speed up can range from 3x to 15x. + (:issue:`21486`, :issue:`21487`, :issue:`20826`) + +* Many operations on :class:`io.BytesIO` are now 50% to 100% faster. + (Contributed by Serhiy Storchaka in :issue:`15381` and David Wilson in + :issue:`22003`.) + +* :func:`marshal.dumps` is now faster (65%-85% with versions 3--4, 20-25% with + versions 0--2 on typical data, and up to 5x in best cases). + (Contributed by Serhiy Storchaka in :issue:`20416` and :issue:`23344`.) + + +Build and C API Changes +======================= + +Changes to Python's build process and to the C API include: + +* New ``calloc`` functions: + + * :c:func:`PyMem_RawCalloc` + * :c:func:`PyMem_Calloc` + * :c:func:`PyObject_Calloc` + * :c:func:`_PyObject_GC_Calloc` + + +Deprecated +========== + +Unsupported Operating Systems +----------------------------- + +* Windows XP - Per :PEP:`11`, Microsoft support of Windows XP has ended. + + +Deprecated Python modules, functions and methods +------------------------------------------------ + +* The :mod:`formatter` module has now graduated to full deprecation and is still + slated for removal in Python 3.6. + +* :mod:`smtpd` has in the past always decoded the DATA portion of email + messages using the ``utf-8`` codec. This can now be controlled by the new + *decode_data* keyword to :class:`~smtpd.SMTPServer`. The default value is + ``True``, but this default is deprecated. Specify the *decode_data* keyword + with an appropriate value to avoid the deprecation warning. + +* Directly assigning values to the :attr:`~http.cookies.Morsel.key`, + :attr:`~http.cookies.Morsel.value` and + :attr:`~http.cookies.Morsel.coded_value` of :class:`~http.cookies.Morsel` + objects is deprecated. Use the :func:`~http.cookies.Morsel.set` method + instead. In addition, the undocumented *LegalChars* parameter of + :func:`~http.cookies.Morsel.set` is deprecated, and is now ignored. + +* Passing a format string as keyword argument *format_string* to the + :meth:`~string.Formatter.format` method of the :class:`string.Formatter` + class has been deprecated. + + +Deprecated functions and types of the C API +------------------------------------------- + +* None yet. + + +Deprecated features +------------------- + +* None yet. + + +Removed +======= + +API and Feature Removals +------------------------ + +The following obsolete and previously deprecated APIs and features have been +removed: + +* The ``__version__`` attribute has been dropped from the email package. The + email code hasn't been shipped separately from the stdlib for a long time, + and the ``__version__`` string was not updated in the last few releases. + +* The internal ``Netrc`` class in the :mod:`ftplib` module was deprecated in + 3.4, and has now been removed. + (Contributed by Matt Chaput in :issue:`6623`.) + +* The concept of ``.pyo`` files has been removed. + +* The JoinableQueue class in the provisional asyncio module was deprecated + in 3.4.4 and is now removed (:issue:`23464`). + + +Porting to Python 3.5 +===================== + +This section lists previously described changes and other bugfixes +that may require changes to your code. + +Changes in the Python API +------------------------- + +* :pep:`475`: Examples of functions which are now retried when interrupted + instead of raising :exc:`InterruptedError` if the signal handler does not + raise an exception: + + - :func:`open`, :func:`os.open`, :func:`io.open` + - functions of the :mod:`faulthandler` module + - :mod:`os` functions: + + * :func:`os.fchdir` + * :func:`os.fchmod` + * :func:`os.fchown` + * :func:`os.fdatasync` + * :func:`os.fstat` + * :func:`os.fstatvfs` + * :func:`os.fsync` + * :func:`os.ftruncate` + * :func:`os.mkfifo` + * :func:`os.mknod` + * :func:`os.posix_fadvise` + * :func:`os.posix_fallocate` + * :func:`os.pread` + * :func:`os.pwrite` + * :func:`os.read` + * :func:`os.readv` + * :func:`os.sendfile` + * :func:`os.wait3` + * :func:`os.wait4` + * :func:`os.wait` + * :func:`os.waitid` + * :func:`os.waitpid` + * :func:`os.write` + * :func:`os.writev` + * special cases: :func:`os.close` and :func:`os.dup2` now ignore + :py:data:`~errno.EINTR` error, the syscall is not retried (see the PEP + for the rationale) + + - :func:`select.select`, :func:`select.poll.poll`, :func:`select.epoll.poll`, + :func:`select.kqueue.control`, :func:`select.devpoll.poll` + - :func:`socket.socket` methods: + + * :meth:`~socket.socket.accept` + * :meth:`~socket.socket.connect` (except for non-blocking sockets) + * :meth:`~socket.socket.recv` + * :meth:`~socket.socket.recvfrom` + * :meth:`~socket.socket.recvmsg` + * :meth:`~socket.socket.send` + * :meth:`~socket.socket.sendall` + * :meth:`~socket.socket.sendmsg` + * :meth:`~socket.socket.sendto` + + - :func:`signal.sigtimedwait`, :func:`signal.sigwaitinfo` + - :func:`time.sleep` + +* Before Python 3.5, a :class:`datetime.time` object was considered to be false + if it represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. + +* :meth:`ssl.SSLSocket.send()` now raises either :exc:`ssl.SSLWantReadError` + or :exc:`ssl.SSLWantWriteError` on a non-blocking socket if the operation + would block. Previously, it would return 0. See :issue:`20951`. + +* The ``__name__`` attribute of generator is now set from the function name, + instead of being set from the code name. Use ``gen.gi_code.co_name`` to + retrieve the code name. Generators also have a new ``__qualname__`` + attribute, the qualified name, which is now used for the representation + of a generator (``repr(gen)``). See :issue:`21205`. + +* The deprecated "strict" mode and argument of :class:`~html.parser.HTMLParser`, + :meth:`HTMLParser.error`, and the :exc:`HTMLParserError` exception have been + removed. (Contributed by Ezio Melotti in :issue:`15114`.) + The *convert_charrefs* argument of :class:`~html.parser.HTMLParser` is + now ``True`` by default. (Contributed by Berker Peksag in :issue:`21047`.) + +* Although it is not formally part of the API, it is worth noting for porting + purposes (ie: fixing tests) that error messages that were previously of the + form "'sometype' does not support the buffer protocol" are now of the form "a + bytes-like object is required, not 'sometype'". (Contributed by Ezio Melotti + in :issue:`16518`.) + +* If the current directory is set to a directory that no longer exists then + :exc:`FileNotFoundError` will no longer be raised and instead + :meth:`~importlib.machinery.FileFinder.find_spec` will return ``None`` + **without** caching ``None`` in :data:`sys.path_importer_cache` which is + different than the typical case (:issue:`22834`). + +* HTTP status code and messages from :mod:`http.client` and :mod:`http.server` + were refactored into a common :class:`~http.HTTPStatus` enum. The values in + :mod:`http.client` and :mod:`http.server` remain available for backwards + compatibility. (Contributed by Demian Brecht in :issue:`21793`.) + +* When an import loader defines :meth:`~importlib.machinery.Loader.exec_module` + it is now expected to also define + :meth:`~importlib.machinery.Loader.create_module` (raises a + :exc:`DeprecationWarning` now, will be an error in Python 3.6). If the loader + inherits from :class:`importlib.abc.Loader` then there is nothing to do, else + simply define :meth:`~importlib.machinery.Loader.create_module` to return + ``None`` (:issue:`23014`). + +* :func:`re.split` always ignored empty pattern matches, so the ``'x*'`` + pattern worked the same as ``'x+'``, and the ``'\b'`` pattern never worked. + Now :func:`re.split` raises a warning if the pattern could match + an empty string. For compatibility use patterns that never match an empty + string (e.g. ``'x+'`` instead of ``'x*'``). Patterns that could only match + an empty string (such as ``'\b'``) now raise an error. + +* The :class:`~http.cookies.Morsel` dict-like interface has been made self + consistent: morsel comparison now takes the :attr:`~http.cookies.Morsel.key` + and :attr:`~http.cookies.Morsel.value` into account, + :meth:`~http.cookies.Morsel.copy` now results in a + :class:`~http.cookies.Morsel` instance rather than a :class:`dict`, and + :meth:`~http.cookies.Morsel.update` will now raise an exception if any of the + keys in the update dictionary are invalid. In addition, the undocumented + *LegalChars* parameter of :func:`~http.cookies.Morsel.set` is deprecated and + is now ignored. (:issue:`2211`) + +* :pep:`488` has removed ``.pyo`` files from Python and introduced the optional + ``opt-`` tag in ``.pyc`` file names. The + :func:`importlib.util.cache_from_source` has gained an *optimization* + parameter to help control the ``opt-`` tag. Because of this, the + *debug_override* parameter of the function is now deprecated. `.pyo` files + are also no longer supported as a file argument to the Python interpreter and + thus serve no purpose when distributed on their own (i.e. sourcless code + distribution). Due to the fact that the magic number for bytecode has changed + in Python 3.5, all old `.pyo` files from previous versions of Python are + invalid regardless of this PEP. + + * The :mod:`socket` module now exports the CAN_RAW_FD_FRAMES constant on linux + 3.6 and greater. + +Changes in the C API +-------------------- + +* The undocumented :c:member:`~PyMemoryViewObject.format` member of the + (non-public) :c:type:`PyMemoryViewObject` structure has been removed. + + All extensions relying on the relevant parts in ``memoryobject.h`` + must be rebuilt. + +* The :c:type:`PyMemAllocator` structure was renamed to + :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. + +* Removed non-documented macro :c:macro:`PyObject_REPR` which leaked references. + Use format character ``%R`` in :c:func:`PyUnicode_FromFormat`-like functions + to format the :func:`repr` of the object. + +* Because the lack of the :attr:`__module__` attribute breaks pickling and + introspection, a deprecation warning now is raised for builtin type without + the :attr:`__module__` attribute. Would be an AttributeError in future. + (:issue:`20204`) diff --git a/Doc/whatsnew/index.rst b/Doc/whatsnew/index.rst index 29902e407c..edb55022bd 100644 --- a/Doc/whatsnew/index.rst +++ b/Doc/whatsnew/index.rst @@ -11,6 +11,7 @@ anyone wishing to stay up-to-date after a new release. .. toctree:: :maxdepth: 2 + 3.5.rst 3.4.rst 3.3.rst 3.2.rst |
