diff options
Diffstat (limited to 'Doc')
387 files changed, 29861 insertions, 13037 deletions
diff --git a/Doc/ACKS.txt b/Doc/ACKS.txt index 73d4e13d55..70063c08fb 100644 --- a/Doc/ACKS.txt +++ b/Doc/ACKS.txt @@ -33,6 +33,7 @@ docs@python.org), and we'll be glad to correct the problem. * Keith Briggs * Ian Bruntlett * Lee Busby + * Arnaud Calmettes * Lorenzo M. Catucci * Carl Cerecke * Mauro Cicognini @@ -57,6 +58,7 @@ docs@python.org), and we'll be glad to correct the problem. * Carl Feynman * Dan Finnie * Hernán Martínez Foffani + * Michael Foord * Stefan Franke * Jim Fulton * Peter Funk @@ -79,6 +81,7 @@ docs@python.org), and we'll be glad to correct the problem. * Travis B. Hartwell * Tim Hatch * Janko Hauser + * Ben Hayden * Thomas Heller * Bernhard Herzog * Magnus L. Hetland @@ -105,6 +108,7 @@ docs@python.org), and we'll be glad to correct the problem. * Robert Kern * Jim Kerr * Jan Kim + * Kamil Kisiel * Greg Kochanski * Guido Kollerie * Peter A. Koren @@ -120,6 +124,7 @@ docs@python.org), and we'll be glad to correct the problem. * Robert Lehmann * Marc-André Lemburg * Ross Light + * Gediminas Liktaras * Ulf A. Lindgren * Everett Lipman * Mirko Liss @@ -141,7 +146,9 @@ docs@python.org), and we'll be glad to correct the problem. * Ross Moore * Sjoerd Mullender * Dale Nagata - * Michal Nowikowski + * Trent Nelson + * Michal Nowikowski + * Steffen Daode Nurpmeso * Ng Pheng Siong * Koray Oner * Tomas Oppelstrup @@ -182,6 +189,7 @@ docs@python.org), and we'll be glad to correct the problem. * Joakim Sernbrant * Justin Sheehy * Charlie Shepherd + * Yue Shuaijie * SilentGhost * Michael Simcich * Ionel Simionescu @@ -200,6 +208,7 @@ docs@python.org), and we'll be glad to correct the problem. * Kalle Svensson * Jim Tittsler * David Turner + * Sandro Tosi * Ville Vainio * Martijn Vries * Charles G. Waldman @@ -213,9 +222,11 @@ docs@python.org), and we'll be glad to correct the problem. * Mats Wichmann * Gerry Wiener * Timothy Wild + * Paul Winkler * Collin Winter * Blake Winton * Dan Wolfe + * Adam Woodbeck * Steven Work * Thomas Wouters * Ka-Ping Yee @@ -223,5 +234,3 @@ docs@python.org), and we'll be glad to correct the problem. * Moshe Zadka * Milan Zamazal * Cheng Zhang - * Trent Nelson - * Michael Foord diff --git a/Doc/Makefile b/Doc/Makefile index 8fcf538573..cb56ea9aa8 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -26,6 +26,7 @@ help: @echo " htmlhelp to make HTML files and a HTML help project" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " text to make plain text files" + @echo " epub to make EPUB files" @echo " changes to make an overview over all changed/added/deprecated items" @echo " linkcheck to check all external links for integrity" @echo " coverage to check documentation coverage for library and C API" @@ -34,12 +35,13 @@ help: @echo " dist to create a \"dist\" directory with archived docs for download" @echo " suspicious to check for suspicious markup in output text" @echo " check to run a check for frequent markup errors" + @echo " serve to serve the documentation on the localhost (8000)" # Note: if you update versions here, do the same in make.bat and README.txt checkout: @if [ ! -d tools/sphinx ]; then \ echo "Checking out Sphinx..."; \ - svn checkout $(SVNROOT)/external/Sphinx-0.6.5/sphinx tools/sphinx; \ + svn checkout $(SVNROOT)/external/Sphinx-1.0.7/sphinx tools/sphinx; \ fi @if [ ! -d tools/docutils ]; then \ echo "Checking out Docutils..."; \ @@ -80,6 +82,10 @@ text: BUILDER = text text: build @echo "Build finished; the text files are in build/text." +epub: BUILDER = epub +epub: build + @echo "Build finished; the epub files are in build/epub." + changes: BUILDER = changes changes: build @echo "The overview file is in build/changes." @@ -107,7 +113,7 @@ doctest: build pydoc-topics: BUILDER = pydoc-topics pydoc-topics: build @echo "Building finished; now copy build/pydoc-topics/topics.py" \ - "to Lib/pydoc_data/topics.py" + "to ../Lib/pydoc_data/topics.py" htmlview: html $(PYTHON) -c "import webbrowser; webbrowser.open('build/html/index.html')" @@ -157,9 +163,23 @@ dist: cp build/latex/docs-pdf.zip dist/python-$(DISTVERSION)-docs-pdf-letter.zip cp build/latex/docs-pdf.tar.bz2 dist/python-$(DISTVERSION)-docs-pdf-letter.tar.bz2 + # archive the epub build + rm -rf build/epub + make epub + mkdir -p dist/python-$(DISTVERSION)-docs-epub + cp -pPR build/epub/*.epub dist/python-$(DISTVERSION)-docs-epub/ + tar -C dist -cf dist/python-$(DISTVERSION)-docs-epub.tar python-$(DISTVERSION)-docs-epub + bzip2 -9 -k dist/python-$(DISTVERSION)-docs-epub.tar + (cd dist; zip -q -r -9 python-$(DISTVERSION)-docs-epub.zip python-$(DISTVERSION)-docs-epub) + rm -r dist/python-$(DISTVERSION)-docs-epub + rm dist/python-$(DISTVERSION)-docs-epub.tar + check: $(PYTHON) tools/rstlint.py -i tools +serve: + ../Tools/scripts/serve.py build/html + # Targets for daily automated doc build # for development releases: always build diff --git a/Doc/README.txt b/Doc/README.txt index 7a561c0d84..f296423fc9 100644 --- a/Doc/README.txt +++ b/Doc/README.txt @@ -54,6 +54,9 @@ Available make targets are: * "text", which builds a plain text file for each source file. + * "epub", which builds an EPUB document, suitable to be viewed on e-book + readers. + * "linkcheck", which checks all external references to see whether they are broken, redirected or malformed, and outputs this information to stdout as well as a plain-text (.txt) file. @@ -78,7 +81,7 @@ Without make You'll need to install the Sphinx package, either by checking it out via :: - svn co http://svn.python.org/projects/external/Sphinx-0.6.5/sphinx tools/sphinx + svn co http://svn.python.org/projects/external/Sphinx-1.0.7/sphinx tools/sphinx or by installing it from PyPI. @@ -132,7 +135,7 @@ The Python source is copyrighted, but you can freely use and copy it as long as you don't change or remove the copyright notice: ---------------------------------------------------------------------- -Copyright (c) 2000-2008 Python Software Foundation. +Copyright (c) 2000-2012 Python Software Foundation. All rights reserved. Copyright (c) 2000 BeOpen.com. diff --git a/Doc/bugs.rst b/Doc/bugs.rst index a9a48c7858..3785ccb72b 100644 --- a/Doc/bugs.rst +++ b/Doc/bugs.rst @@ -57,12 +57,14 @@ were using (including version information as appropriate). Each bug report will be assigned to a developer who will determine what needs to be done to correct the problem. You will receive an update each time action is -taken on the bug. See http://www.python.org/dev/workflow/ for a detailed -description of the issue workflow. +taken on the bug. .. seealso:: + `Python Developer's Guide <http://docs.python.org/devguide/>`_ + Detailed description of the issue workflow and developers tools. + `How to Report Bugs Effectively <http://www.chiark.greenend.org.uk/~sgtatham/bugs.html>`_ Article which goes into some detail about how to create a useful bug report. This describes what kind of information is useful and why it is useful. diff --git a/Doc/c-api/abstract.rst b/Doc/c-api/abstract.rst index aba804d5c3..ad53881112 100644 --- a/Doc/c-api/abstract.rst +++ b/Doc/c-api/abstract.rst @@ -12,7 +12,7 @@ sequence types). When used on object types for which they do not apply, they will raise a Python exception. It is not possible to use these functions on objects that are not properly -initialized, such as a list object that has been created by :cfunc:`PyList_New`, +initialized, such as a list object that has been created by :c:func:`PyList_New`, but whose items have not been set to some non-\ ``NULL`` value yet. .. toctree:: diff --git a/Doc/c-api/allocation.rst b/Doc/c-api/allocation.rst index b64381bf39..e8f60bfcf4 100644 --- a/Doc/c-api/allocation.rst +++ b/Doc/c-api/allocation.rst @@ -6,13 +6,13 @@ Allocating Objects on the Heap ============================== -.. cfunction:: PyObject* _PyObject_New(PyTypeObject *type) +.. c:function:: PyObject* _PyObject_New(PyTypeObject *type) -.. cfunction:: PyVarObject* _PyObject_NewVar(PyTypeObject *type, Py_ssize_t size) +.. c:function:: PyVarObject* _PyObject_NewVar(PyTypeObject *type, Py_ssize_t size) -.. cfunction:: PyObject* PyObject_Init(PyObject *op, PyTypeObject *type) +.. c:function:: PyObject* PyObject_Init(PyObject *op, PyTypeObject *type) Initialize a newly-allocated object *op* with its type and initial reference. Returns the initialized object. If *type* indicates that the @@ -21,13 +21,13 @@ Allocating Objects on the Heap affected. -.. cfunction:: PyVarObject* PyObject_InitVar(PyVarObject *op, PyTypeObject *type, Py_ssize_t size) +.. c:function:: PyVarObject* PyObject_InitVar(PyVarObject *op, PyTypeObject *type, Py_ssize_t size) - This does everything :cfunc:`PyObject_Init` does, and also initializes the + This does everything :c:func:`PyObject_Init` does, and also initializes the length information for a variable-size object. -.. cfunction:: TYPE* PyObject_New(TYPE, PyTypeObject *type) +.. c:function:: TYPE* PyObject_New(TYPE, PyTypeObject *type) Allocate a new Python object using the C structure type *TYPE* and the Python type object *type*. Fields not defined by the Python object header @@ -36,7 +36,7 @@ Allocating Objects on the Heap the type object. -.. cfunction:: TYPE* PyObject_NewVar(TYPE, PyTypeObject *type, Py_ssize_t size) +.. c:function:: TYPE* PyObject_NewVar(TYPE, PyTypeObject *type, Py_ssize_t size) Allocate a new Python object using the C structure type *TYPE* and the Python type object *type*. Fields not defined by the Python object header @@ -48,24 +48,24 @@ Allocating Objects on the Heap improving the memory management efficiency. -.. cfunction:: void PyObject_Del(PyObject *op) +.. c:function:: void PyObject_Del(PyObject *op) - Releases memory allocated to an object using :cfunc:`PyObject_New` or - :cfunc:`PyObject_NewVar`. This is normally called from the + Releases memory allocated to an object using :c:func:`PyObject_New` or + :c:func:`PyObject_NewVar`. This is normally called from the :attr:`tp_dealloc` handler specified in the object's type. The fields of the object should not be accessed after this call as the memory is no longer a valid Python object. -.. cvar:: PyObject _Py_NoneStruct +.. c:var:: PyObject _Py_NoneStruct Object which is visible in Python as ``None``. This should only be accessed - using the :cmacro:`Py_None` macro, which evaluates to a pointer to this + using the :c:macro:`Py_None` macro, which evaluates to a pointer to this object. .. seealso:: - :cfunc:`PyModule_Create` + :c:func:`PyModule_Create` To allocate and create extension modules. diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index 21cebe9086..d4dda7c693 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -9,8 +9,8 @@ These functions are useful when creating your own extensions functions and methods. Additional information and examples are available in :ref:`extending-index`. -The first three of these functions described, :cfunc:`PyArg_ParseTuple`, -:cfunc:`PyArg_ParseTupleAndKeywords`, and :cfunc:`PyArg_Parse`, all use *format +The first three of these functions described, :c:func:`PyArg_ParseTuple`, +:c:func:`PyArg_ParseTupleAndKeywords`, and :c:func:`PyArg_Parse`, all use *format strings* which are used to tell the function about the expected arguments. The format strings use the same syntax for each of these functions. @@ -35,23 +35,23 @@ You don't have to provide raw storage for the returned unicode or bytes area. Also, you won't have to release any memory yourself, except with the ``es``, ``es#``, ``et`` and ``et#`` formats. -However, when a :ctype:`Py_buffer` structure gets filled, the underlying +However, when a :c:type:`Py_buffer` structure gets filled, the underlying buffer is locked so that the caller can subsequently use the buffer even -inside a :ctype:`Py_BEGIN_ALLOW_THREADS` block without the risk of mutable data +inside a :c:type:`Py_BEGIN_ALLOW_THREADS` block without the risk of mutable data being resized or destroyed. As a result, **you have to call** -:cfunc:`PyBuffer_Release` after you have finished processing the data (or +:c:func:`PyBuffer_Release` after you have finished processing the data (or in any early abort case). Unless otherwise stated, buffers are not NUL-terminated. .. note:: For all ``#`` variants of formats (``s#``, ``y#``, etc.), the type of - the length argument (int or :ctype:`Py_ssize_t`) is controlled by - defining the macro :cmacro:`PY_SSIZE_T_CLEAN` before including + the length argument (int or :c:type:`Py_ssize_t`) is controlled by + defining the macro :c:macro:`PY_SSIZE_T_CLEAN` before including :file:`Python.h`. If the macro was defined, length is a - :ctype:`Py_ssize_t` rather than an :ctype:`int`. This behavior will change - in a future Python version to only support :ctype:`Py_ssize_t` and - drop :ctype:`int` support. It is best to always define :cmacro:`PY_SSIZE_T_CLEAN`. + :c:type:`Py_ssize_t` rather than an :c:type:`int`. This behavior will change + in a future Python version to only support :c:type:`Py_ssize_t` and + drop :c:type:`int` support. It is best to always define :c:macro:`PY_SSIZE_T_CLEAN`. ``s`` (:class:`str`) [const char \*] @@ -66,17 +66,17 @@ Unless otherwise stated, buffers are not NUL-terminated. .. note:: This format does not accept bytes-like objects. If you want to accept filesystem paths and convert them to C character strings, it is - preferable to use the ``O&`` format with :cfunc:`PyUnicode_FSConverter` + preferable to use the ``O&`` format with :c:func:`PyUnicode_FSConverter` as *converter*. ``s*`` (:class:`str`, :class:`bytes`, :class:`bytearray` or buffer compatible object) [Py_buffer] This format accepts Unicode objects as well as objects supporting the buffer protocol. - It fills a :ctype:`Py_buffer` structure provided by the caller. + It fills a :c:type:`Py_buffer` structure provided by the caller. In this case the resulting C string may contain embedded NUL bytes. Unicode objects are converted to C strings using ``'utf-8'`` encoding. -``s#`` (:class:`str`, :class:`bytes` or read-only buffer compatible object) [const char \*, int or :ctype:`Py_ssize_t`] +``s#`` (:class:`str`, :class:`bytes` or read-only buffer compatible object) [const char \*, int or :c:type:`Py_ssize_t`] Like ``s*``, except that it doesn't accept mutable buffer-like objects such as :class:`bytearray`. The result is stored into two C variables, the first one a pointer to a C string, the second one its length. @@ -89,7 +89,7 @@ Unless otherwise stated, buffers are not NUL-terminated. ``z*`` (:class:`str`, :class:`bytes`, :class:`bytearray`, buffer compatible object or ``None``) [Py_buffer] Like ``s*``, but the Python object may also be ``None``, in which case the - ``buf`` member of the :ctype:`Py_buffer` structure is set to *NULL*. + ``buf`` member of the :c:type:`Py_buffer` structure is set to *NULL*. ``z#`` (:class:`str`, :class:`bytes`, read-only buffer compatible object or ``None``) [const char \*, int] Like ``s#``, but the Python object may also be ``None``, in which case the C @@ -113,19 +113,21 @@ Unless otherwise stated, buffers are not NUL-terminated. ``S`` (:class:`bytes`) [PyBytesObject \*] Requires that the Python object is a :class:`bytes` object, without attempting any conversion. Raises :exc:`TypeError` if the object is not - a bytes object. The C variable may also be declared as :ctype:`PyObject\*`. + a bytes object. The C variable may also be declared as :c:type:`PyObject\*`. ``Y`` (:class:`bytearray`) [PyByteArrayObject \*] Requires that the Python object is a :class:`bytearray` object, without attempting any conversion. Raises :exc:`TypeError` if the object is not - a :class:`bytearray` object. The C variable may also be declared as :ctype:`PyObject\*`. + a :class:`bytearray` object. The C variable may also be declared as :c:type:`PyObject\*`. ``u`` (:class:`str`) [Py_UNICODE \*] Convert a Python Unicode object to a C pointer to a NUL-terminated buffer of - Unicode characters. You must pass the address of a :ctype:`Py_UNICODE` + Unicode characters. You must pass the address of a :c:type:`Py_UNICODE` pointer variable, which will be filled with the pointer to an existing - Unicode buffer. Please note that the width of a :ctype:`Py_UNICODE` + Unicode buffer. Please note that the width of a :c:type:`Py_UNICODE` character depends on compilation options (it is either 16 or 32 bits). + The Python string must not contain embedded NUL characters; if it does, + a :exc:`TypeError` exception is raised. .. note:: Since ``u`` doesn't give you back the length of the string, and it @@ -138,55 +140,38 @@ Unless otherwise stated, buffers are not NUL-terminated. ``Z`` (:class:`str` or ``None``) [Py_UNICODE \*] Like ``u``, but the Python object may also be ``None``, in which case the - :ctype:`Py_UNICODE` pointer is set to *NULL*. + :c:type:`Py_UNICODE` pointer is set to *NULL*. ``Z#`` (:class:`str` or ``None``) [Py_UNICODE \*, int] Like ``u#``, but the Python object may also be ``None``, in which case the - :ctype:`Py_UNICODE` pointer is set to *NULL*. + :c:type:`Py_UNICODE` pointer is set to *NULL*. ``U`` (:class:`str`) [PyUnicodeObject \*] Requires that the Python object is a Unicode object, without attempting any conversion. Raises :exc:`TypeError` if the object is not a Unicode - object. The C variable may also be declared as :ctype:`PyObject\*`. - -``t#`` (:class:`bytes`, :class:`bytearray` or read-only character buffer) [char \*, int] - Like ``s#``, but accepts any object which implements the read-only buffer - interface. The :ctype:`char\*` variable is set to point to the first byte of - the buffer, and the :ctype:`int` is set to the length of the buffer. Only - single-segment buffer objects are accepted; :exc:`TypeError` is raised for all - others. - -``w`` (:class:`bytearray` or read-write character buffer) [char \*] - Similar to ``y``, but accepts any object which implements the read-write buffer - interface. The caller must determine the length of the buffer by other means, - or use ``w#`` instead. Only single-segment buffer objects are accepted; - :exc:`TypeError` is raised for all others. + object. The C variable may also be declared as :c:type:`PyObject\*`. ``w*`` (:class:`bytearray` or read-write byte-oriented buffer) [Py_buffer] - This is to ``w`` what ``y*`` is to ``y``. - -``w#`` (:class:`bytearray` or read-write character buffer) [char \*, int] - Like ``y#``, but accepts any object which implements the read-write buffer - interface. The :ctype:`char \*` variable is set to point to the first byte - of the buffer, and the :ctype:`int` is set to the length of the buffer. - Only single-segment buffer objects are accepted; :exc:`TypeError` is raised - for all others. + This format accepts any object which implements the read-write buffer + interface. It fills a :c:type:`Py_buffer` structure provided by the caller. + The buffer may contain embedded null bytes. The caller have to call + :c:func:`PyBuffer_Release` when it is done with the buffer. ``es`` (:class:`str`) [const char \*encoding, char \*\*buffer] This variant on ``s`` is used for encoding Unicode into a character buffer. It only works for encoded data without embedded NUL bytes. This format requires two arguments. The first is only used as input, and - must be a :ctype:`const char\*` which points to the name of an encoding as a + must be a :c:type:`const char\*` which points to the name of an encoding as a NUL-terminated string, or *NULL*, in which case ``'utf-8'`` encoding is used. An exception is raised if the named encoding is not known to Python. The - second argument must be a :ctype:`char\*\*`; the value of the pointer it + second argument must be a :c:type:`char\*\*`; the value of the pointer it references will be set to a buffer with the contents of the argument text. The text will be encoded in the encoding specified by the first argument. - :cfunc:`PyArg_ParseTuple` will allocate a buffer of the needed size, copy the + :c:func:`PyArg_ParseTuple` will allocate a buffer of the needed size, copy the encoded data into this buffer and adjust *\*buffer* to reference the newly - allocated storage. The caller is responsible for calling :cfunc:`PyMem_Free` to + allocated storage. The caller is responsible for calling :c:func:`PyMem_Free` to free the allocated buffer after use. ``et`` (:class:`str`, :class:`bytes` or :class:`bytearray`) [const char \*encoding, char \*\*buffer] @@ -200,10 +185,10 @@ Unless otherwise stated, buffers are not NUL-terminated. characters. It requires three arguments. The first is only used as input, and must be a - :ctype:`const char\*` which points to the name of an encoding as a + :c:type:`const char\*` which points to the name of an encoding as a NUL-terminated string, or *NULL*, in which case ``'utf-8'`` encoding is used. An exception is raised if the named encoding is not known to Python. The - second argument must be a :ctype:`char\*\*`; the value of the pointer it + second argument must be a :c:type:`char\*\*`; the value of the pointer it references will be set to a buffer with the contents of the argument text. The text will be encoded in the encoding specified by the first argument. The third argument must be a pointer to an integer; the referenced integer @@ -214,10 +199,10 @@ Unless otherwise stated, buffers are not NUL-terminated. If *\*buffer* points a *NULL* pointer, the function will allocate a buffer of the needed size, copy the encoded data into this buffer and set *\*buffer* to reference the newly allocated storage. The caller is responsible for calling - :cfunc:`PyMem_Free` to free the allocated buffer after usage. + :c:func:`PyMem_Free` to free the allocated buffer after usage. If *\*buffer* points to a non-*NULL* pointer (an already allocated buffer), - :cfunc:`PyArg_ParseTuple` will use this location as the buffer and interpret the + :c:func:`PyArg_ParseTuple` will use this location as the buffer and interpret the initial value of *\*buffer_length* as the buffer size. It will then copy the encoded data into the buffer and NUL-terminate it. If the buffer is not large enough, a :exc:`ValueError` will be set. @@ -235,62 +220,62 @@ Numbers ``b`` (:class:`int`) [unsigned char] Convert a nonnegative Python integer to an unsigned tiny int, stored in a C - :ctype:`unsigned char`. + :c:type:`unsigned char`. ``B`` (:class:`int`) [unsigned char] Convert a Python integer to a tiny int without overflow checking, stored in a C - :ctype:`unsigned char`. + :c:type:`unsigned char`. ``h`` (:class:`int`) [short int] - Convert a Python integer to a C :ctype:`short int`. + Convert a Python integer to a C :c:type:`short int`. ``H`` (:class:`int`) [unsigned short int] - Convert a Python integer to a C :ctype:`unsigned short int`, without overflow + Convert a Python integer to a C :c:type:`unsigned short int`, without overflow checking. ``i`` (:class:`int`) [int] - Convert a Python integer to a plain C :ctype:`int`. + Convert a Python integer to a plain C :c:type:`int`. ``I`` (:class:`int`) [unsigned int] - Convert a Python integer to a C :ctype:`unsigned int`, without overflow + Convert a Python integer to a C :c:type:`unsigned int`, without overflow checking. ``l`` (:class:`int`) [long int] - Convert a Python integer to a C :ctype:`long int`. + Convert a Python integer to a C :c:type:`long int`. ``k`` (:class:`int`) [unsigned long] - Convert a Python integer to a C :ctype:`unsigned long` without + Convert a Python integer to a C :c:type:`unsigned long` without overflow checking. ``L`` (:class:`int`) [PY_LONG_LONG] - Convert a Python integer to a C :ctype:`long long`. This format is only - available on platforms that support :ctype:`long long` (or :ctype:`_int64` on + Convert a Python integer to a C :c:type:`long long`. This format is only + available on platforms that support :c:type:`long long` (or :c:type:`_int64` on Windows). ``K`` (:class:`int`) [unsigned PY_LONG_LONG] - Convert a Python integer to a C :ctype:`unsigned long long` + Convert a Python integer to a C :c:type:`unsigned long long` without overflow checking. This format is only available on platforms that - support :ctype:`unsigned long long` (or :ctype:`unsigned _int64` on Windows). + support :c:type:`unsigned long long` (or :c:type:`unsigned _int64` on Windows). ``n`` (:class:`int`) [Py_ssize_t] - Convert a Python integer to a C :ctype:`Py_ssize_t`. + Convert a Python integer to a C :c:type:`Py_ssize_t`. ``c`` (:class:`bytes` of length 1) [char] Convert a Python byte, represented as a :class:`bytes` object of length 1, - to a C :ctype:`char`. + to a C :c:type:`char`. ``C`` (:class:`str` of length 1) [int] Convert a Python character, represented as a :class:`str` object of - length 1, to a C :ctype:`int`. + length 1, to a C :c:type:`int`. ``f`` (:class:`float`) [float] - Convert a Python floating point number to a C :ctype:`float`. + Convert a Python floating point number to a C :c:type:`float`. ``d`` (:class:`float`) [double] - Convert a Python floating point number to a C :ctype:`double`. + Convert a Python floating point number to a C :c:type:`double`. ``D`` (:class:`complex`) [Py_complex] - Convert a Python complex number to a C :ctype:`Py_complex` structure. + Convert a Python complex number to a C :c:type:`Py_complex` structure. Other objects ------------- @@ -303,20 +288,20 @@ Other objects ``O!`` (object) [*typeobject*, PyObject \*] Store a Python object in a C object pointer. This is similar to ``O``, but takes two C arguments: the first is the address of a Python type object, the - second is the address of the C variable (of type :ctype:`PyObject\*`) into which + second is the address of the C variable (of type :c:type:`PyObject\*`) into which the object pointer is stored. If the Python object does not have the required type, :exc:`TypeError` is raised. ``O&`` (object) [*converter*, *anything*] Convert a Python object to a C variable through a *converter* function. This takes two arguments: the first is a function, the second is the address of a C - variable (of arbitrary type), converted to :ctype:`void \*`. The *converter* + variable (of arbitrary type), converted to :c:type:`void \*`. The *converter* function in turn is called as follows:: status = converter(object, address); where *object* is the Python object to be converted and *address* is the - :ctype:`void\*` argument that was passed to the :cfunc:`PyArg_Parse\*` function. + :c:type:`void\*` argument that was passed to the :c:func:`PyArg_Parse\*` function. The returned *status* should be ``1`` for a successful conversion and ``0`` if the conversion has failed. When the conversion fails, the *converter* function should raise an exception and leave the content of *address* unmodified. @@ -348,13 +333,13 @@ inside nested parentheses. They are: Indicates that the remaining arguments in the Python argument list are optional. The C variables corresponding to optional arguments should be initialized to their default value --- when an optional argument is not specified, - :cfunc:`PyArg_ParseTuple` does not touch the contents of the corresponding C + :c:func:`PyArg_ParseTuple` does not touch the contents of the corresponding C variable(s). ``:`` The list of format units ends here; the string after the colon is used as the function name in error messages (the "associated value" of the exception that - :cfunc:`PyArg_ParseTuple` raises). + :c:func:`PyArg_ParseTuple` raises). ``;`` The list of format units ends here; the string after the semicolon is used as @@ -372,43 +357,52 @@ what is specified for the corresponding format unit in that case. For the conversion to succeed, the *arg* object must match the format and the format must be exhausted. On success, the -:cfunc:`PyArg_Parse\*` functions return true, otherwise they return +:c:func:`PyArg_Parse\*` functions return true, otherwise they return false and raise an appropriate exception. When the -:cfunc:`PyArg_Parse\*` functions fail due to conversion failure in one +:c:func:`PyArg_Parse\*` functions fail due to conversion failure in one of the format units, the variables at the addresses corresponding to that and the following format units are left untouched. API Functions ------------- -.. cfunction:: int PyArg_ParseTuple(PyObject *args, const char *format, ...) +.. c:function:: int PyArg_ParseTuple(PyObject *args, const char *format, ...) Parse the parameters of a function that takes only positional parameters into local variables. Returns true on success; on failure, it returns false and raises the appropriate exception. -.. cfunction:: int PyArg_VaParse(PyObject *args, const char *format, va_list vargs) +.. c:function:: int PyArg_VaParse(PyObject *args, const char *format, va_list vargs) - Identical to :cfunc:`PyArg_ParseTuple`, except that it accepts a va_list rather + Identical to :c:func:`PyArg_ParseTuple`, except that it accepts a va_list rather than a variable number of arguments. -.. cfunction:: int PyArg_ParseTupleAndKeywords(PyObject *args, PyObject *kw, const char *format, char *keywords[], ...) +.. c:function:: int PyArg_ParseTupleAndKeywords(PyObject *args, PyObject *kw, const char *format, char *keywords[], ...) Parse the parameters of a function that takes both positional and keyword parameters into local variables. Returns true on success; on failure, it returns false and raises the appropriate exception. -.. cfunction:: int PyArg_VaParseTupleAndKeywords(PyObject *args, PyObject *kw, const char *format, char *keywords[], va_list vargs) +.. c:function:: int PyArg_VaParseTupleAndKeywords(PyObject *args, PyObject *kw, const char *format, char *keywords[], va_list vargs) - Identical to :cfunc:`PyArg_ParseTupleAndKeywords`, except that it accepts a + Identical to :c:func:`PyArg_ParseTupleAndKeywords`, except that it accepts a va_list rather than a variable number of arguments. +.. c:function:: int PyArg_ValidateKeywordArguments(PyObject *) + + Ensure that the keys in the keywords argument dictionary are strings. This + is only needed if :c:func:`PyArg_ParseTupleAndKeywords` is not used, since the + latter already does this check. + + .. versionadded:: 3.2 + + .. XXX deprecated, will be removed -.. cfunction:: int PyArg_Parse(PyObject *args, const char *format, ...) +.. c:function:: int PyArg_Parse(PyObject *args, const char *format, ...) Function used to deconstruct the argument lists of "old-style" functions --- these are functions which use the :const:`METH_OLDARGS` parameter parsing @@ -418,7 +412,7 @@ API Functions however, and may continue to be used for that purpose. -.. cfunction:: int PyArg_UnpackTuple(PyObject *args, const char *name, Py_ssize_t min, Py_ssize_t max, ...) +.. c:function:: int PyArg_UnpackTuple(PyObject *args, const char *name, Py_ssize_t min, Py_ssize_t max, ...) A simpler form of parameter retrieval which does not use a format string to specify the types of the arguments. Functions which use this method to retrieve @@ -427,7 +421,7 @@ API Functions *args*; it must actually be a tuple. The length of the tuple must be at least *min* and no more than *max*; *min* and *max* may be equal. Additional arguments must be passed to the function, each of which should be a pointer to a - :ctype:`PyObject\*` variable; these will be filled in with the values from + :c:type:`PyObject\*` variable; these will be filled in with the values from *args*; they will contain borrowed references. The variables which correspond to optional parameters not given by *args* will not be filled in; these should be initialized by the caller. This function returns true on success and false if @@ -450,8 +444,8 @@ API Functions return result; } - The call to :cfunc:`PyArg_UnpackTuple` in this example is entirely equivalent to - this call to :cfunc:`PyArg_ParseTuple`:: + The call to :c:func:`PyArg_UnpackTuple` in this example is entirely equivalent to + this call to :c:func:`PyArg_ParseTuple`:: PyArg_ParseTuple(args, "O|O:ref", &object, &callback) @@ -460,14 +454,14 @@ API Functions Building values --------------- -.. cfunction:: PyObject* Py_BuildValue(const char *format, ...) +.. c:function:: PyObject* Py_BuildValue(const char *format, ...) Create a new value based on a format string similar to those accepted by the - :cfunc:`PyArg_Parse\*` family of functions and a sequence of values. Returns + :c:func:`PyArg_Parse\*` family of functions and a sequence of values. Returns the value or *NULL* in the case of an error; an exception will be raised if *NULL* is returned. - :cfunc:`Py_BuildValue` does not always build a tuple. It builds a tuple only if + :c:func:`Py_BuildValue` does not always build a tuple. It builds a tuple only if its format string contains two or more format units. If the format string is empty, it returns ``None``; if it contains exactly one format unit, it returns whatever object is described by that format unit. To force it to return a tuple @@ -476,10 +470,10 @@ Building values When memory buffers are passed as parameters to supply data to build objects, as for the ``s`` and ``s#`` formats, the required data is copied. Buffers provided by the caller are never referenced by the objects created by - :cfunc:`Py_BuildValue`. In other words, if your code invokes :cfunc:`malloc` - and passes the allocated memory to :cfunc:`Py_BuildValue`, your code is - responsible for calling :cfunc:`free` for that memory once - :cfunc:`Py_BuildValue` returns. + :c:func:`Py_BuildValue`. In other words, if your code invokes :c:func:`malloc` + and passes the allocated memory to :c:func:`Py_BuildValue`, your code is + responsible for calling :c:func:`free` for that memory once + :c:func:`Py_BuildValue` returns. In the following description, the quoted form is the format unit; the entry in (round) parentheses is the Python object type that the format unit will return; @@ -522,72 +516,70 @@ Building values and ``None`` is returned. ``U`` (:class:`str` or ``None``) [char \*] - Convert a null-terminated C string to a Python unicode object. If the C string - pointer is *NULL*, ``None`` is used. + Same as ``s``. ``U#`` (:class:`str` or ``None``) [char \*, int] - Convert a C string and its length to a Python unicode object. If the C string - pointer is *NULL*, the length is ignored and ``None`` is returned. + Same as ``s#``. ``i`` (:class:`int`) [int] - Convert a plain C :ctype:`int` to a Python integer object. + Convert a plain C :c:type:`int` to a Python integer object. ``b`` (:class:`int`) [char] - Convert a plain C :ctype:`char` to a Python integer object. + Convert a plain C :c:type:`char` to a Python integer object. ``h`` (:class:`int`) [short int] - Convert a plain C :ctype:`short int` to a Python integer object. + Convert a plain C :c:type:`short int` to a Python integer object. ``l`` (:class:`int`) [long int] - Convert a C :ctype:`long int` to a Python integer object. + Convert a C :c:type:`long int` to a Python integer object. ``B`` (:class:`int`) [unsigned char] - Convert a C :ctype:`unsigned char` to a Python integer object. + Convert a C :c:type:`unsigned char` to a Python integer object. ``H`` (:class:`int`) [unsigned short int] - Convert a C :ctype:`unsigned short int` to a Python integer object. + Convert a C :c:type:`unsigned short int` to a Python integer object. ``I`` (:class:`int`) [unsigned int] - Convert a C :ctype:`unsigned int` to a Python integer object. + Convert a C :c:type:`unsigned int` to a Python integer object. ``k`` (:class:`int`) [unsigned long] - Convert a C :ctype:`unsigned long` to a Python integer object. + Convert a C :c:type:`unsigned long` to a Python integer object. ``L`` (:class:`int`) [PY_LONG_LONG] - Convert a C :ctype:`long long` to a Python integer object. Only available - on platforms that support :ctype:`long long` (or :ctype:`_int64` on + Convert a C :c:type:`long long` to a Python integer object. Only available + on platforms that support :c:type:`long long` (or :c:type:`_int64` on Windows). ``K`` (:class:`int`) [unsigned PY_LONG_LONG] - Convert a C :ctype:`unsigned long long` to a Python integer object. Only - available on platforms that support :ctype:`unsigned long long` (or - :ctype:`unsigned _int64` on Windows). + Convert a C :c:type:`unsigned long long` to a Python integer object. Only + available on platforms that support :c:type:`unsigned long long` (or + :c:type:`unsigned _int64` on Windows). ``n`` (:class:`int`) [Py_ssize_t] - Convert a C :ctype:`Py_ssize_t` to a Python integer. + Convert a C :c:type:`Py_ssize_t` to a Python integer. ``c`` (:class:`bytes` of length 1) [char] - Convert a C :ctype:`int` representing a byte to a Python :class:`bytes` object of + Convert a C :c:type:`int` representing a byte to a Python :class:`bytes` object of length 1. ``C`` (:class:`str` of length 1) [int] - Convert a C :ctype:`int` representing a character to Python :class:`str` + Convert a C :c:type:`int` representing a character to Python :class:`str` object of length 1. ``d`` (:class:`float`) [double] - Convert a C :ctype:`double` to a Python floating point number. + Convert a C :c:type:`double` to a Python floating point number. ``f`` (:class:`float`) [float] - Convert a C :ctype:`float` to a Python floating point number. + Convert a C :c:type:`float` to a Python floating point number. ``D`` (:class:`complex`) [Py_complex \*] - Convert a C :ctype:`Py_complex` structure to a Python complex number. + Convert a C :c:type:`Py_complex` structure to a Python complex number. ``O`` (object) [PyObject \*] Pass a Python object untouched (except for its reference count, which is incremented by one). If the object passed in is a *NULL* pointer, it is assumed that this was caused because the call producing the argument found an error and - set an exception. Therefore, :cfunc:`Py_BuildValue` will return *NULL* but won't + set an exception. Therefore, :c:func:`Py_BuildValue` will return *NULL* but won't raise an exception. If no exception has been raised yet, :exc:`SystemError` is set. @@ -601,7 +593,7 @@ Building values ``O&`` (object) [*converter*, *anything*] Convert *anything* to a Python object through a *converter* function. The - function is called with *anything* (which should be compatible with :ctype:`void + function is called with *anything* (which should be compatible with :c:type:`void \*`) as its argument and should return a "new" Python object, or *NULL* if an error occurred. @@ -619,7 +611,7 @@ Building values If there is an error in the format string, the :exc:`SystemError` exception is set and *NULL* returned. -.. cfunction:: PyObject* Py_VaBuildValue(const char *format, va_list vargs) +.. c:function:: PyObject* Py_VaBuildValue(const char *format, va_list vargs) - Identical to :cfunc:`Py_BuildValue`, except that it accepts a va_list + Identical to :c:func:`Py_BuildValue`, except that it accepts a va_list rather than a variable number of arguments. diff --git a/Doc/c-api/bool.rst b/Doc/c-api/bool.rst index 4479bc69df..a9fb342f7c 100644 --- a/Doc/c-api/bool.rst +++ b/Doc/c-api/bool.rst @@ -11,36 +11,36 @@ creation and deletion functions don't apply to booleans. The following macros are available, however. -.. cfunction:: int PyBool_Check(PyObject *o) +.. c:function:: int PyBool_Check(PyObject *o) - Return true if *o* is of type :cdata:`PyBool_Type`. + Return true if *o* is of type :c:data:`PyBool_Type`. -.. cvar:: PyObject* Py_False +.. c:var:: PyObject* Py_False The Python ``False`` object. This object has no methods. It needs to be treated just like any other object with respect to reference counts. -.. cvar:: PyObject* Py_True +.. c:var:: PyObject* Py_True The Python ``True`` object. This object has no methods. It needs to be treated just like any other object with respect to reference counts. -.. cmacro:: Py_RETURN_FALSE +.. c:macro:: Py_RETURN_FALSE Return :const:`Py_False` from a function, properly incrementing its reference count. -.. cmacro:: Py_RETURN_TRUE +.. c:macro:: Py_RETURN_TRUE Return :const:`Py_True` from a function, properly incrementing its reference count. -.. cfunction:: PyObject* PyBool_FromLong(long v) +.. c:function:: PyObject* PyBool_FromLong(long v) Return a new reference to :const:`Py_True` or :const:`Py_False` depending on the truth value of *v*. diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst index 37866b7dbd..d98ece3e98 100644 --- a/Doc/c-api/buffer.rst +++ b/Doc/c-api/buffer.rst @@ -50,12 +50,12 @@ selectively allow or reject exporting of read-write and read-only buffers. There are two ways for a consumer of the buffer interface to acquire a buffer over a target object: -* call :cfunc:`PyObject_GetBuffer` with the right parameters; +* call :c:func:`PyObject_GetBuffer` with the right parameters; -* call :cfunc:`PyArg_ParseTuple` (or one of its siblings) with one of the +* call :c:func:`PyArg_ParseTuple` (or one of its siblings) with one of the ``y*``, ``w*`` or ``s*`` :ref:`format codes <arg-parsing>`. -In both cases, :cfunc:`PyBuffer_Release` must be called when the buffer +In both cases, :c:func:`PyBuffer_Release` must be called when the buffer isn't needed anymore. Failure to do so could lead to various issues such as resource leaks. @@ -73,55 +73,55 @@ operating system library, or it could be used to pass around structured data in its native, in-memory format. Contrary to most data types exposed by the Python interpreter, buffers -are not :ctype:`PyObject` pointers but rather simple C structures. This +are not :c:type:`PyObject` pointers but rather simple C structures. This allows them to be created and copied very simply. When a generic wrapper around a buffer is needed, a :ref:`memoryview <memoryview-objects>` object can be created. -.. ctype:: Py_buffer +.. c:type:: Py_buffer - .. cmember:: void *buf + .. c:member:: void *buf A pointer to the start of the memory for the object. - .. cmember:: Py_ssize_t len + .. c:member:: Py_ssize_t len :noindex: The total length of the memory in bytes. - .. cmember:: int readonly + .. c:member:: int readonly An indicator of whether the buffer is read only. - .. cmember:: const char *format + .. c:member:: const char *format :noindex: A *NULL* terminated string in :mod:`struct` module style syntax giving the contents of the elements available through the buffer. If this is *NULL*, ``"B"`` (unsigned bytes) is assumed. - .. cmember:: int ndim + .. c:member:: int ndim The number of dimensions the memory represents as a multi-dimensional - array. If it is 0, :cdata:`strides` and :cdata:`suboffsets` must be + array. If it is 0, :c:data:`strides` and :c:data:`suboffsets` must be *NULL*. - .. cmember:: Py_ssize_t *shape + .. c:member:: Py_ssize_t *shape - An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim` giving the + An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim` giving the shape of the memory as a multi-dimensional array. Note that ``((*shape)[0] * ... * (*shape)[ndims-1])*itemsize`` should be equal to - :cdata:`len`. + :c:data:`len`. - .. cmember:: Py_ssize_t *strides + .. c:member:: Py_ssize_t *strides - An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim` giving the + An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim` giving the number of bytes to skip to get to a new element in each dimension. - .. cmember:: Py_ssize_t *suboffsets + .. c:member:: Py_ssize_t *suboffsets - An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim`. If these + An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim`. If these suboffset numbers are greater than or equal to 0, then the value stored along the indicated dimension is a pointer and the suboffset value dictates how many bytes to add to the pointer after de-referencing. A @@ -146,16 +146,16 @@ can be created. } - .. cmember:: Py_ssize_t itemsize + .. c:member:: Py_ssize_t itemsize This is a storage for the itemsize (in bytes) of each element of the shared memory. It is technically un-necessary as it can be obtained - using :cfunc:`PyBuffer_SizeFromFormat`, however an exporter may know + using :c:func:`PyBuffer_SizeFromFormat`, however an exporter may know this information without parsing the format string and it is necessary to know the itemsize for proper interpretation of striding. Therefore, storing it is more convenient and faster. - .. cmember:: void *internal + .. c:member:: void *internal This is for use internally by the exporting object. For example, this might be re-cast as an integer by the exporter and used to store flags @@ -168,32 +168,32 @@ Buffer-related functions ======================== -.. cfunction:: int PyObject_CheckBuffer(PyObject *obj) +.. c:function:: int PyObject_CheckBuffer(PyObject *obj) Return 1 if *obj* supports the buffer interface otherwise 0. When 1 is - returned, it doesn't guarantee that :cfunc:`PyObject_GetBuffer` will + returned, it doesn't guarantee that :c:func:`PyObject_GetBuffer` will succeed. -.. cfunction:: int PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags) +.. c:function:: int PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags) Export a view over some internal data from the target object *obj*. *obj* must not be NULL, and *view* must point to an existing - :ctype:`Py_buffer` structure allocated by the caller (most uses of + :c:type:`Py_buffer` structure allocated by the caller (most uses of this function will simply declare a local variable of type - :ctype:`Py_buffer`). The *flags* argument is a bit field indicating + :c:type:`Py_buffer`). The *flags* argument is a bit field indicating what kind of buffer is requested. The buffer interface allows for complicated memory layout possibilities; however, some callers won't want to handle all the complexity and instead request a simple - view of the target object (using :cmacro:`PyBUF_SIMPLE` for a read-only - view and :cmacro:`PyBUF_WRITABLE` for a read-write view). + view of the target object (using :c:macro:`PyBUF_SIMPLE` for a read-only + view and :c:macro:`PyBUF_WRITABLE` for a read-write view). Some exporters may not be able to share memory in every possible way and may need to raise errors to signal to some consumers that something is just not possible. These errors should be a :exc:`BufferError` unless there is another error that is actually causing the problem. The exporter can use flags information to simplify how much of the - :cdata:`Py_buffer` structure is filled in with non-default values and/or + :c:data:`Py_buffer` structure is filled in with non-default values and/or raise an error if the object can't support a simpler view of its memory. On success, 0 is returned and the *view* structure is filled with useful @@ -202,7 +202,7 @@ Buffer-related functions The following are the possible values to the *flags* arguments. - .. cmacro:: PyBUF_SIMPLE + .. c:macro:: PyBUF_SIMPLE This is the default flag. The returned buffer exposes a read-only memory area. The format of data is assumed to be raw unsigned bytes, @@ -210,47 +210,45 @@ Buffer-related functions constant. It never needs to be '|'d to the others. The exporter will raise an error if it cannot provide such a contiguous buffer of bytes. - .. cmacro:: PyBUF_WRITABLE + .. c:macro:: PyBUF_WRITABLE - Like :cmacro:`PyBUF_SIMPLE`, but the returned buffer is writable. If + Like :c:macro:`PyBUF_SIMPLE`, but the returned buffer is writable. If the exporter doesn't support writable buffers, an error is raised. - .. cmacro:: PyBUF_STRIDES + .. c:macro:: PyBUF_STRIDES - This implies :cmacro:`PyBUF_ND`. The returned buffer must provide + This implies :c:macro:`PyBUF_ND`. The returned buffer must provide strides information (i.e. the strides cannot be NULL). This would be used when the consumer can handle strided, discontiguous arrays. Handling strides automatically assumes you can handle shape. The exporter can raise an error if a strided representation of the data is not possible (i.e. without the suboffsets). - .. cmacro:: PyBUF_ND + .. c:macro:: PyBUF_ND The returned buffer must provide shape information. The memory will be assumed C-style contiguous (last dimension varies the fastest). The exporter may raise an error if it cannot provide this kind of contiguous buffer. If this is not given then shape will be *NULL*. - .. cmacro:: PyBUF_C_CONTIGUOUS - - .. cmacro:: PyBUF_F_CONTIGUOUS - - .. cmacro:: PyBUF_ANY_CONTIGUOUS + .. c:macro:: PyBUF_C_CONTIGUOUS + PyBUF_F_CONTIGUOUS + PyBUF_ANY_CONTIGUOUS These flags indicate that the contiguity returned buffer must be respectively, C-contiguous (last dimension varies the fastest), Fortran contiguous (first dimension varies the fastest) or either one. All of - these flags imply :cmacro:`PyBUF_STRIDES` and guarantee that the + these flags imply :c:macro:`PyBUF_STRIDES` and guarantee that the strides buffer info structure will be filled in correctly. - .. cmacro:: PyBUF_INDIRECT + .. c:macro:: PyBUF_INDIRECT This flag indicates the returned buffer must have suboffsets information (which can be NULL if no suboffsets are needed). This can be used when the consumer can handle indirect array referencing implied - by these suboffsets. This implies :cmacro:`PyBUF_STRIDES`. + by these suboffsets. This implies :c:macro:`PyBUF_STRIDES`. - .. cmacro:: PyBUF_FORMAT + .. c:macro:: PyBUF_FORMAT The returned buffer must have true format information if this flag is provided. This would be used when the consumer is going to be checking @@ -259,68 +257,68 @@ Buffer-related functions explicitly requested then the format must be returned as *NULL* (which means ``'B'``, or unsigned bytes). - .. cmacro:: PyBUF_STRIDED + .. c:macro:: PyBUF_STRIDED This is equivalent to ``(PyBUF_STRIDES | PyBUF_WRITABLE)``. - .. cmacro:: PyBUF_STRIDED_RO + .. c:macro:: PyBUF_STRIDED_RO This is equivalent to ``(PyBUF_STRIDES)``. - .. cmacro:: PyBUF_RECORDS + .. c:macro:: PyBUF_RECORDS This is equivalent to ``(PyBUF_STRIDES | PyBUF_FORMAT | PyBUF_WRITABLE)``. - .. cmacro:: PyBUF_RECORDS_RO + .. c:macro:: PyBUF_RECORDS_RO This is equivalent to ``(PyBUF_STRIDES | PyBUF_FORMAT)``. - .. cmacro:: PyBUF_FULL + .. c:macro:: PyBUF_FULL This is equivalent to ``(PyBUF_INDIRECT | PyBUF_FORMAT | PyBUF_WRITABLE)``. - .. cmacro:: PyBUF_FULL_RO + .. c:macro:: PyBUF_FULL_RO This is equivalent to ``(PyBUF_INDIRECT | PyBUF_FORMAT)``. - .. cmacro:: PyBUF_CONTIG + .. c:macro:: PyBUF_CONTIG This is equivalent to ``(PyBUF_ND | PyBUF_WRITABLE)``. - .. cmacro:: PyBUF_CONTIG_RO + .. c:macro:: PyBUF_CONTIG_RO This is equivalent to ``(PyBUF_ND)``. -.. cfunction:: void PyBuffer_Release(Py_buffer *view) +.. c:function:: void PyBuffer_Release(Py_buffer *view) Release the buffer *view*. This should be called when the buffer is no longer being used as it may free memory from it. -.. cfunction:: Py_ssize_t PyBuffer_SizeFromFormat(const char *) +.. c:function:: Py_ssize_t PyBuffer_SizeFromFormat(const char *) - Return the implied :cdata:`~Py_buffer.itemsize` from the struct-stype - :cdata:`~Py_buffer.format`. + Return the implied :c:data:`~Py_buffer.itemsize` from the struct-stype + :c:data:`~Py_buffer.format`. -.. cfunction:: int PyBuffer_IsContiguous(Py_buffer *view, char fortran) +.. c:function:: int PyBuffer_IsContiguous(Py_buffer *view, char fortran) Return 1 if the memory defined by the *view* is C-style (*fortran* is ``'C'``) or Fortran-style (*fortran* is ``'F'``) contiguous or either one (*fortran* is ``'A'``). Return 0 otherwise. -.. cfunction:: void PyBuffer_FillContiguousStrides(int ndim, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize, char fortran) +.. c:function:: void PyBuffer_FillContiguousStrides(int ndim, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize, char fortran) Fill the *strides* array with byte-strides of a contiguous (C-style if *fortran* is ``'C'`` or Fortran-style if *fortran* is ``'F'``) array of the given shape with the given number of bytes per element. -.. cfunction:: int PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len, int readonly, int infoflags) +.. c:function:: int PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len, int readonly, int infoflags) Fill in a buffer-info structure, *view*, correctly for an exporter that can only share a contiguous chunk of memory of "unsigned bytes" of the given diff --git a/Doc/c-api/bytearray.rst b/Doc/c-api/bytearray.rst index 5e11d8a571..95ded96eb6 100644 --- a/Doc/c-api/bytearray.rst +++ b/Doc/c-api/bytearray.rst @@ -8,27 +8,27 @@ Byte Array Objects .. index:: object: bytearray -.. ctype:: PyByteArrayObject +.. c:type:: PyByteArrayObject - This subtype of :ctype:`PyObject` represents a Python bytearray object. + This subtype of :c:type:`PyObject` represents a Python bytearray object. -.. cvar:: PyTypeObject PyByteArray_Type +.. c:var:: PyTypeObject PyByteArray_Type - This instance of :ctype:`PyTypeObject` represents the Python bytearray type; + This instance of :c:type:`PyTypeObject` represents the Python bytearray type; it is the same object as :class:`bytearray` in the Python layer. Type check macros ^^^^^^^^^^^^^^^^^ -.. cfunction:: int PyByteArray_Check(PyObject *o) +.. c:function:: int PyByteArray_Check(PyObject *o) Return true if the object *o* is a bytearray object or an instance of a subtype of the bytearray type. -.. cfunction:: int PyByteArray_CheckExact(PyObject *o) +.. c:function:: int PyByteArray_CheckExact(PyObject *o) Return true if the object *o* is a bytearray object, but not an instance of a subtype of the bytearray type. @@ -37,7 +37,7 @@ Type check macros Direct API functions ^^^^^^^^^^^^^^^^^^^^ -.. cfunction:: PyObject* PyByteArray_FromObject(PyObject *o) +.. c:function:: PyObject* PyByteArray_FromObject(PyObject *o) Return a new bytearray object from any object, *o*, that implements the buffer protocol. @@ -45,29 +45,29 @@ Direct API functions .. XXX expand about the buffer protocol, at least somewhere -.. cfunction:: PyObject* PyByteArray_FromStringAndSize(const char *string, Py_ssize_t len) +.. c:function:: PyObject* PyByteArray_FromStringAndSize(const char *string, Py_ssize_t len) Create a new bytearray object from *string* and its length, *len*. On failure, *NULL* is returned. -.. cfunction:: PyObject* PyByteArray_Concat(PyObject *a, PyObject *b) +.. c:function:: PyObject* PyByteArray_Concat(PyObject *a, PyObject *b) Concat bytearrays *a* and *b* and return a new bytearray with the result. -.. cfunction:: Py_ssize_t PyByteArray_Size(PyObject *bytearray) +.. c:function:: Py_ssize_t PyByteArray_Size(PyObject *bytearray) Return the size of *bytearray* after checking for a *NULL* pointer. -.. cfunction:: char* PyByteArray_AsString(PyObject *bytearray) +.. c:function:: char* PyByteArray_AsString(PyObject *bytearray) Return the contents of *bytearray* as a char array after checking for a *NULL* pointer. -.. cfunction:: int PyByteArray_Resize(PyObject *bytearray, Py_ssize_t len) +.. c:function:: int PyByteArray_Resize(PyObject *bytearray, Py_ssize_t len) Resize the internal buffer of *bytearray* to *len*. @@ -76,11 +76,11 @@ Macros These macros trade safety for speed and they don't check pointers. -.. cfunction:: char* PyByteArray_AS_STRING(PyObject *bytearray) +.. c:function:: char* PyByteArray_AS_STRING(PyObject *bytearray) - Macro version of :cfunc:`PyByteArray_AsString`. + Macro version of :c:func:`PyByteArray_AsString`. -.. cfunction:: Py_ssize_t PyByteArray_GET_SIZE(PyObject *bytearray) +.. c:function:: Py_ssize_t PyByteArray_GET_SIZE(PyObject *bytearray) - Macro version of :cfunc:`PyByteArray_Size`. + Macro version of :c:func:`PyByteArray_Size`. diff --git a/Doc/c-api/bytes.rst b/Doc/c-api/bytes.rst index abd8744108..12ec80ca5e 100644 --- a/Doc/c-api/bytes.rst +++ b/Doc/c-api/bytes.rst @@ -11,46 +11,46 @@ called with a non-bytes parameter. .. index:: object: bytes -.. ctype:: PyBytesObject +.. c:type:: PyBytesObject - This subtype of :ctype:`PyObject` represents a Python bytes object. + This subtype of :c:type:`PyObject` represents a Python bytes object. -.. cvar:: PyTypeObject PyBytes_Type +.. c:var:: PyTypeObject PyBytes_Type - This instance of :ctype:`PyTypeObject` represents the Python bytes type; it + This instance of :c:type:`PyTypeObject` represents the Python bytes type; it is the same object as :class:`bytes` in the Python layer. -.. cfunction:: int PyBytes_Check(PyObject *o) +.. c:function:: int PyBytes_Check(PyObject *o) Return true if the object *o* is a bytes object or an instance of a subtype of the bytes type. -.. cfunction:: int PyBytes_CheckExact(PyObject *o) +.. c:function:: int PyBytes_CheckExact(PyObject *o) Return true if the object *o* is a bytes object, but not an instance of a subtype of the bytes type. -.. cfunction:: PyObject* PyBytes_FromString(const char *v) +.. c:function:: PyObject* PyBytes_FromString(const char *v) Return a new bytes object with a copy of the string *v* as value on success, and *NULL* on failure. The parameter *v* must not be *NULL*; it will not be checked. -.. cfunction:: PyObject* PyBytes_FromStringAndSize(const char *v, Py_ssize_t len) +.. c:function:: PyObject* PyBytes_FromStringAndSize(const char *v, Py_ssize_t len) Return a new bytes object with a copy of the string *v* as value and length *len* on success, and *NULL* on failure. If *v* is *NULL*, the contents of the bytes object are uninitialized. -.. cfunction:: PyObject* PyBytes_FromFormat(const char *format, ...) +.. c:function:: PyObject* PyBytes_FromFormat(const char *format, ...) - Take a C :cfunc:`printf`\ -style *format* string and a variable number of + Take a C :c:func:`printf`\ -style *format* string and a variable number of arguments, calculate the size of the resulting Python bytes object and return a bytes object with the values formatted into it. The variable arguments must be C types and must correspond exactly to the format characters in the @@ -110,44 +110,44 @@ called with a non-bytes parameter. copied as-is to the result string, and any extra arguments discarded. -.. cfunction:: PyObject* PyBytes_FromFormatV(const char *format, va_list vargs) +.. c:function:: PyObject* PyBytes_FromFormatV(const char *format, va_list vargs) - Identical to :func:`PyBytes_FromFormat` except that it takes exactly two + Identical to :c:func:`PyBytes_FromFormat` except that it takes exactly two arguments. -.. cfunction:: PyObject* PyBytes_FromObject(PyObject *o) +.. c:function:: PyObject* PyBytes_FromObject(PyObject *o) Return the bytes representation of object *o* that implements the buffer protocol. -.. cfunction:: Py_ssize_t PyBytes_Size(PyObject *o) +.. c:function:: Py_ssize_t PyBytes_Size(PyObject *o) Return the length of the bytes in bytes object *o*. -.. cfunction:: Py_ssize_t PyBytes_GET_SIZE(PyObject *o) +.. c:function:: Py_ssize_t PyBytes_GET_SIZE(PyObject *o) - Macro form of :cfunc:`PyBytes_Size` but without error checking. + Macro form of :c:func:`PyBytes_Size` but without error checking. -.. cfunction:: char* PyBytes_AsString(PyObject *o) +.. c:function:: char* PyBytes_AsString(PyObject *o) Return a NUL-terminated representation of the contents of *o*. The pointer refers to the internal buffer of *o*, not a copy. The data must not be modified in any way, unless the string was just created using ``PyBytes_FromStringAndSize(NULL, size)``. It must not be deallocated. If - *o* is not a string object at all, :cfunc:`PyBytes_AsString` returns *NULL* + *o* is not a string object at all, :c:func:`PyBytes_AsString` returns *NULL* and raises :exc:`TypeError`. -.. cfunction:: char* PyBytes_AS_STRING(PyObject *string) +.. c:function:: char* PyBytes_AS_STRING(PyObject *string) - Macro form of :cfunc:`PyBytes_AsString` but without error checking. + Macro form of :c:func:`PyBytes_AsString` but without error checking. -.. cfunction:: int PyBytes_AsStringAndSize(PyObject *obj, char **buffer, Py_ssize_t *length) +.. c:function:: int PyBytes_AsStringAndSize(PyObject *obj, char **buffer, Py_ssize_t *length) Return a NUL-terminated representation of the contents of the object *obj* through the output variables *buffer* and *length*. @@ -158,11 +158,11 @@ called with a non-bytes parameter. The buffer refers to an internal string buffer of *obj*, not a copy. The data must not be modified in any way, unless the string was just created using ``PyBytes_FromStringAndSize(NULL, size)``. It must not be deallocated. If - *string* is not a string object at all, :cfunc:`PyBytes_AsStringAndSize` + *string* is not a string object at all, :c:func:`PyBytes_AsStringAndSize` returns ``-1`` and raises :exc:`TypeError`. -.. cfunction:: void PyBytes_Concat(PyObject **bytes, PyObject *newpart) +.. c:function:: void PyBytes_Concat(PyObject **bytes, PyObject *newpart) Create a new bytes object in *\*bytes* containing the contents of *newpart* appended to *bytes*; the caller will own the new reference. The reference to @@ -171,14 +171,14 @@ called with a non-bytes parameter. of *\*bytes* will be set to *NULL*; the appropriate exception will be set. -.. cfunction:: void PyBytes_ConcatAndDel(PyObject **bytes, PyObject *newpart) +.. c:function:: void PyBytes_ConcatAndDel(PyObject **bytes, PyObject *newpart) Create a new string object in *\*bytes* containing the contents of *newpart* appended to *bytes*. This version decrements the reference count of *newpart*. -.. cfunction:: int _PyBytes_Resize(PyObject **bytes, Py_ssize_t newsize) +.. c:function:: int _PyBytes_Resize(PyObject **bytes, Py_ssize_t newsize) A way to resize a bytes object even though it is "immutable". Only use this to build up a brand new bytes object; don't use this if the bytes may already diff --git a/Doc/c-api/capsule.rst b/Doc/c-api/capsule.rst index 29393146bd..6f6250f671 100644 --- a/Doc/c-api/capsule.rst +++ b/Doc/c-api/capsule.rst @@ -10,33 +10,33 @@ Capsules Refer to :ref:`using-capsules` for more information on using these objects. -.. ctype:: PyCapsule +.. c:type:: PyCapsule - This subtype of :ctype:`PyObject` represents an opaque value, useful for C - extension modules who need to pass an opaque value (as a :ctype:`void\*` + This subtype of :c:type:`PyObject` represents an opaque value, useful for C + extension modules who need to pass an opaque value (as a :c:type:`void\*` pointer) through Python code to other C code. It is often used to make a C function pointer defined in one module available to other modules, so the regular import mechanism can be used to access C APIs defined in dynamically loaded modules. -.. ctype:: PyCapsule_Destructor +.. c:type:: PyCapsule_Destructor The type of a destructor callback for a capsule. Defined as:: typedef void (*PyCapsule_Destructor)(PyObject *); - See :cfunc:`PyCapsule_New` for the semantics of PyCapsule_Destructor + See :c:func:`PyCapsule_New` for the semantics of PyCapsule_Destructor callbacks. -.. cfunction:: int PyCapsule_CheckExact(PyObject *p) +.. c:function:: int PyCapsule_CheckExact(PyObject *p) - Return true if its argument is a :ctype:`PyCapsule`. + Return true if its argument is a :c:type:`PyCapsule`. -.. cfunction:: PyObject* PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor) +.. c:function:: PyObject* PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor) - Create a :ctype:`PyCapsule` encapsulating the *pointer*. The *pointer* + Create a :c:type:`PyCapsule` encapsulating the *pointer*. The *pointer* argument may not be *NULL*. On failure, set an exception and return *NULL*. @@ -50,91 +50,91 @@ Refer to :ref:`using-capsules` for more information on using these objects. If this capsule will be stored as an attribute of a module, the *name* should be specified as ``modulename.attributename``. This will enable other modules - to import the capsule using :cfunc:`PyCapsule_Import`. + to import the capsule using :c:func:`PyCapsule_Import`. -.. cfunction:: void* PyCapsule_GetPointer(PyObject *capsule, const char *name) +.. c:function:: void* PyCapsule_GetPointer(PyObject *capsule, const char *name) Retrieve the *pointer* stored in the capsule. On failure, set an exception and return *NULL*. The *name* parameter must compare exactly to the name stored in the capsule. If the name stored in the capsule is *NULL*, the *name* passed in must also - be *NULL*. Python uses the C function :cfunc:`strcmp` to compare capsule + be *NULL*. Python uses the C function :c:func:`strcmp` to compare capsule names. -.. cfunction:: PyCapsule_Destructor PyCapsule_GetDestructor(PyObject *capsule) +.. c:function:: PyCapsule_Destructor PyCapsule_GetDestructor(PyObject *capsule) Return the current destructor stored in the capsule. On failure, set an exception and return *NULL*. It is legal for a capsule to have a *NULL* destructor. This makes a *NULL* - return code somewhat ambiguous; use :cfunc:`PyCapsule_IsValid` or - :cfunc:`PyErr_Occurred` to disambiguate. + return code somewhat ambiguous; use :c:func:`PyCapsule_IsValid` or + :c:func:`PyErr_Occurred` to disambiguate. -.. cfunction:: void* PyCapsule_GetContext(PyObject *capsule) +.. c:function:: void* PyCapsule_GetContext(PyObject *capsule) Return the current context stored in the capsule. On failure, set an exception and return *NULL*. It is legal for a capsule to have a *NULL* context. This makes a *NULL* - return code somewhat ambiguous; use :cfunc:`PyCapsule_IsValid` or - :cfunc:`PyErr_Occurred` to disambiguate. + return code somewhat ambiguous; use :c:func:`PyCapsule_IsValid` or + :c:func:`PyErr_Occurred` to disambiguate. -.. cfunction:: const char* PyCapsule_GetName(PyObject *capsule) +.. c:function:: const char* PyCapsule_GetName(PyObject *capsule) Return the current name stored in the capsule. On failure, set an exception and return *NULL*. It is legal for a capsule to have a *NULL* name. This makes a *NULL* return - code somewhat ambiguous; use :cfunc:`PyCapsule_IsValid` or - :cfunc:`PyErr_Occurred` to disambiguate. + code somewhat ambiguous; use :c:func:`PyCapsule_IsValid` or + :c:func:`PyErr_Occurred` to disambiguate. -.. cfunction:: void* PyCapsule_Import(const char *name, int no_block) +.. c:function:: void* PyCapsule_Import(const char *name, int no_block) Import a pointer to a C object from a capsule attribute in a module. The *name* parameter should specify the full name to the attribute, as in ``module.attribute``. The *name* stored in the capsule must match this string exactly. If *no_block* is true, import the module without blocking - (using :cfunc:`PyImport_ImportModuleNoBlock`). If *no_block* is false, - import the module conventionally (using :cfunc:`PyImport_ImportModule`). + (using :c:func:`PyImport_ImportModuleNoBlock`). If *no_block* is false, + import the module conventionally (using :c:func:`PyImport_ImportModule`). Return the capsule's internal *pointer* on success. On failure, set an - exception and return *NULL*. However, if :cfunc:`PyCapsule_Import` failed to + exception and return *NULL*. However, if :c:func:`PyCapsule_Import` failed to import the module, and *no_block* was true, no exception is set. -.. cfunction:: int PyCapsule_IsValid(PyObject *capsule, const char *name) +.. c:function:: int PyCapsule_IsValid(PyObject *capsule, const char *name) Determines whether or not *capsule* is a valid capsule. A valid capsule is - non-*NULL*, passes :cfunc:`PyCapsule_CheckExact`, has a non-*NULL* pointer + non-*NULL*, passes :c:func:`PyCapsule_CheckExact`, has a non-*NULL* pointer stored in it, and its internal name matches the *name* parameter. (See - :cfunc:`PyCapsule_GetPointer` for information on how capsule names are + :c:func:`PyCapsule_GetPointer` for information on how capsule names are compared.) - In other words, if :cfunc:`PyCapsule_IsValid` returns a true value, calls to - any of the accessors (any function starting with :cfunc:`PyCapsule_Get`) are + In other words, if :c:func:`PyCapsule_IsValid` returns a true value, calls to + any of the accessors (any function starting with :c:func:`PyCapsule_Get`) are guaranteed to succeed. Return a nonzero value if the object is valid and matches the name passed in. Return 0 otherwise. This function will not fail. -.. cfunction:: int PyCapsule_SetContext(PyObject *capsule, void *context) +.. c:function:: int PyCapsule_SetContext(PyObject *capsule, void *context) Set the context pointer inside *capsule* to *context*. Return 0 on success. Return nonzero and set an exception on failure. -.. cfunction:: int PyCapsule_SetDestructor(PyObject *capsule, PyCapsule_Destructor destructor) +.. c:function:: int PyCapsule_SetDestructor(PyObject *capsule, PyCapsule_Destructor destructor) Set the destructor inside *capsule* to *destructor*. Return 0 on success. Return nonzero and set an exception on failure. -.. cfunction:: int PyCapsule_SetName(PyObject *capsule, const char *name) +.. c:function:: int PyCapsule_SetName(PyObject *capsule, const char *name) Set the name inside *capsule* to *name*. If non-*NULL*, the name must outlive the capsule. If the previous *name* stored in the capsule was not @@ -142,7 +142,7 @@ Refer to :ref:`using-capsules` for more information on using these objects. Return 0 on success. Return nonzero and set an exception on failure. -.. cfunction:: int PyCapsule_SetPointer(PyObject *capsule, void *pointer) +.. c:function:: int PyCapsule_SetPointer(PyObject *capsule, void *pointer) Set the void pointer inside *capsule* to *pointer*. The pointer may not be *NULL*. diff --git a/Doc/c-api/cell.rst b/Doc/c-api/cell.rst index 3562ed9816..427259cc24 100644 --- a/Doc/c-api/cell.rst +++ b/Doc/c-api/cell.rst @@ -15,39 +15,39 @@ generated byte-code; these are not automatically de-referenced when accessed. Cell objects are not likely to be useful elsewhere. -.. ctype:: PyCellObject +.. c:type:: PyCellObject The C structure used for cell objects. -.. cvar:: PyTypeObject PyCell_Type +.. c:var:: PyTypeObject PyCell_Type The type object corresponding to cell objects. -.. cfunction:: int PyCell_Check(ob) +.. c:function:: int PyCell_Check(ob) Return true if *ob* is a cell object; *ob* must not be *NULL*. -.. cfunction:: PyObject* PyCell_New(PyObject *ob) +.. c:function:: PyObject* PyCell_New(PyObject *ob) Create and return a new cell object containing the value *ob*. The parameter may be *NULL*. -.. cfunction:: PyObject* PyCell_Get(PyObject *cell) +.. c:function:: PyObject* PyCell_Get(PyObject *cell) Return the contents of the cell *cell*. -.. cfunction:: PyObject* PyCell_GET(PyObject *cell) +.. c:function:: PyObject* PyCell_GET(PyObject *cell) Return the contents of the cell *cell*, but without checking that *cell* is non-*NULL* and a cell object. -.. cfunction:: int PyCell_Set(PyObject *cell, PyObject *value) +.. c:function:: int PyCell_Set(PyObject *cell, PyObject *value) Set the contents of the cell object *cell* to *value*. This releases the reference to any current content of the cell. *value* may be *NULL*. *cell* @@ -55,7 +55,7 @@ Cell objects are not likely to be useful elsewhere. success, ``0`` will be returned. -.. cfunction:: void PyCell_SET(PyObject *cell, PyObject *value) +.. c:function:: void PyCell_SET(PyObject *cell, PyObject *value) Sets the value of the cell object *cell* to *value*. No reference counts are adjusted, and no checks are made for safety; *cell* must be non-*NULL* and must diff --git a/Doc/c-api/cobject.rst b/Doc/c-api/cobject.rst deleted file mode 100644 index ee65a9833c..0000000000 --- a/Doc/c-api/cobject.rst +++ /dev/null @@ -1,59 +0,0 @@ -.. highlightlang:: c - -.. _cobjects: - -CObjects --------- - -.. index:: object: CObject - - -.. warning:: - - The CObject API is deprecated as of Python 3.1. Please switch to the new - :ref:`capsules` API. - -.. ctype:: PyCObject - - This subtype of :ctype:`PyObject` represents an opaque value, useful for C - extension modules who need to pass an opaque value (as a :ctype:`void\*` - pointer) through Python code to other C code. It is often used to make a C - function pointer defined in one module available to other modules, so the - regular import mechanism can be used to access C APIs defined in dynamically - loaded modules. - - -.. cfunction:: int PyCObject_Check(PyObject *p) - - Return true if its argument is a :ctype:`PyCObject`. - - -.. cfunction:: PyObject* PyCObject_FromVoidPtr(void* cobj, void (*destr)(void *)) - - Create a :ctype:`PyCObject` from the ``void *`` *cobj*. The *destr* function - will be called when the object is reclaimed, unless it is *NULL*. - - -.. cfunction:: PyObject* PyCObject_FromVoidPtrAndDesc(void* cobj, void* desc, void (*destr)(void *, void *)) - - Create a :ctype:`PyCObject` from the :ctype:`void \*` *cobj*. The *destr* - function will be called when the object is reclaimed. The *desc* argument can - be used to pass extra callback data for the destructor function. - - -.. cfunction:: void* PyCObject_AsVoidPtr(PyObject* self) - - Return the object :ctype:`void \*` that the :ctype:`PyCObject` *self* was - created with. - - -.. cfunction:: void* PyCObject_GetDesc(PyObject* self) - - Return the description :ctype:`void \*` that the :ctype:`PyCObject` *self* was - created with. - - -.. cfunction:: int PyCObject_SetVoidPtr(PyObject* self, void* cobj) - - Set the void pointer inside *self* to *cobj*. The :ctype:`PyCObject` must not - have an associated destructor. Return true on success, false on failure. diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst new file mode 100644 index 0000000000..6932bb1964 --- /dev/null +++ b/Doc/c-api/code.rst @@ -0,0 +1,50 @@ +.. highlightlang:: c + +.. _codeobjects: + +Code Objects +------------ + +.. sectionauthor:: Jeffrey Yasskin <jyasskin@gmail.com> + + +.. index:: + object: code + +Code objects are a low-level detail of the CPython implementation. +Each one represents a chunk of executable code that hasn't yet been +bound into a function. + +.. c:type:: PyCodeObject + + The C structure of the objects used to describe code objects. The + fields of this type are subject to change at any time. + + +.. c:var:: PyTypeObject PyCode_Type + + This is an instance of :c:type:`PyTypeObject` representing the Python + :class:`code` type. + + +.. c:function:: int PyCode_Check(PyObject *co) + + Return true if *co* is a :class:`code` object + +.. c:function:: int PyCode_GetNumFree(PyObject *co) + + Return the number of free variables in *co*. + +.. c:function:: PyCodeObject *PyCode_New(int argcount, int kwonlyargcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, int firstlineno, PyObject *lnotab) + + Return a new code object. If you need a dummy code object to + create a frame, use :c:func:`PyCode_NewEmpty` instead. Calling + :c:func:`PyCode_New` directly can bind you to a precise Python + version since the definition of the bytecode changes often. + + +.. c:function:: int PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno) + + Return a new empty code object with the specified filename, + function name, and first line number. It is illegal to + :func:`exec` or :func:`eval` the resulting code object. diff --git a/Doc/c-api/codec.rst b/Doc/c-api/codec.rst index 2597d380af..8207ae044d 100644 --- a/Doc/c-api/codec.rst +++ b/Doc/c-api/codec.rst @@ -3,19 +3,19 @@ Codec registry and support functions ==================================== -.. cfunction:: int PyCodec_Register(PyObject *search_function) +.. c:function:: int PyCodec_Register(PyObject *search_function) Register a new codec search function. As side effect, this tries to load the :mod:`encodings` package, if not yet done, to make sure that it is always first in the list of search functions. -.. cfunction:: int PyCodec_KnownEncoding(const char *encoding) +.. c:function:: int PyCodec_KnownEncoding(const char *encoding) Return ``1`` or ``0`` depending on whether there is a registered codec for the given *encoding*. -.. cfunction:: PyObject* PyCodec_Encode(PyObject *object, const char *encoding, const char *errors) +.. c:function:: PyObject* PyCodec_Encode(PyObject *object, const char *encoding, const char *errors) Generic codec based encoding API. @@ -24,7 +24,7 @@ Codec registry and support functions be *NULL* to use the default method defined for the codec. Raises a :exc:`LookupError` if no encoder can be found. -.. cfunction:: PyObject* PyCodec_Decode(PyObject *object, const char *encoding, const char *errors) +.. c:function:: PyObject* PyCodec_Decode(PyObject *object, const char *encoding, const char *errors) Generic codec based decoding API. @@ -42,27 +42,27 @@ lower-case characters, which makes encodings looked up through this mechanism effectively case-insensitive. If no codec is found, a :exc:`KeyError` is set and *NULL* returned. -.. cfunction:: PyObject* PyCodec_Encoder(const char *encoding) +.. c:function:: PyObject* PyCodec_Encoder(const char *encoding) Get an encoder function for the given *encoding*. -.. cfunction:: PyObject* PyCodec_Decoder(const char *encoding) +.. c:function:: PyObject* PyCodec_Decoder(const char *encoding) Get a decoder function for the given *encoding*. -.. cfunction:: PyObject* PyCodec_IncrementalEncoder(const char *encoding, const char *errors) +.. c:function:: PyObject* PyCodec_IncrementalEncoder(const char *encoding, const char *errors) Get an :class:`IncrementalEncoder` object for the given *encoding*. -.. cfunction:: PyObject* PyCodec_IncrementalDecoder(const char *encoding, const char *errors) +.. c:function:: PyObject* PyCodec_IncrementalDecoder(const char *encoding, const char *errors) Get an :class:`IncrementalDecoder` object for the given *encoding*. -.. cfunction:: PyObject* PyCodec_StreamReader(const char *encoding, PyObject *stream, const char *errors) +.. c:function:: PyObject* PyCodec_StreamReader(const char *encoding, PyObject *stream, const char *errors) Get a :class:`StreamReader` factory function for the given *encoding*. -.. cfunction:: PyObject* PyCodec_StreamWriter(const char *encoding, PyObject *stream, const char *errors) +.. c:function:: PyObject* PyCodec_StreamWriter(const char *encoding, PyObject *stream, const char *errors) Get a :class:`StreamWriter` factory function for the given *encoding*. @@ -70,7 +70,7 @@ and *NULL* returned. Registry API for Unicode encoding error handlers ------------------------------------------------ -.. cfunction:: int PyCodec_RegisterError(const char *name, PyObject *error) +.. c:function:: int PyCodec_RegisterError(const char *name, PyObject *error) Register the error handling callback function *error* under the given *name*. This callback function will be called by a codec when it encounters @@ -89,29 +89,29 @@ Registry API for Unicode encoding error handlers Return ``0`` on success, ``-1`` on error. -.. cfunction:: PyObject* PyCodec_LookupError(const char *name) +.. c:function:: PyObject* PyCodec_LookupError(const char *name) Lookup the error handling callback function registered under *name*. As a special case *NULL* can be passed, in which case the error handling callback for "strict" will be returned. -.. cfunction:: PyObject* PyCodec_StrictErrors(PyObject *exc) +.. c:function:: PyObject* PyCodec_StrictErrors(PyObject *exc) Raise *exc* as an exception. -.. cfunction:: PyObject* PyCodec_IgnoreErrors(PyObject *exc) +.. c:function:: PyObject* PyCodec_IgnoreErrors(PyObject *exc) Ignore the unicode error, skipping the faulty input. -.. cfunction:: PyObject* PyCodec_ReplaceErrors(PyObject *exc) +.. c:function:: PyObject* PyCodec_ReplaceErrors(PyObject *exc) Replace the unicode encode error with ``?`` or ``U+FFFD``. -.. cfunction:: PyObject* PyCodec_XMLCharRefReplaceErrors(PyObject *exc) +.. c:function:: PyObject* PyCodec_XMLCharRefReplaceErrors(PyObject *exc) Replace the unicode encode error with XML character references. -.. cfunction:: PyObject* PyCodec_BackslashReplaceErrors(PyObject *exc) +.. c:function:: PyObject* PyCodec_BackslashReplaceErrors(PyObject *exc) Replace the unicode encode error with backslash escapes (``\x``, ``\u`` and ``\U``). diff --git a/Doc/c-api/complex.rst b/Doc/c-api/complex.rst index 0bcdf4d41a..fc63b57a85 100644 --- a/Doc/c-api/complex.rst +++ b/Doc/c-api/complex.rst @@ -21,7 +21,7 @@ them as results do so *by value* rather than dereferencing them through pointers. This is consistent throughout the API. -.. ctype:: Py_complex +.. c:type:: Py_complex The C structure which corresponds to the value portion of a Python complex number object. Most of the functions for dealing with complex number objects @@ -34,93 +34,99 @@ pointers. This is consistent throughout the API. } Py_complex; -.. cfunction:: Py_complex _Py_c_sum(Py_complex left, Py_complex right) +.. c:function:: Py_complex _Py_c_sum(Py_complex left, Py_complex right) - Return the sum of two complex numbers, using the C :ctype:`Py_complex` + Return the sum of two complex numbers, using the C :c:type:`Py_complex` representation. -.. cfunction:: Py_complex _Py_c_diff(Py_complex left, Py_complex right) +.. c:function:: Py_complex _Py_c_diff(Py_complex left, Py_complex right) Return the difference between two complex numbers, using the C - :ctype:`Py_complex` representation. + :c:type:`Py_complex` representation. -.. cfunction:: Py_complex _Py_c_neg(Py_complex complex) +.. c:function:: Py_complex _Py_c_neg(Py_complex complex) Return the negation of the complex number *complex*, using the C - :ctype:`Py_complex` representation. + :c:type:`Py_complex` representation. -.. cfunction:: Py_complex _Py_c_prod(Py_complex left, Py_complex right) +.. c:function:: Py_complex _Py_c_prod(Py_complex left, Py_complex right) - Return the product of two complex numbers, using the C :ctype:`Py_complex` + Return the product of two complex numbers, using the C :c:type:`Py_complex` representation. -.. cfunction:: Py_complex _Py_c_quot(Py_complex dividend, Py_complex divisor) +.. c:function:: Py_complex _Py_c_quot(Py_complex dividend, Py_complex divisor) - Return the quotient of two complex numbers, using the C :ctype:`Py_complex` + Return the quotient of two complex numbers, using the C :c:type:`Py_complex` representation. + If *divisor* is null, this method returns zero and sets + :c:data:`errno` to :c:data:`EDOM`. -.. cfunction:: Py_complex _Py_c_pow(Py_complex num, Py_complex exp) - Return the exponentiation of *num* by *exp*, using the C :ctype:`Py_complex` +.. c:function:: Py_complex _Py_c_pow(Py_complex num, Py_complex exp) + + Return the exponentiation of *num* by *exp*, using the C :c:type:`Py_complex` representation. + If *num* is null and *exp* is not a positive real number, + this method returns zero and sets :c:data:`errno` to :c:data:`EDOM`. + Complex Numbers as Python Objects ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. ctype:: PyComplexObject +.. c:type:: PyComplexObject - This subtype of :ctype:`PyObject` represents a Python complex number object. + This subtype of :c:type:`PyObject` represents a Python complex number object. -.. cvar:: PyTypeObject PyComplex_Type +.. c:var:: PyTypeObject PyComplex_Type - This instance of :ctype:`PyTypeObject` represents the Python complex number + This instance of :c:type:`PyTypeObject` represents the Python complex number type. It is the same object as :class:`complex` in the Python layer. -.. cfunction:: int PyComplex_Check(PyObject *p) +.. c:function:: int PyComplex_Check(PyObject *p) - Return true if its argument is a :ctype:`PyComplexObject` or a subtype of - :ctype:`PyComplexObject`. + Return true if its argument is a :c:type:`PyComplexObject` or a subtype of + :c:type:`PyComplexObject`. -.. cfunction:: int PyComplex_CheckExact(PyObject *p) +.. c:function:: int PyComplex_CheckExact(PyObject *p) - Return true if its argument is a :ctype:`PyComplexObject`, but not a subtype of - :ctype:`PyComplexObject`. + Return true if its argument is a :c:type:`PyComplexObject`, but not a subtype of + :c:type:`PyComplexObject`. -.. cfunction:: PyObject* PyComplex_FromCComplex(Py_complex v) +.. c:function:: PyObject* PyComplex_FromCComplex(Py_complex v) - Create a new Python complex number object from a C :ctype:`Py_complex` value. + Create a new Python complex number object from a C :c:type:`Py_complex` value. -.. cfunction:: PyObject* PyComplex_FromDoubles(double real, double imag) +.. c:function:: PyObject* PyComplex_FromDoubles(double real, double imag) - Return a new :ctype:`PyComplexObject` object from *real* and *imag*. + Return a new :c:type:`PyComplexObject` object from *real* and *imag*. -.. cfunction:: double PyComplex_RealAsDouble(PyObject *op) +.. c:function:: double PyComplex_RealAsDouble(PyObject *op) - Return the real part of *op* as a C :ctype:`double`. + Return the real part of *op* as a C :c:type:`double`. -.. cfunction:: double PyComplex_ImagAsDouble(PyObject *op) +.. c:function:: double PyComplex_ImagAsDouble(PyObject *op) - Return the imaginary part of *op* as a C :ctype:`double`. + Return the imaginary part of *op* as a C :c:type:`double`. -.. cfunction:: Py_complex PyComplex_AsCComplex(PyObject *op) +.. c:function:: Py_complex PyComplex_AsCComplex(PyObject *op) - Return the :ctype:`Py_complex` value of the complex number *op*. + Return the :c:type:`Py_complex` value of the complex number *op*. If *op* is not a Python complex number object but has a :meth:`__complex__` method, this method will first be called to convert *op* to a Python complex - number object. + number object. Upon failure, this method returns ``-1.0`` as a real value. diff --git a/Doc/c-api/concrete.rst b/Doc/c-api/concrete.rst index d72859989f..65904ee49d 100644 --- a/Doc/c-api/concrete.rst +++ b/Doc/c-api/concrete.rst @@ -11,7 +11,7 @@ The functions in this chapter are specific to certain Python object types. Passing them an object of the wrong type is not a good idea; if you receive an object from a Python program and you are not sure that it has the right type, you must perform a type check first; for example, to check that an object is a -dictionary, use :cfunc:`PyDict_Check`. The chapter is structured like the +dictionary, use :c:func:`PyDict_Check`. The chapter is structured like the "family tree" of Python object types. .. warning:: @@ -102,7 +102,7 @@ Other Objects memoryview.rst weakref.rst capsule.rst - cobject.rst cell.rst gen.rst datetime.rst + code.rst diff --git a/Doc/c-api/conversion.rst b/Doc/c-api/conversion.rst index ad3f2fa16b..dfc0a3aace 100644 --- a/Doc/c-api/conversion.rst +++ b/Doc/c-api/conversion.rst @@ -8,20 +8,20 @@ String conversion and formatting Functions for number conversion and formatted string output. -.. cfunction:: int PyOS_snprintf(char *str, size_t size, const char *format, ...) +.. c:function:: int PyOS_snprintf(char *str, size_t size, const char *format, ...) Output not more than *size* bytes to *str* according to the format string *format* and the extra arguments. See the Unix man page :manpage:`snprintf(2)`. -.. cfunction:: int PyOS_vsnprintf(char *str, size_t size, const char *format, va_list va) +.. c:function:: int PyOS_vsnprintf(char *str, size_t size, const char *format, va_list va) Output not more than *size* bytes to *str* according to the format string *format* and the variable argument list *va*. Unix man page :manpage:`vsnprintf(2)`. -:cfunc:`PyOS_snprintf` and :cfunc:`PyOS_vsnprintf` wrap the Standard C library -functions :cfunc:`snprintf` and :cfunc:`vsnprintf`. Their purpose is to +:c:func:`PyOS_snprintf` and :c:func:`PyOS_vsnprintf` wrap the Standard C library +functions :c:func:`snprintf` and :c:func:`vsnprintf`. Their purpose is to guarantee consistent behavior in corner cases, which the Standard C functions do not. @@ -30,7 +30,7 @@ never write more than *size* bytes (including the trailing ``'\0'``) into str. Both functions require that ``str != NULL``, ``size > 0`` and ``format != NULL``. -If the platform doesn't have :cfunc:`vsnprintf` and the buffer size needed to +If the platform doesn't have :c:func:`vsnprintf` and the buffer size needed to avoid truncation exceeds *size* by more than 512 bytes, Python aborts with a *Py_FatalError*. @@ -51,24 +51,9 @@ The return value (*rv*) for these functions should be interpreted as follows: The following functions provide locale-independent string to number conversions. -.. cfunction:: double PyOS_ascii_strtod(const char *nptr, char **endptr) +.. c:function:: double PyOS_string_to_double(const char *s, char **endptr, PyObject *overflow_exception) - Convert a string to a :ctype:`double`. This function behaves like the Standard C - function :cfunc:`strtod` does in the C locale. It does this without changing the - current locale, since that would not be thread-safe. - - :cfunc:`PyOS_ascii_strtod` should typically be used for reading configuration - files or other non-user input that should be locale independent. - - See the Unix man page :manpage:`strtod(2)` for details. - - .. deprecated:: 3.1 - Use :cfunc:`PyOS_string_to_double` instead. - - -.. cfunction:: double PyOS_string_to_double(const char *s, char **endptr, PyObject *overflow_exception) - - Convert a string ``s`` to a :ctype:`double`, raising a Python + Convert a string ``s`` to a :c:type:`double`, raising a Python exception on failure. The set of accepted strings corresponds to the set of strings accepted by Python's :func:`float` constructor, except that ``s`` must not have leading or trailing whitespace. @@ -100,23 +85,9 @@ The following functions provide locale-independent string to number conversions. .. versionadded:: 3.1 -.. cfunction:: char* PyOS_ascii_formatd(char *buffer, size_t buf_len, const char *format, double d) - - Convert a :ctype:`double` to a string using the ``'.'`` as the decimal - separator. *format* is a :cfunc:`printf`\ -style format string specifying the - number format. Allowed conversion characters are ``'e'``, ``'E'``, ``'f'``, - ``'F'``, ``'g'`` and ``'G'``. - - The return value is a pointer to *buffer* with the converted string or NULL if - the conversion failed. +.. c:function:: char* PyOS_double_to_string(double val, char format_code, int precision, int flags, int *ptype) - .. deprecated:: 3.1 - Use :cfunc:`PyOS_double_to_string` instead. - - -.. cfunction:: char* PyOS_double_to_string(double val, char format_code, int precision, int flags, int *ptype) - - Convert a :ctype:`double` *val* to a string using supplied + Convert a :c:type:`double` *val* to a string using supplied *format_code*, *precision*, and *flags*. *format_code* must be one of ``'e'``, ``'E'``, ``'f'``, ``'F'``, @@ -134,7 +105,7 @@ The following functions provide locale-independent string to number conversions. like an integer. * *Py_DTSF_ALT* means to apply "alternate" formatting rules. See the - documentation for the :cfunc:`PyOS_snprintf` ``'#'`` specifier for + documentation for the :c:func:`PyOS_snprintf` ``'#'`` specifier for details. If *ptype* is non-NULL, then the value it points to will be set to one of @@ -143,28 +114,18 @@ The following functions provide locale-independent string to number conversions. The return value is a pointer to *buffer* with the converted string or *NULL* if the conversion failed. The caller is responsible for freeing the - returned string by calling :cfunc:`PyMem_Free`. + returned string by calling :c:func:`PyMem_Free`. .. versionadded:: 3.1 -.. cfunction:: double PyOS_ascii_atof(const char *nptr) - - Convert a string to a :ctype:`double` in a locale-independent way. - - See the Unix man page :manpage:`atof(2)` for details. - - .. deprecated:: 3.1 - Use :cfunc:`PyOS_string_to_double` instead. - - -.. cfunction:: char* PyOS_stricmp(char *s1, char *s2) +.. c:function:: char* PyOS_stricmp(char *s1, char *s2) Case insensitive comparison of strings. The function works almost - identically to :cfunc:`strcmp` except that it ignores the case. + identically to :c:func:`strcmp` except that it ignores the case. -.. cfunction:: char* PyOS_strnicmp(char *s1, char *s2, Py_ssize_t size) +.. c:function:: char* PyOS_strnicmp(char *s1, char *s2, Py_ssize_t size) Case insensitive comparison of strings. The function works almost - identically to :cfunc:`strncmp` except that it ignores the case. + identically to :c:func:`strncmp` except that it ignores the case. diff --git a/Doc/c-api/datetime.rst b/Doc/c-api/datetime.rst index 6515babbde..fcd13954a3 100644 --- a/Doc/c-api/datetime.rst +++ b/Doc/c-api/datetime.rst @@ -8,93 +8,93 @@ DateTime Objects Various date and time objects are supplied by the :mod:`datetime` module. Before using any of these functions, the header file :file:`datetime.h` must be included in your source (note that this is not included by :file:`Python.h`), -and the macro :cmacro:`PyDateTime_IMPORT` must be invoked, usually as part of +and the macro :c:macro:`PyDateTime_IMPORT` must be invoked, usually as part of the module initialisation function. The macro puts a pointer to a C structure -into a static variable, :cdata:`PyDateTimeAPI`, that is used by the following +into a static variable, :c:data:`PyDateTimeAPI`, that is used by the following macros. Type-check macros: -.. cfunction:: int PyDate_Check(PyObject *ob) +.. c:function:: int PyDate_Check(PyObject *ob) - Return true if *ob* is of type :cdata:`PyDateTime_DateType` or a subtype of - :cdata:`PyDateTime_DateType`. *ob* must not be *NULL*. + Return true if *ob* is of type :c:data:`PyDateTime_DateType` or a subtype of + :c:data:`PyDateTime_DateType`. *ob* must not be *NULL*. -.. cfunction:: int PyDate_CheckExact(PyObject *ob) +.. c:function:: int PyDate_CheckExact(PyObject *ob) - Return true if *ob* is of type :cdata:`PyDateTime_DateType`. *ob* must not be + Return true if *ob* is of type :c:data:`PyDateTime_DateType`. *ob* must not be *NULL*. -.. cfunction:: int PyDateTime_Check(PyObject *ob) +.. c:function:: int PyDateTime_Check(PyObject *ob) - Return true if *ob* is of type :cdata:`PyDateTime_DateTimeType` or a subtype of - :cdata:`PyDateTime_DateTimeType`. *ob* must not be *NULL*. + Return true if *ob* is of type :c:data:`PyDateTime_DateTimeType` or a subtype of + :c:data:`PyDateTime_DateTimeType`. *ob* must not be *NULL*. -.. cfunction:: int PyDateTime_CheckExact(PyObject *ob) +.. c:function:: int PyDateTime_CheckExact(PyObject *ob) - Return true if *ob* is of type :cdata:`PyDateTime_DateTimeType`. *ob* must not + Return true if *ob* is of type :c:data:`PyDateTime_DateTimeType`. *ob* must not be *NULL*. -.. cfunction:: int PyTime_Check(PyObject *ob) +.. c:function:: int PyTime_Check(PyObject *ob) - Return true if *ob* is of type :cdata:`PyDateTime_TimeType` or a subtype of - :cdata:`PyDateTime_TimeType`. *ob* must not be *NULL*. + Return true if *ob* is of type :c:data:`PyDateTime_TimeType` or a subtype of + :c:data:`PyDateTime_TimeType`. *ob* must not be *NULL*. -.. cfunction:: int PyTime_CheckExact(PyObject *ob) +.. c:function:: int PyTime_CheckExact(PyObject *ob) - Return true if *ob* is of type :cdata:`PyDateTime_TimeType`. *ob* must not be + Return true if *ob* is of type :c:data:`PyDateTime_TimeType`. *ob* must not be *NULL*. -.. cfunction:: int PyDelta_Check(PyObject *ob) +.. c:function:: int PyDelta_Check(PyObject *ob) - Return true if *ob* is of type :cdata:`PyDateTime_DeltaType` or a subtype of - :cdata:`PyDateTime_DeltaType`. *ob* must not be *NULL*. + Return true if *ob* is of type :c:data:`PyDateTime_DeltaType` or a subtype of + :c:data:`PyDateTime_DeltaType`. *ob* must not be *NULL*. -.. cfunction:: int PyDelta_CheckExact(PyObject *ob) +.. c:function:: int PyDelta_CheckExact(PyObject *ob) - Return true if *ob* is of type :cdata:`PyDateTime_DeltaType`. *ob* must not be + Return true if *ob* is of type :c:data:`PyDateTime_DeltaType`. *ob* must not be *NULL*. -.. cfunction:: int PyTZInfo_Check(PyObject *ob) +.. c:function:: int PyTZInfo_Check(PyObject *ob) - Return true if *ob* is of type :cdata:`PyDateTime_TZInfoType` or a subtype of - :cdata:`PyDateTime_TZInfoType`. *ob* must not be *NULL*. + Return true if *ob* is of type :c:data:`PyDateTime_TZInfoType` or a subtype of + :c:data:`PyDateTime_TZInfoType`. *ob* must not be *NULL*. -.. cfunction:: int PyTZInfo_CheckExact(PyObject *ob) +.. c:function:: int PyTZInfo_CheckExact(PyObject *ob) - Return true if *ob* is of type :cdata:`PyDateTime_TZInfoType`. *ob* must not be + Return true if *ob* is of type :c:data:`PyDateTime_TZInfoType`. *ob* must not be *NULL*. Macros to create objects: -.. cfunction:: PyObject* PyDate_FromDate(int year, int month, int day) +.. c:function:: PyObject* PyDate_FromDate(int year, int month, int day) Return a ``datetime.date`` object with the specified year, month and day. -.. cfunction:: PyObject* PyDateTime_FromDateAndTime(int year, int month, int day, int hour, int minute, int second, int usecond) +.. c:function:: PyObject* PyDateTime_FromDateAndTime(int year, int month, int day, int hour, int minute, int second, int usecond) Return a ``datetime.datetime`` object with the specified year, month, day, hour, minute, second and microsecond. -.. cfunction:: PyObject* PyTime_FromTime(int hour, int minute, int second, int usecond) +.. c:function:: PyObject* PyTime_FromTime(int hour, int minute, int second, int usecond) Return a ``datetime.time`` object with the specified hour, minute, second and microsecond. -.. cfunction:: PyObject* PyDelta_FromDSU(int days, int seconds, int useconds) +.. c:function:: PyObject* PyDelta_FromDSU(int days, int seconds, int useconds) Return a ``datetime.timedelta`` object representing the given number of days, seconds and microseconds. Normalization is performed so that the resulting @@ -103,82 +103,82 @@ Macros to create objects: Macros to extract fields from date objects. The argument must be an instance of -:cdata:`PyDateTime_Date`, including subclasses (such as -:cdata:`PyDateTime_DateTime`). The argument must not be *NULL*, and the type is +:c:data:`PyDateTime_Date`, including subclasses (such as +:c:data:`PyDateTime_DateTime`). The argument must not be *NULL*, and the type is not checked: -.. cfunction:: int PyDateTime_GET_YEAR(PyDateTime_Date *o) +.. c:function:: int PyDateTime_GET_YEAR(PyDateTime_Date *o) Return the year, as a positive int. -.. cfunction:: int PyDateTime_GET_MONTH(PyDateTime_Date *o) +.. c:function:: int PyDateTime_GET_MONTH(PyDateTime_Date *o) Return the month, as an int from 1 through 12. -.. cfunction:: int PyDateTime_GET_DAY(PyDateTime_Date *o) +.. c:function:: int PyDateTime_GET_DAY(PyDateTime_Date *o) Return the day, as an int from 1 through 31. Macros to extract fields from datetime objects. The argument must be an -instance of :cdata:`PyDateTime_DateTime`, including subclasses. The argument +instance of :c:data:`PyDateTime_DateTime`, including subclasses. The argument must not be *NULL*, and the type is not checked: -.. cfunction:: int PyDateTime_DATE_GET_HOUR(PyDateTime_DateTime *o) +.. c:function:: int PyDateTime_DATE_GET_HOUR(PyDateTime_DateTime *o) Return the hour, as an int from 0 through 23. -.. cfunction:: int PyDateTime_DATE_GET_MINUTE(PyDateTime_DateTime *o) +.. c:function:: int PyDateTime_DATE_GET_MINUTE(PyDateTime_DateTime *o) Return the minute, as an int from 0 through 59. -.. cfunction:: int PyDateTime_DATE_GET_SECOND(PyDateTime_DateTime *o) +.. c:function:: int PyDateTime_DATE_GET_SECOND(PyDateTime_DateTime *o) Return the second, as an int from 0 through 59. -.. cfunction:: int PyDateTime_DATE_GET_MICROSECOND(PyDateTime_DateTime *o) +.. c:function:: int PyDateTime_DATE_GET_MICROSECOND(PyDateTime_DateTime *o) Return the microsecond, as an int from 0 through 999999. Macros to extract fields from time objects. The argument must be an instance of -:cdata:`PyDateTime_Time`, including subclasses. The argument must not be *NULL*, +:c:data:`PyDateTime_Time`, including subclasses. The argument must not be *NULL*, and the type is not checked: -.. cfunction:: int PyDateTime_TIME_GET_HOUR(PyDateTime_Time *o) +.. c:function:: int PyDateTime_TIME_GET_HOUR(PyDateTime_Time *o) Return the hour, as an int from 0 through 23. -.. cfunction:: int PyDateTime_TIME_GET_MINUTE(PyDateTime_Time *o) +.. c:function:: int PyDateTime_TIME_GET_MINUTE(PyDateTime_Time *o) Return the minute, as an int from 0 through 59. -.. cfunction:: int PyDateTime_TIME_GET_SECOND(PyDateTime_Time *o) +.. c:function:: int PyDateTime_TIME_GET_SECOND(PyDateTime_Time *o) Return the second, as an int from 0 through 59. -.. cfunction:: int PyDateTime_TIME_GET_MICROSECOND(PyDateTime_Time *o) +.. c:function:: int PyDateTime_TIME_GET_MICROSECOND(PyDateTime_Time *o) Return the microsecond, as an int from 0 through 999999. Macros for the convenience of modules implementing the DB API: -.. cfunction:: PyObject* PyDateTime_FromTimestamp(PyObject *args) +.. c:function:: PyObject* PyDateTime_FromTimestamp(PyObject *args) Create and return a new ``datetime.datetime`` object given an argument tuple suitable for passing to ``datetime.datetime.fromtimestamp()``. -.. cfunction:: PyObject* PyDate_FromTimestamp(PyObject *args) +.. c:function:: PyObject* PyDate_FromTimestamp(PyObject *args) Create and return a new ``datetime.date`` object given an argument tuple suitable for passing to ``datetime.date.fromtimestamp()``. diff --git a/Doc/c-api/descriptor.rst b/Doc/c-api/descriptor.rst index 5db257072a..c8f6fa5bcd 100644 --- a/Doc/c-api/descriptor.rst +++ b/Doc/c-api/descriptor.rst @@ -10,31 +10,31 @@ found in the dictionary of type objects. .. XXX document these! -.. cvar:: PyTypeObject PyProperty_Type +.. c:var:: PyTypeObject PyProperty_Type The type object for the built-in descriptor types. -.. cfunction:: PyObject* PyDescr_NewGetSet(PyTypeObject *type, struct PyGetSetDef *getset) +.. c:function:: PyObject* PyDescr_NewGetSet(PyTypeObject *type, struct PyGetSetDef *getset) -.. cfunction:: PyObject* PyDescr_NewMember(PyTypeObject *type, struct PyMemberDef *meth) +.. c:function:: PyObject* PyDescr_NewMember(PyTypeObject *type, struct PyMemberDef *meth) -.. cfunction:: PyObject* PyDescr_NewMethod(PyTypeObject *type, struct PyMethodDef *meth) +.. c:function:: PyObject* PyDescr_NewMethod(PyTypeObject *type, struct PyMethodDef *meth) -.. cfunction:: PyObject* PyDescr_NewWrapper(PyTypeObject *type, struct wrapperbase *wrapper, void *wrapped) +.. c:function:: PyObject* PyDescr_NewWrapper(PyTypeObject *type, struct wrapperbase *wrapper, void *wrapped) -.. cfunction:: PyObject* PyDescr_NewClassMethod(PyTypeObject *type, PyMethodDef *method) +.. c:function:: PyObject* PyDescr_NewClassMethod(PyTypeObject *type, PyMethodDef *method) -.. cfunction:: int PyDescr_IsData(PyObject *descr) +.. c:function:: int PyDescr_IsData(PyObject *descr) Return true if the descriptor objects *descr* describes a data attribute, or false if it describes a method. *descr* must be a descriptor object; there is no error checking. -.. cfunction:: PyObject* PyWrapper_New(PyObject *, PyObject *) +.. c:function:: PyObject* PyWrapper_New(PyObject *, PyObject *) diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst index 5d355d1e61..6df84e075a 100644 --- a/Doc/c-api/dict.rst +++ b/Doc/c-api/dict.rst @@ -8,125 +8,125 @@ Dictionary Objects .. index:: object: dictionary -.. ctype:: PyDictObject +.. c:type:: PyDictObject - This subtype of :ctype:`PyObject` represents a Python dictionary object. + This subtype of :c:type:`PyObject` represents a Python dictionary object. -.. cvar:: PyTypeObject PyDict_Type +.. c:var:: PyTypeObject PyDict_Type - This instance of :ctype:`PyTypeObject` represents the Python dictionary + This instance of :c:type:`PyTypeObject` represents the Python dictionary type. This is the same object as :class:`dict` in the Python layer. -.. cfunction:: int PyDict_Check(PyObject *p) +.. c:function:: int PyDict_Check(PyObject *p) Return true if *p* is a dict object or an instance of a subtype of the dict type. -.. cfunction:: int PyDict_CheckExact(PyObject *p) +.. c:function:: int PyDict_CheckExact(PyObject *p) Return true if *p* is a dict object, but not an instance of a subtype of the dict type. -.. cfunction:: PyObject* PyDict_New() +.. c:function:: PyObject* PyDict_New() Return a new empty dictionary, or *NULL* on failure. -.. cfunction:: PyObject* PyDictProxy_New(PyObject *dict) +.. c:function:: PyObject* PyDictProxy_New(PyObject *dict) Return a proxy object for a mapping which enforces read-only behavior. This is normally used to create a proxy to prevent modification of the dictionary for non-dynamic class types. -.. cfunction:: void PyDict_Clear(PyObject *p) +.. c:function:: void PyDict_Clear(PyObject *p) Empty an existing dictionary of all key-value pairs. -.. cfunction:: int PyDict_Contains(PyObject *p, PyObject *key) +.. c:function:: int PyDict_Contains(PyObject *p, PyObject *key) Determine if dictionary *p* contains *key*. If an item in *p* is matches *key*, return ``1``, otherwise return ``0``. On error, return ``-1``. This is equivalent to the Python expression ``key in p``. -.. cfunction:: PyObject* PyDict_Copy(PyObject *p) +.. c:function:: PyObject* PyDict_Copy(PyObject *p) Return a new dictionary that contains the same key-value pairs as *p*. -.. cfunction:: int PyDict_SetItem(PyObject *p, PyObject *key, PyObject *val) +.. c:function:: int PyDict_SetItem(PyObject *p, PyObject *key, PyObject *val) Insert *value* into the dictionary *p* with a key of *key*. *key* must be :term:`hashable`; if it isn't, :exc:`TypeError` will be raised. Return ``0`` on success or ``-1`` on failure. -.. cfunction:: int PyDict_SetItemString(PyObject *p, const char *key, PyObject *val) +.. c:function:: int PyDict_SetItemString(PyObject *p, const char *key, PyObject *val) .. index:: single: PyUnicode_FromString() Insert *value* into the dictionary *p* using *key* as a key. *key* should - be a :ctype:`char\*`. The key object is created using + be a :c:type:`char\*`. The key object is created using ``PyUnicode_FromString(key)``. Return ``0`` on success or ``-1`` on failure. -.. cfunction:: int PyDict_DelItem(PyObject *p, PyObject *key) +.. c:function:: int PyDict_DelItem(PyObject *p, PyObject *key) Remove the entry in dictionary *p* with key *key*. *key* must be hashable; if it isn't, :exc:`TypeError` is raised. Return ``0`` on success or ``-1`` on failure. -.. cfunction:: int PyDict_DelItemString(PyObject *p, char *key) +.. c:function:: int PyDict_DelItemString(PyObject *p, char *key) Remove the entry in dictionary *p* which has a key specified by the string *key*. Return ``0`` on success or ``-1`` on failure. -.. cfunction:: PyObject* PyDict_GetItem(PyObject *p, PyObject *key) +.. c:function:: PyObject* PyDict_GetItem(PyObject *p, PyObject *key) Return the object from dictionary *p* which has a key *key*. Return *NULL* if the key *key* is not present, but *without* setting an exception. -.. cfunction:: PyObject* PyDict_GetItemWithError(PyObject *p, PyObject *key) +.. c:function:: PyObject* PyDict_GetItemWithError(PyObject *p, PyObject *key) - Variant of :cfunc:`PyDict_GetItem` that does not suppress + Variant of :c:func:`PyDict_GetItem` that does not suppress exceptions. Return *NULL* **with** an exception set if an exception occurred. Return *NULL* **without** an exception set if the key wasn't present. -.. cfunction:: PyObject* PyDict_GetItemString(PyObject *p, const char *key) +.. c:function:: PyObject* PyDict_GetItemString(PyObject *p, const char *key) - This is the same as :cfunc:`PyDict_GetItem`, but *key* is specified as a - :ctype:`char\*`, rather than a :ctype:`PyObject\*`. + This is the same as :c:func:`PyDict_GetItem`, but *key* is specified as a + :c:type:`char\*`, rather than a :c:type:`PyObject\*`. -.. cfunction:: PyObject* PyDict_Items(PyObject *p) +.. c:function:: PyObject* PyDict_Items(PyObject *p) - Return a :ctype:`PyListObject` containing all the items from the dictionary. + Return a :c:type:`PyListObject` containing all the items from the dictionary. -.. cfunction:: PyObject* PyDict_Keys(PyObject *p) +.. c:function:: PyObject* PyDict_Keys(PyObject *p) - Return a :ctype:`PyListObject` containing all the keys from the dictionary. + Return a :c:type:`PyListObject` containing all the keys from the dictionary. -.. cfunction:: PyObject* PyDict_Values(PyObject *p) +.. c:function:: PyObject* PyDict_Values(PyObject *p) - Return a :ctype:`PyListObject` containing all the values from the dictionary + Return a :c:type:`PyListObject` containing all the values from the dictionary *p*. -.. cfunction:: Py_ssize_t PyDict_Size(PyObject *p) +.. c:function:: Py_ssize_t PyDict_Size(PyObject *p) .. index:: builtin: len @@ -134,14 +134,14 @@ Dictionary Objects ``len(p)`` on a dictionary. -.. cfunction:: int PyDict_Next(PyObject *p, Py_ssize_t *ppos, PyObject **pkey, PyObject **pvalue) +.. c:function:: int PyDict_Next(PyObject *p, Py_ssize_t *ppos, PyObject **pkey, PyObject **pvalue) Iterate over all key-value pairs in the dictionary *p*. The - :ctype:`Py_ssize_t` referred to by *ppos* must be initialized to ``0`` + :c:type:`Py_ssize_t` referred to by *ppos* must be initialized to ``0`` prior to the first call to this function to start the iteration; the function returns true for each pair in the dictionary, and false once all pairs have been reported. The parameters *pkey* and *pvalue* should either - point to :ctype:`PyObject\*` variables that will be filled in with each key + point to :c:type:`PyObject\*` variables that will be filled in with each key and value, respectively, or may be *NULL*. Any references returned through them are borrowed. *ppos* should not be altered during iteration. Its value represents offsets within the internal dictionary structure, and @@ -180,23 +180,23 @@ Dictionary Objects } -.. cfunction:: int PyDict_Merge(PyObject *a, PyObject *b, int override) +.. c:function:: int PyDict_Merge(PyObject *a, PyObject *b, int override) Iterate over mapping object *b* adding key-value pairs to dictionary *a*. - *b* may be a dictionary, or any object supporting :func:`PyMapping_Keys` - and :func:`PyObject_GetItem`. If *override* is true, existing pairs in *a* + *b* may be a dictionary, or any object supporting :c:func:`PyMapping_Keys` + and :c:func:`PyObject_GetItem`. If *override* is true, existing pairs in *a* will be replaced if a matching key is found in *b*, otherwise pairs will only be added if there is not a matching key in *a*. Return ``0`` on success or ``-1`` if an exception was raised. -.. cfunction:: int PyDict_Update(PyObject *a, PyObject *b) +.. c:function:: int PyDict_Update(PyObject *a, PyObject *b) This is the same as ``PyDict_Merge(a, b, 1)`` in C, or ``a.update(b)`` in Python. Return ``0`` on success or ``-1`` if an exception was raised. -.. cfunction:: int PyDict_MergeFromSeq2(PyObject *a, PyObject *seq2, int override) +.. c:function:: int PyDict_MergeFromSeq2(PyObject *a, PyObject *seq2, int override) Update or merge into dictionary *a*, from the key-value pairs in *seq2*. *seq2* must be an iterable object producing iterable objects of length 2, diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index f465c5822c..6f13c8035a 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -9,12 +9,12 @@ Exception Handling The functions described in this chapter will let you handle and raise Python exceptions. It is important to understand some of the basics of Python -exception handling. It works somewhat like the Unix :cdata:`errno` variable: +exception handling. It works somewhat like the Unix :c:data:`errno` variable: there is a global indicator (per thread) of the last error that occurred. Most functions don't clear this on success, but will set it to indicate the cause of the error on failure. Most functions also return an error indicator, usually *NULL* if they are supposed to return a pointer, or ``-1`` if they return an -integer (exception: the :cfunc:`PyArg_\*` functions return ``1`` for success and +integer (exception: the :c:func:`PyArg_\*` functions return ``1`` for success and ``0`` for failure). When a function must fail because some function it called failed, it generally @@ -35,7 +35,7 @@ in various ways. There is a separate error indicator for each thread. Either alphabetical or some kind of structure. -.. cfunction:: void PyErr_PrintEx(int set_sys_last_vars) +.. c:function:: void PyErr_PrintEx(int set_sys_last_vars) Print a standard traceback to ``sys.stderr`` and clear the error indicator. Call this function only when the error indicator is set. (Otherwise it will @@ -46,35 +46,35 @@ in various ways. There is a separate error indicator for each thread. type, value and traceback of the printed exception, respectively. -.. cfunction:: void PyErr_Print() +.. c:function:: void PyErr_Print() Alias for ``PyErr_PrintEx(1)``. -.. cfunction:: PyObject* PyErr_Occurred() +.. c:function:: PyObject* PyErr_Occurred() Test whether the error indicator is set. If set, return the exception *type* - (the first argument to the last call to one of the :cfunc:`PyErr_Set\*` - functions or to :cfunc:`PyErr_Restore`). If not set, return *NULL*. You do not - own a reference to the return value, so you do not need to :cfunc:`Py_DECREF` + (the first argument to the last call to one of the :c:func:`PyErr_Set\*` + functions or to :c:func:`PyErr_Restore`). If not set, return *NULL*. You do not + own a reference to the return value, so you do not need to :c:func:`Py_DECREF` it. .. note:: Do not compare the return value to a specific exception; use - :cfunc:`PyErr_ExceptionMatches` instead, shown below. (The comparison could + :c:func:`PyErr_ExceptionMatches` instead, shown below. (The comparison could easily fail since the exception may be an instance instead of a class, in the case of a class exception, or it may the a subclass of the expected exception.) -.. cfunction:: int PyErr_ExceptionMatches(PyObject *exc) +.. c:function:: int PyErr_ExceptionMatches(PyObject *exc) Equivalent to ``PyErr_GivenExceptionMatches(PyErr_Occurred(), exc)``. This should only be called when an exception is actually set; a memory access violation will occur if no exception has been raised. -.. cfunction:: int PyErr_GivenExceptionMatches(PyObject *given, PyObject *exc) +.. c:function:: int PyErr_GivenExceptionMatches(PyObject *given, PyObject *exc) Return true if the *given* exception matches the exception in *exc*. If *exc* is a class object, this also returns true when *given* is an instance @@ -82,22 +82,22 @@ in various ways. There is a separate error indicator for each thread. recursively in subtuples) are searched for a match. -.. cfunction:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) +.. c:function:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) - Under certain circumstances, the values returned by :cfunc:`PyErr_Fetch` below + Under certain circumstances, the values returned by :c:func:`PyErr_Fetch` below can be "unnormalized", meaning that ``*exc`` is a class object but ``*val`` is not an instance of the same class. This function can be used to instantiate the class in that case. If the values are already normalized, nothing happens. The delayed normalization is implemented to improve performance. -.. cfunction:: void PyErr_Clear() +.. c:function:: void PyErr_Clear() Clear the error indicator. If the error indicator is not set, there is no effect. -.. cfunction:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) +.. c:function:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) Retrieve the error indicator into three variables whose addresses are passed. If the error indicator is not set, set all three variables to *NULL*. If it is @@ -110,7 +110,7 @@ in various ways. There is a separate error indicator for each thread. by code that needs to save and restore the error indicator temporarily. -.. cfunction:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback) +.. c:function:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback) Set the error indicator from the three objects. If the error indicator is already set, it is cleared first. If the objects are *NULL*, the error @@ -125,107 +125,129 @@ in various ways. There is a separate error indicator for each thread. .. note:: This function is normally only used by code that needs to save and restore the - error indicator temporarily; use :cfunc:`PyErr_Fetch` to save the current + error indicator temporarily; use :c:func:`PyErr_Fetch` to save the current exception state. -.. cfunction:: void PyErr_SetString(PyObject *type, const char *message) +.. c:function:: void PyErr_SetString(PyObject *type, const char *message) This is the most common way to set the error indicator. The first argument specifies the exception type; it is normally one of the standard exceptions, - e.g. :cdata:`PyExc_RuntimeError`. You need not increment its reference count. - The second argument is an error message; it is converted to a string object. + e.g. :c:data:`PyExc_RuntimeError`. You need not increment its reference count. + The second argument is an error message; it is decoded from ``'utf-8``'. -.. cfunction:: void PyErr_SetObject(PyObject *type, PyObject *value) +.. c:function:: void PyErr_SetObject(PyObject *type, PyObject *value) - This function is similar to :cfunc:`PyErr_SetString` but lets you specify an + This function is similar to :c:func:`PyErr_SetString` but lets you specify an arbitrary Python object for the "value" of the exception. -.. cfunction:: PyObject* PyErr_Format(PyObject *exception, const char *format, ...) +.. c:function:: PyObject* PyErr_Format(PyObject *exception, const char *format, ...) This function sets the error indicator and returns *NULL*. *exception* should be a Python exception class. The *format* and subsequent parameters help format the error message; they have the same meaning and - values as in :cfunc:`PyUnicode_FromFormat`. + values as in :c:func:`PyUnicode_FromFormat`. *format* is an ASCII-encoded + string. -.. cfunction:: void PyErr_SetNone(PyObject *type) +.. c:function:: void PyErr_SetNone(PyObject *type) This is a shorthand for ``PyErr_SetObject(type, Py_None)``. -.. cfunction:: int PyErr_BadArgument() +.. c:function:: int PyErr_BadArgument() This is a shorthand for ``PyErr_SetString(PyExc_TypeError, message)``, where *message* indicates that a built-in operation was invoked with an illegal argument. It is mostly for internal use. -.. cfunction:: PyObject* PyErr_NoMemory() +.. c:function:: PyObject* PyErr_NoMemory() This is a shorthand for ``PyErr_SetNone(PyExc_MemoryError)``; it returns *NULL* so an object allocation function can write ``return PyErr_NoMemory();`` when it runs out of memory. -.. cfunction:: PyObject* PyErr_SetFromErrno(PyObject *type) +.. c:function:: PyObject* PyErr_SetFromErrno(PyObject *type) .. index:: single: strerror() This is a convenience function to raise an exception when a C library function - has returned an error and set the C variable :cdata:`errno`. It constructs a - tuple object whose first item is the integer :cdata:`errno` value and whose - second item is the corresponding error message (gotten from :cfunc:`strerror`), + has returned an error and set the C variable :c:data:`errno`. It constructs a + tuple object whose first item is the integer :c:data:`errno` value and whose + second item is the corresponding error message (gotten from :c:func:`strerror`), and then calls ``PyErr_SetObject(type, object)``. On Unix, when the - :cdata:`errno` value is :const:`EINTR`, indicating an interrupted system call, - this calls :cfunc:`PyErr_CheckSignals`, and if that set the error indicator, + :c:data:`errno` value is :const:`EINTR`, indicating an interrupted system call, + this calls :c:func:`PyErr_CheckSignals`, and if that set the error indicator, leaves it set to that. The function always returns *NULL*, so a wrapper function around a system call can write ``return PyErr_SetFromErrno(type);`` when the system call returns an error. -.. cfunction:: PyObject* PyErr_SetFromErrnoWithFilename(PyObject *type, const char *filename) +.. c:function:: PyObject* PyErr_SetFromErrnoWithFilename(PyObject *type, const char *filename) - Similar to :cfunc:`PyErr_SetFromErrno`, with the additional behavior that if + Similar to :c:func:`PyErr_SetFromErrno`, with the additional behavior that if *filename* is not *NULL*, it is passed to the constructor of *type* as a third parameter. In the case of exceptions such as :exc:`IOError` and :exc:`OSError`, this is used to define the :attr:`filename` attribute of the exception instance. + *filename* is decoded from the filesystem encoding + (:func:`sys.getfilesystemencoding`). -.. cfunction:: PyObject* PyErr_SetFromWindowsErr(int ierr) +.. c:function:: PyObject* PyErr_SetFromWindowsErr(int ierr) This is a convenience function to raise :exc:`WindowsError`. If called with - *ierr* of :cdata:`0`, the error code returned by a call to :cfunc:`GetLastError` - is used instead. It calls the Win32 function :cfunc:`FormatMessage` to retrieve - the Windows description of error code given by *ierr* or :cfunc:`GetLastError`, + *ierr* of :c:data:`0`, the error code returned by a call to :c:func:`GetLastError` + is used instead. It calls the Win32 function :c:func:`FormatMessage` to retrieve + the Windows description of error code given by *ierr* or :c:func:`GetLastError`, then it constructs a tuple object whose first item is the *ierr* value and whose second item is the corresponding error message (gotten from - :cfunc:`FormatMessage`), and then calls ``PyErr_SetObject(PyExc_WindowsError, + :c:func:`FormatMessage`), and then calls ``PyErr_SetObject(PyExc_WindowsError, object)``. This function always returns *NULL*. Availability: Windows. -.. cfunction:: PyObject* PyErr_SetExcFromWindowsErr(PyObject *type, int ierr) +.. c:function:: PyObject* PyErr_SetExcFromWindowsErr(PyObject *type, int ierr) - Similar to :cfunc:`PyErr_SetFromWindowsErr`, with an additional parameter + Similar to :c:func:`PyErr_SetFromWindowsErr`, with an additional parameter specifying the exception type to be raised. Availability: Windows. -.. cfunction:: PyObject* PyErr_SetFromWindowsErrWithFilename(int ierr, const char *filename) +.. c:function:: PyObject* PyErr_SetFromWindowsErrWithFilename(int ierr, const char *filename) - Similar to :cfunc:`PyErr_SetFromWindowsErr`, with the additional behavior that + Similar to :c:func:`PyErr_SetFromWindowsErr`, with the additional behavior that if *filename* is not *NULL*, it is passed to the constructor of - :exc:`WindowsError` as a third parameter. Availability: Windows. + :exc:`WindowsError` as a third parameter. *filename* is decoded from the + filesystem encoding (:func:`sys.getfilesystemencoding`). Availability: + Windows. -.. cfunction:: PyObject* PyErr_SetExcFromWindowsErrWithFilename(PyObject *type, int ierr, char *filename) +.. c:function:: PyObject* PyErr_SetExcFromWindowsErrWithFilename(PyObject *type, int ierr, char *filename) - Similar to :cfunc:`PyErr_SetFromWindowsErrWithFilename`, with an additional + Similar to :c:func:`PyErr_SetFromWindowsErrWithFilename`, with an additional parameter specifying the exception type to be raised. Availability: Windows. -.. cfunction:: void PyErr_BadInternalCall() +.. c:function:: void PyErr_SyntaxLocationEx(char *filename, int lineno, int col_offset) + + Set file, line, and offset information for the current exception. If the + current exception is not a :exc:`SyntaxError`, then it sets additional + attributes, which make the exception printing subsystem think the exception + is a :exc:`SyntaxError`. *filename* is decoded from the filesystem encoding + (:func:`sys.getfilesystemencoding`). + +.. versionadded:: 3.2 + + +.. c:function:: void PyErr_SyntaxLocation(char *filename, int lineno) + + Like :c:func:`PyErr_SyntaxLocationExc`, but the col_offset parameter is + omitted. + + +.. c:function:: void PyErr_BadInternalCall() This is a shorthand for ``PyErr_SetString(PyExc_SystemError, message)``, where *message* indicates that an internal operation (e.g. a Python/C API @@ -233,13 +255,13 @@ in various ways. There is a separate error indicator for each thread. use. -.. cfunction:: int PyErr_WarnEx(PyObject *category, char *message, int stacklevel) +.. c:function:: int PyErr_WarnEx(PyObject *category, char *message, int stack_level) Issue a warning message. The *category* argument is a warning category (see - below) or *NULL*; the *message* argument is a message string. *stacklevel* is a + below) or *NULL*; the *message* argument is an UTF-8 encoded string. *stack_level* is a positive number giving a number of stack frames; the warning will be issued from - the currently executing line of code in that stack frame. A *stacklevel* of 1 - is the function calling :cfunc:`PyErr_WarnEx`, 2 is the function above that, + the currently executing line of code in that stack frame. A *stack_level* of 1 + is the function calling :c:func:`PyErr_WarnEx`, 2 is the function above that, and so forth. This function normally prints a warning message to *sys.stderr*; however, it is @@ -251,35 +273,45 @@ in various ways. There is a separate error indicator for each thread. is raised. (It is not possible to determine whether a warning message is actually printed, nor what the reason is for the exception; this is intentional.) If an exception is raised, the caller should do its normal - exception handling (for example, :cfunc:`Py_DECREF` owned references and return + exception handling (for example, :c:func:`Py_DECREF` owned references and return an error value). - Warning categories must be subclasses of :cdata:`Warning`; the default warning - category is :cdata:`RuntimeWarning`. The standard Python warning categories are + Warning categories must be subclasses of :c:data:`Warning`; the default warning + category is :c:data:`RuntimeWarning`. The standard Python warning categories are available as global variables whose names are ``PyExc_`` followed by the Python - exception name. These have the type :ctype:`PyObject\*`; they are all class - objects. Their names are :cdata:`PyExc_Warning`, :cdata:`PyExc_UserWarning`, - :cdata:`PyExc_UnicodeWarning`, :cdata:`PyExc_DeprecationWarning`, - :cdata:`PyExc_SyntaxWarning`, :cdata:`PyExc_RuntimeWarning`, and - :cdata:`PyExc_FutureWarning`. :cdata:`PyExc_Warning` is a subclass of - :cdata:`PyExc_Exception`; the other warning categories are subclasses of - :cdata:`PyExc_Warning`. + exception name. These have the type :c:type:`PyObject\*`; they are all class + objects. Their names are :c:data:`PyExc_Warning`, :c:data:`PyExc_UserWarning`, + :c:data:`PyExc_UnicodeWarning`, :c:data:`PyExc_DeprecationWarning`, + :c:data:`PyExc_SyntaxWarning`, :c:data:`PyExc_RuntimeWarning`, and + :c:data:`PyExc_FutureWarning`. :c:data:`PyExc_Warning` is a subclass of + :c:data:`PyExc_Exception`; the other warning categories are subclasses of + :c:data:`PyExc_Warning`. For information about warning control, see the documentation for the :mod:`warnings` module and the :option:`-W` option in the command line documentation. There is no C API for warning control. -.. cfunction:: int PyErr_WarnExplicit(PyObject *category, const char *message, const char *filename, int lineno, const char *module, PyObject *registry) +.. c:function:: int PyErr_WarnExplicit(PyObject *category, const char *message, const char *filename, int lineno, const char *module, PyObject *registry) Issue a warning message with explicit control over all warning attributes. This is a straightforward wrapper around the Python function :func:`warnings.warn_explicit`, see there for more information. The *module* and *registry* arguments may be set to *NULL* to get the default effect - described there. + described there. *message* and *module* are UTF-8 encoded strings, + *filename* is decoded from the filesystem encoding + (:func:`sys.getfilesystemencoding`). + + +.. c:function:: int PyErr_WarnFormat(PyObject *category, Py_ssize_t stack_level, const char *format, ...) + Function similar to :c:func:`PyErr_WarnEx`, but use + :c:func:`PyUnicode_FromFormat` to format the warning message. *format* is + an ASCII-encoded string. -.. cfunction:: int PyErr_CheckSignals() + .. versionadded:: 3.2 + +.. c:function:: int PyErr_CheckSignals() .. index:: module: signal @@ -296,21 +328,21 @@ in various ways. There is a separate error indicator for each thread. cleared if it was previously set. -.. cfunction:: void PyErr_SetInterrupt() +.. c:function:: void PyErr_SetInterrupt() .. index:: single: SIGINT single: KeyboardInterrupt (built-in exception) This function simulates the effect of a :const:`SIGINT` signal arriving --- the - next time :cfunc:`PyErr_CheckSignals` is called, :exc:`KeyboardInterrupt` will + next time :c:func:`PyErr_CheckSignals` is called, :exc:`KeyboardInterrupt` will be raised. It may be called without holding the interpreter lock. .. % XXX This was described as obsolete, but is used in .. % _thread.interrupt_main() (used from IDLE), so it's still needed. -.. cfunction:: int PySignal_SetWakeupFd(int fd) +.. c:function:: int PySignal_SetWakeupFd(int fd) This utility function specifies a file descriptor to which a ``'\0'`` byte will be written whenever a signal is received. It returns the previous such file @@ -320,13 +352,13 @@ in various ways. There is a separate error indicator for each thread. only be called from the main thread. -.. cfunction:: PyObject* PyErr_NewException(char *name, PyObject *base, PyObject *dict) +.. c:function:: PyObject* PyErr_NewException(char *name, PyObject *base, PyObject *dict) - This utility function creates and returns a new exception object. The *name* + This utility function creates and returns a new exception class. The *name* argument must be the name of the new exception, a C string of the form - ``module.class``. The *base* and *dict* arguments are normally *NULL*. This - creates a class object derived from :exc:`Exception` (accessible in C as - :cdata:`PyExc_Exception`). + ``module.classname``. The *base* and *dict* arguments are normally *NULL*. + This creates a class object derived from :exc:`Exception` (accessible in C as + :c:data:`PyExc_Exception`). The :attr:`__module__` attribute of the new class is set to the first part (up to the last dot) of the *name* argument, and the class name is set to the last @@ -335,7 +367,16 @@ in various ways. There is a separate error indicator for each thread. argument can be used to specify a dictionary of class variables and methods. -.. cfunction:: void PyErr_WriteUnraisable(PyObject *obj) +.. c:function:: PyObject* PyErr_NewExceptionWithDoc(char *name, char *doc, PyObject *base, PyObject *dict) + + Same as :c:func:`PyErr_NewException`, except that the new exception class can + easily be given a docstring: If *doc* is non-*NULL*, it will be used as the + docstring for the exception class. + + .. versionadded:: 3.2 + + +.. c:function:: void PyErr_WriteUnraisable(PyObject *obj) This utility function prints a warning message to ``sys.stderr`` when an exception has been set but it is impossible for the interpreter to actually @@ -350,20 +391,20 @@ in various ways. There is a separate error indicator for each thread. Exception Objects ================= -.. cfunction:: PyObject* PyException_GetTraceback(PyObject *ex) +.. c:function:: PyObject* PyException_GetTraceback(PyObject *ex) Return the traceback associated with the exception as a new reference, as accessible from Python through :attr:`__traceback__`. If there is no traceback associated, this returns *NULL*. -.. cfunction:: int PyException_SetTraceback(PyObject *ex, PyObject *tb) +.. c:function:: int PyException_SetTraceback(PyObject *ex, PyObject *tb) Set the traceback associated with the exception to *tb*. Use ``Py_None`` to clear it. -.. cfunction:: PyObject* PyException_GetContext(PyObject *ex) +.. c:function:: PyObject* PyException_GetContext(PyObject *ex) Return the context (another exception instance during whose handling *ex* was raised) associated with the exception as a new reference, as accessible from @@ -371,14 +412,14 @@ Exception Objects returns *NULL*. -.. cfunction:: void PyException_SetContext(PyObject *ex, PyObject *ctx) +.. c:function:: void PyException_SetContext(PyObject *ex, PyObject *ctx) Set the context associated with the exception to *ctx*. Use *NULL* to clear it. There is no type check to make sure that *ctx* is an exception instance. This steals a reference to *ctx*. -.. cfunction:: PyObject* PyException_GetCause(PyObject *ex) +.. c:function:: PyObject* PyException_GetCause(PyObject *ex) Return the cause (another exception instance set by ``raise ... from ...``) associated with the exception as a new reference, as accessible from Python @@ -386,7 +427,7 @@ Exception Objects *NULL*. -.. cfunction:: void PyException_SetCause(PyObject *ex, PyObject *ctx) +.. c:function:: void PyException_SetCause(PyObject *ex, PyObject *ctx) Set the cause associated with the exception to *ctx*. Use *NULL* to clear it. There is no type check to make sure that *ctx* is an exception instance. @@ -400,71 +441,73 @@ Unicode Exception Objects The following functions are used to create and modify Unicode exceptions from C. -.. cfunction:: PyObject* PyUnicodeDecodeError_Create(const char *encoding, const char *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) +.. c:function:: PyObject* PyUnicodeDecodeError_Create(const char *encoding, const char *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) Create a :class:`UnicodeDecodeError` object with the attributes *encoding*, - *object*, *length*, *start*, *end* and *reason*. + *object*, *length*, *start*, *end* and *reason*. *encoding* and *reason* are + UTF-8 encoded strings. -.. cfunction:: PyObject* PyUnicodeEncodeError_Create(const char *encoding, const Py_UNICODE *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) +.. c:function:: PyObject* PyUnicodeEncodeError_Create(const char *encoding, const Py_UNICODE *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) Create a :class:`UnicodeEncodeError` object with the attributes *encoding*, - *object*, *length*, *start*, *end* and *reason*. + *object*, *length*, *start*, *end* and *reason*. *encoding* and *reason* are + UTF-8 encoded strings. -.. cfunction:: PyObject* PyUnicodeTranslateError_Create(const Py_UNICODE *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) +.. c:function:: PyObject* PyUnicodeTranslateError_Create(const Py_UNICODE *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) Create a :class:`UnicodeTranslateError` object with the attributes *object*, - *length*, *start*, *end* and *reason*. + *length*, *start*, *end* and *reason*. *reason* is an UTF-8 encoded string. -.. cfunction:: PyObject* PyUnicodeDecodeError_GetEncoding(PyObject *exc) - PyObject* PyUnicodeEncodeError_GetEncoding(PyObject *exc) +.. c:function:: PyObject* PyUnicodeDecodeError_GetEncoding(PyObject *exc) + PyObject* PyUnicodeEncodeError_GetEncoding(PyObject *exc) Return the *encoding* attribute of the given exception object. -.. cfunction:: PyObject* PyUnicodeDecodeError_GetObject(PyObject *exc) - PyObject* PyUnicodeEncodeError_GetObject(PyObject *exc) - PyObject* PyUnicodeTranslateError_GetObject(PyObject *exc) +.. c:function:: PyObject* PyUnicodeDecodeError_GetObject(PyObject *exc) + PyObject* PyUnicodeEncodeError_GetObject(PyObject *exc) + PyObject* PyUnicodeTranslateError_GetObject(PyObject *exc) Return the *object* attribute of the given exception object. -.. cfunction:: int PyUnicodeDecodeError_GetStart(PyObject *exc, Py_ssize_t *start) - int PyUnicodeEncodeError_GetStart(PyObject *exc, Py_ssize_t *start) - int PyUnicodeTranslateError_GetStart(PyObject *exc, Py_ssize_t *start) +.. c:function:: int PyUnicodeDecodeError_GetStart(PyObject *exc, Py_ssize_t *start) + int PyUnicodeEncodeError_GetStart(PyObject *exc, Py_ssize_t *start) + int PyUnicodeTranslateError_GetStart(PyObject *exc, Py_ssize_t *start) Get the *start* attribute of the given exception object and place it into *\*start*. *start* must not be *NULL*. Return ``0`` on success, ``-1`` on failure. -.. cfunction:: int PyUnicodeDecodeError_SetStart(PyObject *exc, Py_ssize_t start) - int PyUnicodeEncodeError_SetStart(PyObject *exc, Py_ssize_t start) - int PyUnicodeTranslateError_SetStart(PyObject *exc, Py_ssize_t start) +.. c:function:: int PyUnicodeDecodeError_SetStart(PyObject *exc, Py_ssize_t start) + int PyUnicodeEncodeError_SetStart(PyObject *exc, Py_ssize_t start) + int PyUnicodeTranslateError_SetStart(PyObject *exc, Py_ssize_t start) Set the *start* attribute of the given exception object to *start*. Return ``0`` on success, ``-1`` on failure. -.. cfunction:: int PyUnicodeDecodeError_GetEnd(PyObject *exc, Py_ssize_t *end) - int PyUnicodeEncodeError_GetEnd(PyObject *exc, Py_ssize_t *end) - int PyUnicodeTranslateError_GetEnd(PyObject *exc, Py_ssize_t *end) +.. c:function:: int PyUnicodeDecodeError_GetEnd(PyObject *exc, Py_ssize_t *end) + int PyUnicodeEncodeError_GetEnd(PyObject *exc, Py_ssize_t *end) + int PyUnicodeTranslateError_GetEnd(PyObject *exc, Py_ssize_t *end) Get the *end* attribute of the given exception object and place it into *\*end*. *end* must not be *NULL*. Return ``0`` on success, ``-1`` on failure. -.. cfunction:: int PyUnicodeDecodeError_SetEnd(PyObject *exc, Py_ssize_t end) - int PyUnicodeEncodeError_SetEnd(PyObject *exc, Py_ssize_t end) - int PyUnicodeTranslateError_SetEnd(PyObject *exc, Py_ssize_t end) +.. c:function:: int PyUnicodeDecodeError_SetEnd(PyObject *exc, Py_ssize_t end) + int PyUnicodeEncodeError_SetEnd(PyObject *exc, Py_ssize_t end) + int PyUnicodeTranslateError_SetEnd(PyObject *exc, Py_ssize_t end) Set the *end* attribute of the given exception object to *end*. Return ``0`` on success, ``-1`` on failure. -.. cfunction:: PyObject* PyUnicodeDecodeError_GetReason(PyObject *exc) - PyObject* PyUnicodeEncodeError_GetReason(PyObject *exc) - PyObject* PyUnicodeTranslateError_GetReason(PyObject *exc) +.. c:function:: PyObject* PyUnicodeDecodeError_GetReason(PyObject *exc) + PyObject* PyUnicodeEncodeError_GetReason(PyObject *exc) + PyObject* PyUnicodeTranslateError_GetReason(PyObject *exc) Return the *reason* attribute of the given exception object. -.. cfunction:: int PyUnicodeDecodeError_SetReason(PyObject *exc, const char *reason) - int PyUnicodeEncodeError_SetReason(PyObject *exc, const char *reason) - int PyUnicodeTranslateError_SetReason(PyObject *exc, const char *reason) +.. c:function:: int PyUnicodeDecodeError_SetReason(PyObject *exc, const char *reason) + int PyUnicodeEncodeError_SetReason(PyObject *exc, const char *reason) + int PyUnicodeTranslateError_SetReason(PyObject *exc, const char *reason) Set the *reason* attribute of the given exception object to *reason*. Return ``0`` on success, ``-1`` on failure. @@ -478,12 +521,12 @@ level, both in the core and in extension modules. They are needed if the recursive code does not necessarily invoke Python code (which tracks its recursion depth automatically). -.. cfunction:: int Py_EnterRecursiveCall(char *where) +.. c:function:: int Py_EnterRecursiveCall(char *where) Marks a point where a recursive C-level call is about to be performed. If :const:`USE_STACKCHECK` is defined, this function checks if the the OS - stack overflowed using :cfunc:`PyOS_CheckStack`. In this is the case, it + stack overflowed using :c:func:`PyOS_CheckStack`. In this is the case, it sets a :exc:`MemoryError` and returns a nonzero value. The function then checks if the recursion limit is reached. If this is the @@ -494,10 +537,39 @@ recursion depth automatically). concatenated to the :exc:`RuntimeError` message caused by the recursion depth limit. -.. cfunction:: void Py_LeaveRecursiveCall() +.. c:function:: void Py_LeaveRecursiveCall() + + Ends a :c:func:`Py_EnterRecursiveCall`. Must be called once for each + *successful* invocation of :c:func:`Py_EnterRecursiveCall`. + +Properly implementing :attr:`tp_repr` for container types requires +special recursion handling. In addition to protecting the stack, +:attr:`tp_repr` also needs to track objects to prevent cycles. The +following two functions facilitate this functionality. Effectively, +these are the C equivalent to :func:`reprlib.recursive_repr`. + +.. c:function:: int Py_ReprEnter(PyObject *object) + + Called at the beginning of the :attr:`tp_repr` implementation to + detect cycles. + + If the object has already been processed, the function returns a + positive integer. In that case the :attr:`tp_repr` implementation + should return a string object indicating a cycle. As examples, + :class:`dict` objects return ``{...}`` and :class:`list` objects + return ``[...]``. + + The function will return a negative integer if the recursion limit + is reached. In that case the :attr:`tp_repr` implementation should + typically return ``NULL``. + + Otherwise, the function returns zero and the :attr:`tp_repr` + implementation can continue normally. + +.. c:function:: void Py_ReprLeave(PyObject *object) - Ends a :cfunc:`Py_EnterRecursiveCall`. Must be called once for each - *successful* invocation of :cfunc:`Py_EnterRecursiveCall`. + Ends a :c:func:`Py_ReprEnter`. Must be called once for each + invocation of :c:func:`Py_ReprEnter` that returns zero. .. _standardexceptions: @@ -507,68 +579,68 @@ Standard Exceptions All standard Python exceptions are available as global variables whose names are ``PyExc_`` followed by the Python exception name. These have the type -:ctype:`PyObject\*`; they are all class objects. For completeness, here are all +:c:type:`PyObject\*`; they are all class objects. For completeness, here are all the variables: -+------------------------------------+----------------------------+----------+ -| C Name | Python Name | Notes | -+====================================+============================+==========+ -| :cdata:`PyExc_BaseException` | :exc:`BaseException` | \(1) | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_Exception` | :exc:`Exception` | \(1) | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_ArithmeticError` | :exc:`ArithmeticError` | \(1) | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_LookupError` | :exc:`LookupError` | \(1) | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_AssertionError` | :exc:`AssertionError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_AttributeError` | :exc:`AttributeError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_EOFError` | :exc:`EOFError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_EnvironmentError` | :exc:`EnvironmentError` | \(1) | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_FloatingPointError` | :exc:`FloatingPointError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_IOError` | :exc:`IOError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_ImportError` | :exc:`ImportError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_IndexError` | :exc:`IndexError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_KeyError` | :exc:`KeyError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_KeyboardInterrupt` | :exc:`KeyboardInterrupt` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_MemoryError` | :exc:`MemoryError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_NameError` | :exc:`NameError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_NotImplementedError` | :exc:`NotImplementedError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_OSError` | :exc:`OSError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_OverflowError` | :exc:`OverflowError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_ReferenceError` | :exc:`ReferenceError` | \(2) | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_RuntimeError` | :exc:`RuntimeError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_SyntaxError` | :exc:`SyntaxError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_SystemError` | :exc:`SystemError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_SystemExit` | :exc:`SystemExit` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_TypeError` | :exc:`TypeError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_ValueError` | :exc:`ValueError` | | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_WindowsError` | :exc:`WindowsError` | \(3) | -+------------------------------------+----------------------------+----------+ -| :cdata:`PyExc_ZeroDivisionError` | :exc:`ZeroDivisionError` | | -+------------------------------------+----------------------------+----------+ ++-------------------------------------+----------------------------+----------+ +| C Name | Python Name | Notes | ++=====================================+============================+==========+ +| :c:data:`PyExc_BaseException` | :exc:`BaseException` | \(1) | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_Exception` | :exc:`Exception` | \(1) | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_ArithmeticError` | :exc:`ArithmeticError` | \(1) | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_LookupError` | :exc:`LookupError` | \(1) | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_AssertionError` | :exc:`AssertionError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_AttributeError` | :exc:`AttributeError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_EOFError` | :exc:`EOFError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_EnvironmentError` | :exc:`EnvironmentError` | \(1) | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_FloatingPointError` | :exc:`FloatingPointError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_IOError` | :exc:`IOError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_ImportError` | :exc:`ImportError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_IndexError` | :exc:`IndexError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_KeyError` | :exc:`KeyError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_KeyboardInterrupt` | :exc:`KeyboardInterrupt` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_MemoryError` | :exc:`MemoryError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_NameError` | :exc:`NameError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_NotImplementedError` | :exc:`NotImplementedError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_OSError` | :exc:`OSError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_OverflowError` | :exc:`OverflowError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_ReferenceError` | :exc:`ReferenceError` | \(2) | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_RuntimeError` | :exc:`RuntimeError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_SyntaxError` | :exc:`SyntaxError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_SystemError` | :exc:`SystemError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_SystemExit` | :exc:`SystemExit` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_TypeError` | :exc:`TypeError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_ValueError` | :exc:`ValueError` | | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_WindowsError` | :exc:`WindowsError` | \(3) | ++-------------------------------------+----------------------------+----------+ +| :c:data:`PyExc_ZeroDivisionError` | :exc:`ZeroDivisionError` | | ++-------------------------------------+----------------------------+----------+ .. index:: single: PyExc_BaseException diff --git a/Doc/c-api/file.rst b/Doc/c-api/file.rst index cc851e6be8..c5a4a594b2 100644 --- a/Doc/c-api/file.rst +++ b/Doc/c-api/file.rst @@ -8,7 +8,7 @@ File Objects .. index:: object: file These APIs are a minimal emulation of the Python 2 C API for built-in file -objects, which used to rely on the buffered I/O (:ctype:`FILE\*`) support +objects, which used to rely on the buffered I/O (:c:type:`FILE\*`) support from the C standard library. In Python 3, files and streams use the new :mod:`io` module, which defines several layers over the low-level unbuffered I/O of the operating system. The functions described below are @@ -17,13 +17,14 @@ error reporting in the interpreter; third-party code is advised to access the :mod:`io` APIs instead. -.. cfunction:: PyFile_FromFd(int fd, char *name, char *mode, int buffering, char *encoding, char *errors, char *newline, int closefd) +.. c:function:: PyFile_FromFd(int fd, char *name, char *mode, int buffering, char *encoding, char *errors, char *newline, int closefd) Create a Python file object from the file descriptor of an already opened file *fd*. The arguments *name*, *encoding*, *errors* and *newline* can be *NULL* to use the defaults; *buffering* can be *-1* to use the - default. Return *NULL* on failure. For a more comprehensive description of - the arguments, please refer to the :func:`io.open` function documentation. + default. *name* is ignored and kept for backward compatibility. Return + *NULL* on failure. For a more comprehensive description of the arguments, + please refer to the :func:`io.open` function documentation. .. warning:: @@ -31,17 +32,20 @@ the :mod:`io` APIs instead. OS-level file descriptors can produce various issues (such as unexpected ordering of data). + .. versionchanged:: 3.2 + Ignore *name* attribute. -.. cfunction:: int PyObject_AsFileDescriptor(PyObject *p) - Return the file descriptor associated with *p* as an :ctype:`int`. If the +.. c:function:: int PyObject_AsFileDescriptor(PyObject *p) + + Return the file descriptor associated with *p* as an :c:type:`int`. If the object is an integer, its value is returned. If not, the object's :meth:`fileno` method is called if it exists; the method must return an integer, which is returned as the file descriptor value. Sets an exception and returns ``-1`` on failure. -.. cfunction:: PyObject* PyFile_GetLine(PyObject *p, int n) +.. c:function:: PyObject* PyFile_GetLine(PyObject *p, int n) .. index:: single: EOFError (built-in exception) @@ -55,7 +59,7 @@ the :mod:`io` APIs instead. raised if the end of the file is reached immediately. -.. cfunction:: int PyFile_WriteObject(PyObject *obj, PyObject *p, int flags) +.. c:function:: int PyFile_WriteObject(PyObject *obj, PyObject *p, int flags) .. index:: single: Py_PRINT_RAW @@ -65,7 +69,7 @@ the :mod:`io` APIs instead. appropriate exception will be set. -.. cfunction:: int PyFile_WriteString(const char *s, PyObject *p) +.. c:function:: int PyFile_WriteString(const char *s, PyObject *p) Write string *s* to file object *p*. Return ``0`` on success or ``-1`` on failure; the appropriate exception will be set. diff --git a/Doc/c-api/float.rst b/Doc/c-api/float.rst index 93769aa1f4..27a75e3e0c 100644 --- a/Doc/c-api/float.rst +++ b/Doc/c-api/float.rst @@ -8,70 +8,72 @@ Floating Point Objects .. index:: object: floating point -.. ctype:: PyFloatObject +.. c:type:: PyFloatObject - This subtype of :ctype:`PyObject` represents a Python floating point object. + This subtype of :c:type:`PyObject` represents a Python floating point object. -.. cvar:: PyTypeObject PyFloat_Type +.. c:var:: PyTypeObject PyFloat_Type - This instance of :ctype:`PyTypeObject` represents the Python floating point + This instance of :c:type:`PyTypeObject` represents the Python floating point type. This is the same object as :class:`float` in the Python layer. -.. cfunction:: int PyFloat_Check(PyObject *p) +.. c:function:: int PyFloat_Check(PyObject *p) - Return true if its argument is a :ctype:`PyFloatObject` or a subtype of - :ctype:`PyFloatObject`. + Return true if its argument is a :c:type:`PyFloatObject` or a subtype of + :c:type:`PyFloatObject`. -.. cfunction:: int PyFloat_CheckExact(PyObject *p) +.. c:function:: int PyFloat_CheckExact(PyObject *p) - Return true if its argument is a :ctype:`PyFloatObject`, but not a subtype of - :ctype:`PyFloatObject`. + Return true if its argument is a :c:type:`PyFloatObject`, but not a subtype of + :c:type:`PyFloatObject`. -.. cfunction:: PyObject* PyFloat_FromString(PyObject *str) +.. c:function:: PyObject* PyFloat_FromString(PyObject *str) - Create a :ctype:`PyFloatObject` object based on the string value in *str*, or + Create a :c:type:`PyFloatObject` object based on the string value in *str*, or *NULL* on failure. -.. cfunction:: PyObject* PyFloat_FromDouble(double v) +.. c:function:: PyObject* PyFloat_FromDouble(double v) - Create a :ctype:`PyFloatObject` object from *v*, or *NULL* on failure. + Create a :c:type:`PyFloatObject` object from *v*, or *NULL* on failure. -.. cfunction:: double PyFloat_AsDouble(PyObject *pyfloat) +.. c:function:: double PyFloat_AsDouble(PyObject *pyfloat) - Return a C :ctype:`double` representation of the contents of *pyfloat*. If + Return a C :c:type:`double` representation of the contents of *pyfloat*. If *pyfloat* is not a Python floating point object but has a :meth:`__float__` method, this method will first be called to convert *pyfloat* into a float. + This method returns ``-1.0`` upon failure, so one should call + :c:func:`PyErr_Occurred` to check for errors. -.. cfunction:: double PyFloat_AS_DOUBLE(PyObject *pyfloat) +.. c:function:: double PyFloat_AS_DOUBLE(PyObject *pyfloat) - Return a C :ctype:`double` representation of the contents of *pyfloat*, but + Return a C :c:type:`double` representation of the contents of *pyfloat*, but without error checking. -.. cfunction:: PyObject* PyFloat_GetInfo(void) +.. c:function:: PyObject* PyFloat_GetInfo(void) Return a structseq instance which contains information about the precision, minimum and maximum values of a float. It's a thin wrapper around the header file :file:`float.h`. -.. cfunction:: double PyFloat_GetMax() +.. c:function:: double PyFloat_GetMax() - Return the maximum representable finite float *DBL_MAX* as C :ctype:`double`. + Return the maximum representable finite float *DBL_MAX* as C :c:type:`double`. -.. cfunction:: double PyFloat_GetMin() +.. c:function:: double PyFloat_GetMin() - Return the minimum normalized positive float *DBL_MIN* as C :ctype:`double`. + Return the minimum normalized positive float *DBL_MIN* as C :c:type:`double`. -.. cfunction:: int PyFloat_ClearFreeList() +.. c:function:: int PyFloat_ClearFreeList() Clear the float free list. Return the number of items that could not be freed. diff --git a/Doc/c-api/function.rst b/Doc/c-api/function.rst index 3512fe2206..31805fd0ad 100644 --- a/Doc/c-api/function.rst +++ b/Doc/c-api/function.rst @@ -10,26 +10,26 @@ Function Objects There are a few functions specific to Python functions. -.. ctype:: PyFunctionObject +.. c:type:: PyFunctionObject The C structure used for functions. -.. cvar:: PyTypeObject PyFunction_Type +.. c:var:: PyTypeObject PyFunction_Type .. index:: single: MethodType (in module types) - This is an instance of :ctype:`PyTypeObject` and represents the Python function + This is an instance of :c:type:`PyTypeObject` and represents the Python function type. It is exposed to Python programmers as ``types.FunctionType``. -.. cfunction:: int PyFunction_Check(PyObject *o) +.. c:function:: int PyFunction_Check(PyObject *o) - Return true if *o* is a function object (has type :cdata:`PyFunction_Type`). + Return true if *o* is a function object (has type :c:data:`PyFunction_Type`). The parameter must not be *NULL*. -.. cfunction:: PyObject* PyFunction_New(PyObject *code, PyObject *globals) +.. c:function:: PyObject* PyFunction_New(PyObject *code, PyObject *globals) Return a new function object associated with the code object *code*. *globals* must be a dictionary with the global variables accessible to the function. @@ -38,30 +38,30 @@ There are a few functions specific to Python functions. object, the argument defaults and closure are set to *NULL*. -.. cfunction:: PyObject* PyFunction_GetCode(PyObject *op) +.. c:function:: PyObject* PyFunction_GetCode(PyObject *op) Return the code object associated with the function object *op*. -.. cfunction:: PyObject* PyFunction_GetGlobals(PyObject *op) +.. c:function:: PyObject* PyFunction_GetGlobals(PyObject *op) Return the globals dictionary associated with the function object *op*. -.. cfunction:: PyObject* PyFunction_GetModule(PyObject *op) +.. c:function:: PyObject* PyFunction_GetModule(PyObject *op) Return the *__module__* attribute of the function object *op*. This is normally a string containing the module name, but can be set to any other object by Python code. -.. cfunction:: PyObject* PyFunction_GetDefaults(PyObject *op) +.. c:function:: PyObject* PyFunction_GetDefaults(PyObject *op) Return the argument default values of the function object *op*. This can be a tuple of arguments or *NULL*. -.. cfunction:: int PyFunction_SetDefaults(PyObject *op, PyObject *defaults) +.. c:function:: int PyFunction_SetDefaults(PyObject *op, PyObject *defaults) Set the argument default values for the function object *op*. *defaults* must be *Py_None* or a tuple. @@ -69,13 +69,13 @@ There are a few functions specific to Python functions. Raises :exc:`SystemError` and returns ``-1`` on failure. -.. cfunction:: PyObject* PyFunction_GetClosure(PyObject *op) +.. c:function:: PyObject* PyFunction_GetClosure(PyObject *op) Return the closure associated with the function object *op*. This can be *NULL* or a tuple of cell objects. -.. cfunction:: int PyFunction_SetClosure(PyObject *op, PyObject *closure) +.. c:function:: int PyFunction_SetClosure(PyObject *op, PyObject *closure) Set the closure associated with the function object *op*. *closure* must be *Py_None* or a tuple of cell objects. @@ -83,13 +83,13 @@ There are a few functions specific to Python functions. Raises :exc:`SystemError` and returns ``-1`` on failure. -.. cfunction:: PyObject *PyFunction_GetAnnotations(PyObject *op) +.. c:function:: PyObject *PyFunction_GetAnnotations(PyObject *op) Return the annotations of the function object *op*. This can be a mutable dictionary or *NULL*. -.. cfunction:: int PyFunction_SetAnnotations(PyObject *op, PyObject *annotations) +.. c:function:: int PyFunction_SetAnnotations(PyObject *op, PyObject *annotations) Set the annotations for the function object *op*. *annotations* must be a dictionary or *Py_None*. diff --git a/Doc/c-api/gcsupport.rst b/Doc/c-api/gcsupport.rst index 1a280c823a..3875ff2c65 100644 --- a/Doc/c-api/gcsupport.rst +++ b/Doc/c-api/gcsupport.rst @@ -27,32 +27,32 @@ include the :const:`Py_TPFLAGS_HAVE_GC` and provide an implementation of the Constructors for container types must conform to two rules: -#. The memory for the object must be allocated using :cfunc:`PyObject_GC_New` - or :cfunc:`PyObject_GC_NewVar`. +#. The memory for the object must be allocated using :c:func:`PyObject_GC_New` + or :c:func:`PyObject_GC_NewVar`. #. Once all the fields which may contain references to other containers are - initialized, it must call :cfunc:`PyObject_GC_Track`. + initialized, it must call :c:func:`PyObject_GC_Track`. -.. cfunction:: TYPE* PyObject_GC_New(TYPE, PyTypeObject *type) +.. c:function:: TYPE* PyObject_GC_New(TYPE, PyTypeObject *type) - Analogous to :cfunc:`PyObject_New` but for container objects with the + Analogous to :c:func:`PyObject_New` but for container objects with the :const:`Py_TPFLAGS_HAVE_GC` flag set. -.. cfunction:: TYPE* PyObject_GC_NewVar(TYPE, PyTypeObject *type, Py_ssize_t size) +.. c:function:: TYPE* PyObject_GC_NewVar(TYPE, PyTypeObject *type, Py_ssize_t size) - Analogous to :cfunc:`PyObject_NewVar` but for container objects with the + Analogous to :c:func:`PyObject_NewVar` but for container objects with the :const:`Py_TPFLAGS_HAVE_GC` flag set. -.. cfunction:: TYPE* PyObject_GC_Resize(TYPE, PyVarObject *op, Py_ssize_t newsize) +.. c:function:: TYPE* PyObject_GC_Resize(TYPE, PyVarObject *op, Py_ssize_t newsize) - Resize an object allocated by :cfunc:`PyObject_NewVar`. Returns the + Resize an object allocated by :c:func:`PyObject_NewVar`. Returns the resized object or *NULL* on failure. -.. cfunction:: void PyObject_GC_Track(PyObject *op) +.. c:function:: void PyObject_GC_Track(PyObject *op) Adds the object *op* to the set of container objects tracked by the collector. The collector can run at unexpected times so objects must be @@ -61,44 +61,44 @@ Constructors for container types must conform to two rules: end of the constructor. -.. cfunction:: void _PyObject_GC_TRACK(PyObject *op) +.. c:function:: void _PyObject_GC_TRACK(PyObject *op) - A macro version of :cfunc:`PyObject_GC_Track`. It should not be used for + A macro version of :c:func:`PyObject_GC_Track`. It should not be used for extension modules. Similarly, the deallocator for the object must conform to a similar pair of rules: #. Before fields which refer to other containers are invalidated, - :cfunc:`PyObject_GC_UnTrack` must be called. + :c:func:`PyObject_GC_UnTrack` must be called. -#. The object's memory must be deallocated using :cfunc:`PyObject_GC_Del`. +#. The object's memory must be deallocated using :c:func:`PyObject_GC_Del`. -.. cfunction:: void PyObject_GC_Del(void *op) +.. c:function:: void PyObject_GC_Del(void *op) - Releases memory allocated to an object using :cfunc:`PyObject_GC_New` or - :cfunc:`PyObject_GC_NewVar`. + Releases memory allocated to an object using :c:func:`PyObject_GC_New` or + :c:func:`PyObject_GC_NewVar`. -.. cfunction:: void PyObject_GC_UnTrack(void *op) +.. c:function:: void PyObject_GC_UnTrack(void *op) Remove the object *op* from the set of container objects tracked by the - collector. Note that :cfunc:`PyObject_GC_Track` can be called again on + collector. Note that :c:func:`PyObject_GC_Track` can be called again on this object to add it back to the set of tracked objects. The deallocator (:attr:`tp_dealloc` handler) should call this for the object before any of the fields used by the :attr:`tp_traverse` handler become invalid. -.. cfunction:: void _PyObject_GC_UNTRACK(PyObject *op) +.. c:function:: void _PyObject_GC_UNTRACK(PyObject *op) - A macro version of :cfunc:`PyObject_GC_UnTrack`. It should not be used for + A macro version of :c:func:`PyObject_GC_UnTrack`. It should not be used for extension modules. The :attr:`tp_traverse` handler accepts a function parameter of this type: -.. ctype:: int (*visitproc)(PyObject *object, void *arg) +.. c:type:: int (*visitproc)(PyObject *object, void *arg) Type of the visitor function passed to the :attr:`tp_traverse` handler. The function should be called with an object to traverse as *object* and @@ -110,7 +110,7 @@ The :attr:`tp_traverse` handler accepts a function parameter of this type: The :attr:`tp_traverse` handler must have the following type: -.. ctype:: int (*traverseproc)(PyObject *self, visitproc visit, void *arg) +.. c:type:: int (*traverseproc)(PyObject *self, visitproc visit, void *arg) Traversal function for a container object. Implementations must call the *visit* function for each object directly contained by *self*, with the @@ -119,12 +119,12 @@ The :attr:`tp_traverse` handler must have the following type: object argument. If *visit* returns a non-zero value that value should be returned immediately. -To simplify writing :attr:`tp_traverse` handlers, a :cfunc:`Py_VISIT` macro is +To simplify writing :attr:`tp_traverse` handlers, a :c:func:`Py_VISIT` macro is provided. In order to use this macro, the :attr:`tp_traverse` implementation must name its arguments exactly *visit* and *arg*: -.. cfunction:: void Py_VISIT(PyObject *o) +.. c:function:: void Py_VISIT(PyObject *o) Call the *visit* callback, with arguments *o* and *arg*. If *visit* returns a non-zero value, then return it. Using this macro, :attr:`tp_traverse` @@ -138,15 +138,15 @@ must name its arguments exactly *visit* and *arg*: return 0; } -The :attr:`tp_clear` handler must be of the :ctype:`inquiry` type, or *NULL* +The :attr:`tp_clear` handler must be of the :c:type:`inquiry` type, or *NULL* if the object is immutable. -.. ctype:: int (*inquiry)(PyObject *self) +.. c:type:: int (*inquiry)(PyObject *self) Drop references that may have created reference cycles. Immutable objects do not have to define this method since they can never directly create reference cycles. Note that the object must still be valid after calling - this method (don't just call :cfunc:`Py_DECREF` on a reference). The + this method (don't just call :c:func:`Py_DECREF` on a reference). The collector will call this method if it detects that this object is involved in a reference cycle. diff --git a/Doc/c-api/gen.rst b/Doc/c-api/gen.rst index 0d3789a25f..33cd27a5aa 100644 --- a/Doc/c-api/gen.rst +++ b/Doc/c-api/gen.rst @@ -7,31 +7,31 @@ Generator Objects Generator objects are what Python uses to implement generator iterators. They are normally created by iterating over a function that yields values, rather -than explicitly calling :cfunc:`PyGen_New`. +than explicitly calling :c:func:`PyGen_New`. -.. ctype:: PyGenObject +.. c:type:: PyGenObject The C structure used for generator objects. -.. cvar:: PyTypeObject PyGen_Type +.. c:var:: PyTypeObject PyGen_Type The type object corresponding to generator objects -.. cfunction:: int PyGen_Check(ob) +.. c:function:: int PyGen_Check(ob) Return true if *ob* is a generator object; *ob* must not be *NULL*. -.. cfunction:: int PyGen_CheckExact(ob) +.. c:function:: int PyGen_CheckExact(ob) Return true if *ob*'s type is *PyGen_Type* is a generator object; *ob* must not be *NULL*. -.. cfunction:: PyObject* PyGen_New(PyFrameObject *frame) +.. c:function:: PyObject* PyGen_New(PyFrameObject *frame) Create and return a new generator object based on the *frame* object. A reference to *frame* is stolen by this function. The parameter must not be diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst index ff79112edf..cf48363e10 100644 --- a/Doc/c-api/import.rst +++ b/Doc/c-api/import.rst @@ -6,14 +6,14 @@ Importing Modules ================= -.. cfunction:: PyObject* PyImport_ImportModule(const char *name) +.. c:function:: PyObject* PyImport_ImportModule(const char *name) .. index:: single: package variable; __all__ single: __all__ (package variable) single: modules (in module sys) - This is a simplified interface to :cfunc:`PyImport_ImportModuleEx` below, + This is a simplified interface to :c:func:`PyImport_ImportModuleEx` below, leaving the *globals* and *locals* arguments set to *NULL* and *level* set to 0. When the *name* argument contains a dot (when it specifies a submodule of a package), the @@ -28,18 +28,18 @@ Importing Modules This function always uses absolute imports. -.. cfunction:: PyObject* PyImport_ImportModuleNoBlock(const char *name) +.. c:function:: PyObject* PyImport_ImportModuleNoBlock(const char *name) - This version of :cfunc:`PyImport_ImportModule` does not block. It's intended + This version of :c:func:`PyImport_ImportModule` does not block. It's intended to be used in C functions that import other modules to execute a function. The import may block if another thread holds the import lock. The function - :cfunc:`PyImport_ImportModuleNoBlock` never blocks. It first tries to fetch - the module from sys.modules and falls back to :cfunc:`PyImport_ImportModule` + :c:func:`PyImport_ImportModuleNoBlock` never blocks. It first tries to fetch + the module from sys.modules and falls back to :c:func:`PyImport_ImportModule` unless the lock is held, in which case the function will raise an :exc:`ImportError`. -.. cfunction:: PyObject* PyImport_ImportModuleEx(char *name, PyObject *globals, PyObject *locals, PyObject *fromlist) +.. c:function:: PyObject* PyImport_ImportModuleEx(char *name, PyObject *globals, PyObject *locals, PyObject *fromlist) .. index:: builtin: __import__ @@ -54,10 +54,10 @@ Importing Modules was given. Failing imports remove incomplete module objects, like with - :cfunc:`PyImport_ImportModule`. + :c:func:`PyImport_ImportModule`. -.. cfunction:: PyObject* PyImport_ImportModuleLevel(char *name, PyObject *globals, PyObject *locals, PyObject *fromlist, int level) +.. c:function:: PyObject* PyImport_ImportModuleLevel(char *name, PyObject *globals, PyObject *locals, PyObject *fromlist, int level) Import a module. This is best described by referring to the built-in Python function :func:`__import__`, as the standard :func:`__import__` function calls @@ -69,7 +69,7 @@ Importing Modules top-level package, unless a non-empty *fromlist* was given. -.. cfunction:: PyObject* PyImport_Import(PyObject *name) +.. c:function:: PyObject* PyImport_Import(PyObject *name) This is a higher-level interface that calls the current "import hook function" (with an explicit *level* of 0, meaning absolute import). It @@ -80,13 +80,13 @@ Importing Modules This function always uses absolute imports. -.. cfunction:: PyObject* PyImport_ReloadModule(PyObject *m) +.. c:function:: PyObject* PyImport_ReloadModule(PyObject *m) Reload a module. Return a new reference to the reloaded module, or *NULL* with an exception set on failure (the module still exists in this case). -.. cfunction:: PyObject* PyImport_AddModule(const char *name) +.. c:function:: PyObject* PyImport_AddModule(const char *name) Return the module object corresponding to a module name. The *name* argument may be of the form ``package.module``. First check the modules dictionary if @@ -96,12 +96,12 @@ Importing Modules .. note:: This function does not load or import the module; if the module wasn't already - loaded, you will get an empty module object. Use :cfunc:`PyImport_ImportModule` + loaded, you will get an empty module object. Use :c:func:`PyImport_ImportModule` or one of its variants to import a module. Package structures implied by a dotted name for *name* are not created if not already present. -.. cfunction:: PyObject* PyImport_ExecCodeModule(char *name, PyObject *co) +.. c:function:: PyObject* PyImport_ExecCodeModule(char *name, PyObject *co) .. index:: builtin: compile @@ -110,32 +110,61 @@ Importing Modules :func:`compile`, load the module. Return a new reference to the module object, or *NULL* with an exception set if an error occurred. *name* is removed from :attr:`sys.modules` in error cases, even if *name* was already - in :attr:`sys.modules` on entry to :cfunc:`PyImport_ExecCodeModule`. Leaving + in :attr:`sys.modules` on entry to :c:func:`PyImport_ExecCodeModule`. Leaving incompletely initialized modules in :attr:`sys.modules` is dangerous, as imports of such modules have no way to know that the module object is an unknown (and probably damaged with respect to the module author's intents) state. + The module's :attr:`__file__` attribute will be set to the code object's + :c:member:`co_filename`. + This function will reload the module if it was already imported. See - :cfunc:`PyImport_ReloadModule` for the intended way to reload a module. + :c:func:`PyImport_ReloadModule` for the intended way to reload a module. If *name* points to a dotted name of the form ``package.module``, any package structures not already created will still not be created. + See also :c:func:`PyImport_ExecCodeModuleEx` and + :c:func:`PyImport_ExecCodeModuleWithPathnames`. + + +.. c:function:: PyObject* PyImport_ExecCodeModuleEx(char *name, PyObject *co, char *pathname) + + Like :c:func:`PyImport_ExecCodeModule`, but the :attr:`__file__` attribute of + the module object is set to *pathname* if it is non-``NULL``. + + See also :c:func:`PyImport_ExecCodeModuleWithPathnames`. + -.. cfunction:: long PyImport_GetMagicNumber() +.. c:function:: PyObject* PyImport_ExecCodeModuleWithPathnames(char *name, PyObject *co, char *pathname, char *cpathname) + + Like :c:func:`PyImport_ExecCodeModuleEx`, but the :attr:`__cached__` + attribute of the module object is set to *cpathname* if it is + non-``NULL``. Of the three functions, this is the preferred one to use. + + .. versionadded:: 3.2 + +.. c:function:: long PyImport_GetMagicNumber() Return the magic number for Python bytecode files (a.k.a. :file:`.pyc` and :file:`.pyo` files). The magic number should be present in the first four bytes of the bytecode file, in little-endian byte order. -.. cfunction:: PyObject* PyImport_GetModuleDict() +.. c:function:: const char * PyImport_GetMagicTag() + + Return the magic tag string for :pep:`3147` format Python bytecode file + names. + + .. versionadded:: 3.2 + +.. c:function:: PyObject* PyImport_GetModuleDict() Return the dictionary used for the module administration (a.k.a. ``sys.modules``). Note that this is a per-interpreter variable. -.. cfunction:: PyObject* PyImport_GetImporter(PyObject *path) +.. c:function:: PyObject* PyImport_GetImporter(PyObject *path) Return an importer object for a :data:`sys.path`/:attr:`pkg.__path__` item *path*, possibly by fetching it from the :data:`sys.path_importer_cache` @@ -146,41 +175,41 @@ Importing Modules to the importer object. -.. cfunction:: void _PyImport_Init() +.. c:function:: void _PyImport_Init() Initialize the import mechanism. For internal use only. -.. cfunction:: void PyImport_Cleanup() +.. c:function:: void PyImport_Cleanup() Empty the module table. For internal use only. -.. cfunction:: void _PyImport_Fini() +.. c:function:: void _PyImport_Fini() Finalize the import mechanism. For internal use only. -.. cfunction:: PyObject* _PyImport_FindExtension(char *, char *) +.. c:function:: PyObject* _PyImport_FindExtension(char *, char *) For internal use only. -.. cfunction:: PyObject* _PyImport_FixupExtension(char *, char *) +.. c:function:: PyObject* _PyImport_FixupExtension(char *, char *) For internal use only. -.. cfunction:: int PyImport_ImportFrozenModule(char *name) +.. c:function:: int PyImport_ImportFrozenModule(char *name) Load a frozen module named *name*. Return ``1`` for success, ``0`` if the module is not found, and ``-1`` with an exception set if the initialization failed. To access the imported module on a successful load, use - :cfunc:`PyImport_ImportModule`. (Note the misnomer --- this function would + :c:func:`PyImport_ImportModule`. (Note the misnomer --- this function would reload the module if it was already imported.) -.. ctype:: struct _frozen +.. c:type:: struct _frozen .. index:: single: freeze utility @@ -196,30 +225,30 @@ Importing Modules }; -.. cvar:: struct _frozen* PyImport_FrozenModules +.. c:var:: struct _frozen* PyImport_FrozenModules - This pointer is initialized to point to an array of :ctype:`struct _frozen` + This pointer is initialized to point to an array of :c:type:`struct _frozen` records, terminated by one whose members are all *NULL* or zero. When a frozen module is imported, it is searched in this table. Third-party code could play tricks with this to provide a dynamically created collection of frozen modules. -.. cfunction:: int PyImport_AppendInittab(const char *name, PyObject* (*initfunc)(void)) +.. c:function:: int PyImport_AppendInittab(const char *name, PyObject* (*initfunc)(void)) Add a single module to the existing table of built-in modules. This is a - convenience wrapper around :cfunc:`PyImport_ExtendInittab`, returning ``-1`` if + convenience wrapper around :c:func:`PyImport_ExtendInittab`, returning ``-1`` if the table could not be extended. The new module can be imported by the name *name*, and uses the function *initfunc* as the initialization function called on the first attempted import. This should be called before - :cfunc:`Py_Initialize`. + :c:func:`Py_Initialize`. -.. ctype:: struct _inittab +.. c:type:: struct _inittab Structure describing a single entry in the list of built-in modules. Each of these structures gives the name and initialization function for a module built into the interpreter. Programs which embed Python may use an array of these - structures in conjunction with :cfunc:`PyImport_ExtendInittab` to provide + structures in conjunction with :c:func:`PyImport_ExtendInittab` to provide additional built-in modules. The structure is defined in :file:`Include/import.h` as:: @@ -229,11 +258,11 @@ Importing Modules }; -.. cfunction:: int PyImport_ExtendInittab(struct _inittab *newtab) +.. c:function:: int PyImport_ExtendInittab(struct _inittab *newtab) Add a collection of modules to the table of built-in modules. The *newtab* array must end with a sentinel entry which contains *NULL* for the :attr:`name` field; failure to provide the sentinel value can result in a memory fault. Returns ``0`` on success or ``-1`` if insufficient memory could be allocated to extend the internal table. In the event of failure, no modules are added to the - internal table. This should be called before :cfunc:`Py_Initialize`. + internal table. This should be called before :c:func:`Py_Initialize`. diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index b2fa3ee6e1..7507e3b52d 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -12,13 +12,11 @@ Initializing and finalizing the interpreter =========================================== -.. cfunction:: void Py_Initialize() +.. c:function:: void Py_Initialize() .. index:: single: Py_SetProgramName() single: PyEval_InitThreads() - single: PyEval_ReleaseLock() - single: PyEval_AcquireLock() single: modules (in module sys) single: path (in module sys) module: builtins @@ -31,38 +29,37 @@ Initializing and finalizing the interpreter Initialize the Python interpreter. In an application embedding Python, this should be called before using any other Python/C API functions; with the - exception of :cfunc:`Py_SetProgramName`, :cfunc:`PyEval_InitThreads`, - :cfunc:`PyEval_ReleaseLock`, and :cfunc:`PyEval_AcquireLock`. This initializes + exception of :c:func:`Py_SetProgramName`, :c:func:`Py_SetPythonHome` and :c:func:`Py_SetPath`. This initializes the table of loaded modules (``sys.modules``), and creates the fundamental modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. It also initializes the module search path (``sys.path``). It does not set ``sys.argv``; use - :cfunc:`PySys_SetArgvEx` for that. This is a no-op when called for a second time - (without calling :cfunc:`Py_Finalize` first). There is no return value; it is a + :c:func:`PySys_SetArgvEx` for that. This is a no-op when called for a second time + (without calling :c:func:`Py_Finalize` first). There is no return value; it is a fatal error if the initialization fails. -.. cfunction:: void Py_InitializeEx(int initsigs) +.. c:function:: void Py_InitializeEx(int initsigs) - This function works like :cfunc:`Py_Initialize` if *initsigs* is 1. If + This function works like :c:func:`Py_Initialize` if *initsigs* is 1. If *initsigs* is 0, it skips initialization registration of signal handlers, which might be useful when Python is embedded. -.. cfunction:: int Py_IsInitialized() +.. c:function:: int Py_IsInitialized() Return true (nonzero) when the Python interpreter has been initialized, false - (zero) if not. After :cfunc:`Py_Finalize` is called, this returns false until - :cfunc:`Py_Initialize` is called again. + (zero) if not. After :c:func:`Py_Finalize` is called, this returns false until + :c:func:`Py_Initialize` is called again. -.. cfunction:: void Py_Finalize() +.. c:function:: void Py_Finalize() - Undo all initializations made by :cfunc:`Py_Initialize` and subsequent use of + Undo all initializations made by :c:func:`Py_Initialize` and subsequent use of Python/C API functions, and destroy all sub-interpreters (see - :cfunc:`Py_NewInterpreter` below) that were created and not yet destroyed since - the last call to :cfunc:`Py_Initialize`. Ideally, this frees all memory + :c:func:`Py_NewInterpreter` below) that were created and not yet destroyed since + the last call to :c:func:`Py_Initialize`. Ideally, this frees all memory allocated by the Python interpreter. This is a no-op when called for a second - time (without calling :cfunc:`Py_Initialize` again first). There is no return + time (without calling :c:func:`Py_Initialize` again first). There is no return value; errors during finalization are ignored. This function is provided for a number of reasons. An embedding application @@ -81,26 +78,26 @@ Initializing and finalizing the interpreter please report it). Memory tied up in circular references between objects is not freed. Some memory allocated by extension modules may not be freed. Some extensions may not work properly if their initialization routine is called more - than once; this can happen if an application calls :cfunc:`Py_Initialize` and - :cfunc:`Py_Finalize` more than once. + than once; this can happen if an application calls :c:func:`Py_Initialize` and + :c:func:`Py_Finalize` more than once. Process-wide parameters ======================= -.. cfunction:: void Py_SetProgramName(wchar_t *name) +.. c:function:: void Py_SetProgramName(wchar_t *name) .. index:: single: Py_Initialize() single: main() single: Py_GetPath() - This function should be called before :cfunc:`Py_Initialize` is called for + This function should be called before :c:func:`Py_Initialize` is called for the first time, if it is called at all. It tells the interpreter the value - of the ``argv[0]`` argument to the :cfunc:`main` function of the program + of the ``argv[0]`` argument to the :c:func:`main` function of the program (converted to wide characters). - This is used by :cfunc:`Py_GetPath` and some other functions below to find + This is used by :c:func:`Py_GetPath` and some other functions below to find the Python run-time libraries relative to the interpreter executable. The default value is ``'python'``. The argument should point to a zero-terminated wide character string in static storage whose contents will not @@ -108,37 +105,37 @@ Process-wide parameters interpreter will change the contents of this storage. -.. cfunction:: wchar* Py_GetProgramName() +.. c:function:: wchar* Py_GetProgramName() .. index:: single: Py_SetProgramName() - Return the program name set with :cfunc:`Py_SetProgramName`, or the default. + Return the program name set with :c:func:`Py_SetProgramName`, or the default. The returned string points into static storage; the caller should not modify its value. -.. cfunction:: wchar_t* Py_GetPrefix() +.. c:function:: wchar_t* Py_GetPrefix() Return the *prefix* for installed platform-independent files. This is derived through a number of complicated rules from the program name set with - :cfunc:`Py_SetProgramName` and some environment variables; for example, if the + :c:func:`Py_SetProgramName` and some environment variables; for example, if the program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The returned string points into static storage; the caller should not modify its value. This corresponds to the :makevar:`prefix` variable in the top-level - :file:`Makefile` and the :option:`--prefix` argument to the :program:`configure` + :file:`Makefile` and the ``--prefix`` argument to the :program:`configure` script at build time. The value is available to Python code as ``sys.prefix``. It is only useful on Unix. See also the next function. -.. cfunction:: wchar_t* Py_GetExecPrefix() +.. c:function:: wchar_t* Py_GetExecPrefix() Return the *exec-prefix* for installed platform-*dependent* files. This is derived through a number of complicated rules from the program name set with - :cfunc:`Py_SetProgramName` and some environment variables; for example, if the + :c:func:`Py_SetProgramName` and some environment variables; for example, if the program name is ``'/usr/local/bin/python'``, the exec-prefix is ``'/usr/local'``. The returned string points into static storage; the caller should not modify its value. This corresponds to the :makevar:`exec_prefix` - variable in the top-level :file:`Makefile` and the :option:`--exec-prefix` + variable in the top-level :file:`Makefile` and the ``--exec-prefix`` argument to the :program:`configure` script at build time. The value is available to Python code as ``sys.exec_prefix``. It is only useful on Unix. @@ -165,7 +162,7 @@ Process-wide parameters platform. -.. cfunction:: wchar_t* Py_GetProgramFullPath() +.. c:function:: wchar_t* Py_GetProgramFullPath() .. index:: single: Py_SetProgramName() @@ -173,19 +170,20 @@ Process-wide parameters Return the full program name of the Python executable; this is computed as a side-effect of deriving the default module search path from the program name - (set by :cfunc:`Py_SetProgramName` above). The returned string points into + (set by :c:func:`Py_SetProgramName` above). The returned string points into static storage; the caller should not modify its value. The value is available to Python code as ``sys.executable``. -.. cfunction:: wchar_t* Py_GetPath() +.. c:function:: wchar_t* Py_GetPath() .. index:: triple: module; search; path single: path (in module sys) + single: Py_SetPath() Return the default module search path; this is computed from the program name - (set by :cfunc:`Py_SetProgramName` above) and some environment variables. + (set by :c:func:`Py_SetProgramName` above) and some environment variables. The returned string consists of a series of directory names separated by a platform dependent delimiter character. The delimiter character is ``':'`` on Unix and Mac OS X, ``';'`` on Windows. The returned string points into @@ -197,7 +195,26 @@ Process-wide parameters .. XXX should give the exact rules -.. cfunction:: const char* Py_GetVersion() +.. c:function:: void Py_SetPath(const wchar_t *) + + .. index:: + triple: module; search; path + single: path (in module sys) + single: Py_GetPath() + + Set the default module search path. If this function is called before + :c:func:`Py_Initialize`, then :c:func:`Py_GetPath` won't attempt to compute a + default search path but uses the one provided instead. This is useful if + Python is embedded by an application that has full knowledge of the location + of all modules. The path components should be separated by semicolons. + + This also causes :data:`sys.executable` to be set only to the raw program + name (see :c:func:`Py_SetProgramName`) and for :data:`sys.prefix` and + :data:`sys.exec_prefix` to be empty. It is up to the caller to modify these + if required after calling :c:func:`Py_Initialize`. + + +.. c:function:: const char* Py_GetVersion() Return the version of this Python interpreter. This is a string that looks something like :: @@ -212,7 +229,7 @@ Process-wide parameters modify its value. The value is available to Python code as :data:`sys.version`. -.. cfunction:: const char* Py_GetPlatform() +.. c:function:: const char* Py_GetPlatform() .. index:: single: platform (in module sys) @@ -225,7 +242,7 @@ Process-wide parameters to Python code as ``sys.platform``. -.. cfunction:: const char* Py_GetCopyright() +.. c:function:: const char* Py_GetCopyright() Return the official copyright string for the current Python version, for example @@ -237,7 +254,7 @@ Process-wide parameters value. The value is available to Python code as ``sys.copyright``. -.. cfunction:: const char* Py_GetCompiler() +.. c:function:: const char* Py_GetCompiler() Return an indication of the compiler used to build the current Python version, in square brackets, for example:: @@ -251,7 +268,7 @@ Process-wide parameters ``sys.version``. -.. cfunction:: const char* Py_GetBuildInfo() +.. c:function:: const char* Py_GetBuildInfo() Return information about the sequence number and build date and time of the current Python interpreter instance, for example :: @@ -265,7 +282,7 @@ Process-wide parameters ``sys.version``. -.. cfunction:: void PySys_SetArgvEx(int argc, wchar_t **argv, int updatepath) +.. c:function:: void PySys_SetArgvEx(int argc, wchar_t **argv, int updatepath) .. index:: single: main() @@ -273,12 +290,12 @@ Process-wide parameters single: argv (in module sys) Set :data:`sys.argv` based on *argc* and *argv*. These parameters are - similar to those passed to the program's :cfunc:`main` function with the + similar to those passed to the program's :c:func:`main` function with the difference that the first entry should refer to the script file to be executed rather than the executable hosting the Python interpreter. If there isn't a script that will be run, the first entry in *argv* can be an empty string. If this function fails to initialize :data:`sys.argv`, a fatal - condition is signalled using :cfunc:`Py_FatalError`. + condition is signalled using :c:func:`Py_FatalError`. If *updatepath* is zero, this is all the function does. If *updatepath* is non-zero, the function also modifies :data:`sys.path` according to the @@ -300,7 +317,7 @@ Process-wide parameters On versions before 3.1.3, you can achieve the same effect by manually popping the first :data:`sys.path` element after having called - :cfunc:`PySys_SetArgv`, for example using:: + :c:func:`PySys_SetArgv`, for example using:: PyRun_SimpleString("import sys; sys.path.pop(0)\n"); @@ -310,12 +327,12 @@ Process-wide parameters check w/ Guido. -.. cfunction:: void PySys_SetArgv(int argc, wchar_t **argv) +.. c:function:: void PySys_SetArgv(int argc, wchar_t **argv) - This function works like :cfunc:`PySys_SetArgvEx` with *updatepath* set to 1. + This function works like :c:func:`PySys_SetArgvEx` with *updatepath* set to 1. -.. cfunction:: void Py_SetPythonHome(wchar_t *home) +.. c:function:: void Py_SetPythonHome(wchar_t *home) Set the default "home" directory, that is, the location of the standard Python libraries. See :envvar:`PYTHONHOME` for the meaning of the @@ -327,10 +344,10 @@ Process-wide parameters this storage. -.. cfunction:: w_char* Py_GetPythonHome() +.. c:function:: w_char* Py_GetPythonHome() Return the default "home", that is, the value set by a previous call to - :cfunc:`Py_SetPythonHome`, or the value of the :envvar:`PYTHONHOME` + :c:func:`Py_SetPythonHome`, or the value of the :envvar:`PYTHONHOME` environment variable if it is set. @@ -352,12 +369,12 @@ operations could cause problems in a multi-threaded program: for example, when two threads simultaneously increment the reference count of the same object, the reference count could end up being incremented only once instead of twice. -.. index:: single: setcheckinterval() (in module sys) +.. index:: single: setswitchinterval() (in module sys) Therefore, the rule exists that only the thread that has acquired the :term:`GIL` may operate on Python objects or call Python/C API functions. In order to emulate concurrency of execution, the interpreter regularly -tries to switch threads (see :func:`sys.setcheckinterval`). The lock is also +tries to switch threads (see :func:`sys.setswitchinterval`). The lock is also released around potentially blocking I/O operations like reading or writing a file, so that other Python threads can run in the meantime. @@ -366,9 +383,9 @@ a file, so that other Python threads can run in the meantime. single: PyThreadState The Python interpreter keeps some thread-specific bookkeeping information -inside a data structure called :ctype:`PyThreadState`. There's also one -global variable pointing to the current :ctype:`PyThreadState`: it can -be retrieved using :cfunc:`PyThreadState_Get`. +inside a data structure called :c:type:`PyThreadState`. There's also one +global variable pointing to the current :c:type:`PyThreadState`: it can +be retrieved using :c:func:`PyThreadState_Get`. Releasing the GIL from extension code ------------------------------------- @@ -392,8 +409,8 @@ This is so common that a pair of macros exists to simplify it:: single: Py_BEGIN_ALLOW_THREADS single: Py_END_ALLOW_THREADS -The :cmacro:`Py_BEGIN_ALLOW_THREADS` macro opens a new block and declares a -hidden local variable; the :cmacro:`Py_END_ALLOW_THREADS` macro closes the +The :c:macro:`Py_BEGIN_ALLOW_THREADS` macro opens a new block and declares a +hidden local variable; the :c:macro:`Py_END_ALLOW_THREADS` macro closes the block. These two macros are still available when Python is compiled without thread support (they simply have an empty expansion). @@ -443,7 +460,7 @@ storing their thread state pointer, before you can start using the Python/C API. When you are done, you should reset the thread state pointer, release the GIL, and finally free the thread state data structure. -The :cfunc:`PyGILState_Ensure` and :cfunc:`PyGILState_Release` functions do +The :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` functions do all of the above automatically. The typical idiom for calling into Python from a C thread is:: @@ -457,14 +474,14 @@ from a C thread is:: /* Release the thread. No Python API allowed beyond this point. */ PyGILState_Release(gstate); -Note that the :cfunc:`PyGILState_\*` functions assume there is only one global -interpreter (created automatically by :cfunc:`Py_Initialize`). Python +Note that the :c:func:`PyGILState_\*` functions assume there is only one global +interpreter (created automatically by :c:func:`Py_Initialize`). Python supports the creation of additional interpreters (using -:cfunc:`Py_NewInterpreter`), but mixing multiple interpreters and the -:cfunc:`PyGILState_\*` API is unsupported. +:c:func:`Py_NewInterpreter`), but mixing multiple interpreters and the +:c:func:`PyGILState_\*` API is unsupported. Another important thing to note about threads is their behaviour in the face -of the C :cfunc:`fork` call. On most systems with :cfunc:`fork`, after a +of the C :c:func:`fork` call. On most systems with :c:func:`fork`, after a process forks only the thread that issued the fork will exist. That also means any locks held by other threads will never be released. Python solves this for :func:`os.fork` by acquiring the locks it uses internally before @@ -472,12 +489,12 @@ the fork, and releasing them afterwards. In addition, it resets any :ref:`lock-objects` in the child. When extending or embedding Python, there is no way to inform Python of additional (non-Python) locks that need to be acquired before or reset after a fork. OS facilities such as -:cfunc:`pthread_atfork` would need to be used to accomplish the same thing. -Additionally, when extending or embedding Python, calling :cfunc:`fork` +:c:func:`pthread_atfork` would need to be used to accomplish the same thing. +Additionally, when extending or embedding Python, calling :c:func:`fork` directly rather than through :func:`os.fork` (and returning to or calling into Python) may result in a deadlock by one of Python's internal locks being held by a thread that is defunct after the fork. -:cfunc:`PyOS_AfterFork` tries to reset the necessary locks, but is not +:c:func:`PyOS_AfterFork` tries to reset the necessary locks, but is not always able to. @@ -487,7 +504,7 @@ High-level API These are the most commonly used types and functions when writing C extension code, or when embedding the Python interpreter: -.. ctype:: PyInterpreterState +.. c:type:: PyInterpreterState This data structure represents the state shared by a number of cooperating threads. Threads belonging to the same interpreter share their module @@ -500,31 +517,30 @@ code, or when embedding the Python interpreter: interpreter they belong. -.. ctype:: PyThreadState +.. c:type:: PyThreadState This data structure represents the state of a single thread. The only public - data member is :ctype:`PyInterpreterState \*`:attr:`interp`, which points to + data member is :c:type:`PyInterpreterState \*`:attr:`interp`, which points to this thread's interpreter state. -.. cfunction:: void PyEval_InitThreads() +.. c:function:: void PyEval_InitThreads() .. index:: - single: PyEval_ReleaseLock() + single: PyEval_AcquireThread() single: PyEval_ReleaseThread() single: PyEval_SaveThread() single: PyEval_RestoreThread() Initialize and acquire the global interpreter lock. It should be called in the main thread before creating a second thread or engaging in any other thread - operations such as :cfunc:`PyEval_ReleaseLock` or - ``PyEval_ReleaseThread(tstate)``. It is not needed before calling - :cfunc:`PyEval_SaveThread` or :cfunc:`PyEval_RestoreThread`. + operations such as ``PyEval_ReleaseThread(tstate)``. It is not needed before + calling :c:func:`PyEval_SaveThread` or :c:func:`PyEval_RestoreThread`. - .. index:: single: Py_Initialize() + This is a no-op when called for a second time. - This is a no-op when called for a second time. It is safe to call this function - before calling :cfunc:`Py_Initialize`. + .. versionchanged:: 3.2 + This function cannot be called before :c:func:`Py_Initialize()` anymore. .. index:: module: _thread @@ -537,7 +553,7 @@ code, or when embedding the Python interpreter: when this function initializes the global interpreter lock, it also acquires it. Before the Python :mod:`_thread` module creates a new thread, knowing that either it has the lock or the lock hasn't been created yet, it calls - :cfunc:`PyEval_InitThreads`. When this call returns, it is guaranteed that + :c:func:`PyEval_InitThreads`. When this call returns, it is guaranteed that the lock has been created and that the calling thread has acquired it. It is **not** safe to call this function when it is unknown which thread (if @@ -546,15 +562,15 @@ code, or when embedding the Python interpreter: This function is not available when thread support is disabled at compile time. -.. cfunction:: int PyEval_ThreadsInitialized() +.. c:function:: int PyEval_ThreadsInitialized() - Returns a non-zero value if :cfunc:`PyEval_InitThreads` has been called. This + Returns a non-zero value if :c:func:`PyEval_InitThreads` has been called. This function can be called without holding the GIL, and therefore can be used to avoid calls to the locking API when running single-threaded. This function is not available when thread support is disabled at compile time. -.. cfunction:: PyThreadState* PyEval_SaveThread() +.. c:function:: PyThreadState* PyEval_SaveThread() Release the global interpreter lock (if it has been created and thread support is enabled) and reset the thread state to *NULL*, returning the @@ -563,7 +579,7 @@ code, or when embedding the Python interpreter: when thread support is disabled at compile time.) -.. cfunction:: void PyEval_RestoreThread(PyThreadState *tstate) +.. c:function:: void PyEval_RestoreThread(PyThreadState *tstate) Acquire the global interpreter lock (if it has been created and thread support is enabled) and set the thread state to *tstate*, which must not be @@ -572,23 +588,23 @@ code, or when embedding the Python interpreter: when thread support is disabled at compile time.) -.. cfunction:: PyThreadState* PyThreadState_Get() +.. c:function:: PyThreadState* PyThreadState_Get() Return the current thread state. The global interpreter lock must be held. When the current thread state is *NULL*, this issues a fatal error (so that the caller needn't check for *NULL*). -.. cfunction:: PyThreadState* PyThreadState_Swap(PyThreadState *tstate) +.. c:function:: PyThreadState* PyThreadState_Swap(PyThreadState *tstate) Swap the current thread state with the thread state given by the argument *tstate*, which may be *NULL*. The global interpreter lock must be held and is not released. -.. cfunction:: void PyEval_ReInitThreads() +.. c:function:: void PyEval_ReInitThreads() - This function is called from :cfunc:`PyOS_AfterFork` to ensure that newly + This function is called from :c:func:`PyOS_AfterFork` to ensure that newly created child processes don't hold locks referring to threads which are not running in the child process. @@ -596,71 +612,79 @@ code, or when embedding the Python interpreter: The following functions use thread-local storage, and are not compatible with sub-interpreters: -.. cfunction:: PyGILState_STATE PyGILState_Ensure() +.. c:function:: PyGILState_STATE PyGILState_Ensure() Ensure that the current thread is ready to call the Python C API regardless of the current state of Python, or of the global interpreter lock. This may be called as many times as desired by a thread as long as each call is - matched with a call to :cfunc:`PyGILState_Release`. In general, other - thread-related APIs may be used between :cfunc:`PyGILState_Ensure` and - :cfunc:`PyGILState_Release` calls as long as the thread state is restored to + matched with a call to :c:func:`PyGILState_Release`. In general, other + thread-related APIs may be used between :c:func:`PyGILState_Ensure` and + :c:func:`PyGILState_Release` calls as long as the thread state is restored to its previous state before the Release(). For example, normal usage of the - :cmacro:`Py_BEGIN_ALLOW_THREADS` and :cmacro:`Py_END_ALLOW_THREADS` macros is + :c:macro:`Py_BEGIN_ALLOW_THREADS` and :c:macro:`Py_END_ALLOW_THREADS` macros is acceptable. The return value is an opaque "handle" to the thread state when - :cfunc:`PyGILState_Ensure` was called, and must be passed to - :cfunc:`PyGILState_Release` to ensure Python is left in the same state. Even + :c:func:`PyGILState_Ensure` was called, and must be passed to + :c:func:`PyGILState_Release` to ensure Python is left in the same state. Even though recursive calls are allowed, these handles *cannot* be shared - each - unique call to :cfunc:`PyGILState_Ensure` must save the handle for its call - to :cfunc:`PyGILState_Release`. + unique call to :c:func:`PyGILState_Ensure` must save the handle for its call + to :c:func:`PyGILState_Release`. When the function returns, the current thread will hold the GIL and be able to call arbitrary Python code. Failure is a fatal error. -.. cfunction:: void PyGILState_Release(PyGILState_STATE) +.. c:function:: void PyGILState_Release(PyGILState_STATE) Release any resources previously acquired. After this call, Python's state will - be the same as it was prior to the corresponding :cfunc:`PyGILState_Ensure` call + be the same as it was prior to the corresponding :c:func:`PyGILState_Ensure` call (but generally this state will be unknown to the caller, hence the use of the GILState API). - Every call to :cfunc:`PyGILState_Ensure` must be matched by a call to - :cfunc:`PyGILState_Release` on the same thread. + Every call to :c:func:`PyGILState_Ensure` must be matched by a call to + :c:func:`PyGILState_Release` on the same thread. + + +.. c:function:: PyThreadState PyGILState_GetThisThreadState() + + Get the current thread state for this thread. May return ``NULL`` if no + GILState API has been used on the current thread. Note that the main thread + always has such a thread-state, even if no auto-thread-state call has been + made on the main thread. This is mainly a helper/diagnostic function. The following macros are normally used without a trailing semicolon; look for example usage in the Python source distribution. -.. cmacro:: Py_BEGIN_ALLOW_THREADS +.. c:macro:: Py_BEGIN_ALLOW_THREADS This macro expands to ``{ PyThreadState *_save; _save = PyEval_SaveThread();``. Note that it contains an opening brace; it must be matched with a following - :cmacro:`Py_END_ALLOW_THREADS` macro. See above for further discussion of this + :c:macro:`Py_END_ALLOW_THREADS` macro. See above for further discussion of this macro. It is a no-op when thread support is disabled at compile time. -.. cmacro:: Py_END_ALLOW_THREADS +.. c:macro:: Py_END_ALLOW_THREADS This macro expands to ``PyEval_RestoreThread(_save); }``. Note that it contains a closing brace; it must be matched with an earlier - :cmacro:`Py_BEGIN_ALLOW_THREADS` macro. See above for further discussion of + :c:macro:`Py_BEGIN_ALLOW_THREADS` macro. See above for further discussion of this macro. It is a no-op when thread support is disabled at compile time. -.. cmacro:: Py_BLOCK_THREADS +.. c:macro:: Py_BLOCK_THREADS This macro expands to ``PyEval_RestoreThread(_save);``: it is equivalent to - :cmacro:`Py_END_ALLOW_THREADS` without the closing brace. It is a no-op when + :c:macro:`Py_END_ALLOW_THREADS` without the closing brace. It is a no-op when thread support is disabled at compile time. -.. cmacro:: Py_UNBLOCK_THREADS +.. c:macro:: Py_UNBLOCK_THREADS This macro expands to ``_save = PyEval_SaveThread();``: it is equivalent to - :cmacro:`Py_BEGIN_ALLOW_THREADS` without the opening brace and variable + :c:macro:`Py_BEGIN_ALLOW_THREADS` without the opening brace and variable declaration. It is a no-op when thread support is disabled at compile time. @@ -672,47 +696,47 @@ at compile time, and must be called only when the global interpreter lock has been created. -.. cfunction:: PyInterpreterState* PyInterpreterState_New() +.. c:function:: PyInterpreterState* PyInterpreterState_New() Create a new interpreter state object. The global interpreter lock need not be held, but may be held if it is necessary to serialize calls to this function. -.. cfunction:: void PyInterpreterState_Clear(PyInterpreterState *interp) +.. c:function:: void PyInterpreterState_Clear(PyInterpreterState *interp) Reset all information in an interpreter state object. The global interpreter lock must be held. -.. cfunction:: void PyInterpreterState_Delete(PyInterpreterState *interp) +.. c:function:: void PyInterpreterState_Delete(PyInterpreterState *interp) Destroy an interpreter state object. The global interpreter lock need not be held. The interpreter state must have been reset with a previous call to - :cfunc:`PyInterpreterState_Clear`. + :c:func:`PyInterpreterState_Clear`. -.. cfunction:: PyThreadState* PyThreadState_New(PyInterpreterState *interp) +.. c:function:: PyThreadState* PyThreadState_New(PyInterpreterState *interp) Create a new thread state object belonging to the given interpreter object. The global interpreter lock need not be held, but may be held if it is necessary to serialize calls to this function. -.. cfunction:: void PyThreadState_Clear(PyThreadState *tstate) +.. c:function:: void PyThreadState_Clear(PyThreadState *tstate) Reset all information in a thread state object. The global interpreter lock must be held. -.. cfunction:: void PyThreadState_Delete(PyThreadState *tstate) +.. c:function:: void PyThreadState_Delete(PyThreadState *tstate) Destroy a thread state object. The global interpreter lock need not be held. The thread state must have been reset with a previous call to - :cfunc:`PyThreadState_Clear`. + :c:func:`PyThreadState_Clear`. -.. cfunction:: PyObject* PyThreadState_GetDict() +.. c:function:: PyObject* PyThreadState_GetDict() Return a dictionary in which extensions can store thread-specific state information. Each extension should use a unique key to use to store state in @@ -721,7 +745,7 @@ been created. the caller should assume no current thread state is available. -.. cfunction:: int PyThreadState_SetAsyncExc(long id, PyObject *exc) +.. c:function:: int PyThreadState_SetAsyncExc(long id, PyObject *exc) Asynchronously raise an exception in a thread. The *id* argument is the thread id of the target thread; *exc* is the exception object to be raised. This @@ -732,18 +756,18 @@ been created. exception (if any) for the thread is cleared. This raises no exceptions. -.. cfunction:: void PyEval_AcquireThread(PyThreadState *tstate) +.. c:function:: void PyEval_AcquireThread(PyThreadState *tstate) Acquire the global interpreter lock and set the current thread state to *tstate*, which should not be *NULL*. The lock must have been created earlier. If this thread already has the lock, deadlock ensues. - :cfunc:`PyEval_RestoreThread` is a higher-level function which is always + :c:func:`PyEval_RestoreThread` is a higher-level function which is always available (even when thread support isn't enabled or when threads have not been initialized). -.. cfunction:: void PyEval_ReleaseThread(PyThreadState *tstate) +.. c:function:: void PyEval_ReleaseThread(PyThreadState *tstate) Reset the current thread state to *NULL* and release the global interpreter lock. The lock must have been created earlier and must be held by the current @@ -751,29 +775,29 @@ been created. that it represents the current thread state --- if it isn't, a fatal error is reported. - :cfunc:`PyEval_SaveThread` is a higher-level function which is always + :c:func:`PyEval_SaveThread` is a higher-level function which is always available (even when thread support isn't enabled or when threads have not been initialized). -.. cfunction:: void PyEval_AcquireLock() +.. c:function:: void PyEval_AcquireLock() Acquire the global interpreter lock. The lock must have been created earlier. If this thread already has the lock, a deadlock ensues. - .. warning:: - This function does not change the current thread state. Please use - :cfunc:`PyEval_RestoreThread` or :cfunc:`PyEval_AcquireThread` + .. deprecated:: 3.2 + This function does not update the current thread state. Please use + :c:func:`PyEval_RestoreThread` or :c:func:`PyEval_AcquireThread` instead. -.. cfunction:: void PyEval_ReleaseLock() +.. c:function:: void PyEval_ReleaseLock() Release the global interpreter lock. The lock must have been created earlier. - .. warning:: - This function does not change the current thread state. Please use - :cfunc:`PyEval_SaveThread` or :cfunc:`PyEval_ReleaseThread` + .. deprecated:: 3.2 + This function does not update the current thread state. Please use + :c:func:`PyEval_SaveThread` or :c:func:`PyEval_ReleaseThread` instead. @@ -784,11 +808,11 @@ While in most uses, you will only embed a single Python interpreter, there are cases where you need to create several independent interpreters in the same process and perhaps even in the same thread. Sub-interpreters allow you to do that. You can switch between sub-interpreters using the -:cfunc:`PyThreadState_Swap` function. You can create and destroy them +:c:func:`PyThreadState_Swap` function. You can create and destroy them using the following functions: -.. cfunction:: PyThreadState* Py_NewInterpreter() +.. c:function:: PyThreadState* Py_NewInterpreter() .. index:: module: builtins @@ -830,13 +854,13 @@ using the following functions: and filled with the contents of this copy; the extension's ``init`` function is not called. Note that this is different from what happens when an extension is imported after the interpreter has been completely re-initialized by calling - :cfunc:`Py_Finalize` and :cfunc:`Py_Initialize`; in that case, the extension's + :c:func:`Py_Finalize` and :c:func:`Py_Initialize`; in that case, the extension's ``initmodule`` function *is* called again. .. index:: single: close() (in module os) -.. cfunction:: void Py_EndInterpreter(PyThreadState *tstate) +.. c:function:: void Py_EndInterpreter(PyThreadState *tstate) .. index:: single: Py_Finalize() @@ -845,7 +869,7 @@ using the following functions: states below. When the call returns, the current thread state is *NULL*. All thread states associated with this interpreter are destroyed. (The global interpreter lock must be held before calling this function and is still held - when it returns.) :cfunc:`Py_Finalize` will destroy all sub-interpreters that + when it returns.) :c:func:`Py_Finalize` will destroy all sub-interpreters that haven't been explicitly destroyed at that point. @@ -866,11 +890,11 @@ instances or classes between sub-interpreters, since import operations executed by such objects may affect the wrong (sub-)interpreter's dictionary of loaded modules. -Also note that combining this functionality with :cfunc:`PyGILState_\*` APIs +Also note that combining this functionality with :c:func:`PyGILState_\*` APIs is delicate, because these APIs assume a bijection between Python thread states and OS-level threads, an assumption broken by the presence of sub-interpreters. It is highly recommended that you don't switch sub-interpreters between a pair -of matching :cfunc:`PyGILState_Ensure` and :cfunc:`PyGILState_Release` calls. +of matching :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` calls. Furthermore, extensions (such as :mod:`ctypes`) using these APIs to allow calling of Python code from non-Python created threads will probably be broken when using sub-interpreters. @@ -892,7 +916,7 @@ a worker thread and the actual call than made at the earliest convenience by the main thread where it has possession of the global interpreter lock and can perform any Python API calls. -.. cfunction:: int Py_AddPendingCall(int (*func)(void *), void *arg) +.. c:function:: int Py_AddPendingCall(int (*func)(void *), void *arg) .. index:: single: Py_AddPendingCall() @@ -937,10 +961,10 @@ events reported to the trace function are the same as had been reported to the Python-level trace functions in previous versions. -.. ctype:: int (*Py_tracefunc)(PyObject *obj, PyFrameObject *frame, int what, PyObject *arg) +.. c:type:: int (*Py_tracefunc)(PyObject *obj, PyFrameObject *frame, int what, PyObject *arg) - The type of the trace function registered using :cfunc:`PyEval_SetProfile` and - :cfunc:`PyEval_SetTrace`. The first parameter is the object passed to the + The type of the trace function registered using :c:func:`PyEval_SetProfile` and + :c:func:`PyEval_SetTrace`. The first parameter is the object passed to the registration function as *obj*, *frame* is the frame object to which the event pertains, *what* is one of the constants :const:`PyTrace_CALL`, :const:`PyTrace_EXCEPTION`, :const:`PyTrace_LINE`, :const:`PyTrace_RETURN`, @@ -968,18 +992,18 @@ Python-level trace functions in previous versions. +------------------------------+--------------------------------------+ -.. cvar:: int PyTrace_CALL +.. c:var:: int PyTrace_CALL - The value of the *what* parameter to a :ctype:`Py_tracefunc` function when a new + The value of the *what* parameter to a :c:type:`Py_tracefunc` function when a new call to a function or method is being reported, or a new entry into a generator. Note that the creation of the iterator for a generator function is not reported as there is no control transfer to the Python bytecode in the corresponding frame. -.. cvar:: int PyTrace_EXCEPTION +.. c:var:: int PyTrace_EXCEPTION - The value of the *what* parameter to a :ctype:`Py_tracefunc` function when an + The value of the *what* parameter to a :c:type:`Py_tracefunc` function when an exception has been raised. The callback function is called with this value for *what* when after any bytecode is processed after which the exception becomes set within the frame being executed. The effect of this is that as exception @@ -988,37 +1012,37 @@ Python-level trace functions in previous versions. these events; they are not needed by the profiler. -.. cvar:: int PyTrace_LINE +.. c:var:: int PyTrace_LINE The value passed as the *what* parameter to a trace function (but not a profiling function) when a line-number event is being reported. -.. cvar:: int PyTrace_RETURN +.. c:var:: int PyTrace_RETURN - The value for the *what* parameter to :ctype:`Py_tracefunc` functions when a + The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a call is returning without propagating an exception. -.. cvar:: int PyTrace_C_CALL +.. c:var:: int PyTrace_C_CALL - The value for the *what* parameter to :ctype:`Py_tracefunc` functions when a C + The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a C function is about to be called. -.. cvar:: int PyTrace_C_EXCEPTION +.. c:var:: int PyTrace_C_EXCEPTION - The value for the *what* parameter to :ctype:`Py_tracefunc` functions when a C + The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a C function has raised an exception. -.. cvar:: int PyTrace_C_RETURN +.. c:var:: int PyTrace_C_RETURN - The value for the *what* parameter to :ctype:`Py_tracefunc` functions when a C + The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a C function has returned. -.. cfunction:: void PyEval_SetProfile(Py_tracefunc func, PyObject *obj) +.. c:function:: void PyEval_SetProfile(Py_tracefunc func, PyObject *obj) Set the profiler function to *func*. The *obj* parameter is passed to the function as its first parameter, and may be any Python object, or *NULL*. If @@ -1028,13 +1052,13 @@ Python-level trace functions in previous versions. events. -.. cfunction:: void PyEval_SetTrace(Py_tracefunc func, PyObject *obj) +.. c:function:: void PyEval_SetTrace(Py_tracefunc func, PyObject *obj) Set the tracing function to *func*. This is similar to - :cfunc:`PyEval_SetProfile`, except the tracing function does receive line-number + :c:func:`PyEval_SetProfile`, except the tracing function does receive line-number events. -.. cfunction:: PyObject* PyEval_GetCallStats(PyObject *self) +.. c:function:: PyObject* PyEval_GetCallStats(PyObject *self) Return a tuple of function call counts. There are constants defined for the positions within the tuple: @@ -1086,25 +1110,25 @@ Advanced Debugger Support These functions are only intended to be used by advanced debugging tools. -.. cfunction:: PyInterpreterState* PyInterpreterState_Head() +.. c:function:: PyInterpreterState* PyInterpreterState_Head() Return the interpreter state object at the head of the list of all such objects. -.. cfunction:: PyInterpreterState* PyInterpreterState_Next(PyInterpreterState *interp) +.. c:function:: PyInterpreterState* PyInterpreterState_Next(PyInterpreterState *interp) Return the next interpreter state object after *interp* from the list of all such objects. -.. cfunction:: PyThreadState * PyInterpreterState_ThreadHead(PyInterpreterState *interp) +.. c:function:: PyThreadState * PyInterpreterState_ThreadHead(PyInterpreterState *interp) - Return the a pointer to the first :ctype:`PyThreadState` object in the list of + Return the a pointer to the first :c:type:`PyThreadState` object in the list of threads associated with the interpreter *interp*. -.. cfunction:: PyThreadState* PyThreadState_Next(PyThreadState *tstate) +.. c:function:: PyThreadState* PyThreadState_Next(PyThreadState *tstate) Return the next thread state object after *tstate* from the list of all such - objects belonging to the same :ctype:`PyInterpreterState` object. + objects belonging to the same :c:type:`PyInterpreterState` object. diff --git a/Doc/c-api/intro.rst b/Doc/c-api/intro.rst index c008b5c93d..e136816688 100644 --- a/Doc/c-api/intro.rst +++ b/Doc/c-api/intro.rst @@ -88,15 +88,15 @@ Objects, Types and Reference Counts .. index:: object: type Most Python/C API functions have one or more arguments as well as a return value -of type :ctype:`PyObject\*`. This type is a pointer to an opaque data type +of type :c:type:`PyObject\*`. This type is a pointer to an opaque data type representing an arbitrary Python object. Since all Python object types are treated the same way by the Python language in most situations (e.g., assignments, scope rules, and argument passing), it is only fitting that they should be represented by a single C type. Almost all Python objects live on the heap: you never declare an automatic or static variable of type -:ctype:`PyObject`, only pointer variables of type :ctype:`PyObject\*` can be +:c:type:`PyObject`, only pointer variables of type :c:type:`PyObject\*` can be declared. The sole exception are the type objects; since these must never be -deallocated, they are typically static :ctype:`PyTypeObject` objects. +deallocated, they are typically static :c:type:`PyTypeObject` objects. All Python objects (even Python integers) have a :dfn:`type` and a :dfn:`reference count`. An object's type determines what kind of object it is @@ -127,8 +127,8 @@ that.") single: Py_DECREF() Reference counts are always manipulated explicitly. The normal way is to use -the macro :cfunc:`Py_INCREF` to increment an object's reference count by one, -and :cfunc:`Py_DECREF` to decrement it by one. The :cfunc:`Py_DECREF` macro +the macro :c:func:`Py_INCREF` to increment an object's reference count by one, +and :c:func:`Py_DECREF` to decrement it by one. The :c:func:`Py_DECREF` macro is considerably more complex than the incref one, since it must check whether the reference count becomes zero and then cause the object's deallocator to be called. The deallocator is a function pointer contained in the object's type @@ -159,13 +159,13 @@ for a while without incrementing its reference count. Some other operation might conceivably remove the object from the list, decrementing its reference count and possible deallocating it. The real danger is that innocent-looking operations may invoke arbitrary Python code which could do this; there is a code -path which allows control to flow back to the user from a :cfunc:`Py_DECREF`, so +path which allows control to flow back to the user from a :c:func:`Py_DECREF`, so almost any operation is potentially dangerous. A safe approach is to always use the generic operations (functions whose name begins with ``PyObject_``, ``PyNumber_``, ``PySequence_`` or ``PyMapping_``). These operations always increment the reference count of the object they return. -This leaves the caller with the responsibility to call :cfunc:`Py_DECREF` when +This leaves the caller with the responsibility to call :c:func:`Py_DECREF` when they are done with the result; this soon becomes second nature. @@ -180,7 +180,7 @@ to objects (objects are not owned: they are always shared). "Owning a reference" means being responsible for calling Py_DECREF on it when the reference is no longer needed. Ownership can also be transferred, meaning that the code that receives ownership of the reference then becomes responsible for -eventually decref'ing it by calling :cfunc:`Py_DECREF` or :cfunc:`Py_XDECREF` +eventually decref'ing it by calling :c:func:`Py_DECREF` or :c:func:`Py_XDECREF` when it's no longer needed---or passing on this responsibility (usually to its caller). When a function passes ownership of a reference on to its caller, the caller is said to receive a *new* reference. When no ownership is transferred, @@ -198,7 +198,7 @@ responsible for it any longer. single: PyTuple_SetItem() Few functions steal references; the two notable exceptions are -:cfunc:`PyList_SetItem` and :cfunc:`PyTuple_SetItem`, which steal a reference +:c:func:`PyList_SetItem` and :c:func:`PyTuple_SetItem`, which steal a reference to the item (but not to the tuple or list into which the item is put!). These functions were designed to steal a reference because of a common idiom for populating a tuple or list with newly created objects; for example, the code to @@ -212,21 +212,21 @@ error handling for the moment; a better way to code this is shown below):: PyTuple_SetItem(t, 1, PyLong_FromLong(2L)); PyTuple_SetItem(t, 2, PyString_FromString("three")); -Here, :cfunc:`PyLong_FromLong` returns a new reference which is immediately -stolen by :cfunc:`PyTuple_SetItem`. When you want to keep using an object -although the reference to it will be stolen, use :cfunc:`Py_INCREF` to grab +Here, :c:func:`PyLong_FromLong` returns a new reference which is immediately +stolen by :c:func:`PyTuple_SetItem`. When you want to keep using an object +although the reference to it will be stolen, use :c:func:`Py_INCREF` to grab another reference before calling the reference-stealing function. -Incidentally, :cfunc:`PyTuple_SetItem` is the *only* way to set tuple items; -:cfunc:`PySequence_SetItem` and :cfunc:`PyObject_SetItem` refuse to do this +Incidentally, :c:func:`PyTuple_SetItem` is the *only* way to set tuple items; +:c:func:`PySequence_SetItem` and :c:func:`PyObject_SetItem` refuse to do this since tuples are an immutable data type. You should only use -:cfunc:`PyTuple_SetItem` for tuples that you are creating yourself. +:c:func:`PyTuple_SetItem` for tuples that you are creating yourself. -Equivalent code for populating a list can be written using :cfunc:`PyList_New` -and :cfunc:`PyList_SetItem`. +Equivalent code for populating a list can be written using :c:func:`PyList_New` +and :c:func:`PyList_SetItem`. However, in practice, you will rarely use these ways of creating and populating -a tuple or list. There's a generic function, :cfunc:`Py_BuildValue`, that can +a tuple or list. There's a generic function, :c:func:`Py_BuildValue`, that can create most common objects from C values, directed by a :dfn:`format string`. For example, the above two blocks of code could be replaced by the following (which also takes care of the error checking):: @@ -236,7 +236,7 @@ For example, the above two blocks of code could be replaced by the following tuple = Py_BuildValue("(iis)", 1, 2, "three"); list = Py_BuildValue("[iis]", 1, 2, "three"); -It is much more common to use :cfunc:`PyObject_SetItem` and friends with items +It is much more common to use :c:func:`PyObject_SetItem` and friends with items whose references you are only borrowing, like arguments that were passed in to the function you are writing. In that case, their behaviour regarding reference counts is much saner, since you don't have to increment a reference count so you @@ -246,17 +246,19 @@ sets all items of a list (actually, any mutable sequence) to a given item:: int set_all(PyObject *target, PyObject *item) { - int i, n; + Py_ssize_t i, n; n = PyObject_Length(target); if (n < 0) return -1; for (i = 0; i < n; i++) { - PyObject *index = PyLong_FromLong(i); + PyObject *index = PyLong_FromSsize_t(i); if (!index) return -1; - if (PyObject_SetItem(target, index, item) < 0) + if (PyObject_SetItem(target, index, item) < 0) { + Py_DECREF(index); return -1; + } Py_DECREF(index); } return 0; @@ -270,15 +272,15 @@ for that reference, many functions that return a reference to an object give you ownership of the reference. The reason is simple: in many cases, the returned object is created on the fly, and the reference you get is the only reference to the object. Therefore, the generic functions that return object -references, like :cfunc:`PyObject_GetItem` and :cfunc:`PySequence_GetItem`, +references, like :c:func:`PyObject_GetItem` and :c:func:`PySequence_GetItem`, always return a new reference (the caller becomes the owner of the reference). It is important to realize that whether you own a reference returned by a function depends on which function you call only --- *the plumage* (the type of the object passed as an argument to the function) *doesn't enter into it!* -Thus, if you extract an item from a list using :cfunc:`PyList_GetItem`, you +Thus, if you extract an item from a list using :c:func:`PyList_GetItem`, you don't own the reference --- but if you obtain the same item from the same list -using :cfunc:`PySequence_GetItem` (which happens to take exactly the same +using :c:func:`PySequence_GetItem` (which happens to take exactly the same arguments), you do own a reference to the returned object. .. index:: @@ -286,14 +288,14 @@ arguments), you do own a reference to the returned object. single: PySequence_GetItem() Here is an example of how you could write a function that computes the sum of -the items in a list of integers; once using :cfunc:`PyList_GetItem`, and once -using :cfunc:`PySequence_GetItem`. :: +the items in a list of integers; once using :c:func:`PyList_GetItem`, and once +using :c:func:`PySequence_GetItem`. :: long sum_list(PyObject *list) { - int i, n; - long total = 0; + Py_ssize_t i, n; + long total = 0, value; PyObject *item; n = PyList_Size(list); @@ -302,7 +304,11 @@ using :cfunc:`PySequence_GetItem`. :: for (i = 0; i < n; i++) { item = PyList_GetItem(list, i); /* Can't fail */ if (!PyLong_Check(item)) continue; /* Skip non-integers */ - total += PyLong_AsLong(item); + value = PyLong_AsLong(item); + if (value == -1 && PyErr_Occurred()) + /* Integer too big to fit in a C long, bail out */ + return -1; + total += value; } return total; } @@ -314,8 +320,8 @@ using :cfunc:`PySequence_GetItem`. :: long sum_sequence(PyObject *sequence) { - int i, n; - long total = 0; + Py_ssize_t i, n; + long total = 0, value; PyObject *item; n = PySequence_Length(sequence); if (n < 0) @@ -324,9 +330,17 @@ using :cfunc:`PySequence_GetItem`. :: item = PySequence_GetItem(sequence, i); if (item == NULL) return -1; /* Not a sequence, or other failure */ - if (PyLong_Check(item)) - total += PyLong_AsLong(item); - Py_DECREF(item); /* Discard reference ownership */ + if (PyLong_Check(item)) { + value = PyLong_AsLong(item); + Py_DECREF(item); + if (value == -1 && PyErr_Occurred()) + /* Integer too big to fit in a C long, bail out */ + return -1; + total += value; + } + else { + Py_DECREF(item); /* Discard reference ownership */ + } } return total; } @@ -340,8 +354,8 @@ Types ----- There are few other data types that play a significant role in the Python/C -API; most are simple C types such as :ctype:`int`, :ctype:`long`, -:ctype:`double` and :ctype:`char\*`. A few structure types are used to +API; most are simple C types such as :c:type:`int`, :c:type:`long`, +:c:type:`double` and :c:type:`char\*`. A few structure types are used to describe static tables used to list the functions exported by a module or the data attributes of a new object type, and another is used to describe the value of a complex number. These will be discussed together with the functions that @@ -370,7 +384,7 @@ indicator is either *NULL* or ``-1``, depending on the function's return type. A few functions return a Boolean true/false result, with false indicating an error. Very few functions return no explicit error indicator or have an ambiguous return value, and require explicit testing for errors with -:cfunc:`PyErr_Occurred`. These exceptions are always explicitly documented. +:c:func:`PyErr_Occurred`. These exceptions are always explicitly documented. .. index:: single: PyErr_SetString() @@ -379,11 +393,11 @@ ambiguous return value, and require explicit testing for errors with Exception state is maintained in per-thread storage (this is equivalent to using global storage in an unthreaded application). A thread can be in one of two states: an exception has occurred, or not. The function -:cfunc:`PyErr_Occurred` can be used to check for this: it returns a borrowed +:c:func:`PyErr_Occurred` can be used to check for this: it returns a borrowed reference to the exception type object when an exception has occurred, and *NULL* otherwise. There are a number of functions to set the exception state: -:cfunc:`PyErr_SetString` is the most common (though not the most general) -function to set the exception state, and :cfunc:`PyErr_Clear` clears the +:c:func:`PyErr_SetString` is the most common (though not the most general) +function to set the exception state, and :c:func:`PyErr_Clear` clears the exception state. The full exception state consists of three objects (all of which can be @@ -419,7 +433,7 @@ and lose important information about the exact cause of the error. .. index:: single: sum_sequence() A simple example of detecting exceptions and passing them on is shown in the -:cfunc:`sum_sequence` example above. It so happens that that example doesn't +:c:func:`sum_sequence` example above. It so happens that that example doesn't need to clean up any owned references when it detects an error. The following example function shows some error cleanup. First, to remind you why you like Python, we show the equivalent Python code:: @@ -486,10 +500,10 @@ Here is the corresponding C code, in all its glory:: single: Py_XDECREF() This example represents an endorsed use of the ``goto`` statement in C! -It illustrates the use of :cfunc:`PyErr_ExceptionMatches` and -:cfunc:`PyErr_Clear` to handle specific exceptions, and the use of -:cfunc:`Py_XDECREF` to dispose of owned references that may be *NULL* (note the -``'X'`` in the name; :cfunc:`Py_DECREF` would crash when confronted with a +It illustrates the use of :c:func:`PyErr_ExceptionMatches` and +:c:func:`PyErr_Clear` to handle specific exceptions, and the use of +:c:func:`Py_XDECREF` to dispose of owned references that may be *NULL* (note the +``'X'`` in the name; :c:func:`Py_DECREF` would crash when confronted with a *NULL* reference). It is important that the variables used to hold owned references are initialized to *NULL* for this to work; likewise, the proposed return value is initialized to ``-1`` (failure) and only set to success after @@ -514,20 +528,20 @@ interpreter can only be used after the interpreter has been initialized. triple: module; search; path single: path (in module sys) -The basic initialization function is :cfunc:`Py_Initialize`. This initializes +The basic initialization function is :c:func:`Py_Initialize`. This initializes the table of loaded modules, and creates the fundamental modules :mod:`builtins`, :mod:`__main__`, and :mod:`sys`. It also initializes the module search path (``sys.path``). -.. index:: single: PySys_SetArgv() +.. index:: single: PySys_SetArgvEx() -:cfunc:`Py_Initialize` does not set the "script argument list" (``sys.argv``). -If this variable is needed by Python code that will be executed later, it must -be set explicitly with a call to ``PySys_SetArgv(argc, argv)`` subsequent to -the call to :cfunc:`Py_Initialize`. +:c:func:`Py_Initialize` does not set the "script argument list" (``sys.argv``). +If this variable is needed by Python code that will be executed later, it must +be set explicitly with a call to ``PySys_SetArgvEx(argc, argv, updatepath)`` +after the call to :c:func:`Py_Initialize`. On most systems (in particular, on Unix and Windows, although the details are -slightly different), :cfunc:`Py_Initialize` calculates the module search path +slightly different), :c:func:`Py_Initialize` calculates the module search path based upon its best guess for the location of the standard Python interpreter executable, assuming that the Python library is found in a fixed location relative to the Python interpreter executable. In particular, it looks for a @@ -551,22 +565,22 @@ front of the standard path by setting :envvar:`PYTHONPATH`. single: Py_GetProgramFullPath() The embedding application can steer the search by calling -``Py_SetProgramName(file)`` *before* calling :cfunc:`Py_Initialize`. Note that +``Py_SetProgramName(file)`` *before* calling :c:func:`Py_Initialize`. Note that :envvar:`PYTHONHOME` still overrides this and :envvar:`PYTHONPATH` is still inserted in front of the standard path. An application that requires total -control has to provide its own implementation of :cfunc:`Py_GetPath`, -:cfunc:`Py_GetPrefix`, :cfunc:`Py_GetExecPrefix`, and -:cfunc:`Py_GetProgramFullPath` (all defined in :file:`Modules/getpath.c`). +control has to provide its own implementation of :c:func:`Py_GetPath`, +:c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`, and +:c:func:`Py_GetProgramFullPath` (all defined in :file:`Modules/getpath.c`). .. index:: single: Py_IsInitialized() Sometimes, it is desirable to "uninitialize" Python. For instance, the application may want to start over (make another call to -:cfunc:`Py_Initialize`) or the application is simply done with its use of +:c:func:`Py_Initialize`) or the application is simply done with its use of Python and wants to free memory allocated by Python. This can be accomplished -by calling :cfunc:`Py_Finalize`. The function :cfunc:`Py_IsInitialized` returns +by calling :c:func:`Py_Finalize`. The function :c:func:`Py_IsInitialized` returns true if Python is currently in the initialized state. More information about -these functions is given in a later chapter. Notice that :cfunc:`Py_Finalize` +these functions is given in a later chapter. Notice that :c:func:`Py_Finalize` does *not* free all memory allocated by the Python interpreter, e.g. memory allocated by extension modules currently cannot be released. @@ -586,11 +600,11 @@ available that support tracing of reference counts, debugging the memory allocator, or low-level profiling of the main interpreter loop. Only the most frequently-used builds will be described in the remainder of this section. -Compiling the interpreter with the :cmacro:`Py_DEBUG` macro defined produces -what is generally meant by "a debug build" of Python. :cmacro:`Py_DEBUG` is -enabled in the Unix build by adding :option:`--with-pydebug` to the -:file:`configure` command. It is also implied by the presence of the -not-Python-specific :cmacro:`_DEBUG` macro. When :cmacro:`Py_DEBUG` is enabled +Compiling the interpreter with the :c:macro:`Py_DEBUG` macro defined produces +what is generally meant by "a debug build" of Python. :c:macro:`Py_DEBUG` is +enabled in the Unix build by adding ``--with-pydebug`` to the +:file:`./configure` command. It is also implied by the presence of the +not-Python-specific :c:macro:`_DEBUG` macro. When :c:macro:`Py_DEBUG` is enabled in the Unix build, compiler optimization is disabled. In addition to the reference count debugging described below, the following @@ -619,11 +633,11 @@ extra checks are performed: There may be additional checks not mentioned here. -Defining :cmacro:`Py_TRACE_REFS` enables reference tracing. When defined, a +Defining :c:macro:`Py_TRACE_REFS` enables reference tracing. When defined, a circular doubly linked list of active objects is maintained by adding two extra -fields to every :ctype:`PyObject`. Total allocations are tracked as well. Upon +fields to every :c:type:`PyObject`. Total allocations are tracked as well. Upon exit, all existing references are printed. (In interactive mode this happens -after every statement run by the interpreter.) Implied by :cmacro:`Py_DEBUG`. +after every statement run by the interpreter.) Implied by :c:macro:`Py_DEBUG`. Please refer to :file:`Misc/SpecialBuilds.txt` in the Python source distribution for more detailed information. diff --git a/Doc/c-api/iter.rst b/Doc/c-api/iter.rst index ba7e9e3a18..3f0f554dad 100644 --- a/Doc/c-api/iter.rst +++ b/Doc/c-api/iter.rst @@ -7,12 +7,12 @@ Iterator Protocol There are only a couple of functions specifically for working with iterators. -.. cfunction:: int PyIter_Check(PyObject *o) +.. c:function:: int PyIter_Check(PyObject *o) Return true if the object *o* supports the iterator protocol. -.. cfunction:: PyObject* PyIter_Next(PyObject *o) +.. c:function:: PyObject* PyIter_Next(PyObject *o) Return the next value from the iteration *o*. If the object is an iterator, this retrieves the next value from the iteration, and returns *NULL* with no diff --git a/Doc/c-api/iterator.rst b/Doc/c-api/iterator.rst index 86650800a4..82cb4eba8a 100644 --- a/Doc/c-api/iterator.rst +++ b/Doc/c-api/iterator.rst @@ -12,37 +12,37 @@ the callable for each item in the sequence, and ending the iteration when the sentinel value is returned. -.. cvar:: PyTypeObject PySeqIter_Type +.. c:var:: PyTypeObject PySeqIter_Type - Type object for iterator objects returned by :cfunc:`PySeqIter_New` and the + Type object for iterator objects returned by :c:func:`PySeqIter_New` and the one-argument form of the :func:`iter` built-in function for built-in sequence types. -.. cfunction:: int PySeqIter_Check(op) +.. c:function:: int PySeqIter_Check(op) - Return true if the type of *op* is :cdata:`PySeqIter_Type`. + Return true if the type of *op* is :c:data:`PySeqIter_Type`. -.. cfunction:: PyObject* PySeqIter_New(PyObject *seq) +.. c:function:: PyObject* PySeqIter_New(PyObject *seq) Return an iterator that works with a general sequence object, *seq*. The iteration ends when the sequence raises :exc:`IndexError` for the subscripting operation. -.. cvar:: PyTypeObject PyCallIter_Type +.. c:var:: PyTypeObject PyCallIter_Type - Type object for iterator objects returned by :cfunc:`PyCallIter_New` and the + Type object for iterator objects returned by :c:func:`PyCallIter_New` and the two-argument form of the :func:`iter` built-in function. -.. cfunction:: int PyCallIter_Check(op) +.. c:function:: int PyCallIter_Check(op) - Return true if the type of *op* is :cdata:`PyCallIter_Type`. + Return true if the type of *op* is :c:data:`PyCallIter_Type`. -.. cfunction:: PyObject* PyCallIter_New(PyObject *callable, PyObject *sentinel) +.. c:function:: PyObject* PyCallIter_New(PyObject *callable, PyObject *sentinel) Return a new iterator. The first parameter, *callable*, can be any Python callable object that can be called with no parameters; each call to it should diff --git a/Doc/c-api/list.rst b/Doc/c-api/list.rst index 4e3d500b69..feb9015b6d 100644 --- a/Doc/c-api/list.rst +++ b/Doc/c-api/list.rst @@ -8,30 +8,30 @@ List Objects .. index:: object: list -.. ctype:: PyListObject +.. c:type:: PyListObject - This subtype of :ctype:`PyObject` represents a Python list object. + This subtype of :c:type:`PyObject` represents a Python list object. -.. cvar:: PyTypeObject PyList_Type +.. c:var:: PyTypeObject PyList_Type - This instance of :ctype:`PyTypeObject` represents the Python list type. + This instance of :c:type:`PyTypeObject` represents the Python list type. This is the same object as :class:`list` in the Python layer. -.. cfunction:: int PyList_Check(PyObject *p) +.. c:function:: int PyList_Check(PyObject *p) Return true if *p* is a list object or an instance of a subtype of the list type. -.. cfunction:: int PyList_CheckExact(PyObject *p) +.. c:function:: int PyList_CheckExact(PyObject *p) Return true if *p* is a list object, but not an instance of a subtype of the list type. -.. cfunction:: PyObject* PyList_New(Py_ssize_t len) +.. c:function:: PyObject* PyList_New(Py_ssize_t len) Return a new list of length *len* on success, or *NULL* on failure. @@ -39,11 +39,11 @@ List Objects If *len* is greater than zero, the returned list object's items are set to ``NULL``. Thus you cannot use abstract API functions such as - :cfunc:`PySequence_SetItem` or expose the object to Python code before - setting all items to a real object with :cfunc:`PyList_SetItem`. + :c:func:`PySequence_SetItem` or expose the object to Python code before + setting all items to a real object with :c:func:`PyList_SetItem`. -.. cfunction:: Py_ssize_t PyList_Size(PyObject *list) +.. c:function:: Py_ssize_t PyList_Size(PyObject *list) .. index:: builtin: len @@ -51,12 +51,12 @@ List Objects ``len(list)`` on a list object. -.. cfunction:: Py_ssize_t PyList_GET_SIZE(PyObject *list) +.. c:function:: Py_ssize_t PyList_GET_SIZE(PyObject *list) - Macro form of :cfunc:`PyList_Size` without error checking. + Macro form of :c:func:`PyList_Size` without error checking. -.. cfunction:: PyObject* PyList_GetItem(PyObject *list, Py_ssize_t index) +.. c:function:: PyObject* PyList_GetItem(PyObject *list, Py_ssize_t index) Return the object at position *index* in the list pointed to by *list*. The position must be positive, indexing from the end of the list is not @@ -64,12 +64,12 @@ List Objects :exc:`IndexError` exception. -.. cfunction:: PyObject* PyList_GET_ITEM(PyObject *list, Py_ssize_t i) +.. c:function:: PyObject* PyList_GET_ITEM(PyObject *list, Py_ssize_t i) - Macro form of :cfunc:`PyList_GetItem` without error checking. + Macro form of :c:func:`PyList_GetItem` without error checking. -.. cfunction:: int PyList_SetItem(PyObject *list, Py_ssize_t index, PyObject *item) +.. c:function:: int PyList_SetItem(PyObject *list, Py_ssize_t index, PyObject *item) Set the item at index *index* in list to *item*. Return ``0`` on success or ``-1`` on failure. @@ -80,34 +80,34 @@ List Objects an item already in the list at the affected position. -.. cfunction:: void PyList_SET_ITEM(PyObject *list, Py_ssize_t i, PyObject *o) +.. c:function:: void PyList_SET_ITEM(PyObject *list, Py_ssize_t i, PyObject *o) - Macro form of :cfunc:`PyList_SetItem` without error checking. This is + Macro form of :c:func:`PyList_SetItem` without error checking. This is normally only used to fill in new lists where there is no previous content. .. note:: This macro "steals" a reference to *item*, and, unlike - :cfunc:`PyList_SetItem`, does *not* discard a reference to any item that + :c:func:`PyList_SetItem`, does *not* discard a reference to any item that is being replaced; any reference in *list* at position *i* will be leaked. -.. cfunction:: int PyList_Insert(PyObject *list, Py_ssize_t index, PyObject *item) +.. c:function:: int PyList_Insert(PyObject *list, Py_ssize_t index, PyObject *item) Insert the item *item* into list *list* in front of index *index*. Return ``0`` if successful; return ``-1`` and set an exception if unsuccessful. Analogous to ``list.insert(index, item)``. -.. cfunction:: int PyList_Append(PyObject *list, PyObject *item) +.. c:function:: int PyList_Append(PyObject *list, PyObject *item) Append the object *item* at the end of list *list*. Return ``0`` if successful; return ``-1`` and set an exception if unsuccessful. Analogous to ``list.append(item)``. -.. cfunction:: PyObject* PyList_GetSlice(PyObject *list, Py_ssize_t low, Py_ssize_t high) +.. c:function:: PyObject* PyList_GetSlice(PyObject *list, Py_ssize_t low, Py_ssize_t high) Return a list of the objects in *list* containing the objects *between* *low* and *high*. Return *NULL* and set an exception if unsuccessful. Analogous @@ -115,7 +115,7 @@ List Objects supported. -.. cfunction:: int PyList_SetSlice(PyObject *list, Py_ssize_t low, Py_ssize_t high, PyObject *itemlist) +.. c:function:: int PyList_SetSlice(PyObject *list, Py_ssize_t low, Py_ssize_t high, PyObject *itemlist) Set the slice of *list* between *low* and *high* to the contents of *itemlist*. Analogous to ``list[low:high] = itemlist``. The *itemlist* may @@ -124,19 +124,19 @@ List Objects slicing from Python, are not supported. -.. cfunction:: int PyList_Sort(PyObject *list) +.. c:function:: int PyList_Sort(PyObject *list) Sort the items of *list* in place. Return ``0`` on success, ``-1`` on failure. This is equivalent to ``list.sort()``. -.. cfunction:: int PyList_Reverse(PyObject *list) +.. c:function:: int PyList_Reverse(PyObject *list) Reverse the items of *list* in place. Return ``0`` on success, ``-1`` on failure. This is the equivalent of ``list.reverse()``. -.. cfunction:: PyObject* PyList_AsTuple(PyObject *list) +.. c:function:: PyObject* PyList_AsTuple(PyObject *list) .. index:: builtin: tuple diff --git a/Doc/c-api/long.rst b/Doc/c-api/long.rst index 9f8e3c5f64..b2295e0fad 100644 --- a/Doc/c-api/long.rst +++ b/Doc/c-api/long.rst @@ -10,32 +10,32 @@ Integer Objects All integers are implemented as "long" integer objects of arbitrary size. -.. ctype:: PyLongObject +.. c:type:: PyLongObject - This subtype of :ctype:`PyObject` represents a Python integer object. + This subtype of :c:type:`PyObject` represents a Python integer object. -.. cvar:: PyTypeObject PyLong_Type +.. c:var:: PyTypeObject PyLong_Type - This instance of :ctype:`PyTypeObject` represents the Python integer type. + This instance of :c:type:`PyTypeObject` represents the Python integer type. This is the same object as :class:`int` in the Python layer. -.. cfunction:: int PyLong_Check(PyObject *p) +.. c:function:: int PyLong_Check(PyObject *p) - Return true if its argument is a :ctype:`PyLongObject` or a subtype of - :ctype:`PyLongObject`. + Return true if its argument is a :c:type:`PyLongObject` or a subtype of + :c:type:`PyLongObject`. -.. cfunction:: int PyLong_CheckExact(PyObject *p) +.. c:function:: int PyLong_CheckExact(PyObject *p) - Return true if its argument is a :ctype:`PyLongObject`, but not a subtype of - :ctype:`PyLongObject`. + Return true if its argument is a :c:type:`PyLongObject`, but not a subtype of + :c:type:`PyLongObject`. -.. cfunction:: PyObject* PyLong_FromLong(long v) +.. c:function:: PyObject* PyLong_FromLong(long v) - Return a new :ctype:`PyLongObject` object from *v*, or *NULL* on failure. + Return a new :c:type:`PyLongObject` object from *v*, or *NULL* on failure. The current implementation keeps an array of integer objects for all integers between ``-5`` and ``256``, when you create an int in that range you actually @@ -44,47 +44,47 @@ All integers are implemented as "long" integer objects of arbitrary size. undefined. :-) -.. cfunction:: PyObject* PyLong_FromUnsignedLong(unsigned long v) +.. c:function:: PyObject* PyLong_FromUnsignedLong(unsigned long v) - Return a new :ctype:`PyLongObject` object from a C :ctype:`unsigned long`, or + Return a new :c:type:`PyLongObject` object from a C :c:type:`unsigned long`, or *NULL* on failure. -.. cfunction:: PyObject* PyLong_FromSsize_t(Py_ssize_t v) +.. c:function:: PyObject* PyLong_FromSsize_t(Py_ssize_t v) - Return a new :ctype:`PyLongObject` object from a C :ctype:`Py_ssize_t`, or + Return a new :c:type:`PyLongObject` object from a C :c:type:`Py_ssize_t`, or *NULL* on failure. -.. cfunction:: PyObject* PyLong_FromSize_t(size_t v) +.. c:function:: PyObject* PyLong_FromSize_t(size_t v) - Return a new :ctype:`PyLongObject` object from a C :ctype:`size_t`, or + Return a new :c:type:`PyLongObject` object from a C :c:type:`size_t`, or *NULL* on failure. -.. cfunction:: PyObject* PyLong_FromLongLong(PY_LONG_LONG v) +.. c:function:: PyObject* PyLong_FromLongLong(PY_LONG_LONG v) - Return a new :ctype:`PyLongObject` object from a C :ctype:`long long`, or *NULL* + Return a new :c:type:`PyLongObject` object from a C :c:type:`long long`, or *NULL* on failure. -.. cfunction:: PyObject* PyLong_FromUnsignedLongLong(unsigned PY_LONG_LONG v) +.. c:function:: PyObject* PyLong_FromUnsignedLongLong(unsigned PY_LONG_LONG v) - Return a new :ctype:`PyLongObject` object from a C :ctype:`unsigned long long`, + Return a new :c:type:`PyLongObject` object from a C :c:type:`unsigned long long`, or *NULL* on failure. -.. cfunction:: PyObject* PyLong_FromDouble(double v) +.. c:function:: PyObject* PyLong_FromDouble(double v) - Return a new :ctype:`PyLongObject` object from the integer part of *v*, or + Return a new :c:type:`PyLongObject` object from the integer part of *v*, or *NULL* on failure. -.. cfunction:: PyObject* PyLong_FromString(char *str, char **pend, int base) +.. c:function:: PyObject* PyLong_FromString(char *str, char **pend, int base) - Return a new :ctype:`PyLongObject` based on the string value in *str*, which + Return a new :c:type:`PyLongObject` based on the string value in *str*, which is interpreted according to the radix in *base*. If *pend* is non-*NULL*, - ``*pend`` will point to the first character in *str* which follows the + *\*pend* will point to the first character in *str* which follows the representation of the number. If *base* is ``0``, the radix will be determined based on the leading characters of *str*: if *str* starts with ``'0x'`` or ``'0X'``, radix 16 will be used; if *str* starts with ``'0o'`` or @@ -94,85 +94,101 @@ All integers are implemented as "long" integer objects of arbitrary size. ignored. If there are no digits, :exc:`ValueError` will be raised. -.. cfunction:: PyObject* PyLong_FromUnicode(Py_UNICODE *u, Py_ssize_t length, int base) +.. c:function:: PyObject* PyLong_FromUnicode(Py_UNICODE *u, Py_ssize_t length, int base) Convert a sequence of Unicode digits to a Python integer value. The Unicode - string is first encoded to a byte string using :cfunc:`PyUnicode_EncodeDecimal` - and then converted using :cfunc:`PyLong_FromString`. + string is first encoded to a byte string using :c:func:`PyUnicode_EncodeDecimal` + and then converted using :c:func:`PyLong_FromString`. -.. cfunction:: PyObject* PyLong_FromVoidPtr(void *p) +.. c:function:: PyObject* PyLong_FromVoidPtr(void *p) Create a Python integer from the pointer *p*. The pointer value can be - retrieved from the resulting value using :cfunc:`PyLong_AsVoidPtr`. + retrieved from the resulting value using :c:func:`PyLong_AsVoidPtr`. .. XXX alias PyLong_AS_LONG (for now) -.. cfunction:: long PyLong_AsLong(PyObject *pylong) +.. c:function:: long PyLong_AsLong(PyObject *pylong) .. index:: single: LONG_MAX single: OverflowError (built-in exception) - Return a C :ctype:`long` representation of the contents of *pylong*. If + Return a C :c:type:`long` representation of the contents of *pylong*. If *pylong* is greater than :const:`LONG_MAX`, raise an :exc:`OverflowError`, and return -1. Convert non-long objects automatically to long first, and return -1 if that raises exceptions. -.. cfunction:: long PyLong_AsLongAndOverflow(PyObject *pylong, int* overflow) +.. c:function:: long PyLong_AsLongAndOverflow(PyObject *pylong, int *overflow) - Return a C :ctype:`long` representation of the contents of *pylong*. If - *pylong* is greater than :const:`LONG_MAX`, return -1 and - set `*overflow` to 1 (for overflow) or -1 (for underflow). - If an exception is set because of type errors, also return -1. + Return a C :c:type:`long` representation of the contents of + *pylong*. If *pylong* is greater than :const:`LONG_MAX` or less + than :const:`LONG_MIN`, set *\*overflow* to ``1`` or ``-1``, + respectively, and return ``-1``; otherwise, set *\*overflow* to + ``0``. If any other exception occurs (for example a TypeError or + MemoryError), then ``-1`` will be returned and *\*overflow* will + be ``0``. -.. cfunction:: Py_ssize_t PyLong_AsSsize_t(PyObject *pylong) +.. c:function:: PY_LONG_LONG PyLong_AsLongLongAndOverflow(PyObject *pylong, int *overflow) + + Return a C :c:type:`long long` representation of the contents of + *pylong*. If *pylong* is greater than :const:`PY_LLONG_MAX` or less + than :const:`PY_LLONG_MIN`, set *\*overflow* to ``1`` or ``-1``, + respectively, and return ``-1``; otherwise, set *\*overflow* to + ``0``. If any other exception occurs (for example a TypeError or + MemoryError), then ``-1`` will be returned and *\*overflow* will + be ``0``. + + .. versionadded:: 3.2 + + +.. c:function:: Py_ssize_t PyLong_AsSsize_t(PyObject *pylong) .. index:: single: PY_SSIZE_T_MAX single: OverflowError (built-in exception) - Return a C :ctype:`Py_ssize_t` representation of the contents of *pylong*. + Return a C :c:type:`Py_ssize_t` representation of the contents of *pylong*. If *pylong* is greater than :const:`PY_SSIZE_T_MAX`, an :exc:`OverflowError` is raised and ``-1`` will be returned. -.. cfunction:: unsigned long PyLong_AsUnsignedLong(PyObject *pylong) +.. c:function:: unsigned long PyLong_AsUnsignedLong(PyObject *pylong) .. index:: single: ULONG_MAX single: OverflowError (built-in exception) - Return a C :ctype:`unsigned long` representation of the contents of *pylong*. + Return a C :c:type:`unsigned long` representation of the contents of *pylong*. If *pylong* is greater than :const:`ULONG_MAX`, an :exc:`OverflowError` is raised. -.. cfunction:: size_t PyLong_AsSize_t(PyObject *pylong) +.. c:function:: size_t PyLong_AsSize_t(PyObject *pylong) - Return a :ctype:`size_t` representation of the contents of *pylong*. If - *pylong* is greater than the maximum value for a :ctype:`size_t`, an + Return a :c:type:`size_t` representation of the contents of *pylong*. If + *pylong* is greater than the maximum value for a :c:type:`size_t`, an :exc:`OverflowError` is raised. -.. cfunction:: PY_LONG_LONG PyLong_AsLongLong(PyObject *pylong) +.. c:function:: PY_LONG_LONG PyLong_AsLongLong(PyObject *pylong) .. index:: single: OverflowError (built-in exception) - Return a C :ctype:`long long` from a Python integer. If *pylong* - cannot be represented as a :ctype:`long long`, an + Return a C :c:type:`long long` from a Python integer. If *pylong* + cannot be represented as a :c:type:`long long`, an :exc:`OverflowError` is raised and ``-1`` is returned. -.. cfunction:: unsigned PY_LONG_LONG PyLong_AsUnsignedLongLong(PyObject *pylong) +.. c:function:: unsigned PY_LONG_LONG PyLong_AsUnsignedLongLong(PyObject *pylong) .. index:: single: OverflowError (built-in exception) - Return a C :ctype:`unsigned long long` from a Python integer. If - *pylong* cannot be represented as an :ctype:`unsigned long long`, + Return a C :c:type:`unsigned long long` from a Python integer. If + *pylong* cannot be represented as an :c:type:`unsigned long long`, an :exc:`OverflowError` is raised and ``(unsigned long long)-1`` is returned. @@ -180,28 +196,28 @@ All integers are implemented as "long" integer objects of arbitrary size. A negative *pylong* now raises :exc:`OverflowError`, not :exc:`TypeError`. -.. cfunction:: unsigned long PyLong_AsUnsignedLongMask(PyObject *io) +.. c:function:: unsigned long PyLong_AsUnsignedLongMask(PyObject *io) - Return a C :ctype:`unsigned long` from a Python integer, without checking for + Return a C :c:type:`unsigned long` from a Python integer, without checking for overflow. -.. cfunction:: unsigned PY_LONG_LONG PyLong_AsUnsignedLongLongMask(PyObject *io) +.. c:function:: unsigned PY_LONG_LONG PyLong_AsUnsignedLongLongMask(PyObject *io) - Return a C :ctype:`unsigned long long` from a Python integer, without + Return a C :c:type:`unsigned long long` from a Python integer, without checking for overflow. -.. cfunction:: double PyLong_AsDouble(PyObject *pylong) +.. c:function:: double PyLong_AsDouble(PyObject *pylong) - Return a C :ctype:`double` representation of the contents of *pylong*. If - *pylong* cannot be approximately represented as a :ctype:`double`, an + Return a C :c:type:`double` representation of the contents of *pylong*. If + *pylong* cannot be approximately represented as a :c:type:`double`, an :exc:`OverflowError` exception is raised and ``-1.0`` will be returned. -.. cfunction:: void* PyLong_AsVoidPtr(PyObject *pylong) +.. c:function:: void* PyLong_AsVoidPtr(PyObject *pylong) - Convert a Python integer *pylong* to a C :ctype:`void` pointer. + Convert a Python integer *pylong* to a C :c:type:`void` pointer. If *pylong* cannot be converted, an :exc:`OverflowError` will be raised. This - is only assured to produce a usable :ctype:`void` pointer for values created - with :cfunc:`PyLong_FromVoidPtr`. + is only assured to produce a usable :c:type:`void` pointer for values created + with :c:func:`PyLong_FromVoidPtr`. diff --git a/Doc/c-api/mapping.rst b/Doc/c-api/mapping.rst index 5b2de14783..0ef29616e5 100644 --- a/Doc/c-api/mapping.rst +++ b/Doc/c-api/mapping.rst @@ -6,13 +6,13 @@ Mapping Protocol ================ -.. cfunction:: int PyMapping_Check(PyObject *o) +.. c:function:: int PyMapping_Check(PyObject *o) Return ``1`` if the object provides mapping protocol, and ``0`` otherwise. This function always succeeds. -.. cfunction:: Py_ssize_t PyMapping_Size(PyObject *o) +.. c:function:: Py_ssize_t PyMapping_Size(PyObject *o) Py_ssize_t PyMapping_Length(PyObject *o) .. index:: builtin: len @@ -22,58 +22,58 @@ Mapping Protocol expression ``len(o)``. -.. cfunction:: int PyMapping_DelItemString(PyObject *o, char *key) +.. c:function:: int PyMapping_DelItemString(PyObject *o, char *key) Remove the mapping for object *key* from the object *o*. Return ``-1`` on failure. This is equivalent to the Python statement ``del o[key]``. -.. cfunction:: int PyMapping_DelItem(PyObject *o, PyObject *key) +.. c:function:: int PyMapping_DelItem(PyObject *o, PyObject *key) Remove the mapping for object *key* from the object *o*. Return ``-1`` on failure. This is equivalent to the Python statement ``del o[key]``. -.. cfunction:: int PyMapping_HasKeyString(PyObject *o, char *key) +.. c:function:: int PyMapping_HasKeyString(PyObject *o, char *key) On success, return ``1`` if the mapping object has the key *key* and ``0`` otherwise. This is equivalent to the Python expression ``key in o``. This function always succeeds. -.. cfunction:: int PyMapping_HasKey(PyObject *o, PyObject *key) +.. c:function:: int PyMapping_HasKey(PyObject *o, PyObject *key) Return ``1`` if the mapping object has the key *key* and ``0`` otherwise. This is equivalent to the Python expression ``key in o``. This function always succeeds. -.. cfunction:: PyObject* PyMapping_Keys(PyObject *o) +.. c:function:: PyObject* PyMapping_Keys(PyObject *o) On success, return a list of the keys in object *o*. On failure, return *NULL*. This is equivalent to the Python expression ``list(o.keys())``. -.. cfunction:: PyObject* PyMapping_Values(PyObject *o) +.. c:function:: PyObject* PyMapping_Values(PyObject *o) On success, return a list of the values in object *o*. On failure, return *NULL*. This is equivalent to the Python expression ``list(o.values())``. -.. cfunction:: PyObject* PyMapping_Items(PyObject *o) +.. c:function:: PyObject* PyMapping_Items(PyObject *o) On success, return a list of the items in object *o*, where each item is a tuple containing a key-value pair. On failure, return *NULL*. This is equivalent to the Python expression ``list(o.items())``. -.. cfunction:: PyObject* PyMapping_GetItemString(PyObject *o, char *key) +.. c:function:: PyObject* PyMapping_GetItemString(PyObject *o, char *key) Return element of *o* corresponding to the object *key* or *NULL* on failure. This is the equivalent of the Python expression ``o[key]``. -.. cfunction:: int PyMapping_SetItemString(PyObject *o, char *key, PyObject *v) +.. c:function:: int PyMapping_SetItemString(PyObject *o, char *key, PyObject *v) Map the object *key* to the value *v* in object *o*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``o[key] = v``. diff --git a/Doc/c-api/marshal.rst b/Doc/c-api/marshal.rst index 04b0b88e6f..da402a8086 100644 --- a/Doc/c-api/marshal.rst +++ b/Doc/c-api/marshal.rst @@ -19,20 +19,20 @@ unmarshalling. Version 2 uses a binary format for floating point numbers. *Py_MARSHAL_VERSION* indicates the current file format (currently 2). -.. cfunction:: void PyMarshal_WriteLongToFile(long value, FILE *file, int version) +.. c:function:: void PyMarshal_WriteLongToFile(long value, FILE *file, int version) - Marshal a :ctype:`long` integer, *value*, to *file*. This will only write + Marshal a :c:type:`long` integer, *value*, to *file*. This will only write the least-significant 32 bits of *value*; regardless of the size of the - native :ctype:`long` type. *version* indicates the file format. + native :c:type:`long` type. *version* indicates the file format. -.. cfunction:: void PyMarshal_WriteObjectToFile(PyObject *value, FILE *file, int version) +.. c:function:: void PyMarshal_WriteObjectToFile(PyObject *value, FILE *file, int version) Marshal a Python object, *value*, to *file*. *version* indicates the file format. -.. cfunction:: PyObject* PyMarshal_WriteObjectToString(PyObject *value, int version) +.. c:function:: PyObject* PyMarshal_WriteObjectToString(PyObject *value, int version) Return a string object containing the marshalled representation of *value*. *version* indicates the file format. @@ -47,31 +47,31 @@ no error. What's the right way to tell? Should only non-negative values be written using these routines? -.. cfunction:: long PyMarshal_ReadLongFromFile(FILE *file) +.. c:function:: long PyMarshal_ReadLongFromFile(FILE *file) - Return a C :ctype:`long` from the data stream in a :ctype:`FILE\*` opened + Return a C :c:type:`long` from the data stream in a :c:type:`FILE\*` opened for reading. Only a 32-bit value can be read in using this function, - regardless of the native size of :ctype:`long`. + regardless of the native size of :c:type:`long`. -.. cfunction:: int PyMarshal_ReadShortFromFile(FILE *file) +.. c:function:: int PyMarshal_ReadShortFromFile(FILE *file) - Return a C :ctype:`short` from the data stream in a :ctype:`FILE\*` opened + Return a C :c:type:`short` from the data stream in a :c:type:`FILE\*` opened for reading. Only a 16-bit value can be read in using this function, - regardless of the native size of :ctype:`short`. + regardless of the native size of :c:type:`short`. -.. cfunction:: PyObject* PyMarshal_ReadObjectFromFile(FILE *file) +.. c:function:: PyObject* PyMarshal_ReadObjectFromFile(FILE *file) - Return a Python object from the data stream in a :ctype:`FILE\*` opened for + Return a Python object from the data stream in a :c:type:`FILE\*` opened for reading. On error, sets the appropriate exception (:exc:`EOFError` or :exc:`TypeError`) and returns *NULL*. -.. cfunction:: PyObject* PyMarshal_ReadLastObjectFromFile(FILE *file) +.. c:function:: PyObject* PyMarshal_ReadLastObjectFromFile(FILE *file) - Return a Python object from the data stream in a :ctype:`FILE\*` opened for - reading. Unlike :cfunc:`PyMarshal_ReadObjectFromFile`, this function + Return a Python object from the data stream in a :c:type:`FILE\*` opened for + reading. Unlike :c:func:`PyMarshal_ReadObjectFromFile`, this function assumes that no further objects will be read from the file, allowing it to aggressively load file data into memory so that the de-serialization can operate from data in memory rather than reading a byte at a time from the @@ -80,7 +80,7 @@ written using these routines? (:exc:`EOFError` or :exc:`TypeError`) and returns *NULL*. -.. cfunction:: PyObject* PyMarshal_ReadObjectFromString(char *string, Py_ssize_t len) +.. c:function:: PyObject* PyMarshal_ReadObjectFromString(char *string, Py_ssize_t len) Return a Python object from the data stream in a character buffer containing *len* bytes pointed to by *string*. On error, sets the diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 81d7cd96cf..b80b3d53d9 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -47,8 +47,8 @@ API functions listed in this document. single: free() To avoid memory corruption, extension writers should never try to operate on -Python objects with the functions exported by the C library: :cfunc:`malloc`, -:cfunc:`calloc`, :cfunc:`realloc` and :cfunc:`free`. This will result in mixed +Python objects with the functions exported by the C library: :c:func:`malloc`, +:c:func:`calloc`, :c:func:`realloc` and :c:func:`free`. This will result in mixed calls between the C allocator and the Python memory manager with fatal consequences, because they implement different algorithms and operate on different heaps. However, one may safely allocate and release memory blocks @@ -94,65 +94,65 @@ behavior when requesting zero bytes, are available for allocating and releasing memory from the Python heap: -.. cfunction:: void* PyMem_Malloc(size_t n) +.. c:function:: void* PyMem_Malloc(size_t n) - Allocates *n* bytes and returns a pointer of type :ctype:`void\*` to the + Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the request fails. Requesting zero bytes returns - a distinct non-*NULL* pointer if possible, as if :cfunc:`PyMem_Malloc(1)` had + a distinct non-*NULL* pointer if possible, as if :c:func:`PyMem_Malloc(1)` had been called instead. The memory will not have been initialized in any way. -.. cfunction:: void* PyMem_Realloc(void *p, size_t n) +.. c:function:: void* PyMem_Realloc(void *p, size_t n) Resizes the memory block pointed to by *p* to *n* bytes. The contents will be unchanged to the minimum of the old and the new sizes. If *p* is *NULL*, the - call is equivalent to :cfunc:`PyMem_Malloc(n)`; else if *n* is equal to zero, + call is equivalent to :c:func:`PyMem_Malloc(n)`; else if *n* is equal to zero, the memory block is resized but is not freed, and the returned pointer is non-*NULL*. Unless *p* is *NULL*, it must have been returned by a previous call - to :cfunc:`PyMem_Malloc` or :cfunc:`PyMem_Realloc`. If the request fails, - :cfunc:`PyMem_Realloc` returns *NULL* and *p* remains a valid pointer to the + to :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. If the request fails, + :c:func:`PyMem_Realloc` returns *NULL* and *p* remains a valid pointer to the previous memory area. -.. cfunction:: void PyMem_Free(void *p) +.. c:function:: void PyMem_Free(void *p) Frees the memory block pointed to by *p*, which must have been returned by a - previous call to :cfunc:`PyMem_Malloc` or :cfunc:`PyMem_Realloc`. Otherwise, or - if :cfunc:`PyMem_Free(p)` has been called before, undefined behavior occurs. If + previous call to :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. Otherwise, or + if :c:func:`PyMem_Free(p)` has been called before, undefined behavior occurs. If *p* is *NULL*, no operation is performed. The following type-oriented macros are provided for convenience. Note that *TYPE* refers to any C type. -.. cfunction:: TYPE* PyMem_New(TYPE, size_t n) +.. c:function:: TYPE* PyMem_New(TYPE, size_t n) - Same as :cfunc:`PyMem_Malloc`, but allocates ``(n * sizeof(TYPE))`` bytes of - memory. Returns a pointer cast to :ctype:`TYPE\*`. The memory will not have + Same as :c:func:`PyMem_Malloc`, but allocates ``(n * sizeof(TYPE))`` bytes of + memory. Returns a pointer cast to :c:type:`TYPE\*`. The memory will not have been initialized in any way. -.. cfunction:: TYPE* PyMem_Resize(void *p, TYPE, size_t n) +.. c:function:: TYPE* PyMem_Resize(void *p, TYPE, size_t n) - Same as :cfunc:`PyMem_Realloc`, but the memory block is resized to ``(n * - sizeof(TYPE))`` bytes. Returns a pointer cast to :ctype:`TYPE\*`. On return, + Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n * + sizeof(TYPE))`` bytes. Returns a pointer cast to :c:type:`TYPE\*`. On return, *p* will be a pointer to the new memory area, or *NULL* in the event of failure. This is a C preprocessor macro; p is always reassigned. Save the original value of p to avoid losing memory when handling errors. -.. cfunction:: void PyMem_Del(void *p) +.. c:function:: void PyMem_Del(void *p) - Same as :cfunc:`PyMem_Free`. + Same as :c:func:`PyMem_Free`. In addition, the following macro sets are provided for calling the Python memory allocator directly, without involving the C API functions listed above. However, note that their use does not preserve binary compatibility across Python versions and is therefore deprecated in extension modules. -:cfunc:`PyMem_MALLOC`, :cfunc:`PyMem_REALLOC`, :cfunc:`PyMem_FREE`. +:c:func:`PyMem_MALLOC`, :c:func:`PyMem_REALLOC`, :c:func:`PyMem_FREE`. -:cfunc:`PyMem_NEW`, :cfunc:`PyMem_RESIZE`, :cfunc:`PyMem_DEL`. +:c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`. .. _memoryexamples: @@ -201,8 +201,8 @@ allocators operating on different heaps. :: free(buf1); /* Fatal -- should be PyMem_Del() */ In addition to the functions aimed at handling raw memory blocks from the Python -heap, objects in Python are allocated and released with :cfunc:`PyObject_New`, -:cfunc:`PyObject_NewVar` and :cfunc:`PyObject_Del`. +heap, objects in Python are allocated and released with :c:func:`PyObject_New`, +:c:func:`PyObject_NewVar` and :c:func:`PyObject_Del`. These will be explained in the next chapter on defining and implementing new object types in C. diff --git a/Doc/c-api/memoryview.rst b/Doc/c-api/memoryview.rst index 9003d3e26d..6b49cdf70a 100644 --- a/Doc/c-api/memoryview.rst +++ b/Doc/c-api/memoryview.rst @@ -13,37 +13,37 @@ A :class:`memoryview` object exposes the C level :ref:`buffer interface any other object. -.. cfunction:: PyObject *PyMemoryView_FromObject(PyObject *obj) +.. c:function:: PyObject *PyMemoryView_FromObject(PyObject *obj) Create a memoryview object from an object that provides the buffer interface. If *obj* supports writable buffer exports, the memoryview object will be - readable and writable, other it will be read-only. + readable and writable, otherwise it will be read-only. -.. cfunction:: PyObject *PyMemoryView_FromBuffer(Py_buffer *view) +.. c:function:: PyObject *PyMemoryView_FromBuffer(Py_buffer *view) Create a memoryview object wrapping the given buffer structure *view*. The memoryview object then owns the buffer represented by *view*, which - means you shouldn't try to call :cfunc:`PyBuffer_Release` yourself: it + means you shouldn't try to call :c:func:`PyBuffer_Release` yourself: it will be done on deallocation of the memoryview object. -.. cfunction:: PyObject *PyMemoryView_GetContiguous(PyObject *obj, int buffertype, char order) +.. c:function:: PyObject *PyMemoryView_GetContiguous(PyObject *obj, int buffertype, char order) Create a memoryview object to a contiguous chunk of memory (in either 'C' or 'F'ortran *order*) from an object that defines the buffer interface. If memory is contiguous, the memoryview object points to the - original memory. Otherwise copy is made and the memoryview points to a + original memory. Otherwise, a copy is made and the memoryview points to a new bytes object. -.. cfunction:: int PyMemoryView_Check(PyObject *obj) +.. c:function:: int PyMemoryView_Check(PyObject *obj) Return true if the object *obj* is a memoryview object. It is not currently allowed to create subclasses of :class:`memoryview`. -.. cfunction:: Py_buffer *PyMemoryView_GET_BUFFER(PyObject *obj) +.. c:function:: Py_buffer *PyMemoryView_GET_BUFFER(PyObject *obj) Return a pointer to the buffer structure wrapped by the given memoryview object. The object **must** be a memoryview instance; diff --git a/Doc/c-api/method.rst b/Doc/c-api/method.rst index d8b2ed89ee..acc81e4814 100644 --- a/Doc/c-api/method.rst +++ b/Doc/c-api/method.rst @@ -7,38 +7,38 @@ Instance Method Objects .. index:: object: instancemethod -An instance method is a wrapper for a :cdata:`PyCFunction` and the new way -to bind a :cdata:`PyCFunction` to a class object. It replaces the former call +An instance method is a wrapper for a :c:data:`PyCFunction` and the new way +to bind a :c:data:`PyCFunction` to a class object. It replaces the former call ``PyMethod_New(func, NULL, class)``. -.. cvar:: PyTypeObject PyInstanceMethod_Type +.. c:var:: PyTypeObject PyInstanceMethod_Type - This instance of :ctype:`PyTypeObject` represents the Python instance + This instance of :c:type:`PyTypeObject` represents the Python instance method type. It is not exposed to Python programs. -.. cfunction:: int PyInstanceMethod_Check(PyObject *o) +.. c:function:: int PyInstanceMethod_Check(PyObject *o) Return true if *o* is an instance method object (has type - :cdata:`PyInstanceMethod_Type`). The parameter must not be *NULL*. + :c:data:`PyInstanceMethod_Type`). The parameter must not be *NULL*. -.. cfunction:: PyObject* PyInstanceMethod_New(PyObject *func) +.. c:function:: PyObject* PyInstanceMethod_New(PyObject *func) Return a new instance method object, with *func* being any callable object - *func* is is the function that will be called when the instance method is + *func* is the function that will be called when the instance method is called. -.. cfunction:: PyObject* PyInstanceMethod_Function(PyObject *im) +.. c:function:: PyObject* PyInstanceMethod_Function(PyObject *im) Return the function object associated with the instance method *im*. -.. cfunction:: PyObject* PyInstanceMethod_GET_FUNCTION(PyObject *im) +.. c:function:: PyObject* PyInstanceMethod_GET_FUNCTION(PyObject *im) - Macro version of :cfunc:`PyInstanceMethod_Function` which avoids error checking. + Macro version of :c:func:`PyInstanceMethod_Function` which avoids error checking. .. _method-objects: @@ -53,48 +53,48 @@ an user-defined class. Unbound methods (methods bound to a class object) are no longer available. -.. cvar:: PyTypeObject PyMethod_Type +.. c:var:: PyTypeObject PyMethod_Type .. index:: single: MethodType (in module types) - This instance of :ctype:`PyTypeObject` represents the Python method type. This + This instance of :c:type:`PyTypeObject` represents the Python method type. This is exposed to Python programs as ``types.MethodType``. -.. cfunction:: int PyMethod_Check(PyObject *o) +.. c:function:: int PyMethod_Check(PyObject *o) - Return true if *o* is a method object (has type :cdata:`PyMethod_Type`). The + Return true if *o* is a method object (has type :c:data:`PyMethod_Type`). The parameter must not be *NULL*. -.. cfunction:: PyObject* PyMethod_New(PyObject *func, PyObject *self) +.. c:function:: PyObject* PyMethod_New(PyObject *func, PyObject *self) Return a new method object, with *func* being any callable object and *self* - the instance the method should be bound. *func* is is the function that will + the instance the method should be bound. *func* is the function that will be called when the method is called. *self* must not be *NULL*. -.. cfunction:: PyObject* PyMethod_Function(PyObject *meth) +.. c:function:: PyObject* PyMethod_Function(PyObject *meth) Return the function object associated with the method *meth*. -.. cfunction:: PyObject* PyMethod_GET_FUNCTION(PyObject *meth) +.. c:function:: PyObject* PyMethod_GET_FUNCTION(PyObject *meth) - Macro version of :cfunc:`PyMethod_Function` which avoids error checking. + Macro version of :c:func:`PyMethod_Function` which avoids error checking. -.. cfunction:: PyObject* PyMethod_Self(PyObject *meth) +.. c:function:: PyObject* PyMethod_Self(PyObject *meth) Return the instance associated with the method *meth*. -.. cfunction:: PyObject* PyMethod_GET_SELF(PyObject *meth) +.. c:function:: PyObject* PyMethod_GET_SELF(PyObject *meth) - Macro version of :cfunc:`PyMethod_Self` which avoids error checking. + Macro version of :c:func:`PyMethod_Self` which avoids error checking. -.. cfunction:: int PyMethod_ClearFreeList() +.. c:function:: int PyMethod_ClearFreeList() Clear the free list. Return the total number of freed items. diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index 12816c9fd8..ffd68e3084 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -10,26 +10,26 @@ Module Objects There are only a few functions special to module objects. -.. cvar:: PyTypeObject PyModule_Type +.. c:var:: PyTypeObject PyModule_Type .. index:: single: ModuleType (in module types) - This instance of :ctype:`PyTypeObject` represents the Python module type. This + This instance of :c:type:`PyTypeObject` represents the Python module type. This is exposed to Python programs as ``types.ModuleType``. -.. cfunction:: int PyModule_Check(PyObject *p) +.. c:function:: int PyModule_Check(PyObject *p) Return true if *p* is a module object, or a subtype of a module object. -.. cfunction:: int PyModule_CheckExact(PyObject *p) +.. c:function:: int PyModule_CheckExact(PyObject *p) Return true if *p* is a module object, but not a subtype of - :cdata:`PyModule_Type`. + :c:data:`PyModule_Type`. -.. cfunction:: PyObject* PyModule_New(const char *name) +.. c:function:: PyObject* PyModule_New(const char *name) .. index:: single: __name__ (module attribute) @@ -41,18 +41,18 @@ There are only a few functions special to module objects. the caller is responsible for providing a :attr:`__file__` attribute. -.. cfunction:: PyObject* PyModule_GetDict(PyObject *module) +.. c:function:: PyObject* PyModule_GetDict(PyObject *module) .. index:: single: __dict__ (module attribute) Return the dictionary object that implements *module*'s namespace; this object is the same as the :attr:`__dict__` attribute of the module object. This function never fails. It is recommended extensions use other - :cfunc:`PyModule_\*` and :cfunc:`PyObject_\*` functions rather than directly + :c:func:`PyModule_\*` and :c:func:`PyObject_\*` functions rather than directly manipulate a module's :attr:`__dict__`. -.. cfunction:: char* PyModule_GetName(PyObject *module) +.. c:function:: char* PyModule_GetName(PyObject *module) .. index:: single: __name__ (module attribute) @@ -62,29 +62,42 @@ There are only a few functions special to module objects. or if it is not a string, :exc:`SystemError` is raised and *NULL* is returned. -.. cfunction:: char* PyModule_GetFilename(PyObject *module) +.. c:function:: char* PyModule_GetFilename(PyObject *module) + + Similar to :c:func:`PyModule_GetFilenameObject` but return the filename + encoded to 'utf-8'. + + .. deprecated:: 3.2 + :c:func:`PyModule_GetFilename` raises :c:type:`UnicodeEncodeError` on + unencodable filenames, use :c:func:`PyModule_GetFilenameObject` instead. + + +.. c:function:: PyObject* PyModule_GetFilenameObject(PyObject *module) .. index:: single: __file__ (module attribute) single: SystemError (built-in exception) Return the name of the file from which *module* was loaded using *module*'s - :attr:`__file__` attribute. If this is not defined, or if it is not a string, - raise :exc:`SystemError` and return *NULL*. + :attr:`__file__` attribute. If this is not defined, or if it is not a + unicode string, raise :exc:`SystemError` and return *NULL*; otherwise return + a reference to a :c:type:`PyUnicodeObject`. + + .. versionadded:: 3.2 -.. cfunction:: void* PyModule_GetState(PyObject *module) +.. c:function:: void* PyModule_GetState(PyObject *module) Return the "state" of the module, that is, a pointer to the block of memory allocated at module creation time, or *NULL*. See - :cmember:`PyModuleDef.m_size`. + :c:member:`PyModuleDef.m_size`. -.. cfunction:: PyModuleDef* PyModule_GetDef(PyObject *module) +.. c:function:: PyModuleDef* PyModule_GetDef(PyObject *module) - Return a pointer to the :ctype:`PyModuleDef` struct from which the module was + Return a pointer to the :c:type:`PyModuleDef` struct from which the module was created, or *NULL* if the module wasn't created with - :cfunc:`PyModule_Create`. + :c:func:`PyModule_Create`. Initializing C modules @@ -92,14 +105,14 @@ Initializing C modules These functions are usually used in the module initialization function. -.. cfunction:: PyObject* PyModule_Create(PyModuleDef *module) +.. c:function:: PyObject* PyModule_Create(PyModuleDef *module) Create a new module object, given the definition in *module*. This behaves - like :cfunc:`PyModule_Create2` with *module_api_version* set to + like :c:func:`PyModule_Create2` with *module_api_version* set to :const:`PYTHON_API_VERSION`. -.. cfunction:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version) +.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version) Create a new module object, given the definition in *module*, assuming the API version *module_api_version*. If that version does not match the version @@ -107,89 +120,89 @@ These functions are usually used in the module initialization function. .. note:: - Most uses of this function should be using :cfunc:`PyModule_Create` + Most uses of this function should be using :c:func:`PyModule_Create` instead; only use this if you are sure you need it. -.. ctype:: PyModuleDef +.. c:type:: PyModuleDef This struct holds all information that is needed to create a module object. There is usually only one static variable of that type for each module, which - is statically initialized and then passed to :cfunc:`PyModule_Create` in the + is statically initialized and then passed to :c:func:`PyModule_Create` in the module initialization function. - .. cmember:: PyModuleDef_Base m_base + .. c:member:: PyModuleDef_Base m_base Always initialize this member to :const:`PyModuleDef_HEAD_INIT`. - .. cmember:: char* m_name + .. c:member:: char* m_name Name for the new module. - .. cmember:: char* m_doc + .. c:member:: char* m_doc Docstring for the module; usually a docstring variable created with - :cfunc:`PyDoc_STRVAR` is used. + :c:func:`PyDoc_STRVAR` is used. - .. cmember:: Py_ssize_t m_size + .. c:member:: Py_ssize_t m_size If the module object needs additional memory, this should be set to the number of bytes to allocate; a pointer to the block of memory can be - retrieved with :cfunc:`PyModule_GetState`. If no memory is needed, set + retrieved with :c:func:`PyModule_GetState`. If no memory is needed, set this to ``-1``. This memory should be used, rather than static globals, to hold per-module state, since it is then safe for use in multiple sub-interpreters. It is - freed when the module object is deallocated, after the :cmember:`m_free` + freed when the module object is deallocated, after the :c:member:`m_free` function has been called, if present. - .. cmember:: PyMethodDef* m_methods + .. c:member:: PyMethodDef* m_methods A pointer to a table of module-level functions, described by - :ctype:`PyMethodDef` values. Can be *NULL* if no functions are present. + :c:type:`PyMethodDef` values. Can be *NULL* if no functions are present. - .. cmember:: inquiry m_reload + .. c:member:: inquiry m_reload Currently unused, should be *NULL*. - .. cmember:: traverseproc m_traverse + .. c:member:: traverseproc m_traverse A traversal function to call during GC traversal of the module object, or *NULL* if not needed. - .. cmember:: inquiry m_clear + .. c:member:: inquiry m_clear A clear function to call during GC clearing of the module object, or *NULL* if not needed. - .. cmember:: freefunc m_free + .. c:member:: freefunc m_free A function to call during deallocation of the module object, or *NULL* if not needed. -.. cfunction:: int PyModule_AddObject(PyObject *module, const char *name, PyObject *value) +.. c:function:: int PyModule_AddObject(PyObject *module, const char *name, PyObject *value) Add an object to *module* as *name*. This is a convenience function which can be used from the module's initialization function. This steals a reference to *value*. Return ``-1`` on error, ``0`` on success. -.. cfunction:: int PyModule_AddIntConstant(PyObject *module, const char *name, long value) +.. c:function:: int PyModule_AddIntConstant(PyObject *module, const char *name, long value) Add an integer constant to *module* as *name*. This convenience function can be used from the module's initialization function. Return ``-1`` on error, ``0`` on success. -.. cfunction:: int PyModule_AddStringConstant(PyObject *module, const char *name, const char *value) +.. c:function:: int PyModule_AddStringConstant(PyObject *module, const char *name, const char *value) Add a string constant to *module* as *name*. This convenience function can be used from the module's initialization function. The string *value* must be null-terminated. Return ``-1`` on error, ``0`` on success. -.. cfunction:: int PyModule_AddIntMacro(PyObject *module, macro) +.. c:function:: int PyModule_AddIntMacro(PyObject *module, macro) Add an int constant to *module*. The name and the value are taken from *macro*. For example ``PyModule_AddIntMacro(module, AF_INET)`` adds the int @@ -197,6 +210,6 @@ These functions are usually used in the module initialization function. Return ``-1`` on error, ``0`` on success. -.. cfunction:: int PyModule_AddStringMacro(PyObject *module, macro) +.. c:function:: int PyModule_AddStringMacro(PyObject *module, macro) Add a string constant to *module*. diff --git a/Doc/c-api/none.rst b/Doc/c-api/none.rst index 70e2c04bd0..b9ac26921d 100644 --- a/Doc/c-api/none.rst +++ b/Doc/c-api/none.rst @@ -7,20 +7,20 @@ The None Object .. index:: object: None -Note that the :ctype:`PyTypeObject` for ``None`` is not directly exposed in the +Note that the :c:type:`PyTypeObject` for ``None`` is not directly exposed in the Python/C API. Since ``None`` is a singleton, testing for object identity (using -``==`` in C) is sufficient. There is no :cfunc:`PyNone_Check` function for the +``==`` in C) is sufficient. There is no :c:func:`PyNone_Check` function for the same reason. -.. cvar:: PyObject* Py_None +.. c:var:: PyObject* Py_None The Python ``None`` object, denoting lack of value. This object has no methods. It needs to be treated just like any other object with respect to reference counts. -.. cmacro:: Py_RETURN_NONE +.. c:macro:: Py_RETURN_NONE - Properly handle returning :cdata:`Py_None` from within a C function (that is, + Properly handle returning :c:data:`Py_None` from within a C function (that is, increment the reference count of None and return it.) diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst index 57ef293ded..21951c38c0 100644 --- a/Doc/c-api/number.rst +++ b/Doc/c-api/number.rst @@ -6,37 +6,37 @@ Number Protocol =============== -.. cfunction:: int PyNumber_Check(PyObject *o) +.. c:function:: int PyNumber_Check(PyObject *o) Returns ``1`` if the object *o* provides numeric protocols, and false otherwise. This function always succeeds. -.. cfunction:: PyObject* PyNumber_Add(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_Add(PyObject *o1, PyObject *o2) Returns the result of adding *o1* and *o2*, or *NULL* on failure. This is the equivalent of the Python expression ``o1 + o2``. -.. cfunction:: PyObject* PyNumber_Subtract(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_Subtract(PyObject *o1, PyObject *o2) Returns the result of subtracting *o2* from *o1*, or *NULL* on failure. This is the equivalent of the Python expression ``o1 - o2``. -.. cfunction:: PyObject* PyNumber_Multiply(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_Multiply(PyObject *o1, PyObject *o2) Returns the result of multiplying *o1* and *o2*, or *NULL* on failure. This is the equivalent of the Python expression ``o1 * o2``. -.. cfunction:: PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2) Return the floor of *o1* divided by *o2*, or *NULL* on failure. This is equivalent to the "classic" division of integers. -.. cfunction:: PyObject* PyNumber_TrueDivide(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_TrueDivide(PyObject *o1, PyObject *o2) Return a reasonable approximation for the mathematical value of *o1* divided by *o2*, or *NULL* on failure. The return value is "approximate" because binary @@ -45,13 +45,13 @@ Number Protocol passed two integers. -.. cfunction:: PyObject* PyNumber_Remainder(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_Remainder(PyObject *o1, PyObject *o2) Returns the remainder of dividing *o1* by *o2*, or *NULL* on failure. This is the equivalent of the Python expression ``o1 % o2``. -.. cfunction:: PyObject* PyNumber_Divmod(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_Divmod(PyObject *o1, PyObject *o2) .. index:: builtin: divmod @@ -59,29 +59,29 @@ Number Protocol the equivalent of the Python expression ``divmod(o1, o2)``. -.. cfunction:: PyObject* PyNumber_Power(PyObject *o1, PyObject *o2, PyObject *o3) +.. c:function:: PyObject* PyNumber_Power(PyObject *o1, PyObject *o2, PyObject *o3) .. index:: builtin: pow See the built-in function :func:`pow`. Returns *NULL* on failure. This is the equivalent of the Python expression ``pow(o1, o2, o3)``, where *o3* is optional. - If *o3* is to be ignored, pass :cdata:`Py_None` in its place (passing *NULL* for + If *o3* is to be ignored, pass :c:data:`Py_None` in its place (passing *NULL* for *o3* would cause an illegal memory access). -.. cfunction:: PyObject* PyNumber_Negative(PyObject *o) +.. c:function:: PyObject* PyNumber_Negative(PyObject *o) Returns the negation of *o* on success, or *NULL* on failure. This is the equivalent of the Python expression ``-o``. -.. cfunction:: PyObject* PyNumber_Positive(PyObject *o) +.. c:function:: PyObject* PyNumber_Positive(PyObject *o) Returns *o* on success, or *NULL* on failure. This is the equivalent of the Python expression ``+o``. -.. cfunction:: PyObject* PyNumber_Absolute(PyObject *o) +.. c:function:: PyObject* PyNumber_Absolute(PyObject *o) .. index:: builtin: abs @@ -89,71 +89,71 @@ Number Protocol of the Python expression ``abs(o)``. -.. cfunction:: PyObject* PyNumber_Invert(PyObject *o) +.. c:function:: PyObject* PyNumber_Invert(PyObject *o) Returns the bitwise negation of *o* on success, or *NULL* on failure. This is the equivalent of the Python expression ``~o``. -.. cfunction:: PyObject* PyNumber_Lshift(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_Lshift(PyObject *o1, PyObject *o2) Returns the result of left shifting *o1* by *o2* on success, or *NULL* on failure. This is the equivalent of the Python expression ``o1 << o2``. -.. cfunction:: PyObject* PyNumber_Rshift(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_Rshift(PyObject *o1, PyObject *o2) Returns the result of right shifting *o1* by *o2* on success, or *NULL* on failure. This is the equivalent of the Python expression ``o1 >> o2``. -.. cfunction:: PyObject* PyNumber_And(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_And(PyObject *o1, PyObject *o2) Returns the "bitwise and" of *o1* and *o2* on success and *NULL* on failure. This is the equivalent of the Python expression ``o1 & o2``. -.. cfunction:: PyObject* PyNumber_Xor(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_Xor(PyObject *o1, PyObject *o2) Returns the "bitwise exclusive or" of *o1* by *o2* on success, or *NULL* on failure. This is the equivalent of the Python expression ``o1 ^ o2``. -.. cfunction:: PyObject* PyNumber_Or(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_Or(PyObject *o1, PyObject *o2) Returns the "bitwise or" of *o1* and *o2* on success, or *NULL* on failure. This is the equivalent of the Python expression ``o1 | o2``. -.. cfunction:: PyObject* PyNumber_InPlaceAdd(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_InPlaceAdd(PyObject *o1, PyObject *o2) Returns the result of adding *o1* and *o2*, or *NULL* on failure. The operation is done *in-place* when *o1* supports it. This is the equivalent of the Python statement ``o1 += o2``. -.. cfunction:: PyObject* PyNumber_InPlaceSubtract(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_InPlaceSubtract(PyObject *o1, PyObject *o2) Returns the result of subtracting *o2* from *o1*, or *NULL* on failure. The operation is done *in-place* when *o1* supports it. This is the equivalent of the Python statement ``o1 -= o2``. -.. cfunction:: PyObject* PyNumber_InPlaceMultiply(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_InPlaceMultiply(PyObject *o1, PyObject *o2) Returns the result of multiplying *o1* and *o2*, or *NULL* on failure. The operation is done *in-place* when *o1* supports it. This is the equivalent of the Python statement ``o1 *= o2``. -.. cfunction:: PyObject* PyNumber_InPlaceFloorDivide(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_InPlaceFloorDivide(PyObject *o1, PyObject *o2) Returns the mathematical floor of dividing *o1* by *o2*, or *NULL* on failure. The operation is done *in-place* when *o1* supports it. This is the equivalent of the Python statement ``o1 //= o2``. -.. cfunction:: PyObject* PyNumber_InPlaceTrueDivide(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_InPlaceTrueDivide(PyObject *o1, PyObject *o2) Return a reasonable approximation for the mathematical value of *o1* divided by *o2*, or *NULL* on failure. The return value is "approximate" because binary @@ -162,72 +162,60 @@ Number Protocol passed two integers. The operation is done *in-place* when *o1* supports it. -.. cfunction:: PyObject* PyNumber_InPlaceRemainder(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_InPlaceRemainder(PyObject *o1, PyObject *o2) Returns the remainder of dividing *o1* by *o2*, or *NULL* on failure. The operation is done *in-place* when *o1* supports it. This is the equivalent of the Python statement ``o1 %= o2``. -.. cfunction:: PyObject* PyNumber_InPlacePower(PyObject *o1, PyObject *o2, PyObject *o3) +.. c:function:: PyObject* PyNumber_InPlacePower(PyObject *o1, PyObject *o2, PyObject *o3) .. index:: builtin: pow See the built-in function :func:`pow`. Returns *NULL* on failure. The operation is done *in-place* when *o1* supports it. This is the equivalent of the Python - statement ``o1 **= o2`` when o3 is :cdata:`Py_None`, or an in-place variant of - ``pow(o1, o2, o3)`` otherwise. If *o3* is to be ignored, pass :cdata:`Py_None` + statement ``o1 **= o2`` when o3 is :c:data:`Py_None`, or an in-place variant of + ``pow(o1, o2, o3)`` otherwise. If *o3* is to be ignored, pass :c:data:`Py_None` in its place (passing *NULL* for *o3* would cause an illegal memory access). -.. cfunction:: PyObject* PyNumber_InPlaceLshift(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_InPlaceLshift(PyObject *o1, PyObject *o2) Returns the result of left shifting *o1* by *o2* on success, or *NULL* on failure. The operation is done *in-place* when *o1* supports it. This is the equivalent of the Python statement ``o1 <<= o2``. -.. cfunction:: PyObject* PyNumber_InPlaceRshift(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_InPlaceRshift(PyObject *o1, PyObject *o2) Returns the result of right shifting *o1* by *o2* on success, or *NULL* on failure. The operation is done *in-place* when *o1* supports it. This is the equivalent of the Python statement ``o1 >>= o2``. -.. cfunction:: PyObject* PyNumber_InPlaceAnd(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_InPlaceAnd(PyObject *o1, PyObject *o2) Returns the "bitwise and" of *o1* and *o2* on success and *NULL* on failure. The operation is done *in-place* when *o1* supports it. This is the equivalent of the Python statement ``o1 &= o2``. -.. cfunction:: PyObject* PyNumber_InPlaceXor(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_InPlaceXor(PyObject *o1, PyObject *o2) Returns the "bitwise exclusive or" of *o1* by *o2* on success, or *NULL* on failure. The operation is done *in-place* when *o1* supports it. This is the equivalent of the Python statement ``o1 ^= o2``. -.. cfunction:: PyObject* PyNumber_InPlaceOr(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PyNumber_InPlaceOr(PyObject *o1, PyObject *o2) Returns the "bitwise or" of *o1* and *o2* on success, or *NULL* on failure. The operation is done *in-place* when *o1* supports it. This is the equivalent of the Python statement ``o1 |= o2``. -.. cfunction:: PyObject* PyNumber_Int(PyObject *o) - - Returns the *o* converted to an integer object on success, or *NULL* on - failure. This is the equivalent of the Python expression ``int(o)``. - - .. note:: - - This function is defined in the transitional :file:`intobject.h` - header file. It will be removed completely in Python 3.1. Use - the :cfunc:`PyNumber_Long` function instead. - - -.. cfunction:: PyObject* PyNumber_Long(PyObject *o) +.. c:function:: PyObject* PyNumber_Long(PyObject *o) .. index:: builtin: int @@ -235,7 +223,7 @@ Number Protocol failure. This is the equivalent of the Python expression ``int(o)``. -.. cfunction:: PyObject* PyNumber_Float(PyObject *o) +.. c:function:: PyObject* PyNumber_Float(PyObject *o) .. index:: builtin: float @@ -243,25 +231,27 @@ Number Protocol This is the equivalent of the Python expression ``float(o)``. -.. cfunction:: PyObject* PyNumber_Index(PyObject *o) +.. c:function:: PyObject* PyNumber_Index(PyObject *o) Returns the *o* converted to a Python int on success or *NULL* with a :exc:`TypeError` exception raised on failure. -.. cfunction:: PyObject* PyNumber_ToBase(PyObject *n, int base) +.. c:function:: PyObject* PyNumber_ToBase(PyObject *n, int base) - Returns the integer *n* converted to *base* as a string with a base - marker of ``'0b'``, ``'0o'``, or ``'0x'`` if applicable. When - *base* is not 2, 8, 10, or 16, the format is ``'x#num'`` where x is the - base. If *n* is not an int object, it is converted with - :cfunc:`PyNumber_Index` first. + Returns the integer *n* converted to base *base* as a string. The *base* + argument must be one of 2, 8, 10, or 16. For base 2, 8, or 16, the + returned string is prefixed with a base marker of ``'0b'``, ``'0o'``, or + ``'0x'``, respectively. If *n* is not a Python int, it is converted with + :c:func:`PyNumber_Index` first. -.. cfunction:: Py_ssize_t PyNumber_AsSsize_t(PyObject *o, PyObject *exc) +.. c:function:: Py_ssize_t PyNumber_AsSsize_t(PyObject *o, PyObject *exc) Returns *o* converted to a Py_ssize_t value if *o* can be interpreted as an - integer. If *o* can be converted to a Python int but the attempt to + integer. If the call fails, an exception is raised and -1 is returned. + + If *o* can be converted to a Python int but the attempt to convert to a Py_ssize_t value would raise an :exc:`OverflowError`, then the *exc* argument is the type of exception that will be raised (usually :exc:`IndexError` or :exc:`OverflowError`). If *exc* is *NULL*, then the @@ -269,7 +259,7 @@ Number Protocol integer or *PY_SSIZE_T_MAX* for a positive integer. -.. cfunction:: int PyIndex_Check(PyObject *o) +.. c:function:: int PyIndex_Check(PyObject *o) Returns True if *o* is an index integer (has the nb_index slot of the tp_as_number structure filled in). diff --git a/Doc/c-api/objbuffer.rst b/Doc/c-api/objbuffer.rst index 728d38343f..e7f4fde002 100644 --- a/Doc/c-api/objbuffer.rst +++ b/Doc/c-api/objbuffer.rst @@ -12,13 +12,13 @@ around the :ref:`new buffer protocol <bufferobjects>`, but they don't give you control over the lifetime of the resources acquired when a buffer is exported. -Therefore, it is recommended that you call :cfunc:`PyObject_GetBuffer` +Therefore, it is recommended that you call :c:func:`PyObject_GetBuffer` (or the ``y*`` or ``w*`` :ref:`format codes <arg-parsing>` with the -:cfunc:`PyArg_ParseTuple` family of functions) to get a buffer view over -an object, and :cfunc:`PyBuffer_Release` when the buffer view can be released. +:c:func:`PyArg_ParseTuple` family of functions) to get a buffer view over +an object, and :c:func:`PyBuffer_Release` when the buffer view can be released. -.. cfunction:: int PyObject_AsCharBuffer(PyObject *obj, const char **buffer, Py_ssize_t *buffer_len) +.. c:function:: int PyObject_AsCharBuffer(PyObject *obj, const char **buffer, Py_ssize_t *buffer_len) Returns a pointer to a read-only memory location usable as character-based input. The *obj* argument must support the single-segment character buffer @@ -27,7 +27,7 @@ an object, and :cfunc:`PyBuffer_Release` when the buffer view can be released. :exc:`TypeError` on error. -.. cfunction:: int PyObject_AsReadBuffer(PyObject *obj, const void **buffer, Py_ssize_t *buffer_len) +.. c:function:: int PyObject_AsReadBuffer(PyObject *obj, const void **buffer, Py_ssize_t *buffer_len) Returns a pointer to a read-only memory location containing arbitrary data. The *obj* argument must support the single-segment readable buffer @@ -36,13 +36,13 @@ an object, and :cfunc:`PyBuffer_Release` when the buffer view can be released. :exc:`TypeError` on error. -.. cfunction:: int PyObject_CheckReadBuffer(PyObject *o) +.. c:function:: int PyObject_CheckReadBuffer(PyObject *o) Returns ``1`` if *o* supports the single-segment readable buffer interface. Otherwise returns ``0``. -.. cfunction:: int PyObject_AsWriteBuffer(PyObject *obj, void **buffer, Py_ssize_t *buffer_len) +.. c:function:: int PyObject_AsWriteBuffer(PyObject *obj, void **buffer, Py_ssize_t *buffer_len) Returns a pointer to a writable memory location. The *obj* argument must support the single-segment, character buffer interface. On success, diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst index fd3a6d2380..d0d45ad001 100644 --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -6,7 +6,7 @@ Object Protocol =============== -.. cfunction:: int PyObject_Print(PyObject *o, FILE *fp, int flags) +.. c:function:: int PyObject_Print(PyObject *o, FILE *fp, int flags) Print an object *o*, on file *fp*. Returns ``-1`` on error. The flags argument is used to enable certain printing options. The only option currently supported @@ -14,35 +14,35 @@ Object Protocol instead of the :func:`repr`. -.. cfunction:: int PyObject_HasAttr(PyObject *o, PyObject *attr_name) +.. c:function:: int PyObject_HasAttr(PyObject *o, PyObject *attr_name) Returns ``1`` if *o* has the attribute *attr_name*, and ``0`` otherwise. This is equivalent to the Python expression ``hasattr(o, attr_name)``. This function always succeeds. -.. cfunction:: int PyObject_HasAttrString(PyObject *o, const char *attr_name) +.. c:function:: int PyObject_HasAttrString(PyObject *o, const char *attr_name) Returns ``1`` if *o* has the attribute *attr_name*, and ``0`` otherwise. This is equivalent to the Python expression ``hasattr(o, attr_name)``. This function always succeeds. -.. cfunction:: PyObject* PyObject_GetAttr(PyObject *o, PyObject *attr_name) +.. c:function:: PyObject* PyObject_GetAttr(PyObject *o, PyObject *attr_name) Retrieve an attribute named *attr_name* from object *o*. Returns the attribute value on success, or *NULL* on failure. This is the equivalent of the Python expression ``o.attr_name``. -.. cfunction:: PyObject* PyObject_GetAttrString(PyObject *o, const char *attr_name) +.. c:function:: PyObject* PyObject_GetAttrString(PyObject *o, const char *attr_name) Retrieve an attribute named *attr_name* from object *o*. Returns the attribute value on success, or *NULL* on failure. This is the equivalent of the Python expression ``o.attr_name``. -.. cfunction:: PyObject* PyObject_GenericGetAttr(PyObject *o, PyObject *name) +.. c:function:: PyObject* PyObject_GenericGetAttr(PyObject *o, PyObject *name) Generic attribute getter function that is meant to be put into a type object's ``tp_getattro`` slot. It looks for a descriptor in the dictionary @@ -52,21 +52,21 @@ Object Protocol descriptors don't. Otherwise, an :exc:`AttributeError` is raised. -.. cfunction:: int PyObject_SetAttr(PyObject *o, PyObject *attr_name, PyObject *v) +.. c:function:: int PyObject_SetAttr(PyObject *o, PyObject *attr_name, PyObject *v) Set the value of the attribute named *attr_name*, for object *o*, to the value *v*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``o.attr_name = v``. -.. cfunction:: int PyObject_SetAttrString(PyObject *o, const char *attr_name, PyObject *v) +.. c:function:: int PyObject_SetAttrString(PyObject *o, const char *attr_name, PyObject *v) Set the value of the attribute named *attr_name*, for object *o*, to the value *v*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``o.attr_name = v``. -.. cfunction:: int PyObject_GenericSetAttr(PyObject *o, PyObject *name, PyObject *value) +.. c:function:: int PyObject_GenericSetAttr(PyObject *o, PyObject *name, PyObject *value) Generic attribute setter function that is meant to be put into a type object's ``tp_setattro`` slot. It looks for a data descriptor in the @@ -76,19 +76,19 @@ Object Protocol an :exc:`AttributeError` is raised and ``-1`` is returned. -.. cfunction:: int PyObject_DelAttr(PyObject *o, PyObject *attr_name) +.. c:function:: int PyObject_DelAttr(PyObject *o, PyObject *attr_name) Delete attribute named *attr_name*, for object *o*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``del o.attr_name``. -.. cfunction:: int PyObject_DelAttrString(PyObject *o, const char *attr_name) +.. c:function:: int PyObject_DelAttrString(PyObject *o, const char *attr_name) Delete attribute named *attr_name*, for object *o*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``del o.attr_name``. -.. cfunction:: PyObject* PyObject_RichCompare(PyObject *o1, PyObject *o2, int opid) +.. c:function:: PyObject* PyObject_RichCompare(PyObject *o1, PyObject *o2, int opid) Compare the values of *o1* and *o2* using the operation specified by *opid*, which must be one of :const:`Py_LT`, :const:`Py_LE`, :const:`Py_EQ`, @@ -98,7 +98,7 @@ Object Protocol to *opid*. Returns the value of the comparison on success, or *NULL* on failure. -.. cfunction:: int PyObject_RichCompareBool(PyObject *o1, PyObject *o2, int opid) +.. c:function:: int PyObject_RichCompareBool(PyObject *o1, PyObject *o2, int opid) Compare the values of *o1* and *o2* using the operation specified by *opid*, which must be one of :const:`Py_LT`, :const:`Py_LE`, :const:`Py_EQ`, @@ -109,10 +109,10 @@ Object Protocol *opid*. .. note:: - If *o1* and *o2* are the same object, :cfunc:`PyObject_RichCompareBool` + If *o1* and *o2* are the same object, :c:func:`PyObject_RichCompareBool` will always return ``1`` for :const:`Py_EQ` and ``0`` for :const:`Py_NE`. -.. cfunction:: PyObject* PyObject_Repr(PyObject *o) +.. c:function:: PyObject* PyObject_Repr(PyObject *o) .. index:: builtin: repr @@ -121,18 +121,18 @@ Object Protocol Python expression ``repr(o)``. Called by the :func:`repr` built-in function. -.. cfunction:: PyObject* PyObject_ASCII(PyObject *o) +.. c:function:: PyObject* PyObject_ASCII(PyObject *o) .. index:: builtin: ascii - As :cfunc:`PyObject_Repr`, compute a string representation of object *o*, but + As :c:func:`PyObject_Repr`, compute a string representation of object *o*, but escape the non-ASCII characters in the string returned by - :cfunc:`PyObject_Repr` with ``\x``, ``\u`` or ``\U`` escapes. This generates - a string similar to that returned by :cfunc:`PyObject_Repr` in Python 2. + :c:func:`PyObject_Repr` with ``\x``, ``\u`` or ``\U`` escapes. This generates + a string similar to that returned by :c:func:`PyObject_Repr` in Python 2. Called by the :func:`ascii` built-in function. -.. cfunction:: PyObject* PyObject_Str(PyObject *o) +.. c:function:: PyObject* PyObject_Str(PyObject *o) .. index:: builtin: str @@ -141,20 +141,21 @@ Object Protocol Python expression ``str(o)``. Called by the :func:`str` built-in function and, therefore, by the :func:`print` function. -.. cfunction:: PyObject* PyObject_Bytes(PyObject *o) +.. c:function:: PyObject* PyObject_Bytes(PyObject *o) .. index:: builtin: bytes - Compute a bytes representation of object *o*. *NULL* is returned on failure - and a bytes object on success. This is equivalent to the Python expression - ``bytes(o)``. + Compute a bytes representation of object *o*. *NULL* is returned on + failure and a bytes object on success. This is equivalent to the Python + expression ``bytes(o)``, when *o* is not an integer. Unlike ``bytes(o)``, + a TypeError is raised when *o* is an integer instead of a zero-initialized + bytes object. - -.. cfunction:: int PyObject_IsInstance(PyObject *inst, PyObject *cls) +.. c:function:: int PyObject_IsInstance(PyObject *inst, PyObject *cls) Returns ``1`` if *inst* is an instance of the class *cls* or a subclass of *cls*, or ``0`` if not. On error, returns ``-1`` and sets an exception. If - *cls* is a type object rather than a class object, :cfunc:`PyObject_IsInstance` + *cls* is a type object rather than a class object, :c:func:`PyObject_IsInstance` returns ``1`` if *inst* is of type *cls*. If *cls* is a tuple, the check will be done against every entry in *cls*. The result will be ``1`` when at least one of the checks returns ``1``, otherwise it will be ``0``. If *inst* is not a @@ -170,13 +171,13 @@ of. If :class:`A` and :class:`B` are class objects, :class:`B` is a subclass of :class:`A` if it inherits from :class:`A` either directly or indirectly. If either is not a class object, a more general mechanism is used to determine the class relationship of the two objects. When testing if *B* is a subclass of -*A*, if *A* is *B*, :cfunc:`PyObject_IsSubclass` returns true. If *A* and *B* +*A*, if *A* is *B*, :c:func:`PyObject_IsSubclass` returns true. If *A* and *B* are different objects, *B*'s :attr:`__bases__` attribute is searched in a depth-first fashion for *A* --- the presence of the :attr:`__bases__` attribute is considered sufficient for this determination. -.. cfunction:: int PyObject_IsSubclass(PyObject *derived, PyObject *cls) +.. c:function:: int PyObject_IsSubclass(PyObject *derived, PyObject *cls) Returns ``1`` if the class *derived* is identical to or derived from the class *cls*, otherwise returns ``0``. In case of an error, returns ``-1``. If *cls* @@ -186,13 +187,13 @@ is considered sufficient for this determination. this function uses the generic algorithm described above. -.. cfunction:: int PyCallable_Check(PyObject *o) +.. c:function:: int PyCallable_Check(PyObject *o) Determine if the object *o* is callable. Return ``1`` if the object is callable and ``0`` otherwise. This function always succeeds. -.. cfunction:: PyObject* PyObject_Call(PyObject *callable_object, PyObject *args, PyObject *kw) +.. c:function:: PyObject* PyObject_Call(PyObject *callable_object, PyObject *args, PyObject *kw) Call a callable Python object *callable_object*, with arguments given by the tuple *args*, and named arguments given by the dictionary *kw*. If no named @@ -202,7 +203,7 @@ is considered sufficient for this determination. ``callable_object(*args, **kw)``. -.. cfunction:: PyObject* PyObject_CallObject(PyObject *callable_object, PyObject *args) +.. c:function:: PyObject* PyObject_CallObject(PyObject *callable_object, PyObject *args) Call a callable Python object *callable_object*, with arguments given by the tuple *args*. If no arguments are needed, then *args* may be *NULL*. Returns @@ -210,54 +211,58 @@ is considered sufficient for this determination. of the Python expression ``callable_object(*args)``. -.. cfunction:: PyObject* PyObject_CallFunction(PyObject *callable, char *format, ...) +.. c:function:: PyObject* PyObject_CallFunction(PyObject *callable, char *format, ...) Call a callable Python object *callable*, with a variable number of C arguments. - The C arguments are described using a :cfunc:`Py_BuildValue` style format + The C arguments are described using a :c:func:`Py_BuildValue` style format string. The format may be *NULL*, indicating that no arguments are provided. Returns the result of the call on success, or *NULL* on failure. This is the equivalent of the Python expression ``callable(*args)``. Note that if you only - pass :ctype:`PyObject \*` args, :cfunc:`PyObject_CallFunctionObjArgs` is a + pass :c:type:`PyObject \*` args, :c:func:`PyObject_CallFunctionObjArgs` is a faster alternative. -.. cfunction:: PyObject* PyObject_CallMethod(PyObject *o, char *method, char *format, ...) +.. c:function:: PyObject* PyObject_CallMethod(PyObject *o, char *method, char *format, ...) Call the method named *method* of object *o* with a variable number of C - arguments. The C arguments are described by a :cfunc:`Py_BuildValue` format + arguments. The C arguments are described by a :c:func:`Py_BuildValue` format string that should produce a tuple. The format may be *NULL*, indicating that no arguments are provided. Returns the result of the call on success, or *NULL* on failure. This is the equivalent of the Python expression ``o.method(args)``. - Note that if you only pass :ctype:`PyObject \*` args, - :cfunc:`PyObject_CallMethodObjArgs` is a faster alternative. + Note that if you only pass :c:type:`PyObject \*` args, + :c:func:`PyObject_CallMethodObjArgs` is a faster alternative. -.. cfunction:: PyObject* PyObject_CallFunctionObjArgs(PyObject *callable, ..., NULL) +.. c:function:: PyObject* PyObject_CallFunctionObjArgs(PyObject *callable, ..., NULL) Call a callable Python object *callable*, with a variable number of - :ctype:`PyObject\*` arguments. The arguments are provided as a variable number + :c:type:`PyObject\*` arguments. The arguments are provided as a variable number of parameters followed by *NULL*. Returns the result of the call on success, or *NULL* on failure. -.. cfunction:: PyObject* PyObject_CallMethodObjArgs(PyObject *o, PyObject *name, ..., NULL) +.. c:function:: PyObject* PyObject_CallMethodObjArgs(PyObject *o, PyObject *name, ..., NULL) Calls a method of the object *o*, where the name of the method is given as a Python string object in *name*. It is called with a variable number of - :ctype:`PyObject\*` arguments. The arguments are provided as a variable number + :c:type:`PyObject\*` arguments. The arguments are provided as a variable number of parameters followed by *NULL*. Returns the result of the call on success, or *NULL* on failure. -.. cfunction:: long PyObject_Hash(PyObject *o) +.. c:function:: Py_hash_t PyObject_Hash(PyObject *o) .. index:: builtin: hash Compute and return the hash value of an object *o*. On failure, return ``-1``. This is the equivalent of the Python expression ``hash(o)``. + .. versionchanged:: 3.2 + The return type is now Py_hash_t. This is a signed integer the same size + as Py_ssize_t. + -.. cfunction:: long PyObject_HashNotImplemented(PyObject *o) +.. c:function:: Py_hash_t PyObject_HashNotImplemented(PyObject *o) Set a :exc:`TypeError` indicating that ``type(o)`` is not hashable and return ``-1``. This function receives special treatment when stored in a ``tp_hash`` slot, @@ -265,21 +270,21 @@ is considered sufficient for this determination. hashable. -.. cfunction:: int PyObject_IsTrue(PyObject *o) +.. c:function:: int PyObject_IsTrue(PyObject *o) Returns ``1`` if the object *o* is considered to be true, and ``0`` otherwise. This is equivalent to the Python expression ``not not o``. On failure, return ``-1``. -.. cfunction:: int PyObject_Not(PyObject *o) +.. c:function:: int PyObject_Not(PyObject *o) Returns ``0`` if the object *o* is considered to be true, and ``1`` otherwise. This is equivalent to the Python expression ``not o``. On failure, return ``-1``. -.. cfunction:: PyObject* PyObject_Type(PyObject *o) +.. c:function:: PyObject* PyObject_Type(PyObject *o) .. index:: builtin: type @@ -288,17 +293,17 @@ is considered sufficient for this determination. is equivalent to the Python expression ``type(o)``. This function increments the reference count of the return value. There's really no reason to use this function instead of the common expression ``o->ob_type``, which returns a - pointer of type :ctype:`PyTypeObject\*`, except when the incremented reference + pointer of type :c:type:`PyTypeObject\*`, except when the incremented reference count is needed. -.. cfunction:: int PyObject_TypeCheck(PyObject *o, PyTypeObject *type) +.. c:function:: int PyObject_TypeCheck(PyObject *o, PyTypeObject *type) Return true if the object *o* is of type *type* or a subtype of *type*. Both parameters must be non-*NULL*. -.. cfunction:: Py_ssize_t PyObject_Length(PyObject *o) +.. c:function:: Py_ssize_t PyObject_Length(PyObject *o) Py_ssize_t PyObject_Size(PyObject *o) .. index:: builtin: len @@ -308,34 +313,34 @@ is considered sufficient for this determination. returned. This is the equivalent to the Python expression ``len(o)``. -.. cfunction:: PyObject* PyObject_GetItem(PyObject *o, PyObject *key) +.. c:function:: PyObject* PyObject_GetItem(PyObject *o, PyObject *key) Return element of *o* corresponding to the object *key* or *NULL* on failure. This is the equivalent of the Python expression ``o[key]``. -.. cfunction:: int PyObject_SetItem(PyObject *o, PyObject *key, PyObject *v) +.. c:function:: int PyObject_SetItem(PyObject *o, PyObject *key, PyObject *v) Map the object *key* to the value *v*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``o[key] = v``. -.. cfunction:: int PyObject_DelItem(PyObject *o, PyObject *key) +.. c:function:: int PyObject_DelItem(PyObject *o, PyObject *key) Delete the mapping for *key* from *o*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``del o[key]``. -.. cfunction:: PyObject* PyObject_Dir(PyObject *o) +.. c:function:: PyObject* PyObject_Dir(PyObject *o) This is equivalent to the Python expression ``dir(o)``, returning a (possibly empty) list of strings appropriate for the object argument, or *NULL* if there was an error. If the argument is *NULL*, this is like the Python ``dir()``, returning the names of the current locals; in this case, if no execution frame - is active then *NULL* is returned but :cfunc:`PyErr_Occurred` will return false. + is active then *NULL* is returned but :c:func:`PyErr_Occurred` will return false. -.. cfunction:: PyObject* PyObject_GetIter(PyObject *o) +.. c:function:: PyObject* PyObject_GetIter(PyObject *o) This is equivalent to the Python expression ``iter(o)``. It returns a new iterator for the object argument, or the object itself if the object is already diff --git a/Doc/c-api/refcounting.rst b/Doc/c-api/refcounting.rst index c0f4ca12dc..4f512ecdbe 100644 --- a/Doc/c-api/refcounting.rst +++ b/Doc/c-api/refcounting.rst @@ -11,22 +11,22 @@ The macros in this section are used for managing reference counts of Python objects. -.. cfunction:: void Py_INCREF(PyObject *o) +.. c:function:: void Py_INCREF(PyObject *o) Increment the reference count for object *o*. The object must not be *NULL*; if - you aren't sure that it isn't *NULL*, use :cfunc:`Py_XINCREF`. + you aren't sure that it isn't *NULL*, use :c:func:`Py_XINCREF`. -.. cfunction:: void Py_XINCREF(PyObject *o) +.. c:function:: void Py_XINCREF(PyObject *o) Increment the reference count for object *o*. The object may be *NULL*, in which case the macro has no effect. -.. cfunction:: void Py_DECREF(PyObject *o) +.. c:function:: void Py_DECREF(PyObject *o) Decrement the reference count for object *o*. The object must not be *NULL*; if - you aren't sure that it isn't *NULL*, use :cfunc:`Py_XDECREF`. If the reference + you aren't sure that it isn't *NULL*, use :c:func:`Py_XDECREF`. If the reference count reaches zero, the object's type's deallocation function (which must not be *NULL*) is invoked. @@ -36,25 +36,25 @@ objects. when a class instance with a :meth:`__del__` method is deallocated). While exceptions in such code are not propagated, the executed code has free access to all Python global variables. This means that any object that is reachable from - a global variable should be in a consistent state before :cfunc:`Py_DECREF` is + a global variable should be in a consistent state before :c:func:`Py_DECREF` is invoked. For example, code to delete an object from a list should copy a reference to the deleted object in a temporary variable, update the list data - structure, and then call :cfunc:`Py_DECREF` for the temporary variable. + structure, and then call :c:func:`Py_DECREF` for the temporary variable. -.. cfunction:: void Py_XDECREF(PyObject *o) +.. c:function:: void Py_XDECREF(PyObject *o) Decrement the reference count for object *o*. The object may be *NULL*, in which case the macro has no effect; otherwise the effect is the same as for - :cfunc:`Py_DECREF`, and the same warning applies. + :c:func:`Py_DECREF`, and the same warning applies. -.. cfunction:: void Py_CLEAR(PyObject *o) +.. c:function:: void Py_CLEAR(PyObject *o) Decrement the reference count for object *o*. The object may be *NULL*, in which case the macro has no effect; otherwise the effect is the same as for - :cfunc:`Py_DECREF`, except that the argument is also set to *NULL*. The warning - for :cfunc:`Py_DECREF` does not apply with respect to the object passed because + :c:func:`Py_DECREF`, except that the argument is also set to *NULL*. The warning + for :c:func:`Py_DECREF` does not apply with respect to the object passed because the macro carefully uses a temporary variable and sets the argument to *NULL* before decrementing its reference count. @@ -64,10 +64,10 @@ objects. The following functions are for runtime dynamic embedding of Python: ``Py_IncRef(PyObject *o)``, ``Py_DecRef(PyObject *o)``. They are -simply exported function versions of :cfunc:`Py_XINCREF` and -:cfunc:`Py_XDECREF`, respectively. +simply exported function versions of :c:func:`Py_XINCREF` and +:c:func:`Py_XDECREF`, respectively. The following functions or macros are only for use within the interpreter core: -:cfunc:`_Py_Dealloc`, :cfunc:`_Py_ForgetReference`, :cfunc:`_Py_NewReference`, -as well as the global variable :cdata:`_Py_RefTotal`. +:c:func:`_Py_Dealloc`, :c:func:`_Py_ForgetReference`, :c:func:`_Py_NewReference`, +as well as the global variable :c:data:`_Py_RefTotal`. diff --git a/Doc/c-api/reflection.rst b/Doc/c-api/reflection.rst index 4f46c7fae8..96893652f7 100644 --- a/Doc/c-api/reflection.rst +++ b/Doc/c-api/reflection.rst @@ -5,40 +5,45 @@ Reflection ========== -.. cfunction:: PyObject* PyEval_GetBuiltins() +.. c:function:: PyObject* PyEval_GetBuiltins() Return a dictionary of the builtins in the current execution frame, or the interpreter of the thread state if no frame is currently executing. -.. cfunction:: PyObject* PyEval_GetLocals() +.. c:function:: PyObject* PyEval_GetLocals() Return a dictionary of the local variables in the current execution frame, or *NULL* if no frame is currently executing. -.. cfunction:: PyObject* PyEval_GetGlobals() +.. c:function:: PyObject* PyEval_GetGlobals() Return a dictionary of the global variables in the current execution frame, or *NULL* if no frame is currently executing. -.. cfunction:: PyFrameObject* PyEval_GetFrame() +.. c:function:: PyFrameObject* PyEval_GetFrame() Return the current thread state's frame, which is *NULL* if no frame is currently executing. -.. cfunction:: const char* PyEval_GetFuncName(PyObject *func) +.. c:function:: int PyFrame_GetLineNumber(PyFrameObject *frame) + + Return the line number that *frame* is currently executing. + + +.. c:function:: const char* PyEval_GetFuncName(PyObject *func) Return the name of *func* if it is a function, class or instance object, else the name of *func*\s type. -.. cfunction:: const char* PyEval_GetFuncDesc(PyObject *func) +.. c:function:: const char* PyEval_GetFuncDesc(PyObject *func) Return a description string, depending on the type of *func*. Return values include "()" for functions and methods, " constructor", " instance", and " object". Concatenated with the result of - :cfunc:`PyEval_GetFuncName`, the result will be a description of + :c:func:`PyEval_GetFuncName`, the result will be a description of *func*. diff --git a/Doc/c-api/sequence.rst b/Doc/c-api/sequence.rst index d8631770fb..0297ba3588 100644 --- a/Doc/c-api/sequence.rst +++ b/Doc/c-api/sequence.rst @@ -6,13 +6,13 @@ Sequence Protocol ================= -.. cfunction:: int PySequence_Check(PyObject *o) +.. c:function:: int PySequence_Check(PyObject *o) Return ``1`` if the object provides sequence protocol, and ``0`` otherwise. This function always succeeds. -.. cfunction:: Py_ssize_t PySequence_Size(PyObject *o) +.. c:function:: Py_ssize_t PySequence_Size(PyObject *o) Py_ssize_t PySequence_Length(PyObject *o) .. index:: builtin: len @@ -22,96 +22,96 @@ Sequence Protocol Python expression ``len(o)``. -.. cfunction:: PyObject* PySequence_Concat(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PySequence_Concat(PyObject *o1, PyObject *o2) Return the concatenation of *o1* and *o2* on success, and *NULL* on failure. This is the equivalent of the Python expression ``o1 + o2``. -.. cfunction:: PyObject* PySequence_Repeat(PyObject *o, Py_ssize_t count) +.. c:function:: PyObject* PySequence_Repeat(PyObject *o, Py_ssize_t count) Return the result of repeating sequence object *o* *count* times, or *NULL* on failure. This is the equivalent of the Python expression ``o * count``. -.. cfunction:: PyObject* PySequence_InPlaceConcat(PyObject *o1, PyObject *o2) +.. c:function:: PyObject* PySequence_InPlaceConcat(PyObject *o1, PyObject *o2) Return the concatenation of *o1* and *o2* on success, and *NULL* on failure. The operation is done *in-place* when *o1* supports it. This is the equivalent of the Python expression ``o1 += o2``. -.. cfunction:: PyObject* PySequence_InPlaceRepeat(PyObject *o, Py_ssize_t count) +.. c:function:: PyObject* PySequence_InPlaceRepeat(PyObject *o, Py_ssize_t count) Return the result of repeating sequence object *o* *count* times, or *NULL* on failure. The operation is done *in-place* when *o* supports it. This is the equivalent of the Python expression ``o *= count``. -.. cfunction:: PyObject* PySequence_GetItem(PyObject *o, Py_ssize_t i) +.. c:function:: PyObject* PySequence_GetItem(PyObject *o, Py_ssize_t i) Return the *i*\ th element of *o*, or *NULL* on failure. This is the equivalent of the Python expression ``o[i]``. -.. cfunction:: PyObject* PySequence_GetSlice(PyObject *o, Py_ssize_t i1, Py_ssize_t i2) +.. c:function:: PyObject* PySequence_GetSlice(PyObject *o, Py_ssize_t i1, Py_ssize_t i2) Return the slice of sequence object *o* between *i1* and *i2*, or *NULL* on failure. This is the equivalent of the Python expression ``o[i1:i2]``. -.. cfunction:: int PySequence_SetItem(PyObject *o, Py_ssize_t i, PyObject *v) +.. c:function:: int PySequence_SetItem(PyObject *o, Py_ssize_t i, PyObject *v) Assign object *v* to the *i*\ th element of *o*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``o[i] = v``. This function *does not* steal a reference to *v*. -.. cfunction:: int PySequence_DelItem(PyObject *o, Py_ssize_t i) +.. c:function:: int PySequence_DelItem(PyObject *o, Py_ssize_t i) Delete the *i*\ th element of object *o*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``del o[i]``. -.. cfunction:: int PySequence_SetSlice(PyObject *o, Py_ssize_t i1, Py_ssize_t i2, PyObject *v) +.. c:function:: int PySequence_SetSlice(PyObject *o, Py_ssize_t i1, Py_ssize_t i2, PyObject *v) Assign the sequence object *v* to the slice in sequence object *o* from *i1* to *i2*. This is the equivalent of the Python statement ``o[i1:i2] = v``. -.. cfunction:: int PySequence_DelSlice(PyObject *o, Py_ssize_t i1, Py_ssize_t i2) +.. c:function:: int PySequence_DelSlice(PyObject *o, Py_ssize_t i1, Py_ssize_t i2) Delete the slice in sequence object *o* from *i1* to *i2*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``del o[i1:i2]``. -.. cfunction:: Py_ssize_t PySequence_Count(PyObject *o, PyObject *value) +.. c:function:: Py_ssize_t PySequence_Count(PyObject *o, PyObject *value) Return the number of occurrences of *value* in *o*, that is, return the number of keys for which ``o[key] == value``. On failure, return ``-1``. This is equivalent to the Python expression ``o.count(value)``. -.. cfunction:: int PySequence_Contains(PyObject *o, PyObject *value) +.. c:function:: int PySequence_Contains(PyObject *o, PyObject *value) Determine if *o* contains *value*. If an item in *o* is equal to *value*, return ``1``, otherwise return ``0``. On error, return ``-1``. This is equivalent to the Python expression ``value in o``. -.. cfunction:: Py_ssize_t PySequence_Index(PyObject *o, PyObject *value) +.. c:function:: Py_ssize_t PySequence_Index(PyObject *o, PyObject *value) Return the first index *i* for which ``o[i] == value``. On error, return ``-1``. This is equivalent to the Python expression ``o.index(value)``. -.. cfunction:: PyObject* PySequence_List(PyObject *o) +.. c:function:: PyObject* PySequence_List(PyObject *o) Return a list object with the same contents as the arbitrary sequence *o*. The returned list is guaranteed to be new. -.. cfunction:: PyObject* PySequence_Tuple(PyObject *o) +.. c:function:: PyObject* PySequence_Tuple(PyObject *o) .. index:: builtin: tuple @@ -121,42 +121,42 @@ Sequence Protocol equivalent to the Python expression ``tuple(o)``. -.. cfunction:: PyObject* PySequence_Fast(PyObject *o, const char *m) +.. c:function:: PyObject* PySequence_Fast(PyObject *o, const char *m) Returns the sequence *o* as a tuple, unless it is already a tuple or list, in - which case *o* is returned. Use :cfunc:`PySequence_Fast_GET_ITEM` to access the + which case *o* is returned. Use :c:func:`PySequence_Fast_GET_ITEM` to access the members of the result. Returns *NULL* on failure. If the object is not a sequence, raises :exc:`TypeError` with *m* as the message text. -.. cfunction:: PyObject* PySequence_Fast_GET_ITEM(PyObject *o, Py_ssize_t i) +.. c:function:: PyObject* PySequence_Fast_GET_ITEM(PyObject *o, Py_ssize_t i) Return the *i*\ th element of *o*, assuming that *o* was returned by - :cfunc:`PySequence_Fast`, *o* is not *NULL*, and that *i* is within bounds. + :c:func:`PySequence_Fast`, *o* is not *NULL*, and that *i* is within bounds. -.. cfunction:: PyObject** PySequence_Fast_ITEMS(PyObject *o) +.. c:function:: PyObject** PySequence_Fast_ITEMS(PyObject *o) Return the underlying array of PyObject pointers. Assumes that *o* was returned - by :cfunc:`PySequence_Fast` and *o* is not *NULL*. + by :c:func:`PySequence_Fast` and *o* is not *NULL*. Note, if a list gets resized, the reallocation may relocate the items array. So, only use the underlying array pointer in contexts where the sequence cannot change. -.. cfunction:: PyObject* PySequence_ITEM(PyObject *o, Py_ssize_t i) +.. c:function:: PyObject* PySequence_ITEM(PyObject *o, Py_ssize_t i) Return the *i*\ th element of *o* or *NULL* on failure. Macro form of - :cfunc:`PySequence_GetItem` but without checking that - :cfunc:`PySequence_Check(o)` is true and without adjustment for negative + :c:func:`PySequence_GetItem` but without checking that + :c:func:`PySequence_Check` on *o* is true and without adjustment for negative indices. -.. cfunction:: Py_ssize_t PySequence_Fast_GET_SIZE(PyObject *o) +.. c:function:: Py_ssize_t PySequence_Fast_GET_SIZE(PyObject *o) Returns the length of *o*, assuming that *o* was returned by - :cfunc:`PySequence_Fast` and that *o* is not *NULL*. The size can also be - gotten by calling :cfunc:`PySequence_Size` on *o*, but - :cfunc:`PySequence_Fast_GET_SIZE` is faster because it can assume *o* is a list + :c:func:`PySequence_Fast` and that *o* is not *NULL*. The size can also be + gotten by calling :c:func:`PySequence_Size` on *o*, but + :c:func:`PySequence_Fast_GET_SIZE` is faster because it can assume *o* is a list or tuple. diff --git a/Doc/c-api/set.rst b/Doc/c-api/set.rst index 4348108421..66b47c4f71 100644 --- a/Doc/c-api/set.rst +++ b/Doc/c-api/set.rst @@ -14,20 +14,20 @@ Set Objects This section details the public API for :class:`set` and :class:`frozenset` objects. Any functionality not listed below is best accessed using the either -the abstract object protocol (including :cfunc:`PyObject_CallMethod`, -:cfunc:`PyObject_RichCompareBool`, :cfunc:`PyObject_Hash`, -:cfunc:`PyObject_Repr`, :cfunc:`PyObject_IsTrue`, :cfunc:`PyObject_Print`, and -:cfunc:`PyObject_GetIter`) or the abstract number protocol (including -:cfunc:`PyNumber_And`, :cfunc:`PyNumber_Subtract`, :cfunc:`PyNumber_Or`, -:cfunc:`PyNumber_Xor`, :cfunc:`PyNumber_InPlaceAnd`, -:cfunc:`PyNumber_InPlaceSubtract`, :cfunc:`PyNumber_InPlaceOr`, and -:cfunc:`PyNumber_InPlaceXor`). +the abstract object protocol (including :c:func:`PyObject_CallMethod`, +:c:func:`PyObject_RichCompareBool`, :c:func:`PyObject_Hash`, +:c:func:`PyObject_Repr`, :c:func:`PyObject_IsTrue`, :c:func:`PyObject_Print`, and +:c:func:`PyObject_GetIter`) or the abstract number protocol (including +:c:func:`PyNumber_And`, :c:func:`PyNumber_Subtract`, :c:func:`PyNumber_Or`, +:c:func:`PyNumber_Xor`, :c:func:`PyNumber_InPlaceAnd`, +:c:func:`PyNumber_InPlaceSubtract`, :c:func:`PyNumber_InPlaceOr`, and +:c:func:`PyNumber_InPlaceXor`). -.. ctype:: PySetObject +.. c:type:: PySetObject - This subtype of :ctype:`PyObject` is used to hold the internal data for both - :class:`set` and :class:`frozenset` objects. It is like a :ctype:`PyDictObject` + This subtype of :c:type:`PyObject` is used to hold the internal data for both + :class:`set` and :class:`frozenset` objects. It is like a :c:type:`PyDictObject` in that it is a fixed size for small sets (much like tuple storage) and will point to a separate, variable sized block of memory for medium and large sized sets (much like list storage). None of the fields of this structure should be @@ -35,49 +35,49 @@ the abstract object protocol (including :cfunc:`PyObject_CallMethod`, the documented API rather than by manipulating the values in the structure. -.. cvar:: PyTypeObject PySet_Type +.. c:var:: PyTypeObject PySet_Type - This is an instance of :ctype:`PyTypeObject` representing the Python + This is an instance of :c:type:`PyTypeObject` representing the Python :class:`set` type. -.. cvar:: PyTypeObject PyFrozenSet_Type +.. c:var:: PyTypeObject PyFrozenSet_Type - This is an instance of :ctype:`PyTypeObject` representing the Python + This is an instance of :c:type:`PyTypeObject` representing the Python :class:`frozenset` type. The following type check macros work on pointers to any Python object. Likewise, the constructor functions work with any iterable Python object. -.. cfunction:: int PySet_Check(PyObject *p) +.. c:function:: int PySet_Check(PyObject *p) Return true if *p* is a :class:`set` object or an instance of a subtype. -.. cfunction:: int PyFrozenSet_Check(PyObject *p) +.. c:function:: int PyFrozenSet_Check(PyObject *p) Return true if *p* is a :class:`frozenset` object or an instance of a subtype. -.. cfunction:: int PyAnySet_Check(PyObject *p) +.. c:function:: int PyAnySet_Check(PyObject *p) Return true if *p* is a :class:`set` object, a :class:`frozenset` object, or an instance of a subtype. -.. cfunction:: int PyAnySet_CheckExact(PyObject *p) +.. c:function:: int PyAnySet_CheckExact(PyObject *p) Return true if *p* is a :class:`set` object or a :class:`frozenset` object but not an instance of a subtype. -.. cfunction:: int PyFrozenSet_CheckExact(PyObject *p) +.. c:function:: int PyFrozenSet_CheckExact(PyObject *p) Return true if *p* is a :class:`frozenset` object but not an instance of a subtype. -.. cfunction:: PyObject* PySet_New(PyObject *iterable) +.. c:function:: PyObject* PySet_New(PyObject *iterable) Return a new :class:`set` containing objects returned by the *iterable*. The *iterable* may be *NULL* to create a new empty set. Return the new set on @@ -86,7 +86,7 @@ the constructor functions work with any iterable Python object. (``c=set(s)``). -.. cfunction:: PyObject* PyFrozenSet_New(PyObject *iterable) +.. c:function:: PyObject* PyFrozenSet_New(PyObject *iterable) Return a new :class:`frozenset` containing objects returned by the *iterable*. The *iterable* may be *NULL* to create a new empty frozenset. Return the new @@ -98,7 +98,7 @@ The following functions and macros are available for instances of :class:`set` or :class:`frozenset` or instances of their subtypes. -.. cfunction:: Py_ssize_t PySet_Size(PyObject *anyset) +.. c:function:: Py_ssize_t PySet_Size(PyObject *anyset) .. index:: builtin: len @@ -107,12 +107,12 @@ or :class:`frozenset` or instances of their subtypes. :class:`set`, :class:`frozenset`, or an instance of a subtype. -.. cfunction:: Py_ssize_t PySet_GET_SIZE(PyObject *anyset) +.. c:function:: Py_ssize_t PySet_GET_SIZE(PyObject *anyset) - Macro form of :cfunc:`PySet_Size` without error checking. + Macro form of :c:func:`PySet_Size` without error checking. -.. cfunction:: int PySet_Contains(PyObject *anyset, PyObject *key) +.. c:function:: int PySet_Contains(PyObject *anyset, PyObject *key) Return 1 if found, 0 if not found, and -1 if an error is encountered. Unlike the Python :meth:`__contains__` method, this function does not automatically @@ -121,10 +121,10 @@ or :class:`frozenset` or instances of their subtypes. :class:`set`, :class:`frozenset`, or an instance of a subtype. -.. cfunction:: int PySet_Add(PyObject *set, PyObject *key) +.. c:function:: int PySet_Add(PyObject *set, PyObject *key) Add *key* to a :class:`set` instance. Also works with :class:`frozenset` - instances (like :cfunc:`PyTuple_SetItem` it can be used to fill-in the values + instances (like :c:func:`PyTuple_SetItem` it can be used to fill-in the values of brand new frozensets before they are exposed to other code). Return 0 on success or -1 on failure. Raise a :exc:`TypeError` if the *key* is unhashable. Raise a :exc:`MemoryError` if there is no room to grow. Raise a @@ -136,7 +136,7 @@ The following functions are available for instances of :class:`set` or its subtypes but not for instances of :class:`frozenset` or its subtypes. -.. cfunction:: int PySet_Discard(PyObject *set, PyObject *key) +.. c:function:: int PySet_Discard(PyObject *set, PyObject *key) Return 1 if found and removed, 0 if not found (no action taken), and -1 if an error is encountered. Does not raise :exc:`KeyError` for missing keys. Raise a @@ -146,7 +146,7 @@ subtypes but not for instances of :class:`frozenset` or its subtypes. instance of :class:`set` or its subtype. -.. cfunction:: PyObject* PySet_Pop(PyObject *set) +.. c:function:: PyObject* PySet_Pop(PyObject *set) Return a new reference to an arbitrary object in the *set*, and removes the object from the *set*. Return *NULL* on failure. Raise :exc:`KeyError` if the @@ -154,6 +154,6 @@ subtypes but not for instances of :class:`frozenset` or its subtypes. :class:`set` or its subtype. -.. cfunction:: int PySet_Clear(PyObject *set) +.. c:function:: int PySet_Clear(PyObject *set) Empty an existing set of all elements. diff --git a/Doc/c-api/slice.rst b/Doc/c-api/slice.rst index f33cd533c2..e157df216c 100644 --- a/Doc/c-api/slice.rst +++ b/Doc/c-api/slice.rst @@ -6,18 +6,18 @@ Slice Objects ------------- -.. cvar:: PyTypeObject PySlice_Type +.. c:var:: PyTypeObject PySlice_Type The type object for slice objects. This is the same as :class:`slice` in the Python layer. -.. cfunction:: int PySlice_Check(PyObject *ob) +.. c:function:: int PySlice_Check(PyObject *ob) Return true if *ob* is a slice object; *ob* must not be *NULL*. -.. cfunction:: PyObject* PySlice_New(PyObject *start, PyObject *stop, PyObject *step) +.. c:function:: PyObject* PySlice_New(PyObject *start, PyObject *stop, PyObject *step) Return a new slice object with the given values. The *start*, *stop*, and *step* parameters are used as the values of the slice object attributes of @@ -26,7 +26,7 @@ Slice Objects the new object could not be allocated. -.. cfunction:: int PySlice_GetIndices(PySliceObject *slice, Py_ssize_t length, Py_ssize_t *start, Py_ssize_t *stop, Py_ssize_t *step) +.. c:function:: int PySlice_GetIndices(PyObject *slice, Py_ssize_t length, Py_ssize_t *start, Py_ssize_t *stop, Py_ssize_t *step) Retrieve the start, stop and step indices from the slice object *slice*, assuming a sequence of length *length*. Treats indices greater than @@ -38,13 +38,21 @@ Slice Objects You probably do not want to use this function. + .. versionchanged:: 3.2 + The parameter type for the *slice* parameter was ``PySliceObject*`` + before. -.. cfunction:: int PySlice_GetIndicesEx(PySliceObject *slice, Py_ssize_t length, Py_ssize_t *start, Py_ssize_t *stop, Py_ssize_t *step, Py_ssize_t *slicelength) - Usable replacement for :cfunc:`PySlice_GetIndices`. Retrieve the start, +.. c:function:: int PySlice_GetIndicesEx(PyObject *slice, Py_ssize_t length, Py_ssize_t *start, Py_ssize_t *stop, Py_ssize_t *step, Py_ssize_t *slicelength) + + Usable replacement for :c:func:`PySlice_GetIndices`. Retrieve the start, stop, and step indices from the slice object *slice* assuming a sequence of length *length*, and store the length of the slice in *slicelength*. Out of bounds indices are clipped in a manner consistent with the handling of normal slices. Returns 0 on success and -1 on error with exception set. + + .. versionchanged:: 3.2 + The parameter type for the *slice* parameter was ``PySliceObject*`` + before. diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index dbacedf041..bb741fb099 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -11,12 +11,12 @@ are used. All Python objects ultimately share a small number of fields at the beginning of the object's representation in memory. These are represented by the -:ctype:`PyObject` and :ctype:`PyVarObject` types, which are defined, in turn, +:c:type:`PyObject` and :c:type:`PyVarObject` types, which are defined, in turn, by the expansions of some macros also used, whether directly or indirectly, in the definition of all other Python objects. -.. ctype:: PyObject +.. c:type:: PyObject All object types are extensions of this type. This is a type which contains the information Python needs to treat a pointer to an object as an @@ -26,88 +26,88 @@ the definition of all other Python objects. macro. -.. ctype:: PyVarObject +.. c:type:: PyVarObject - This is an extension of :ctype:`PyObject` that adds the :attr:`ob_size` + This is an extension of :c:type:`PyObject` that adds the :attr:`ob_size` field. This is only used for objects that have some notion of *length*. This type does not often appear in the Python/C API. It corresponds to the fields defined by the expansion of the ``PyObject_VAR_HEAD`` macro. -These macros are used in the definition of :ctype:`PyObject` and -:ctype:`PyVarObject`: +These macros are used in the definition of :c:type:`PyObject` and +:c:type:`PyVarObject`: .. XXX need to document PEP 3123 changes here -.. cmacro:: PyObject_HEAD +.. c:macro:: PyObject_HEAD This is a macro which expands to the declarations of the fields of the - :ctype:`PyObject` type; it is used when declaring new types which represent + :c:type:`PyObject` type; it is used when declaring new types which represent objects without a varying length. The specific fields it expands to depend - on the definition of :cmacro:`Py_TRACE_REFS`. By default, that macro is - not defined, and :cmacro:`PyObject_HEAD` expands to:: + on the definition of :c:macro:`Py_TRACE_REFS`. By default, that macro is + not defined, and :c:macro:`PyObject_HEAD` expands to:: Py_ssize_t ob_refcnt; PyTypeObject *ob_type; - When :cmacro:`Py_TRACE_REFS` is defined, it expands to:: + When :c:macro:`Py_TRACE_REFS` is defined, it expands to:: PyObject *_ob_next, *_ob_prev; Py_ssize_t ob_refcnt; PyTypeObject *ob_type; -.. cmacro:: PyObject_VAR_HEAD +.. c:macro:: PyObject_VAR_HEAD This is a macro which expands to the declarations of the fields of the - :ctype:`PyVarObject` type; it is used when declaring new types which + :c:type:`PyVarObject` type; it is used when declaring new types which represent objects with a length that varies from instance to instance. This macro always expands to:: PyObject_HEAD Py_ssize_t ob_size; - Note that :cmacro:`PyObject_HEAD` is part of the expansion, and that its own - expansion varies depending on the definition of :cmacro:`Py_TRACE_REFS`. + Note that :c:macro:`PyObject_HEAD` is part of the expansion, and that its own + expansion varies depending on the definition of :c:macro:`Py_TRACE_REFS`. -.. cmacro:: PyObject_HEAD_INIT(type) +.. c:macro:: PyObject_HEAD_INIT(type) This is a macro which expands to initialization values for a new - :ctype:`PyObject` type. This macro expands to:: + :c:type:`PyObject` type. This macro expands to:: _PyObject_EXTRA_INIT 1, type, -.. cmacro:: PyVarObject_HEAD_INIT(type, size) +.. c:macro:: PyVarObject_HEAD_INIT(type, size) This is a macro which expands to initialization values for a new - :ctype:`PyVarObject` type, including the :attr:`ob_size` field. + :c:type:`PyVarObject` type, including the :attr:`ob_size` field. This macro expands to:: _PyObject_EXTRA_INIT 1, type, size, -.. ctype:: PyCFunction +.. c:type:: PyCFunction Type of the functions used to implement most Python callables in C. - Functions of this type take two :ctype:`PyObject\*` parameters and return + Functions of this type take two :c:type:`PyObject\*` parameters and return one such value. If the return value is *NULL*, an exception shall have been set. If not *NULL*, the return value is interpreted as the return value of the function as exposed in Python. The function must return a new reference. -.. ctype:: PyCFunctionWithKeywords +.. c:type:: PyCFunctionWithKeywords Type of the functions used to implement Python callables in C that take - keyword arguments: they take three :ctype:`PyObject\*` parameters and return - one such value. See :ctype:`PyCFunction` above for the meaning of the return + keyword arguments: they take three :c:type:`PyObject\*` parameters and return + one such value. See :c:type:`PyCFunction` above for the meaning of the return value. -.. ctype:: PyMethodDef +.. c:type:: PyMethodDef Structure used to describe a method of an extension type. This structure has four fields: @@ -128,10 +128,10 @@ These macros are used in the definition of :ctype:`PyObject` and +------------------+-------------+-------------------------------+ The :attr:`ml_meth` is a C function pointer. The functions may be of different -types, but they always return :ctype:`PyObject\*`. If the function is not of -the :ctype:`PyCFunction`, the compiler will require a cast in the method table. -Even though :ctype:`PyCFunction` defines the first parameter as -:ctype:`PyObject\*`, it is common that the method implementation uses a the +types, but they always return :c:type:`PyObject\*`. If the function is not of +the :c:type:`PyCFunction`, the compiler will require a cast in the method table. +Even though :c:type:`PyCFunction` defines the first parameter as +:c:type:`PyObject\*`, it is common that the method implementation uses a the specific C type of the *self* object. The :attr:`ml_flags` field is a bitfield which can include the following flags. @@ -145,27 +145,27 @@ convention flags can be combined with a binding flag. .. data:: METH_VARARGS This is the typical calling convention, where the methods have the type - :ctype:`PyCFunction`. The function expects two :ctype:`PyObject\*` values. + :c:type:`PyCFunction`. The function expects two :c:type:`PyObject\*` values. The first one is the *self* object for methods; for module functions, it is the module object. The second parameter (often called *args*) is a tuple object representing all arguments. This parameter is typically processed - using :cfunc:`PyArg_ParseTuple` or :cfunc:`PyArg_UnpackTuple`. + using :c:func:`PyArg_ParseTuple` or :c:func:`PyArg_UnpackTuple`. .. data:: METH_KEYWORDS - Methods with these flags must be of type :ctype:`PyCFunctionWithKeywords`. + Methods with these flags must be of type :c:type:`PyCFunctionWithKeywords`. The function expects three parameters: *self*, *args*, and a dictionary of all the keyword arguments. The flag is typically combined with :const:`METH_VARARGS`, and the parameters are typically processed using - :cfunc:`PyArg_ParseTupleAndKeywords`. + :c:func:`PyArg_ParseTupleAndKeywords`. .. data:: METH_NOARGS Methods without parameters don't need to check whether arguments are given if they are listed with the :const:`METH_NOARGS` flag. They need to be of type - :ctype:`PyCFunction`. The first parameter is typically named *self* and will + :c:type:`PyCFunction`. The first parameter is typically named *self* and will hold a reference to the module or object instance. In all cases the second parameter will be *NULL*. @@ -173,9 +173,9 @@ convention flags can be combined with a binding flag. .. data:: METH_O Methods with a single object argument can be listed with the :const:`METH_O` - flag, instead of invoking :cfunc:`PyArg_ParseTuple` with a ``"O"`` argument. - They have the type :ctype:`PyCFunction`, with the *self* parameter, and a - :ctype:`PyObject\*` parameter representing the single argument. + flag, instead of invoking :c:func:`PyArg_ParseTuple` with a ``"O"`` argument. + They have the type :c:type:`PyCFunction`, with the *self* parameter, and a + :c:type:`PyObject\*` parameter representing the single argument. These two constants are not used to indicate the calling convention but the @@ -219,7 +219,7 @@ definition with the same method name. than wrapper object calls. -.. ctype:: PyMemberDef +.. c:type:: PyMemberDef Structure which describes an attribute of a type which corresponds to a C struct member. Its fields are: @@ -271,14 +271,14 @@ definition with the same method name. T_PYSSIZET Py_ssize_t =============== ================== - :cmacro:`T_OBJECT` and :cmacro:`T_OBJECT_EX` differ in that - :cmacro:`T_OBJECT` returns ``None`` if the member is *NULL* and - :cmacro:`T_OBJECT_EX` raises an :exc:`AttributeError`. Try to use - :cmacro:`T_OBJECT_EX` over :cmacro:`T_OBJECT` because :cmacro:`T_OBJECT_EX` + :c:macro:`T_OBJECT` and :c:macro:`T_OBJECT_EX` differ in that + :c:macro:`T_OBJECT` returns ``None`` if the member is *NULL* and + :c:macro:`T_OBJECT_EX` raises an :exc:`AttributeError`. Try to use + :c:macro:`T_OBJECT_EX` over :c:macro:`T_OBJECT` because :c:macro:`T_OBJECT_EX` handles use of the :keyword:`del` statement on that attribute more correctly - than :cmacro:`T_OBJECT`. + than :c:macro:`T_OBJECT`. - :attr:`flags` can be 0 for write and read access or :cmacro:`READONLY` for - read-only access. Using :cmacro:`T_STRING` for :attr:`type` implies - :cmacro:`READONLY`. Only :cmacro:`T_OBJECT` and :cmacro:`T_OBJECT_EX` + :attr:`flags` can be 0 for write and read access or :c:macro:`READONLY` for + read-only access. Using :c:macro:`T_STRING` for :attr:`type` implies + :c:macro:`READONLY`. Only :c:macro:`T_OBJECT` and :c:macro:`T_OBJECT_EX` members can be deleted. (They are set to *NULL*). diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index adadfe55da..252bd1ad06 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -6,16 +6,16 @@ Operating System Utilities ========================== -.. cfunction:: int Py_FdIsInteractive(FILE *fp, const char *filename) +.. c:function:: int Py_FdIsInteractive(FILE *fp, const char *filename) Return true (nonzero) if the standard I/O file *fp* with name *filename* is deemed interactive. This is the case for files for which ``isatty(fileno(fp))`` - is true. If the global flag :cdata:`Py_InteractiveFlag` is true, this function + is true. If the global flag :c:data:`Py_InteractiveFlag` is true, this function also returns true if the *filename* pointer is *NULL* or if the name is equal to one of the strings ``'<stdin>'`` or ``'???'``. -.. cfunction:: void PyOS_AfterFork() +.. c:function:: void PyOS_AfterFork() Function to update some internal state after a process fork; this should be called in the new process if the Python interpreter will continue to be used. @@ -23,7 +23,7 @@ Operating System Utilities to be called. -.. cfunction:: int PyOS_CheckStack() +.. c:function:: int PyOS_CheckStack() Return true when the interpreter runs out of stack space. This is a reliable check, but is only available when :const:`USE_STACKCHECK` is defined (currently @@ -32,20 +32,20 @@ Operating System Utilities own code. -.. cfunction:: PyOS_sighandler_t PyOS_getsig(int i) +.. c:function:: PyOS_sighandler_t PyOS_getsig(int i) Return the current signal handler for signal *i*. This is a thin wrapper around - either :cfunc:`sigaction` or :cfunc:`signal`. Do not call those functions - directly! :ctype:`PyOS_sighandler_t` is a typedef alias for :ctype:`void + either :c:func:`sigaction` or :c:func:`signal`. Do not call those functions + directly! :c:type:`PyOS_sighandler_t` is a typedef alias for :c:type:`void (\*)(int)`. -.. cfunction:: PyOS_sighandler_t PyOS_setsig(int i, PyOS_sighandler_t h) +.. c:function:: PyOS_sighandler_t PyOS_setsig(int i, PyOS_sighandler_t h) Set the signal handler for signal *i* to be *h*; return the old signal handler. - This is a thin wrapper around either :cfunc:`sigaction` or :cfunc:`signal`. Do - not call those functions directly! :ctype:`PyOS_sighandler_t` is a typedef - alias for :ctype:`void (\*)(int)`. + This is a thin wrapper around either :c:func:`sigaction` or :c:func:`signal`. Do + not call those functions directly! :c:type:`PyOS_sighandler_t` is a typedef + alias for :c:type:`void (\*)(int)`. .. _systemfunctions: @@ -56,38 +56,42 @@ These are utility functions that make functionality from the :mod:`sys` module accessible to C code. They all work with the current interpreter thread's :mod:`sys` module's dict, which is contained in the internal thread state structure. -.. cfunction:: PyObject *PySys_GetObject(char *name) +.. c:function:: PyObject *PySys_GetObject(char *name) Return the object *name* from the :mod:`sys` module or *NULL* if it does not exist, without setting an exception. -.. cfunction:: FILE *PySys_GetFile(char *name, FILE *def) +.. c:function:: FILE *PySys_GetFile(char *name, FILE *def) - Return the :ctype:`FILE*` associated with the object *name* in the + Return the :c:type:`FILE*` associated with the object *name* in the :mod:`sys` module, or *def* if *name* is not in the module or is not associated - with a :ctype:`FILE*`. + with a :c:type:`FILE*`. -.. cfunction:: int PySys_SetObject(char *name, PyObject *v) +.. c:function:: int PySys_SetObject(char *name, PyObject *v) Set *name* in the :mod:`sys` module to *v* unless *v* is *NULL*, in which case *name* is deleted from the sys module. Returns ``0`` on success, ``-1`` on error. -.. cfunction:: void PySys_ResetWarnOptions() +.. c:function:: void PySys_ResetWarnOptions() Reset :data:`sys.warnoptions` to an empty list. -.. cfunction:: void PySys_AddWarnOption(wchar_t *s) +.. c:function:: void PySys_AddWarnOption(wchar_t *s) Append *s* to :data:`sys.warnoptions`. -.. cfunction:: void PySys_SetPath(wchar_t *path) +.. c:function:: void PySys_AddWarnOptionUnicode(PyObject *unicode) + + Append *unicode* to :data:`sys.warnoptions`. + +.. c:function:: void PySys_SetPath(wchar_t *path) Set :data:`sys.path` to a list object of paths found in *path* which should be a list of paths separated with the platform's search path delimiter (``:`` on Unix, ``;`` on Windows). -.. cfunction:: void PySys_WriteStdout(const char *format, ...) +.. c:function:: void PySys_WriteStdout(const char *format, ...) Write the output string described by *format* to :data:`sys.stdout`. No exceptions are raised, even if truncation occurs (see below). @@ -103,9 +107,40 @@ accessible to C code. They all work with the current interpreter thread's If a problem occurs, or :data:`sys.stdout` is unset, the formatted message is written to the real (C level) *stdout*. -.. cfunction:: void PySys_WriteStderr(const char *format, ...) +.. c:function:: void PySys_WriteStderr(const char *format, ...) + + As :c:func:`PySys_WriteStdout`, but write to :data:`sys.stderr` or *stderr* + instead. + +.. c:function:: void PySys_FormatStdout(const char *format, ...) + + Function similar to PySys_WriteStdout() but format the message using + :c:func:`PyUnicode_FromFormatV` and don't truncate the message to an + arbitrary length. + + .. versionadded:: 3.2 + +.. c:function:: void PySys_FormatStderr(const char *format, ...) + + As :c:func:`PySys_FormatStdout`, but write to :data:`sys.stderr` or *stderr* + instead. + + .. versionadded:: 3.2 + +.. c:function:: void PySys_AddXOption(const wchar_t *s) + + Parse *s* as a set of :option:`-X` options and add them to the current + options mapping as returned by :c:func:`PySys_GetXOptions`. + + .. versionadded:: 3.2 + +.. c:function:: PyObject *PySys_GetXOptions() + + Return the current dictionary of :option:`-X` options, similarly to + :data:`sys._xoptions`. On error, *NULL* is returned and an exception is + set. - As above, but write to :data:`sys.stderr` or *stderr* instead. + .. versionadded:: 3.2 .. _processcontrol: @@ -114,7 +149,7 @@ Process Control =============== -.. cfunction:: void Py_FatalError(const char *message) +.. c:function:: void Py_FatalError(const char *message) .. index:: single: abort() @@ -122,30 +157,30 @@ Process Control This function should only be invoked when a condition is detected that would make it dangerous to continue using the Python interpreter; e.g., when the object administration appears to be corrupted. On Unix, the standard C library - function :cfunc:`abort` is called which will attempt to produce a :file:`core` + function :c:func:`abort` is called which will attempt to produce a :file:`core` file. -.. cfunction:: void Py_Exit(int status) +.. c:function:: void Py_Exit(int status) .. index:: single: Py_Finalize() single: exit() - Exit the current process. This calls :cfunc:`Py_Finalize` and then calls the + Exit the current process. This calls :c:func:`Py_Finalize` and then calls the standard C library function ``exit(status)``. -.. cfunction:: int Py_AtExit(void (*func) ()) +.. c:function:: int Py_AtExit(void (*func) ()) .. index:: single: Py_Finalize() single: cleanup functions - Register a cleanup function to be called by :cfunc:`Py_Finalize`. The cleanup + Register a cleanup function to be called by :c:func:`Py_Finalize`. The cleanup function will be called with no arguments and should return no value. At most 32 cleanup functions can be registered. When the registration is successful, - :cfunc:`Py_AtExit` returns ``0``; on failure, it returns ``-1``. The cleanup + :c:func:`Py_AtExit` returns ``0``; on failure, it returns ``-1``. The cleanup function registered last is called first. Each cleanup function will be called at most once. Since Python's internal finalization will have completed before the cleanup function, no Python APIs should be called by *func*. diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst index 612acc754e..3cbfe5bcaa 100644 --- a/Doc/c-api/tuple.rst +++ b/Doc/c-api/tuple.rst @@ -8,70 +8,70 @@ Tuple Objects .. index:: object: tuple -.. ctype:: PyTupleObject +.. c:type:: PyTupleObject - This subtype of :ctype:`PyObject` represents a Python tuple object. + This subtype of :c:type:`PyObject` represents a Python tuple object. -.. cvar:: PyTypeObject PyTuple_Type +.. c:var:: PyTypeObject PyTuple_Type - This instance of :ctype:`PyTypeObject` represents the Python tuple type; it + This instance of :c:type:`PyTypeObject` represents the Python tuple type; it is the same object as :class:`tuple` in the Python layer. -.. cfunction:: int PyTuple_Check(PyObject *p) +.. c:function:: int PyTuple_Check(PyObject *p) Return true if *p* is a tuple object or an instance of a subtype of the tuple type. -.. cfunction:: int PyTuple_CheckExact(PyObject *p) +.. c:function:: int PyTuple_CheckExact(PyObject *p) Return true if *p* is a tuple object, but not an instance of a subtype of the tuple type. -.. cfunction:: PyObject* PyTuple_New(Py_ssize_t len) +.. c:function:: PyObject* PyTuple_New(Py_ssize_t len) Return a new tuple object of size *len*, or *NULL* on failure. -.. cfunction:: PyObject* PyTuple_Pack(Py_ssize_t n, ...) +.. c:function:: PyObject* PyTuple_Pack(Py_ssize_t n, ...) Return a new tuple object of size *n*, or *NULL* on failure. The tuple values are initialized to the subsequent *n* C arguments pointing to Python objects. ``PyTuple_Pack(2, a, b)`` is equivalent to ``Py_BuildValue("(OO)", a, b)``. -.. cfunction:: Py_ssize_t PyTuple_Size(PyObject *p) +.. c:function:: Py_ssize_t PyTuple_Size(PyObject *p) Take a pointer to a tuple object, and return the size of that tuple. -.. cfunction:: Py_ssize_t PyTuple_GET_SIZE(PyObject *p) +.. c:function:: Py_ssize_t PyTuple_GET_SIZE(PyObject *p) Return the size of the tuple *p*, which must be non-*NULL* and point to a tuple; no error checking is performed. -.. cfunction:: PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos) +.. c:function:: PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos) Return the object at position *pos* in the tuple pointed to by *p*. If *pos* is out of bounds, return *NULL* and sets an :exc:`IndexError` exception. -.. cfunction:: PyObject* PyTuple_GET_ITEM(PyObject *p, Py_ssize_t pos) +.. c:function:: PyObject* PyTuple_GET_ITEM(PyObject *p, Py_ssize_t pos) - Like :cfunc:`PyTuple_GetItem`, but does no checking of its arguments. + Like :c:func:`PyTuple_GetItem`, but does no checking of its arguments. -.. cfunction:: PyObject* PyTuple_GetSlice(PyObject *p, Py_ssize_t low, Py_ssize_t high) +.. c:function:: PyObject* PyTuple_GetSlice(PyObject *p, Py_ssize_t low, Py_ssize_t high) Take a slice of the tuple pointed to by *p* from *low* to *high* and return it as a new tuple. -.. cfunction:: int PyTuple_SetItem(PyObject *p, Py_ssize_t pos, PyObject *o) +.. c:function:: int PyTuple_SetItem(PyObject *p, Py_ssize_t pos, PyObject *o) Insert a reference to object *o* at position *pos* of the tuple pointed to by *p*. Return ``0`` on success. @@ -81,9 +81,9 @@ Tuple Objects This function "steals" a reference to *o*. -.. cfunction:: void PyTuple_SET_ITEM(PyObject *p, Py_ssize_t pos, PyObject *o) +.. c:function:: void PyTuple_SET_ITEM(PyObject *p, Py_ssize_t pos, PyObject *o) - Like :cfunc:`PyTuple_SetItem`, but does no error checking, and should *only* be + Like :c:func:`PyTuple_SetItem`, but does no error checking, and should *only* be used to fill in brand new tuples. .. note:: @@ -91,7 +91,7 @@ Tuple Objects This function "steals" a reference to *o*. -.. cfunction:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) +.. c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) Can be used to resize a tuple. *newsize* will be the new length of the tuple. Because tuples are *supposed* to be immutable, this should only be used if there @@ -105,6 +105,6 @@ Tuple Objects raises :exc:`MemoryError` or :exc:`SystemError`. -.. cfunction:: int PyTuple_ClearFreeList() +.. c:function:: int PyTuple_ClearFreeList() Clear the free list. Return the total number of freed items. diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst index d2a567656e..b3386eaa8b 100644 --- a/Doc/c-api/type.rst +++ b/Doc/c-api/type.rst @@ -8,69 +8,77 @@ Type Objects .. index:: object: type -.. ctype:: PyTypeObject +.. c:type:: PyTypeObject The C structure of the objects used to describe built-in types. -.. cvar:: PyObject* PyType_Type +.. c:var:: PyObject* PyType_Type This is the type object for type objects; it is the same object as :class:`type` in the Python layer. -.. cfunction:: int PyType_Check(PyObject *o) +.. c:function:: int PyType_Check(PyObject *o) Return true if the object *o* is a type object, including instances of types derived from the standard type object. Return false in all other cases. -.. cfunction:: int PyType_CheckExact(PyObject *o) +.. c:function:: int PyType_CheckExact(PyObject *o) Return true if the object *o* is a type object, but not a subtype of the standard type object. Return false in all other cases. -.. cfunction:: unsigned int PyType_ClearCache() +.. c:function:: unsigned int PyType_ClearCache() Clear the internal lookup cache. Return the current version tag. +.. c:function:: long PyType_GetFlags(PyTypeObject* type) -.. cfunction:: void PyType_Modified(PyTypeObject *type) + Return the :attr:`tp_flags` member of *type*. This function is primarily + meant for use with `Py_LIMITED_API`; the individual flag bits are + guaranteed to be stable across Python releases, but access to + :attr:`tp_flags` itself is not part of the limited API. + + .. versionadded:: 3.2 + +.. c:function:: void PyType_Modified(PyTypeObject *type) Invalidate the internal lookup cache for the type and all of its subtypes. This function must be called after any manual modification of the attributes or base classes of the type. -.. cfunction:: int PyType_HasFeature(PyObject *o, int feature) +.. c:function:: int PyType_HasFeature(PyObject *o, int feature) Return true if the type object *o* sets the feature *feature*. Type features are denoted by single bit flags. -.. cfunction:: int PyType_IS_GC(PyObject *o) +.. c:function:: int PyType_IS_GC(PyObject *o) Return true if the type object includes support for the cycle detector; this tests the type flag :const:`Py_TPFLAGS_HAVE_GC`. -.. cfunction:: int PyType_IsSubtype(PyTypeObject *a, PyTypeObject *b) +.. c:function:: int PyType_IsSubtype(PyTypeObject *a, PyTypeObject *b) Return true if *a* is a subtype of *b*. -.. cfunction:: PyObject* PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems) +.. c:function:: PyObject* PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems) XXX: Document. -.. cfunction:: PyObject* PyType_GenericNew(PyTypeObject *type, PyObject *args, PyObject *kwds) +.. c:function:: PyObject* PyType_GenericNew(PyTypeObject *type, PyObject *args, PyObject *kwds) XXX: Document. -.. cfunction:: int PyType_Ready(PyTypeObject *type) +.. c:function:: int PyType_Ready(PyTypeObject *type) Finalize a type object. This should be called on all type objects to finish their initialization. This function is responsible for adding inherited slots diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 7aa827af88..68ca9adaa9 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -6,9 +6,9 @@ Type Objects ============ Perhaps one of the most important structures of the Python object system is the -structure that defines a new type: the :ctype:`PyTypeObject` structure. Type -objects can be handled using any of the :cfunc:`PyObject_\*` or -:cfunc:`PyType_\*` functions, but do not offer much that's interesting to most +structure that defines a new type: the :c:type:`PyTypeObject` structure. Type +objects can be handled using any of the :c:func:`PyObject_\*` or +:c:func:`PyType_\*` functions, but do not offer much that's interesting to most Python applications. These objects are fundamental to how objects behave, so they are very important to the interpreter itself and to any extension module that implements new types. @@ -25,21 +25,21 @@ intintargfunc, intobjargproc, intintobjargproc, objobjargproc, destructor, freefunc, printfunc, getattrfunc, getattrofunc, setattrfunc, setattrofunc, reprfunc, hashfunc -The structure definition for :ctype:`PyTypeObject` can be found in +The structure definition for :c:type:`PyTypeObject` can be found in :file:`Include/object.h`. For convenience of reference, this repeats the definition found there: .. literalinclude:: ../includes/typestruct.h -The type object structure extends the :ctype:`PyVarObject` structure. The +The type object structure extends the :c:type:`PyVarObject` structure. The :attr:`ob_size` field is used for dynamic types (created by :func:`type_new`, -usually called from a class statement). Note that :cdata:`PyType_Type` (the +usually called from a class statement). Note that :c:data:`PyType_Type` (the metatype) initializes :attr:`tp_itemsize`, which means that its instances (i.e. type objects) *must* have the :attr:`ob_size` field. -.. cmember:: PyObject* PyObject._ob_next +.. c:member:: PyObject* PyObject._ob_next PyObject* PyObject._ob_prev These fields are only present when the macro ``Py_TRACE_REFS`` is defined. @@ -54,7 +54,7 @@ type objects) *must* have the :attr:`ob_size` field. These fields are not inherited by subtypes. -.. cmember:: Py_ssize_t PyObject.ob_refcnt +.. c:member:: Py_ssize_t PyObject.ob_refcnt This is the type object's reference count, initialized to ``1`` by the ``PyObject_HEAD_INIT`` macro. Note that for statically allocated type objects, @@ -65,7 +65,7 @@ type objects) *must* have the :attr:`ob_size` field. This field is not inherited by subtypes. -.. cmember:: PyTypeObject* PyObject.ob_type +.. c:member:: PyTypeObject* PyObject.ob_type This is the type's type, in other words its metatype. It is initialized by the argument to the ``PyObject_HEAD_INIT`` macro, and its value should normally be @@ -79,14 +79,14 @@ type objects) *must* have the :attr:`ob_size` field. Foo_Type.ob_type = &PyType_Type; This should be done before any instances of the type are created. - :cfunc:`PyType_Ready` checks if :attr:`ob_type` is *NULL*, and if so, + :c:func:`PyType_Ready` checks if :attr:`ob_type` is *NULL*, and if so, initializes it to the :attr:`ob_type` field of the base class. - :cfunc:`PyType_Ready` will not change this field if it is non-zero. + :c:func:`PyType_Ready` will not change this field if it is non-zero. This field is inherited by subtypes. -.. cmember:: Py_ssize_t PyVarObject.ob_size +.. c:member:: Py_ssize_t PyVarObject.ob_size For statically allocated type objects, this should be initialized to zero. For dynamically allocated type objects, this field has a special internal meaning. @@ -94,7 +94,7 @@ type objects) *must* have the :attr:`ob_size` field. This field is not inherited by subtypes. -.. cmember:: char* PyTypeObject.tp_name +.. c:member:: char* PyTypeObject.tp_name Pointer to a NUL-terminated string containing the name of the type. For types that are accessible as module globals, the string should be the full module @@ -121,7 +121,7 @@ type objects) *must* have the :attr:`ob_size` field. This field is not inherited by subtypes. -.. cmember:: Py_ssize_t PyTypeObject.tp_basicsize +.. c:member:: Py_ssize_t PyTypeObject.tp_basicsize Py_ssize_t PyTypeObject.tp_itemsize These fields allow calculating the size in bytes of instances of the type. @@ -143,7 +143,7 @@ type objects) *must* have the :attr:`ob_size` field. field). The basic size includes the fields in the instance declared by the macro - :cmacro:`PyObject_HEAD` or :cmacro:`PyObject_VAR_HEAD` (whichever is used to + :c:macro:`PyObject_HEAD` or :c:macro:`PyObject_VAR_HEAD` (whichever is used to declare the instance struct) and this in turn includes the :attr:`_ob_prev` and :attr:`_ob_next` fields if they are present. This means that the only correct way to get an initializer for the :attr:`tp_basicsize` is to use the @@ -163,14 +163,14 @@ type objects) *must* have the :attr:`ob_size` field. alignment requirement for ``double``). -.. cmember:: destructor PyTypeObject.tp_dealloc +.. c:member:: destructor PyTypeObject.tp_dealloc A pointer to the instance destructor function. This function must be defined unless the type guarantees that its instances will never be deallocated (as is the case for the singletons ``None`` and ``Ellipsis``). - The destructor function is called by the :cfunc:`Py_DECREF` and - :cfunc:`Py_XDECREF` macros when the new reference count is zero. At this point, + The destructor function is called by the :c:func:`Py_DECREF` and + :c:func:`Py_XDECREF` macros when the new reference count is zero. At this point, the instance is still in existence, but there are no references to it. The destructor function should free all references which the instance owns, free all memory buffers owned by the instance (using the freeing function corresponding @@ -179,15 +179,15 @@ type objects) *must* have the :attr:`ob_size` field. subtypable (doesn't have the :const:`Py_TPFLAGS_BASETYPE` flag bit set), it is permissible to call the object deallocator directly instead of via :attr:`tp_free`. The object deallocator should be the one used to allocate the - instance; this is normally :cfunc:`PyObject_Del` if the instance was allocated - using :cfunc:`PyObject_New` or :cfunc:`PyObject_VarNew`, or - :cfunc:`PyObject_GC_Del` if the instance was allocated using - :cfunc:`PyObject_GC_New` or :cfunc:`PyObject_GC_NewVar`. + instance; this is normally :c:func:`PyObject_Del` if the instance was allocated + using :c:func:`PyObject_New` or :c:func:`PyObject_VarNew`, or + :c:func:`PyObject_GC_Del` if the instance was allocated using + :c:func:`PyObject_GC_New` or :c:func:`PyObject_GC_NewVar`. This field is inherited by subtypes. -.. cmember:: printfunc PyTypeObject.tp_print +.. c:member:: printfunc PyTypeObject.tp_print An optional pointer to the instance print function. @@ -198,7 +198,7 @@ type objects) *must* have the :attr:`ob_size` field. *NULL*. A type should never implement :attr:`tp_print` in a way that produces different output than :attr:`tp_repr` or :attr:`tp_str` would. - The print function is called with the same signature as :cfunc:`PyObject_Print`: + The print function is called with the same signature as :c:func:`PyObject_Print`: ``int tp_print(PyObject *self, FILE *file, int flags)``. The *self* argument is the instance to be printed. The *file* argument is the stdio file to which it is to be printed. The *flags* argument is composed of flag bits. The only flag @@ -216,47 +216,47 @@ type objects) *must* have the :attr:`ob_size` field. This field is inherited by subtypes. -.. cmember:: getattrfunc PyTypeObject.tp_getattr +.. c:member:: getattrfunc PyTypeObject.tp_getattr An optional pointer to the get-attribute-string function. This field is deprecated. When it is defined, it should point to a function that acts the same as the :attr:`tp_getattro` function, but taking a C string instead of a Python string object to give the attribute name. The signature is - the same as for :cfunc:`PyObject_GetAttrString`. + the same as for :c:func:`PyObject_GetAttrString`. This field is inherited by subtypes together with :attr:`tp_getattro`: a subtype inherits both :attr:`tp_getattr` and :attr:`tp_getattro` from its base type when the subtype's :attr:`tp_getattr` and :attr:`tp_getattro` are both *NULL*. -.. cmember:: setattrfunc PyTypeObject.tp_setattr +.. c:member:: setattrfunc PyTypeObject.tp_setattr An optional pointer to the set-attribute-string function. This field is deprecated. When it is defined, it should point to a function that acts the same as the :attr:`tp_setattro` function, but taking a C string instead of a Python string object to give the attribute name. The signature is - the same as for :cfunc:`PyObject_SetAttrString`. + the same as for :c:func:`PyObject_SetAttrString`. This field is inherited by subtypes together with :attr:`tp_setattro`: a subtype inherits both :attr:`tp_setattr` and :attr:`tp_setattro` from its base type when the subtype's :attr:`tp_setattr` and :attr:`tp_setattro` are both *NULL*. -.. cmember:: void* PyTypeObject.tp_reserved +.. c:member:: void* PyTypeObject.tp_reserved Reserved slot, formerly known as tp_compare. -.. cmember:: reprfunc PyTypeObject.tp_repr +.. c:member:: reprfunc PyTypeObject.tp_repr .. index:: builtin: repr An optional pointer to a function that implements the built-in function :func:`repr`. - The signature is the same as for :cfunc:`PyObject_Repr`; it must return a string + The signature is the same as for :c:func:`PyObject_Repr`; it must return a string or a Unicode object. Ideally, this function should return a string that, when passed to :func:`eval`, given a suitable environment, returns an object with the same value. If this is not feasible, it should return a string starting with @@ -269,7 +269,7 @@ type objects) *must* have the :attr:`ob_size` field. This field is inherited by subtypes. -.. cmember:: PyNumberMethods* tp_as_number +.. c:member:: PyNumberMethods* tp_as_number Pointer to an additional structure that contains fields relevant only to objects which implement the number protocol. These fields are documented in @@ -279,7 +279,7 @@ type objects) *must* have the :attr:`ob_size` field. inherited individually. -.. cmember:: PySequenceMethods* tp_as_sequence +.. c:member:: PySequenceMethods* tp_as_sequence Pointer to an additional structure that contains fields relevant only to objects which implement the sequence protocol. These fields are documented @@ -289,7 +289,7 @@ type objects) *must* have the :attr:`ob_size` field. are inherited individually. -.. cmember:: PyMappingMethods* tp_as_mapping +.. c:member:: PyMappingMethods* tp_as_mapping Pointer to an additional structure that contains fields relevant only to objects which implement the mapping protocol. These fields are documented in @@ -299,25 +299,25 @@ type objects) *must* have the :attr:`ob_size` field. are inherited individually. -.. cmember:: hashfunc PyTypeObject.tp_hash +.. c:member:: hashfunc PyTypeObject.tp_hash .. index:: builtin: hash An optional pointer to a function that implements the built-in function :func:`hash`. - The signature is the same as for :cfunc:`PyObject_Hash`; it must return a C - long. The value ``-1`` should not be returned as a normal return value; when an - error occurs during the computation of the hash value, the function should set - an exception and return ``-1``. + The signature is the same as for :c:func:`PyObject_Hash`; it must return a + value of the type Py_hash_t. The value ``-1`` should not be returned as a + normal return value; when an error occurs during the computation of the hash + value, the function should set an exception and return ``-1``. - This field can be set explicitly to :cfunc:`PyObject_HashNotImplemented` to + This field can be set explicitly to :c:func:`PyObject_HashNotImplemented` to block inheritance of the hash method from a parent type. This is interpreted as the equivalent of ``__hash__ = None`` at the Python level, causing ``isinstance(o, collections.Hashable)`` to correctly return ``False``. Note that the converse is also true - setting ``__hash__ = None`` on a class at the Python level will result in the ``tp_hash`` slot being set to - :cfunc:`PyObject_HashNotImplemented`. + :c:func:`PyObject_HashNotImplemented`. When this field is not set, an attempt to take the hash of the object raises :exc:`TypeError`. @@ -328,39 +328,39 @@ type objects) *must* have the :attr:`ob_size` field. :attr:`tp_richcompare` and :attr:`tp_hash` are both *NULL*. -.. cmember:: ternaryfunc PyTypeObject.tp_call +.. c:member:: ternaryfunc PyTypeObject.tp_call An optional pointer to a function that implements calling the object. This should be *NULL* if the object is not callable. The signature is the same as - for :cfunc:`PyObject_Call`. + for :c:func:`PyObject_Call`. This field is inherited by subtypes. -.. cmember:: reprfunc PyTypeObject.tp_str +.. c:member:: reprfunc PyTypeObject.tp_str An optional pointer to a function that implements the built-in operation :func:`str`. (Note that :class:`str` is a type now, and :func:`str` calls the - constructor for that type. This constructor calls :cfunc:`PyObject_Str` to do - the actual work, and :cfunc:`PyObject_Str` will call this handler.) + constructor for that type. This constructor calls :c:func:`PyObject_Str` to do + the actual work, and :c:func:`PyObject_Str` will call this handler.) - The signature is the same as for :cfunc:`PyObject_Str`; it must return a string + The signature is the same as for :c:func:`PyObject_Str`; it must return a string or a Unicode object. This function should return a "friendly" string representation of the object, as this is the representation that will be used, among other things, by the :func:`print` function. - When this field is not set, :cfunc:`PyObject_Repr` is called to return a string + When this field is not set, :c:func:`PyObject_Repr` is called to return a string representation. This field is inherited by subtypes. -.. cmember:: getattrofunc PyTypeObject.tp_getattro +.. c:member:: getattrofunc PyTypeObject.tp_getattro An optional pointer to the get-attribute function. - The signature is the same as for :cfunc:`PyObject_GetAttr`. It is usually - convenient to set this field to :cfunc:`PyObject_GenericGetAttr`, which + The signature is the same as for :c:func:`PyObject_GetAttr`. It is usually + convenient to set this field to :c:func:`PyObject_GenericGetAttr`, which implements the normal way of looking for object attributes. This field is inherited by subtypes together with :attr:`tp_getattr`: a subtype @@ -368,12 +368,12 @@ type objects) *must* have the :attr:`ob_size` field. the subtype's :attr:`tp_getattr` and :attr:`tp_getattro` are both *NULL*. -.. cmember:: setattrofunc PyTypeObject.tp_setattro +.. c:member:: setattrofunc PyTypeObject.tp_setattro An optional pointer to the set-attribute function. - The signature is the same as for :cfunc:`PyObject_SetAttr`. It is usually - convenient to set this field to :cfunc:`PyObject_GenericSetAttr`, which + The signature is the same as for :c:func:`PyObject_SetAttr`. It is usually + convenient to set this field to :c:func:`PyObject_GenericSetAttr`, which implements the normal way of setting object attributes. This field is inherited by subtypes together with :attr:`tp_setattr`: a subtype @@ -381,7 +381,7 @@ type objects) *must* have the :attr:`ob_size` field. the subtype's :attr:`tp_setattr` and :attr:`tp_setattro` are both *NULL*. -.. cmember:: PyBufferProcs* PyTypeObject.tp_as_buffer +.. c:member:: PyBufferProcs* PyTypeObject.tp_as_buffer Pointer to an additional structure that contains fields relevant only to objects which implement the buffer interface. These fields are documented in @@ -391,7 +391,7 @@ type objects) *must* have the :attr:`ob_size` field. inherited individually. -.. cmember:: long PyTypeObject.tp_flags +.. c:member:: long PyTypeObject.tp_flags This field is a bit mask of various flags. Some flags indicate variant semantics for certain situations; others are used to indicate that certain @@ -414,7 +414,7 @@ type objects) *must* have the :attr:`ob_size` field. The following bit masks are currently defined; these can be ORed together using the ``|`` operator to form the value of the :attr:`tp_flags` field. The macro - :cfunc:`PyType_HasFeature` takes a type and a flags value, *tp* and *f*, and + :c:func:`PyType_HasFeature` takes a type and a flags value, *tp* and *f*, and checks whether ``tp->tp_flags & f`` is non-zero. @@ -438,20 +438,20 @@ type objects) *must* have the :attr:`ob_size` field. .. data:: Py_TPFLAGS_READY This bit is set when the type object has been fully initialized by - :cfunc:`PyType_Ready`. + :c:func:`PyType_Ready`. .. data:: Py_TPFLAGS_READYING - This bit is set while :cfunc:`PyType_Ready` is in the process of initializing + This bit is set while :c:func:`PyType_Ready` is in the process of initializing the type object. .. data:: Py_TPFLAGS_HAVE_GC This bit is set when the object supports garbage collection. If this bit - is set, instances must be created using :cfunc:`PyObject_GC_New` and - destroyed using :cfunc:`PyObject_GC_Del`. More information in section + is set, instances must be created using :c:func:`PyObject_GC_New` and + destroyed using :c:func:`PyObject_GC_Del`. More information in section :ref:`supporting-cycle-detection`. This bit also implies that the GC-related fields :attr:`tp_traverse` and :attr:`tp_clear` are present in the type object. @@ -465,7 +465,7 @@ type objects) *must* have the :attr:`ob_size` field. :const:`Py_TPFLAGS_HAVE_VERSION_TAG`. -.. cmember:: char* PyTypeObject.tp_doc +.. c:member:: char* PyTypeObject.tp_doc An optional pointer to a NUL-terminated C string giving the docstring for this type object. This is exposed as the :attr:`__doc__` attribute on the type and @@ -474,7 +474,7 @@ type objects) *must* have the :attr:`ob_size` field. This field is *not* inherited by subtypes. -.. cmember:: traverseproc PyTypeObject.tp_traverse +.. c:member:: traverseproc PyTypeObject.tp_traverse An optional pointer to a traversal function for the garbage collector. This is only used if the :const:`Py_TPFLAGS_HAVE_GC` flag bit is set. More information @@ -483,8 +483,8 @@ type objects) *must* have the :attr:`ob_size` field. The :attr:`tp_traverse` pointer is used by the garbage collector to detect reference cycles. A typical implementation of a :attr:`tp_traverse` function - simply calls :cfunc:`Py_VISIT` on each of the instance's members that are Python - objects. For example, this is function :cfunc:`local_traverse` from the + simply calls :c:func:`Py_VISIT` on each of the instance's members that are Python + objects. For example, this is function :c:func:`local_traverse` from the :mod:`_thread` extension module:: static int @@ -496,7 +496,7 @@ type objects) *must* have the :attr:`ob_size` field. return 0; } - Note that :cfunc:`Py_VISIT` is called only on those members that can participate + Note that :c:func:`Py_VISIT` is called only on those members that can participate in reference cycles. Although there is also a ``self->key`` member, it can only be *NULL* or a Python string and therefore cannot be part of a reference cycle. @@ -504,8 +504,8 @@ type objects) *must* have the :attr:`ob_size` field. debugging aid you may want to visit it anyway just so the :mod:`gc` module's :func:`get_referents` function will include it. - Note that :cfunc:`Py_VISIT` requires the *visit* and *arg* parameters to - :cfunc:`local_traverse` to have these specific names; don't name them just + Note that :c:func:`Py_VISIT` requires the *visit* and *arg* parameters to + :c:func:`local_traverse` to have these specific names; don't name them just anything. This field is inherited by subtypes together with :attr:`tp_clear` and the @@ -514,7 +514,7 @@ type objects) *must* have the :attr:`ob_size` field. the subtype. -.. cmember:: inquiry PyTypeObject.tp_clear +.. c:member:: inquiry PyTypeObject.tp_clear An optional pointer to a clear function for the garbage collector. This is only used if the :const:`Py_TPFLAGS_HAVE_GC` flag bit is set. @@ -543,7 +543,7 @@ type objects) *must* have the :attr:`ob_size` field. return 0; } - The :cfunc:`Py_CLEAR` macro should be used, because clearing references is + The :c:func:`Py_CLEAR` macro should be used, because clearing references is delicate: the reference to the contained object must not be decremented until after the pointer to the contained object is set to *NULL*. This is because decrementing the reference count may cause the contained object to become trash, @@ -552,7 +552,7 @@ type objects) *must* have the :attr:`ob_size` field. contained object). If it's possible for such code to reference *self* again, it's important that the pointer to the contained object be *NULL* at that time, so that *self* knows the contained object can no longer be used. The - :cfunc:`Py_CLEAR` macro performs the operations in a safe order. + :c:func:`Py_CLEAR` macro performs the operations in a safe order. Because the goal of :attr:`tp_clear` functions is to break reference cycles, it's not necessary to clear contained objects like Python strings or Python @@ -569,7 +569,7 @@ type objects) *must* have the :attr:`ob_size` field. the subtype. -.. cmember:: richcmpfunc PyTypeObject.tp_richcompare +.. c:member:: richcmpfunc PyTypeObject.tp_richcompare An optional pointer to the rich comparison function, whose signature is ``PyObject *tp_richcompare(PyObject *a, PyObject *b, int op)``. @@ -591,7 +591,7 @@ type objects) *must* have the :attr:`ob_size` field. *NULL*. The following constants are defined to be used as the third argument for - :attr:`tp_richcompare` and for :cfunc:`PyObject_RichCompare`: + :attr:`tp_richcompare` and for :c:func:`PyObject_RichCompare`: +----------------+------------+ | Constant | Comparison | @@ -610,13 +610,13 @@ type objects) *must* have the :attr:`ob_size` field. +----------------+------------+ -.. cmember:: long PyTypeObject.tp_weaklistoffset +.. c:member:: long PyTypeObject.tp_weaklistoffset If the instances of this type are weakly referenceable, this field is greater than zero and contains the offset in the instance structure of the weak reference list head (ignoring the GC header, if present); this offset is used by - :cfunc:`PyObject_ClearWeakRefs` and the :cfunc:`PyWeakref_\*` functions. The - instance structure needs to include a field of type :ctype:`PyObject\*` which is + :c:func:`PyObject_ClearWeakRefs` and the :c:func:`PyWeakref_\*` functions. The + instance structure needs to include a field of type :c:type:`PyObject\*` which is initialized to *NULL*. Do not confuse this field with :attr:`tp_weaklist`; that is the list head for @@ -641,18 +641,18 @@ type objects) *must* have the :attr:`ob_size` field. :attr:`__weakref__`, the type inherits its :attr:`tp_weaklistoffset` from its base type. -.. cmember:: getiterfunc PyTypeObject.tp_iter +.. c:member:: getiterfunc PyTypeObject.tp_iter An optional pointer to a function that returns an iterator for the object. Its presence normally signals that the instances of this type are iterable (although sequences may be iterable without this function). - This function has the same signature as :cfunc:`PyObject_GetIter`. + This function has the same signature as :c:func:`PyObject_GetIter`. This field is inherited by subtypes. -.. cmember:: iternextfunc PyTypeObject.tp_iternext +.. c:member:: iternextfunc PyTypeObject.tp_iternext An optional pointer to a function that returns the next item in an iterator. When the iterator is exhausted, it must return *NULL*; a :exc:`StopIteration` @@ -664,14 +664,14 @@ type objects) *must* have the :attr:`ob_size` field. function should return the iterator instance itself (not a new iterator instance). - This function has the same signature as :cfunc:`PyIter_Next`. + This function has the same signature as :c:func:`PyIter_Next`. This field is inherited by subtypes. -.. cmember:: struct PyMethodDef* PyTypeObject.tp_methods +.. c:member:: struct PyMethodDef* PyTypeObject.tp_methods - An optional pointer to a static *NULL*-terminated array of :ctype:`PyMethodDef` + An optional pointer to a static *NULL*-terminated array of :c:type:`PyMethodDef` structures, declaring regular methods of this type. For each entry in the array, an entry is added to the type's dictionary (see @@ -681,9 +681,9 @@ type objects) *must* have the :attr:`ob_size` field. different mechanism). -.. cmember:: struct PyMemberDef* PyTypeObject.tp_members +.. c:member:: struct PyMemberDef* PyTypeObject.tp_members - An optional pointer to a static *NULL*-terminated array of :ctype:`PyMemberDef` + An optional pointer to a static *NULL*-terminated array of :c:type:`PyMemberDef` structures, declaring regular data members (fields or slots) of instances of this type. @@ -694,9 +694,9 @@ type objects) *must* have the :attr:`ob_size` field. different mechanism). -.. cmember:: struct PyGetSetDef* PyTypeObject.tp_getset +.. c:member:: struct PyGetSetDef* PyTypeObject.tp_getset - An optional pointer to a static *NULL*-terminated array of :ctype:`PyGetSetDef` + An optional pointer to a static *NULL*-terminated array of :c:type:`PyGetSetDef` structures, declaring computed attributes of instances of this type. For each entry in the array, an entry is added to the type's dictionary (see @@ -721,7 +721,7 @@ type objects) *must* have the :attr:`ob_size` field. } PyGetSetDef; -.. cmember:: PyTypeObject* PyTypeObject.tp_base +.. c:member:: PyTypeObject* PyTypeObject.tp_base An optional pointer to a base type from which type properties are inherited. At this level, only single inheritance is supported; multiple inheritance require @@ -732,21 +732,26 @@ type objects) *must* have the :attr:`ob_size` field. :class:`object`). -.. cmember:: PyObject* PyTypeObject.tp_dict +.. c:member:: PyObject* PyTypeObject.tp_dict - The type's dictionary is stored here by :cfunc:`PyType_Ready`. + The type's dictionary is stored here by :c:func:`PyType_Ready`. This field should normally be initialized to *NULL* before PyType_Ready is called; it may also be initialized to a dictionary containing initial attributes - for the type. Once :cfunc:`PyType_Ready` has initialized the type, extra + for the type. Once :c:func:`PyType_Ready` has initialized the type, extra attributes for the type may be added to this dictionary only if they don't correspond to overloaded operations (like :meth:`__add__`). This field is not inherited by subtypes (though the attributes defined in here are inherited through a different mechanism). + .. warning:: -.. cmember:: descrgetfunc PyTypeObject.tp_descr_get + It is not safe to use :c:func:`PyDict_SetItem` on or otherwise modify + :attr:`tp_dict` with the dictionary C-API. + + +.. c:member:: descrgetfunc PyTypeObject.tp_descr_get An optional pointer to a "descriptor get" function. @@ -759,7 +764,7 @@ type objects) *must* have the :attr:`ob_size` field. This field is inherited by subtypes. -.. cmember:: descrsetfunc PyTypeObject.tp_descr_set +.. c:member:: descrsetfunc PyTypeObject.tp_descr_set An optional pointer to a "descriptor set" function. @@ -772,12 +777,12 @@ type objects) *must* have the :attr:`ob_size` field. .. XXX explain. -.. cmember:: long PyTypeObject.tp_dictoffset +.. c:member:: long PyTypeObject.tp_dictoffset If the instances of this type have a dictionary containing instance variables, this field is non-zero and contains the offset in the instances of the type of the instance variable dictionary; this offset is used by - :cfunc:`PyObject_GenericGetAttr`. + :c:func:`PyObject_GenericGetAttr`. Do not confuse this field with :attr:`tp_dict`; that is the dictionary for attributes of the type object itself. @@ -805,7 +810,7 @@ type objects) *must* have the :attr:`ob_size` field. taken from the type object, and :attr:`ob_size` is taken from the instance. The absolute value is taken because ints use the sign of :attr:`ob_size` to store the sign of the number. (There's never a need to do this calculation - yourself; it is done for you by :cfunc:`_PyObject_GetDictPtr`.) + yourself; it is done for you by :c:func:`_PyObject_GetDictPtr`.) This field is inherited by subtypes, but see the rules listed below. A subtype may override this offset; this means that the subtype instances store the @@ -825,7 +830,7 @@ type objects) *must* have the :attr:`ob_size` field. added as a feature just like :attr:`__weakref__` though.) -.. cmember:: initproc PyTypeObject.tp_init +.. c:member:: initproc PyTypeObject.tp_init An optional pointer to an instance initialization function. @@ -852,7 +857,7 @@ type objects) *must* have the :attr:`ob_size` field. This field is inherited by subtypes. -.. cmember:: allocfunc PyTypeObject.tp_alloc +.. c:member:: allocfunc PyTypeObject.tp_alloc An optional pointer to an instance allocation function. @@ -875,11 +880,11 @@ type objects) *must* have the :attr:`ob_size` field. This field is inherited by static subtypes, but not by dynamic subtypes (subtypes created by a class statement); in the latter, this field is always set - to :cfunc:`PyType_GenericAlloc`, to force a standard heap allocation strategy. + to :c:func:`PyType_GenericAlloc`, to force a standard heap allocation strategy. That is also the recommended value for statically defined types. -.. cmember:: newfunc PyTypeObject.tp_new +.. c:member:: newfunc PyTypeObject.tp_new An optional pointer to an instance creation function. @@ -909,22 +914,22 @@ type objects) *must* have the :attr:`ob_size` field. whose :attr:`tp_base` is *NULL* or ``&PyBaseObject_Type``. -.. cmember:: destructor PyTypeObject.tp_free +.. c:member:: destructor PyTypeObject.tp_free An optional pointer to an instance deallocation function. Its signature is - :ctype:`freefunc`:: + :c:type:`freefunc`:: void tp_free(void *) - An initializer that is compatible with this signature is :cfunc:`PyObject_Free`. + An initializer that is compatible with this signature is :c:func:`PyObject_Free`. This field is inherited by static subtypes, but not by dynamic subtypes (subtypes created by a class statement); in the latter, this field is set to a - deallocator suitable to match :cfunc:`PyType_GenericAlloc` and the value of the + deallocator suitable to match :c:func:`PyType_GenericAlloc` and the value of the :const:`Py_TPFLAGS_HAVE_GC` flag bit. -.. cmember:: inquiry PyTypeObject.tp_is_gc +.. c:member:: inquiry PyTypeObject.tp_is_gc An optional pointer to a function called by the garbage collector. @@ -939,13 +944,13 @@ type objects) *must* have the :attr:`ob_size` field. int tp_is_gc(PyObject *self) (The only example of this are types themselves. The metatype, - :cdata:`PyType_Type`, defines this function to distinguish between statically + :c:data:`PyType_Type`, defines this function to distinguish between statically and dynamically allocated types.) This field is inherited by subtypes. -.. cmember:: PyObject* PyTypeObject.tp_bases +.. c:member:: PyObject* PyTypeObject.tp_bases Tuple of base types. @@ -955,25 +960,25 @@ type objects) *must* have the :attr:`ob_size` field. This field is not inherited. -.. cmember:: PyObject* PyTypeObject.tp_mro +.. c:member:: PyObject* PyTypeObject.tp_mro Tuple containing the expanded set of base types, starting with the type itself and ending with :class:`object`, in Method Resolution Order. - This field is not inherited; it is calculated fresh by :cfunc:`PyType_Ready`. + This field is not inherited; it is calculated fresh by :c:func:`PyType_Ready`. -.. cmember:: PyObject* PyTypeObject.tp_cache +.. c:member:: PyObject* PyTypeObject.tp_cache Unused. Not inherited. Internal use only. -.. cmember:: PyObject* PyTypeObject.tp_subclasses +.. c:member:: PyObject* PyTypeObject.tp_subclasses List of weak references to subclasses. Not inherited. Internal use only. -.. cmember:: PyObject* PyTypeObject.tp_weaklist +.. c:member:: PyObject* PyTypeObject.tp_weaklist Weak reference list head, for weak references to this type object. Not inherited. Internal use only. @@ -984,22 +989,22 @@ documented here for completeness. None of these fields are inherited by subtypes. -.. cmember:: Py_ssize_t PyTypeObject.tp_allocs +.. c:member:: Py_ssize_t PyTypeObject.tp_allocs Number of allocations. -.. cmember:: Py_ssize_t PyTypeObject.tp_frees +.. c:member:: Py_ssize_t PyTypeObject.tp_frees Number of frees. -.. cmember:: Py_ssize_t PyTypeObject.tp_maxalloc +.. c:member:: Py_ssize_t PyTypeObject.tp_maxalloc Maximum simultaneously allocated objects. -.. cmember:: PyTypeObject* PyTypeObject.tp_next +.. c:member:: PyTypeObject* PyTypeObject.tp_next Pointer to the next type object with a non-zero :attr:`tp_allocs` field. @@ -1022,7 +1027,7 @@ Number Object Structures .. sectionauthor:: Amaury Forgeot d'Arc -.. ctype:: PyNumberMethods +.. c:type:: PyNumberMethods This structure holds pointers to the functions which an object uses to implement the number protocol. Each function is used by the function of @@ -1081,8 +1086,8 @@ Number Object Structures .. note:: - The :cdata:`nb_reserved` field should always be ``NULL``. It - was previously called :cdata:`nb_long`, and was renamed in + The :c:data:`nb_reserved` field should always be ``NULL``. It + was previously called :c:data:`nb_long`, and was renamed in Python 3.0.1. @@ -1094,26 +1099,26 @@ Mapping Object Structures .. sectionauthor:: Amaury Forgeot d'Arc -.. ctype:: PyMappingMethods +.. c:type:: PyMappingMethods This structure holds pointers to the functions which an object uses to implement the mapping protocol. It has three members: -.. cmember:: lenfunc PyMappingMethods.mp_length +.. c:member:: lenfunc PyMappingMethods.mp_length - This function is used by :cfunc:`PyMapping_Length` and - :cfunc:`PyObject_Size`, and has the same signature. This slot may be set to + This function is used by :c:func:`PyMapping_Length` and + :c:func:`PyObject_Size`, and has the same signature. This slot may be set to *NULL* if the object has no defined length. -.. cmember:: binaryfunc PyMappingMethods.mp_subscript +.. c:member:: binaryfunc PyMappingMethods.mp_subscript - This function is used by :cfunc:`PyObject_GetItem` and has the same - signature. This slot must be filled for the :cfunc:`PyMapping_Check` + This function is used by :c:func:`PyObject_GetItem` and has the same + signature. This slot must be filled for the :c:func:`PyMapping_Check` function to return ``1``, it can be *NULL* otherwise. -.. cmember:: objobjargproc PyMappingMethods.mp_ass_subscript +.. c:member:: objobjargproc PyMappingMethods.mp_ass_subscript - This function is used by :cfunc:`PyObject_SetItem` and has the same + This function is used by :c:func:`PyObject_SetItem` and has the same signature. If this slot is *NULL*, the object does not support item assignment. @@ -1126,32 +1131,32 @@ Sequence Object Structures .. sectionauthor:: Amaury Forgeot d'Arc -.. ctype:: PySequenceMethods +.. c:type:: PySequenceMethods This structure holds pointers to the functions which an object uses to implement the sequence protocol. -.. cmember:: lenfunc PySequenceMethods.sq_length +.. c:member:: lenfunc PySequenceMethods.sq_length - This function is used by :cfunc:`PySequence_Size` and :cfunc:`PyObject_Size`, + This function is used by :c:func:`PySequence_Size` and :c:func:`PyObject_Size`, and has the same signature. -.. cmember:: binaryfunc PySequenceMethods.sq_concat +.. c:member:: binaryfunc PySequenceMethods.sq_concat - This function is used by :cfunc:`PySequence_Concat` and has the same + This function is used by :c:func:`PySequence_Concat` and has the same signature. It is also used by the ``+`` operator, after trying the numeric addition via the :attr:`tp_as_number.nb_add` slot. -.. cmember:: ssizeargfunc PySequenceMethods.sq_repeat +.. c:member:: ssizeargfunc PySequenceMethods.sq_repeat - This function is used by :cfunc:`PySequence_Repeat` and has the same + This function is used by :c:func:`PySequence_Repeat` and has the same signature. It is also used by the ``*`` operator, after trying numeric multiplication via the :attr:`tp_as_number.nb_mul` slot. -.. cmember:: ssizeargfunc PySequenceMethods.sq_item +.. c:member:: ssizeargfunc PySequenceMethods.sq_item - This function is used by :cfunc:`PySequence_GetItem` and has the same - signature. This slot must be filled for the :cfunc:`PySequence_Check` + This function is used by :c:func:`PySequence_GetItem` and has the same + signature. This slot must be filled for the :c:func:`PySequence_Check` function to return ``1``, it can be *NULL* otherwise. Negative indexes are handled as follows: if the :attr:`sq_length` slot is @@ -1159,27 +1164,27 @@ Sequence Object Structures index which is passed to :attr:`sq_item`. If :attr:`sq_length` is *NULL*, the index is passed as is to the function. -.. cmember:: ssizeobjargproc PySequenceMethods.sq_ass_item +.. c:member:: ssizeobjargproc PySequenceMethods.sq_ass_item - This function is used by :cfunc:`PySequence_SetItem` and has the same + This function is used by :c:func:`PySequence_SetItem` and has the same signature. This slot may be left to *NULL* if the object does not support item assignment. -.. cmember:: objobjproc PySequenceMethods.sq_contains +.. c:member:: objobjproc PySequenceMethods.sq_contains - This function may be used by :cfunc:`PySequence_Contains` and has the same + This function may be used by :c:func:`PySequence_Contains` and has the same signature. This slot may be left to *NULL*, in this case - :cfunc:`PySequence_Contains` simply traverses the sequence until it finds a + :c:func:`PySequence_Contains` simply traverses the sequence until it finds a match. -.. cmember:: binaryfunc PySequenceMethods.sq_inplace_concat +.. c:member:: binaryfunc PySequenceMethods.sq_inplace_concat - This function is used by :cfunc:`PySequence_InPlaceConcat` and has the same + This function is used by :c:func:`PySequence_InPlaceConcat` and has the same signature. It should modify its first operand, and return it. -.. cmember:: ssizeargfunc PySequenceMethods.sq_inplace_repeat +.. c:member:: ssizeargfunc PySequenceMethods.sq_inplace_repeat - This function is used by :cfunc:`PySequence_InPlaceRepeat` and has the same + This function is used by :c:func:`PySequence_InPlaceRepeat` and has the same signature. It should modify its first operand, and return it. .. XXX need to explain precedence between mapping and sequence @@ -1199,40 +1204,40 @@ The :ref:`buffer interface <bufferobjects>` exports a model where an object can data. If an object does not export the buffer interface, then its :attr:`tp_as_buffer` -member in the :ctype:`PyTypeObject` structure should be *NULL*. Otherwise, the -:attr:`tp_as_buffer` will point to a :ctype:`PyBufferProcs` structure. +member in the :c:type:`PyTypeObject` structure should be *NULL*. Otherwise, the +:attr:`tp_as_buffer` will point to a :c:type:`PyBufferProcs` structure. -.. ctype:: PyBufferProcs +.. c:type:: PyBufferProcs Structure used to hold the function pointers which define an implementation of the buffer protocol. - .. cmember:: getbufferproc bf_getbuffer + .. c:member:: getbufferproc bf_getbuffer - This should fill a :ctype:`Py_buffer` with the necessary data for + This should fill a :c:type:`Py_buffer` with the necessary data for exporting the type. The signature of :data:`getbufferproc` is ``int (PyObject *obj, Py_buffer *view, int flags)``. *obj* is the object to - export, *view* is the :ctype:`Py_buffer` struct to fill, and *flags* gives + export, *view* is the :c:type:`Py_buffer` struct to fill, and *flags* gives the conditions the caller wants the memory under. (See - :cfunc:`PyObject_GetBuffer` for all flags.) :cmember:`bf_getbuffer` is + :c:func:`PyObject_GetBuffer` for all flags.) :c:member:`bf_getbuffer` is responsible for filling *view* with the appropriate information. - (:cfunc:`PyBuffer_FillView` can be used in simple cases.) See - :ctype:`Py_buffer`\s docs for what needs to be filled in. + (:c:func:`PyBuffer_FillView` can be used in simple cases.) See + :c:type:`Py_buffer`\s docs for what needs to be filled in. - .. cmember:: releasebufferproc bf_releasebuffer + .. c:member:: releasebufferproc bf_releasebuffer This should release the resources of the buffer. The signature of - :cdata:`releasebufferproc` is ``void (PyObject *obj, Py_buffer *view)``. - If the :cdata:`bf_releasebuffer` function is not provided (i.e. it is + :c:data:`releasebufferproc` is ``void (PyObject *obj, Py_buffer *view)``. + If the :c:data:`bf_releasebuffer` function is not provided (i.e. it is *NULL*), then it does not ever need to be called. The exporter of the buffer interface must make sure that any memory - pointed to in the :ctype:`Py_buffer` structure remains valid until + pointed to in the :c:type:`Py_buffer` structure remains valid until releasebuffer is called. Exporters will need to define a - :cdata:`bf_releasebuffer` function if they can re-allocate their memory, + :c:data:`bf_releasebuffer` function if they can re-allocate their memory, strides, shape, suboffsets, or format variables which they might share through the struct bufferinfo. - See :cfunc:`PyBuffer_Release`. + See :c:func:`PyBuffer_Release`. diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index a91f258e58..35006547c6 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -17,75 +17,75 @@ These are the basic Unicode object types used for the Unicode implementation in Python: -.. ctype:: Py_UNICODE +.. c:type:: Py_UNICODE This type represents the storage type which is used by Python internally as basis for holding Unicode ordinals. Python's default builds use a 16-bit type - for :ctype:`Py_UNICODE` and store Unicode values internally as UCS2. It is also + for :c:type:`Py_UNICODE` and store Unicode values internally as UCS2. It is also possible to build a UCS4 version of Python (most recent Linux distributions come with UCS4 builds of Python). These builds then use a 32-bit type for - :ctype:`Py_UNICODE` and store Unicode data internally as UCS4. On platforms - where :ctype:`wchar_t` is available and compatible with the chosen Python - Unicode build variant, :ctype:`Py_UNICODE` is a typedef alias for - :ctype:`wchar_t` to enhance native platform compatibility. On all other - platforms, :ctype:`Py_UNICODE` is a typedef alias for either :ctype:`unsigned - short` (UCS2) or :ctype:`unsigned long` (UCS4). + :c:type:`Py_UNICODE` and store Unicode data internally as UCS4. On platforms + where :c:type:`wchar_t` is available and compatible with the chosen Python + Unicode build variant, :c:type:`Py_UNICODE` is a typedef alias for + :c:type:`wchar_t` to enhance native platform compatibility. On all other + platforms, :c:type:`Py_UNICODE` is a typedef alias for either :c:type:`unsigned + short` (UCS2) or :c:type:`unsigned long` (UCS4). Note that UCS2 and UCS4 Python builds are not binary compatible. Please keep this in mind when writing extensions or interfaces. -.. ctype:: PyUnicodeObject +.. c:type:: PyUnicodeObject - This subtype of :ctype:`PyObject` represents a Python Unicode object. + This subtype of :c:type:`PyObject` represents a Python Unicode object. -.. cvar:: PyTypeObject PyUnicode_Type +.. c:var:: PyTypeObject PyUnicode_Type - This instance of :ctype:`PyTypeObject` represents the Python Unicode type. It + This instance of :c:type:`PyTypeObject` represents the Python Unicode type. It is exposed to Python code as ``str``. The following APIs are really C macros and can be used to do fast checks and to access internal read-only data of Unicode objects: -.. cfunction:: int PyUnicode_Check(PyObject *o) +.. c:function:: int PyUnicode_Check(PyObject *o) Return true if the object *o* is a Unicode object or an instance of a Unicode subtype. -.. cfunction:: int PyUnicode_CheckExact(PyObject *o) +.. c:function:: int PyUnicode_CheckExact(PyObject *o) Return true if the object *o* is a Unicode object, but not an instance of a subtype. -.. cfunction:: Py_ssize_t PyUnicode_GET_SIZE(PyObject *o) +.. c:function:: Py_ssize_t PyUnicode_GET_SIZE(PyObject *o) - Return the size of the object. *o* has to be a :ctype:`PyUnicodeObject` (not + Return the size of the object. *o* has to be a :c:type:`PyUnicodeObject` (not checked). -.. cfunction:: Py_ssize_t PyUnicode_GET_DATA_SIZE(PyObject *o) +.. c:function:: Py_ssize_t PyUnicode_GET_DATA_SIZE(PyObject *o) Return the size of the object's internal buffer in bytes. *o* has to be a - :ctype:`PyUnicodeObject` (not checked). + :c:type:`PyUnicodeObject` (not checked). -.. cfunction:: Py_UNICODE* PyUnicode_AS_UNICODE(PyObject *o) +.. c:function:: Py_UNICODE* PyUnicode_AS_UNICODE(PyObject *o) - Return a pointer to the internal :ctype:`Py_UNICODE` buffer of the object. *o* - has to be a :ctype:`PyUnicodeObject` (not checked). + Return a pointer to the internal :c:type:`Py_UNICODE` buffer of the object. *o* + has to be a :c:type:`PyUnicodeObject` (not checked). -.. cfunction:: const char* PyUnicode_AS_DATA(PyObject *o) +.. c:function:: const char* PyUnicode_AS_DATA(PyObject *o) Return a pointer to the internal buffer of the object. *o* has to be a - :ctype:`PyUnicodeObject` (not checked). + :c:type:`PyUnicodeObject` (not checked). -.. cfunction:: int PyUnicode_ClearFreeList() +.. c:function:: int PyUnicode_ClearFreeList() Clear the free list. Return the total number of freed items. @@ -98,57 +98,57 @@ are available through these macros which are mapped to C functions depending on the Python configuration. -.. cfunction:: int Py_UNICODE_ISSPACE(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_ISSPACE(Py_UNICODE ch) Return 1 or 0 depending on whether *ch* is a whitespace character. -.. cfunction:: int Py_UNICODE_ISLOWER(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_ISLOWER(Py_UNICODE ch) Return 1 or 0 depending on whether *ch* is a lowercase character. -.. cfunction:: int Py_UNICODE_ISUPPER(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_ISUPPER(Py_UNICODE ch) Return 1 or 0 depending on whether *ch* is an uppercase character. -.. cfunction:: int Py_UNICODE_ISTITLE(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_ISTITLE(Py_UNICODE ch) Return 1 or 0 depending on whether *ch* is a titlecase character. -.. cfunction:: int Py_UNICODE_ISLINEBREAK(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_ISLINEBREAK(Py_UNICODE ch) Return 1 or 0 depending on whether *ch* is a linebreak character. -.. cfunction:: int Py_UNICODE_ISDECIMAL(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_ISDECIMAL(Py_UNICODE ch) Return 1 or 0 depending on whether *ch* is a decimal character. -.. cfunction:: int Py_UNICODE_ISDIGIT(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_ISDIGIT(Py_UNICODE ch) Return 1 or 0 depending on whether *ch* is a digit character. -.. cfunction:: int Py_UNICODE_ISNUMERIC(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_ISNUMERIC(Py_UNICODE ch) Return 1 or 0 depending on whether *ch* is a numeric character. -.. cfunction:: int Py_UNICODE_ISALPHA(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_ISALPHA(Py_UNICODE ch) Return 1 or 0 depending on whether *ch* is an alphabetic character. -.. cfunction:: int Py_UNICODE_ISALNUM(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_ISALNUM(Py_UNICODE ch) Return 1 or 0 depending on whether *ch* is an alphanumeric character. -.. cfunction:: int Py_UNICODE_ISPRINTABLE(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_ISPRINTABLE(Py_UNICODE ch) Return 1 or 0 depending on whether *ch* is a printable character. Nonprintable characters are those characters defined in the Unicode character @@ -162,34 +162,34 @@ the Python configuration. These APIs can be used for fast direct character conversions: -.. cfunction:: Py_UNICODE Py_UNICODE_TOLOWER(Py_UNICODE ch) +.. c:function:: Py_UNICODE Py_UNICODE_TOLOWER(Py_UNICODE ch) Return the character *ch* converted to lower case. -.. cfunction:: Py_UNICODE Py_UNICODE_TOUPPER(Py_UNICODE ch) +.. c:function:: Py_UNICODE Py_UNICODE_TOUPPER(Py_UNICODE ch) Return the character *ch* converted to upper case. -.. cfunction:: Py_UNICODE Py_UNICODE_TOTITLE(Py_UNICODE ch) +.. c:function:: Py_UNICODE Py_UNICODE_TOTITLE(Py_UNICODE ch) Return the character *ch* converted to title case. -.. cfunction:: int Py_UNICODE_TODECIMAL(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_TODECIMAL(Py_UNICODE ch) Return the character *ch* converted to a decimal positive integer. Return ``-1`` if this is not possible. This macro does not raise exceptions. -.. cfunction:: int Py_UNICODE_TODIGIT(Py_UNICODE ch) +.. c:function:: int Py_UNICODE_TODIGIT(Py_UNICODE ch) Return the character *ch* converted to a single digit integer. Return ``-1`` if this is not possible. This macro does not raise exceptions. -.. cfunction:: double Py_UNICODE_TONUMERIC(Py_UNICODE ch) +.. c:function:: double Py_UNICODE_TONUMERIC(Py_UNICODE ch) Return the character *ch* converted to a double. Return ``-1.0`` if this is not possible. This macro does not raise exceptions. @@ -202,7 +202,7 @@ To create Unicode objects and access their basic sequence properties, use these APIs: -.. cfunction:: PyObject* PyUnicode_FromUnicode(const Py_UNICODE *u, Py_ssize_t size) +.. c:function:: PyObject* PyUnicode_FromUnicode(const Py_UNICODE *u, Py_ssize_t size) Create a Unicode object from the Py_UNICODE buffer *u* of the given size. *u* may be *NULL* which causes the contents to be undefined. It is the user's @@ -212,7 +212,7 @@ APIs: is *NULL*. -.. cfunction:: PyObject* PyUnicode_FromStringAndSize(const char *u, Py_ssize_t size) +.. c:function:: PyObject* PyUnicode_FromStringAndSize(const char *u, Py_ssize_t size) Create a Unicode object from the char buffer *u*. The bytes will be interpreted as being UTF-8 encoded. *u* may also be *NULL* which @@ -222,23 +222,26 @@ APIs: the resulting Unicode object is only allowed when *u* is *NULL*. -.. cfunction:: PyObject *PyUnicode_FromString(const char *u) +.. c:function:: PyObject *PyUnicode_FromString(const char *u) Create a Unicode object from an UTF-8 encoded null-terminated char buffer *u*. -.. cfunction:: PyObject* PyUnicode_FromFormat(const char *format, ...) +.. c:function:: PyObject* PyUnicode_FromFormat(const char *format, ...) - Take a C :cfunc:`printf`\ -style *format* string and a variable number of + Take a C :c:func:`printf`\ -style *format* string and a variable number of arguments, calculate the size of the resulting Python unicode string and return a string with the values formatted into it. The variable arguments must be C types and must correspond exactly to the format characters in the *format* - string. The following format characters are allowed: + ASCII-encoded string. The following format characters are allowed: + .. % This should be exactly the same as the table in PyErr_Format. .. % The descriptions for %zd and %zu are wrong, but the truth is complicated .. % because not all compilers support the %z width modifier -- we fake it .. % when necessary via interpolating PY_FORMAT_SIZE_T. + .. % Similar comments apply to the %ll width modifier and + .. % PY_FORMAT_LONG_LONG. +-------------------+---------------------+--------------------------------+ | Format Characters | Type | Comment | @@ -260,6 +263,12 @@ APIs: | :attr:`%lu` | unsigned long | Exactly equivalent to | | | | ``printf("%lu")``. | +-------------------+---------------------+--------------------------------+ + | :attr:`%lld` | long long | Exactly equivalent to | + | | | ``printf("%lld")``. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%llu` | unsigned long long | Exactly equivalent to | + | | | ``printf("%llu")``. | + +-------------------+---------------------+--------------------------------+ | :attr:`%zd` | Py_ssize_t | Exactly equivalent to | | | | ``printf("%zd")``. | +-------------------+---------------------+--------------------------------+ @@ -296,34 +305,64 @@ APIs: | | | *NULL*). | +-------------------+---------------------+--------------------------------+ | :attr:`%S` | PyObject\* | The result of calling | - | | | :func:`PyObject_Str`. | + | | | :c:func:`PyObject_Str`. | +-------------------+---------------------+--------------------------------+ | :attr:`%R` | PyObject\* | The result of calling | - | | | :func:`PyObject_Repr`. | + | | | :c:func:`PyObject_Repr`. | +-------------------+---------------------+--------------------------------+ An unrecognized format character causes all the rest of the format string to be copied as-is to the result string, and any extra arguments discarded. + .. note:: + + The `"%lld"` and `"%llu"` format specifiers are only available + when :const:`HAVE_LONG_LONG` is defined. + + .. versionchanged:: 3.2 + Support for ``"%lld"`` and ``"%llu"`` added. -.. cfunction:: PyObject* PyUnicode_FromFormatV(const char *format, va_list vargs) - Identical to :func:`PyUnicode_FromFormat` except that it takes exactly two +.. c:function:: PyObject* PyUnicode_FromFormatV(const char *format, va_list vargs) + + Identical to :c:func:`PyUnicode_FromFormat` except that it takes exactly two arguments. +.. c:function:: PyObject* PyUnicode_TransformDecimalToASCII(Py_UNICODE *s, Py_ssize_t size) + + Create a Unicode object by replacing all decimal digits in + :c:type:`Py_UNICODE` buffer of the given *size* by ASCII digits 0--9 + according to their decimal value. Return *NULL* if an exception + occurs. + + +.. c:function:: Py_UNICODE* PyUnicode_AsUnicode(PyObject *unicode) -.. cfunction:: Py_UNICODE* PyUnicode_AsUnicode(PyObject *unicode) + Return a read-only pointer to the Unicode object's internal + :c:type:`Py_UNICODE` buffer, *NULL* if *unicode* is not a Unicode object. + Note that the resulting :c:type:`Py_UNICODE*` string may contain embedded + null characters, which would cause the string to be truncated when used in + most C functions. - Return a read-only pointer to the Unicode object's internal :ctype:`Py_UNICODE` - buffer, *NULL* if *unicode* is not a Unicode object. +.. c:function:: Py_UNICODE* PyUnicode_AsUnicodeCopy(PyObject *unicode) -.. cfunction:: Py_ssize_t PyUnicode_GetSize(PyObject *unicode) + Create a copy of a Unicode string ending with a nul character. Return *NULL* + and raise a :exc:`MemoryError` exception on memory allocation failure, + otherwise return a new allocated buffer (use :c:func:`PyMem_Free` to free + the buffer). Note that the resulting :c:type:`Py_UNICODE*` string may contain + embedded null characters, which would cause the string to be truncated when + used in most C functions. + + .. versionadded:: 3.2 + + +.. c:function:: Py_ssize_t PyUnicode_GetSize(PyObject *unicode) Return the length of the Unicode object. -.. cfunction:: PyObject* PyUnicode_FromEncodedObject(PyObject *obj, const char *encoding, const char *errors) +.. c:function:: PyObject* PyUnicode_FromEncodedObject(PyObject *obj, const char *encoding, const char *errors) Coerce an encoded object *obj* to an Unicode object and return a reference with incremented refcount. @@ -340,75 +379,128 @@ APIs: decref'ing the returned objects. -.. cfunction:: PyObject* PyUnicode_FromObject(PyObject *obj) +.. c:function:: PyObject* PyUnicode_FromObject(PyObject *obj) Shortcut for ``PyUnicode_FromEncodedObject(obj, NULL, "strict")`` which is used throughout the interpreter whenever coercion to Unicode is needed. -If the platform supports :ctype:`wchar_t` and provides a header file wchar.h, +If the platform supports :c:type:`wchar_t` and provides a header file wchar.h, Python can interface directly to this type using the following functions. -Support is optimized if Python's own :ctype:`Py_UNICODE` type is identical to -the system's :ctype:`wchar_t`. +Support is optimized if Python's own :c:type:`Py_UNICODE` type is identical to +the system's :c:type:`wchar_t`. File System Encoding """""""""""""""""""" To encode and decode file names and other environment strings, -:cdata:`Py_FileSystemEncoding` should be used as the encoding, and +:c:data:`Py_FileSystemEncoding` should be used as the encoding, and ``"surrogateescape"`` should be used as the error handler (:pep:`383`). To encode file names during argument parsing, the ``"O&"`` converter should be -used, passing :cfunc:`PyUnicode_FSConverter` as the conversion function: +used, passing :c:func:`PyUnicode_FSConverter` as the conversion function: -.. cfunction:: int PyUnicode_FSConverter(PyObject* obj, void* result) +.. c:function:: int PyUnicode_FSConverter(PyObject* obj, void* result) - Convert *obj* into *result*, using :cdata:`Py_FileSystemDefaultEncoding`, - and the ``"surrogateescape"`` error handler. *result* must be a - ``PyObject*``, return a :func:`bytes` object which must be released if it - is no longer used. + ParseTuple converter: encode :class:`str` objects to :class:`bytes` using + :c:func:`PyUnicode_EncodeFSDefault`; :class:`bytes` objects are output as-is. + *result* must be a :c:type:`PyBytesObject*` which must be released when it is + no longer used. .. versionadded:: 3.1 -.. cfunction:: PyObject* PyUnicode_DecodeFSDefaultAndSize(const char *s, Py_ssize_t size) +To decode file names during argument parsing, the ``"O&"`` converter should be +used, passing :c:func:`PyUnicode_FSDecoder` as the conversion function: + +.. c:function:: int PyUnicode_FSDecoder(PyObject* obj, void* result) + + ParseTuple converter: decode :class:`bytes` objects to :class:`str` using + :c:func:`PyUnicode_DecodeFSDefaultAndSize`; :class:`str` objects are output + as-is. *result* must be a :c:type:`PyUnicodeObject*` which must be released + when it is no longer used. + + .. versionadded:: 3.2 + + +.. c:function:: PyObject* PyUnicode_DecodeFSDefaultAndSize(const char *s, Py_ssize_t size) - Decode a null-terminated string using :cdata:`Py_FileSystemDefaultEncoding` - and the ``"surrogateescape"`` error handler. + Decode a string using :c:data:`Py_FileSystemDefaultEncoding` and the + ``'surrogateescape'`` error handler, or ``'strict'`` on Windows. - If :cdata:`Py_FileSystemDefaultEncoding` is not set, fall back to UTF-8. + If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the + locale encoding. - Use :func:`PyUnicode_DecodeFSDefaultAndSize` if you know the string length. + .. versionchanged:: 3.2 + Use ``'strict'`` error handler on Windows. -.. cfunction:: PyObject* PyUnicode_DecodeFSDefault(const char *s) - Decode a string using :cdata:`Py_FileSystemDefaultEncoding` and - the ``"surrogateescape"`` error handler. +.. c:function:: PyObject* PyUnicode_DecodeFSDefault(const char *s) - If :cdata:`Py_FileSystemDefaultEncoding` is not set, fall back to UTF-8. + Decode a null-terminated string using :c:data:`Py_FileSystemDefaultEncoding` + and the ``'surrogateescape'`` error handler, or ``'strict'`` on Windows. + + If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the + locale encoding. + + Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` if you know the string length. + + .. versionchanged:: 3.2 + Use ``'strict'`` error handler on Windows. + + +.. c:function:: PyObject* PyUnicode_EncodeFSDefault(PyObject *unicode) + + Encode a Unicode object to :c:data:`Py_FileSystemDefaultEncoding` with the + ``'surrogateescape'`` error handler, or ``'strict'`` on Windows, and return + :class:`bytes`. Note that the resulting :class:`bytes` object may contain + null bytes. + + If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the + locale encoding. + + .. versionadded:: 3.2 wchar_t Support """"""""""""""" -:ctype:`wchar_t` support for platforms which support it: +:c:type:`wchar_t` support for platforms which support it: -.. cfunction:: PyObject* PyUnicode_FromWideChar(const wchar_t *w, Py_ssize_t size) +.. c:function:: PyObject* PyUnicode_FromWideChar(const wchar_t *w, Py_ssize_t size) - Create a Unicode object from the :ctype:`wchar_t` buffer *w* of the given *size*. + Create a Unicode object from the :c:type:`wchar_t` buffer *w* of the given *size*. Passing -1 as the *size* indicates that the function must itself compute the length, using wcslen. Return *NULL* on failure. -.. cfunction:: Py_ssize_t PyUnicode_AsWideChar(PyUnicodeObject *unicode, wchar_t *w, Py_ssize_t size) +.. c:function:: Py_ssize_t PyUnicode_AsWideChar(PyUnicodeObject *unicode, wchar_t *w, Py_ssize_t size) - Copy the Unicode object contents into the :ctype:`wchar_t` buffer *w*. At most - *size* :ctype:`wchar_t` characters are copied (excluding a possibly trailing - 0-termination character). Return the number of :ctype:`wchar_t` characters - copied or -1 in case of an error. Note that the resulting :ctype:`wchar_t` + Copy the Unicode object contents into the :c:type:`wchar_t` buffer *w*. At most + *size* :c:type:`wchar_t` characters are copied (excluding a possibly trailing + 0-termination character). Return the number of :c:type:`wchar_t` characters + copied or -1 in case of an error. Note that the resulting :c:type:`wchar_t` string may or may not be 0-terminated. It is the responsibility of the caller - to make sure that the :ctype:`wchar_t` string is 0-terminated in case this is - required by the application. + to make sure that the :c:type:`wchar_t` string is 0-terminated in case this is + required by the application. Also, note that the :c:type:`wchar_t*` string + might contain null characters, which would cause the string to be truncated + when used with most C functions. + + +.. c:function:: wchar_t* PyUnicode_AsWideCharString(PyObject *unicode, Py_ssize_t *size) + + Convert the Unicode object to a wide character string. The output string + always ends with a nul character. If *size* is not *NULL*, write the number + of wide characters (excluding the trailing 0-termination character) into + *\*size*. + + Returns a buffer allocated by :c:func:`PyMem_Alloc` (use + :c:func:`PyMem_Free` to free it) on success. On error, returns *NULL*, + *\*size* is undefined and raises a :exc:`MemoryError`. Note that the + resulting :c:type:`wchar_t*` string might contain null characters, which + would cause the string to be truncated when used with most C functions. + + .. versionadded:: 3.2 .. _builtincodecs: @@ -425,8 +517,8 @@ constructor. Setting encoding to *NULL* causes the default encoding to be used which is ASCII. The file system calls should use -:cfunc:`PyUnicode_FSConverter` for encoding file names. This uses the -variable :cdata:`Py_FileSystemDefaultEncoding` internally. This +:c:func:`PyUnicode_FSConverter` for encoding file names. This uses the +variable :c:data:`Py_FileSystemDefaultEncoding` internally. This variable should be treated as read-only: on some systems, it will be a pointer to a static string, on others, it will change at run-time (such as when the application invokes setlocale). @@ -445,7 +537,7 @@ Generic Codecs These are the generic codec APIs: -.. cfunction:: PyObject* PyUnicode_Decode(const char *s, Py_ssize_t size, const char *encoding, const char *errors) +.. c:function:: PyObject* PyUnicode_Decode(const char *s, Py_ssize_t size, const char *encoding, const char *errors) Create a Unicode object by decoding *size* bytes of the encoded string *s*. *encoding* and *errors* have the same meaning as the parameters of the same name @@ -454,16 +546,16 @@ These are the generic codec APIs: the codec. -.. cfunction:: PyObject* PyUnicode_Encode(const Py_UNICODE *s, Py_ssize_t size, const char *encoding, const char *errors) +.. c:function:: PyObject* PyUnicode_Encode(const Py_UNICODE *s, Py_ssize_t size, const char *encoding, const char *errors) - Encode the :ctype:`Py_UNICODE` buffer *s* of the given *size* and return a Python + Encode the :c:type:`Py_UNICODE` buffer *s* of the given *size* and return a Python bytes object. *encoding* and *errors* have the same meaning as the parameters of the same name in the Unicode :meth:`encode` method. The codec to be used is looked up using the Python codec registry. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_AsEncodedString(PyObject *unicode, const char *encoding, const char *errors) +.. c:function:: PyObject* PyUnicode_AsEncodedString(PyObject *unicode, const char *encoding, const char *errors) Encode a Unicode object and return the result as Python bytes object. *encoding* and *errors* have the same meaning as the parameters of the same @@ -478,28 +570,28 @@ UTF-8 Codecs These are the UTF-8 codec APIs: -.. cfunction:: PyObject* PyUnicode_DecodeUTF8(const char *s, Py_ssize_t size, const char *errors) +.. c:function:: PyObject* PyUnicode_DecodeUTF8(const char *s, Py_ssize_t size, const char *errors) Create a Unicode object by decoding *size* bytes of the UTF-8 encoded string *s*. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_DecodeUTF8Stateful(const char *s, Py_ssize_t size, const char *errors, Py_ssize_t *consumed) +.. c:function:: PyObject* PyUnicode_DecodeUTF8Stateful(const char *s, Py_ssize_t size, const char *errors, Py_ssize_t *consumed) - If *consumed* is *NULL*, behave like :cfunc:`PyUnicode_DecodeUTF8`. If + If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF8`. If *consumed* is not *NULL*, trailing incomplete UTF-8 byte sequences will not be treated as an error. Those bytes will not be decoded and the number of bytes that have been decoded will be stored in *consumed*. -.. cfunction:: PyObject* PyUnicode_EncodeUTF8(const Py_UNICODE *s, Py_ssize_t size, const char *errors) +.. c:function:: PyObject* PyUnicode_EncodeUTF8(const Py_UNICODE *s, Py_ssize_t size, const char *errors) - Encode the :ctype:`Py_UNICODE` buffer *s* of the given *size* using UTF-8 and + Encode the :c:type:`Py_UNICODE` buffer *s* of the given *size* using UTF-8 and return a Python bytes object. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_AsUTF8String(PyObject *unicode) +.. c:function:: PyObject* PyUnicode_AsUTF8String(PyObject *unicode) Encode a Unicode object using UTF-8 and return the result as Python bytes object. Error handling is "strict". Return *NULL* if an exception was @@ -512,7 +604,7 @@ UTF-32 Codecs These are the UTF-32 codec APIs: -.. cfunction:: PyObject* PyUnicode_DecodeUTF32(const char *s, Py_ssize_t size, const char *errors, int *byteorder) +.. c:function:: PyObject* PyUnicode_DecodeUTF32(const char *s, Py_ssize_t size, const char *errors, int *byteorder) Decode *size* bytes from a UTF-32 encoded buffer string and return the corresponding Unicode object. *errors* (if non-*NULL*) defines the error @@ -540,16 +632,16 @@ These are the UTF-32 codec APIs: Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_DecodeUTF32Stateful(const char *s, Py_ssize_t size, const char *errors, int *byteorder, Py_ssize_t *consumed) +.. c:function:: PyObject* PyUnicode_DecodeUTF32Stateful(const char *s, Py_ssize_t size, const char *errors, int *byteorder, Py_ssize_t *consumed) - If *consumed* is *NULL*, behave like :cfunc:`PyUnicode_DecodeUTF32`. If - *consumed* is not *NULL*, :cfunc:`PyUnicode_DecodeUTF32Stateful` will not treat + If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF32`. If + *consumed* is not *NULL*, :c:func:`PyUnicode_DecodeUTF32Stateful` will not treat trailing incomplete UTF-32 byte sequences (such as a number of bytes not divisible by four) as an error. Those bytes will not be decoded and the number of bytes that have been decoded will be stored in *consumed*. -.. cfunction:: PyObject* PyUnicode_EncodeUTF32(const Py_UNICODE *s, Py_ssize_t size, const char *errors, int byteorder) +.. c:function:: PyObject* PyUnicode_EncodeUTF32(const Py_UNICODE *s, Py_ssize_t size, const char *errors, int byteorder) Return a Python bytes object holding the UTF-32 encoded value of the Unicode data in *s*. Output is written according to the following byte order:: @@ -567,7 +659,7 @@ These are the UTF-32 codec APIs: Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_AsUTF32String(PyObject *unicode) +.. c:function:: PyObject* PyUnicode_AsUTF32String(PyObject *unicode) Return a Python byte string using the UTF-32 encoding in native byte order. The string always starts with a BOM mark. Error handling is "strict". @@ -580,7 +672,7 @@ UTF-16 Codecs These are the UTF-16 codec APIs: -.. cfunction:: PyObject* PyUnicode_DecodeUTF16(const char *s, Py_ssize_t size, const char *errors, int *byteorder) +.. c:function:: PyObject* PyUnicode_DecodeUTF16(const char *s, Py_ssize_t size, const char *errors, int *byteorder) Decode *size* bytes from a UTF-16 encoded buffer string and return the corresponding Unicode object. *errors* (if non-*NULL*) defines the error @@ -607,16 +699,16 @@ These are the UTF-16 codec APIs: Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_DecodeUTF16Stateful(const char *s, Py_ssize_t size, const char *errors, int *byteorder, Py_ssize_t *consumed) +.. c:function:: PyObject* PyUnicode_DecodeUTF16Stateful(const char *s, Py_ssize_t size, const char *errors, int *byteorder, Py_ssize_t *consumed) - If *consumed* is *NULL*, behave like :cfunc:`PyUnicode_DecodeUTF16`. If - *consumed* is not *NULL*, :cfunc:`PyUnicode_DecodeUTF16Stateful` will not treat + If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF16`. If + *consumed* is not *NULL*, :c:func:`PyUnicode_DecodeUTF16Stateful` will not treat trailing incomplete UTF-16 byte sequences (such as an odd number of bytes or a split surrogate pair) as an error. Those bytes will not be decoded and the number of bytes that have been decoded will be stored in *consumed*. -.. cfunction:: PyObject* PyUnicode_EncodeUTF16(const Py_UNICODE *s, Py_ssize_t size, const char *errors, int byteorder) +.. c:function:: PyObject* PyUnicode_EncodeUTF16(const Py_UNICODE *s, Py_ssize_t size, const char *errors, int byteorder) Return a Python bytes object holding the UTF-16 encoded value of the Unicode data in *s*. Output is written according to the following byte order:: @@ -628,14 +720,14 @@ These are the UTF-16 codec APIs: If byteorder is ``0``, the output string will always start with the Unicode BOM mark (U+FEFF). In the other two modes, no BOM mark is prepended. - If *Py_UNICODE_WIDE* is defined, a single :ctype:`Py_UNICODE` value may get - represented as a surrogate pair. If it is not defined, each :ctype:`Py_UNICODE` + If *Py_UNICODE_WIDE* is defined, a single :c:type:`Py_UNICODE` value may get + represented as a surrogate pair. If it is not defined, each :c:type:`Py_UNICODE` values is interpreted as an UCS-2 character. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_AsUTF16String(PyObject *unicode) +.. c:function:: PyObject* PyUnicode_AsUTF16String(PyObject *unicode) Return a Python byte string using the UTF-16 encoding in native byte order. The string always starts with a BOM mark. Error handling is "strict". @@ -648,23 +740,23 @@ UTF-7 Codecs These are the UTF-7 codec APIs: -.. cfunction:: PyObject* PyUnicode_DecodeUTF7(const char *s, Py_ssize_t size, const char *errors) +.. c:function:: PyObject* PyUnicode_DecodeUTF7(const char *s, Py_ssize_t size, const char *errors) Create a Unicode object by decoding *size* bytes of the UTF-7 encoded string *s*. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_DecodeUTF7Stateful(const char *s, Py_ssize_t size, const char *errors, Py_ssize_t *consumed) +.. c:function:: PyObject* PyUnicode_DecodeUTF7Stateful(const char *s, Py_ssize_t size, const char *errors, Py_ssize_t *consumed) - If *consumed* is *NULL*, behave like :cfunc:`PyUnicode_DecodeUTF7`. If + If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF7`. If *consumed* is not *NULL*, trailing incomplete UTF-7 base-64 sections will not be treated as an error. Those bytes will not be decoded and the number of bytes that have been decoded will be stored in *consumed*. -.. cfunction:: PyObject* PyUnicode_EncodeUTF7(const Py_UNICODE *s, Py_ssize_t size, int base64SetO, int base64WhiteSpace, const char *errors) +.. c:function:: PyObject* PyUnicode_EncodeUTF7(const Py_UNICODE *s, Py_ssize_t size, int base64SetO, int base64WhiteSpace, const char *errors) - Encode the :ctype:`Py_UNICODE` buffer of the given size using UTF-7 and + Encode the :c:type:`Py_UNICODE` buffer of the given size using UTF-7 and return a Python bytes object. Return *NULL* if an exception was raised by the codec. @@ -680,20 +772,20 @@ Unicode-Escape Codecs These are the "Unicode Escape" codec APIs: -.. cfunction:: PyObject* PyUnicode_DecodeUnicodeEscape(const char *s, Py_ssize_t size, const char *errors) +.. c:function:: PyObject* PyUnicode_DecodeUnicodeEscape(const char *s, Py_ssize_t size, const char *errors) Create a Unicode object by decoding *size* bytes of the Unicode-Escape encoded string *s*. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_EncodeUnicodeEscape(const Py_UNICODE *s, Py_ssize_t size) +.. c:function:: PyObject* PyUnicode_EncodeUnicodeEscape(const Py_UNICODE *s, Py_ssize_t size) - Encode the :ctype:`Py_UNICODE` buffer of the given size using Unicode-Escape and + Encode the :c:type:`Py_UNICODE` buffer of the given *size* using Unicode-Escape and return a Python string object. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_AsUnicodeEscapeString(PyObject *unicode) +.. c:function:: PyObject* PyUnicode_AsUnicodeEscapeString(PyObject *unicode) Encode a Unicode object using Unicode-Escape and return the result as Python string object. Error handling is "strict". Return *NULL* if an exception was @@ -706,20 +798,20 @@ Raw-Unicode-Escape Codecs These are the "Raw Unicode Escape" codec APIs: -.. cfunction:: PyObject* PyUnicode_DecodeRawUnicodeEscape(const char *s, Py_ssize_t size, const char *errors) +.. c:function:: PyObject* PyUnicode_DecodeRawUnicodeEscape(const char *s, Py_ssize_t size, const char *errors) Create a Unicode object by decoding *size* bytes of the Raw-Unicode-Escape encoded string *s*. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_EncodeRawUnicodeEscape(const Py_UNICODE *s, Py_ssize_t size, const char *errors) +.. c:function:: PyObject* PyUnicode_EncodeRawUnicodeEscape(const Py_UNICODE *s, Py_ssize_t size, const char *errors) - Encode the :ctype:`Py_UNICODE` buffer of the given *size* using Raw-Unicode-Escape + Encode the :c:type:`Py_UNICODE` buffer of the given *size* using Raw-Unicode-Escape and return a Python string object. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_AsRawUnicodeEscapeString(PyObject *unicode) +.. c:function:: PyObject* PyUnicode_AsRawUnicodeEscapeString(PyObject *unicode) Encode a Unicode object using Raw-Unicode-Escape and return the result as Python string object. Error handling is "strict". Return *NULL* if an exception @@ -733,20 +825,20 @@ These are the Latin-1 codec APIs: Latin-1 corresponds to the first 256 Unicode ordinals and only these are accepted by the codecs during encoding. -.. cfunction:: PyObject* PyUnicode_DecodeLatin1(const char *s, Py_ssize_t size, const char *errors) +.. c:function:: PyObject* PyUnicode_DecodeLatin1(const char *s, Py_ssize_t size, const char *errors) Create a Unicode object by decoding *size* bytes of the Latin-1 encoded string *s*. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_EncodeLatin1(const Py_UNICODE *s, Py_ssize_t size, const char *errors) +.. c:function:: PyObject* PyUnicode_EncodeLatin1(const Py_UNICODE *s, Py_ssize_t size, const char *errors) - Encode the :ctype:`Py_UNICODE` buffer of the given *size* using Latin-1 and + Encode the :c:type:`Py_UNICODE` buffer of the given *size* using Latin-1 and return a Python bytes object. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_AsLatin1String(PyObject *unicode) +.. c:function:: PyObject* PyUnicode_AsLatin1String(PyObject *unicode) Encode a Unicode object using Latin-1 and return the result as Python bytes object. Error handling is "strict". Return *NULL* if an exception was @@ -760,20 +852,20 @@ These are the ASCII codec APIs. Only 7-bit ASCII data is accepted. All other codes generate errors. -.. cfunction:: PyObject* PyUnicode_DecodeASCII(const char *s, Py_ssize_t size, const char *errors) +.. c:function:: PyObject* PyUnicode_DecodeASCII(const char *s, Py_ssize_t size, const char *errors) Create a Unicode object by decoding *size* bytes of the ASCII encoded string *s*. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_EncodeASCII(const Py_UNICODE *s, Py_ssize_t size, const char *errors) +.. c:function:: PyObject* PyUnicode_EncodeASCII(const Py_UNICODE *s, Py_ssize_t size, const char *errors) - Encode the :ctype:`Py_UNICODE` buffer of the given *size* using ASCII and + Encode the :c:type:`Py_UNICODE` buffer of the given *size* using ASCII and return a Python bytes object. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_AsASCIIString(PyObject *unicode) +.. c:function:: PyObject* PyUnicode_AsASCIIString(PyObject *unicode) Encode a Unicode object using ASCII and return the result as Python bytes object. Error handling is "strict". Return *NULL* if an exception was @@ -806,7 +898,7 @@ characters to different code points. These are the mapping codec APIs: -.. cfunction:: PyObject* PyUnicode_DecodeCharmap(const char *s, Py_ssize_t size, PyObject *mapping, const char *errors) +.. c:function:: PyObject* PyUnicode_DecodeCharmap(const char *s, Py_ssize_t size, PyObject *mapping, const char *errors) Create a Unicode object by decoding *size* bytes of the encoded string *s* using the given *mapping* object. Return *NULL* if an exception was raised by the @@ -816,14 +908,14 @@ These are the mapping codec APIs: treated as "undefined mapping". -.. cfunction:: PyObject* PyUnicode_EncodeCharmap(const Py_UNICODE *s, Py_ssize_t size, PyObject *mapping, const char *errors) +.. c:function:: PyObject* PyUnicode_EncodeCharmap(const Py_UNICODE *s, Py_ssize_t size, PyObject *mapping, const char *errors) - Encode the :ctype:`Py_UNICODE` buffer of the given *size* using the given + Encode the :c:type:`Py_UNICODE` buffer of the given *size* using the given *mapping* object and return a Python string object. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_AsCharmapString(PyObject *unicode, PyObject *mapping) +.. c:function:: PyObject* PyUnicode_AsCharmapString(PyObject *unicode, PyObject *mapping) Encode a Unicode object using the given *mapping* object and return the result as Python string object. Error handling is "strict". Return *NULL* if an @@ -832,9 +924,9 @@ These are the mapping codec APIs: The following codec API is special in that maps Unicode to Unicode. -.. cfunction:: PyObject* PyUnicode_TranslateCharmap(const Py_UNICODE *s, Py_ssize_t size, PyObject *table, const char *errors) +.. c:function:: PyObject* PyUnicode_TranslateCharmap(const Py_UNICODE *s, Py_ssize_t size, PyObject *table, const char *errors) - Translate a :ctype:`Py_UNICODE` buffer of the given *size* by applying a + Translate a :c:type:`Py_UNICODE` buffer of the given *size* by applying a character mapping *table* to it and return the resulting Unicode object. Return *NULL* when an exception was raised by the codec. @@ -846,6 +938,7 @@ The following codec API is special in that maps Unicode to Unicode. :exc:`LookupError`) are left untouched and are copied as-is. + MBCS codecs for Windows """"""""""""""""""""""" @@ -854,29 +947,28 @@ use the Win32 MBCS converters to implement the conversions. Note that MBCS (or DBCS) is a class of encodings, not just one. The target encoding is defined by the user settings on the machine running the codec. - -.. cfunction:: PyObject* PyUnicode_DecodeMBCS(const char *s, Py_ssize_t size, const char *errors) +.. c:function:: PyObject* PyUnicode_DecodeMBCS(const char *s, Py_ssize_t size, const char *errors) Create a Unicode object by decoding *size* bytes of the MBCS encoded string *s*. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_DecodeMBCSStateful(const char *s, int size, const char *errors, int *consumed) +.. c:function:: PyObject* PyUnicode_DecodeMBCSStateful(const char *s, int size, const char *errors, int *consumed) - If *consumed* is *NULL*, behave like :cfunc:`PyUnicode_DecodeMBCS`. If - *consumed* is not *NULL*, :cfunc:`PyUnicode_DecodeMBCSStateful` will not decode + If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeMBCS`. If + *consumed* is not *NULL*, :c:func:`PyUnicode_DecodeMBCSStateful` will not decode trailing lead byte and the number of bytes that have been decoded will be stored in *consumed*. -.. cfunction:: PyObject* PyUnicode_EncodeMBCS(const Py_UNICODE *s, Py_ssize_t size, const char *errors) +.. c:function:: PyObject* PyUnicode_EncodeMBCS(const Py_UNICODE *s, Py_ssize_t size, const char *errors) - Encode the :ctype:`Py_UNICODE` buffer of the given *size* using MBCS and return + Encode the :c:type:`Py_UNICODE` buffer of the given *size* using MBCS and return a Python bytes object. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_AsMBCSString(PyObject *unicode) +.. c:function:: PyObject* PyUnicode_AsMBCSString(PyObject *unicode) Encode a Unicode object using MBCS and return the result as Python bytes object. Error handling is "strict". Return *NULL* if an exception was @@ -899,12 +991,12 @@ integers as appropriate. They all return *NULL* or ``-1`` if an exception occurs. -.. cfunction:: PyObject* PyUnicode_Concat(PyObject *left, PyObject *right) +.. c:function:: PyObject* PyUnicode_Concat(PyObject *left, PyObject *right) Concat two strings giving a new Unicode string. -.. cfunction:: PyObject* PyUnicode_Split(PyObject *s, PyObject *sep, Py_ssize_t maxsplit) +.. c:function:: PyObject* PyUnicode_Split(PyObject *s, PyObject *sep, Py_ssize_t maxsplit) Split a string giving a list of Unicode strings. If *sep* is *NULL*, splitting will be done at all whitespace substrings. Otherwise, splits occur at the given @@ -912,14 +1004,14 @@ They all return *NULL* or ``-1`` if an exception occurs. set. Separators are not included in the resulting list. -.. cfunction:: PyObject* PyUnicode_Splitlines(PyObject *s, int keepend) +.. c:function:: PyObject* PyUnicode_Splitlines(PyObject *s, int keepend) Split a Unicode string at line breaks, returning a list of Unicode strings. CRLF is considered to be one line break. If *keepend* is 0, the Line break characters are not included in the resulting strings. -.. cfunction:: PyObject* PyUnicode_Translate(PyObject *str, PyObject *table, const char *errors) +.. c:function:: PyObject* PyUnicode_Translate(PyObject *str, PyObject *table, const char *errors) Translate a string by applying a character mapping table to it and return the resulting Unicode object. @@ -935,20 +1027,20 @@ They all return *NULL* or ``-1`` if an exception occurs. use the default error handling. -.. cfunction:: PyObject* PyUnicode_Join(PyObject *separator, PyObject *seq) +.. c:function:: PyObject* PyUnicode_Join(PyObject *separator, PyObject *seq) Join a sequence of strings using the given *separator* and return the resulting Unicode string. -.. cfunction:: int PyUnicode_Tailmatch(PyObject *str, PyObject *substr, Py_ssize_t start, Py_ssize_t end, int direction) +.. c:function:: int PyUnicode_Tailmatch(PyObject *str, PyObject *substr, Py_ssize_t start, Py_ssize_t end, int direction) Return 1 if *substr* matches ``str[start:end]`` at the given tail end (*direction* == -1 means to do a prefix match, *direction* == 1 a suffix match), 0 otherwise. Return ``-1`` if an error occurred. -.. cfunction:: Py_ssize_t PyUnicode_Find(PyObject *str, PyObject *substr, Py_ssize_t start, Py_ssize_t end, int direction) +.. c:function:: Py_ssize_t PyUnicode_Find(PyObject *str, PyObject *substr, Py_ssize_t start, Py_ssize_t end, int direction) Return the first position of *substr* in ``str[start:end]`` using the given *direction* (*direction* == 1 means to do a forward search, *direction* == -1 a @@ -957,32 +1049,34 @@ They all return *NULL* or ``-1`` if an exception occurs. occurred and an exception has been set. -.. cfunction:: Py_ssize_t PyUnicode_Count(PyObject *str, PyObject *substr, Py_ssize_t start, Py_ssize_t end) +.. c:function:: Py_ssize_t PyUnicode_Count(PyObject *str, PyObject *substr, Py_ssize_t start, Py_ssize_t end) Return the number of non-overlapping occurrences of *substr* in ``str[start:end]``. Return ``-1`` if an error occurred. -.. cfunction:: PyObject* PyUnicode_Replace(PyObject *str, PyObject *substr, PyObject *replstr, Py_ssize_t maxcount) +.. c:function:: PyObject* PyUnicode_Replace(PyObject *str, PyObject *substr, PyObject *replstr, Py_ssize_t maxcount) Replace at most *maxcount* occurrences of *substr* in *str* with *replstr* and return the resulting Unicode object. *maxcount* == -1 means replace all occurrences. -.. cfunction:: int PyUnicode_Compare(PyObject *left, PyObject *right) +.. c:function:: int PyUnicode_Compare(PyObject *left, PyObject *right) Compare two strings and return -1, 0, 1 for less than, equal, and greater than, respectively. -.. cfunction:: int PyUnicode_CompareWithASCIIString(PyObject *uni, char *string) +.. c:function:: int PyUnicode_CompareWithASCIIString(PyObject *uni, char *string) Compare a unicode object, *uni*, with *string* and return -1, 0, 1 for less - than, equal, and greater than, respectively. + than, equal, and greater than, respectively. It is best to pass only + ASCII-encoded strings, but the function interprets the input string as + ISO-8859-1 if it contains non-ASCII characters". -.. cfunction:: int PyUnicode_RichCompare(PyObject *left, PyObject *right, int op) +.. c:function:: int PyUnicode_RichCompare(PyObject *left, PyObject *right, int op) Rich compare two unicode strings and return one of the following: @@ -998,13 +1092,13 @@ They all return *NULL* or ``-1`` if an exception occurs. :const:`Py_NE`, :const:`Py_LT`, and :const:`Py_LE`. -.. cfunction:: PyObject* PyUnicode_Format(PyObject *format, PyObject *args) +.. c:function:: PyObject* PyUnicode_Format(PyObject *format, PyObject *args) Return a new string object from *format* and *args*; this is analogous to ``format % args``. The *args* argument must be a tuple. -.. cfunction:: int PyUnicode_Contains(PyObject *container, PyObject *element) +.. c:function:: int PyUnicode_Contains(PyObject *container, PyObject *element) Check whether *element* is contained in *container* and return true or false accordingly. @@ -1013,7 +1107,7 @@ They all return *NULL* or ``-1`` if an exception occurs. there was an error. -.. cfunction:: void PyUnicode_InternInPlace(PyObject **string) +.. c:function:: void PyUnicode_InternInPlace(PyObject **string) Intern the argument *\*string* in place. The argument must be the address of a pointer variable pointing to a Python unicode string object. If there is an @@ -1026,10 +1120,10 @@ They all return *NULL* or ``-1`` if an exception occurs. if and only if you owned it before the call.) -.. cfunction:: PyObject* PyUnicode_InternFromString(const char *v) +.. c:function:: PyObject* PyUnicode_InternFromString(const char *v) - A combination of :cfunc:`PyUnicode_FromString` and - :cfunc:`PyUnicode_InternInPlace`, returning either a new unicode string object + A combination of :c:func:`PyUnicode_FromString` and + :c:func:`PyUnicode_InternInPlace`, returning either a new unicode string object that has been interned, or a new ("owned") reference to an earlier interned string object with the same value. diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst index 73cd1d23b4..41cdd6b0a4 100644 --- a/Doc/c-api/veryhigh.rst +++ b/Doc/c-api/veryhigh.rst @@ -16,68 +16,67 @@ parameter. The available start symbols are :const:`Py_eval_input`, :const:`Py_file_input`, and :const:`Py_single_input`. These are described following the functions which accept them as parameters. -Note also that several of these functions take :ctype:`FILE\*` parameters. One -particular issue which needs to be handled carefully is that the :ctype:`FILE` +Note also that several of these functions take :c:type:`FILE\*` parameters. One +particular issue which needs to be handled carefully is that the :c:type:`FILE` structure for different C libraries can be different and incompatible. Under Windows (at least), it is possible for dynamically linked extensions to actually -use different libraries, so care should be taken that :ctype:`FILE\*` parameters +use different libraries, so care should be taken that :c:type:`FILE\*` parameters are only passed to these functions if it is certain that they were created by the same library that the Python runtime is using. -.. cfunction:: int Py_Main(int argc, wchar_t **argv) +.. c:function:: int Py_Main(int argc, wchar_t **argv) - The main program for the standard interpreter. This is made - available for programs which embed Python. The *argc* and *argv* - parameters should be prepared exactly as those which are passed to - a C program's :cfunc:`main` function (converted to wchar_t - according to the user's locale). It is important to note that the - argument list may be modified (but the contents of the strings - pointed to by the argument list are not). The return value will be - ``0`` if the interpreter exits normally (i.e. without an - exception), ``1`` if the interpreter exits due to an exception, or - ``2`` if the parameter list does not represent a valid Python - command line. + The main program for the standard interpreter. This is made available for + programs which embed Python. The *argc* and *argv* parameters should be + prepared exactly as those which are passed to a C program's :c:func:`main` + function (converted to wchar_t according to the user's locale). It is + important to note that the argument list may be modified (but the contents of + the strings pointed to by the argument list are not). The return value will + be ``0`` if the interpreter exits normally (i.e., without an exception), + ``1`` if the interpreter exits due to an exception, or ``2`` if the parameter + list does not represent a valid Python command line. Note that if an otherwise unhandled :exc:`SystemExit` is raised, this function will not return ``1``, but exit the process, as long as ``Py_InspectFlag`` is not set. -.. cfunction:: int PyRun_AnyFile(FILE *fp, const char *filename) +.. c:function:: int PyRun_AnyFile(FILE *fp, const char *filename) - This is a simplified interface to :cfunc:`PyRun_AnyFileExFlags` below, leaving + This is a simplified interface to :c:func:`PyRun_AnyFileExFlags` below, leaving *closeit* set to ``0`` and *flags* set to *NULL*. -.. cfunction:: int PyRun_AnyFileFlags(FILE *fp, const char *filename, PyCompilerFlags *flags) +.. c:function:: int PyRun_AnyFileFlags(FILE *fp, const char *filename, PyCompilerFlags *flags) - This is a simplified interface to :cfunc:`PyRun_AnyFileExFlags` below, leaving + This is a simplified interface to :c:func:`PyRun_AnyFileExFlags` below, leaving the *closeit* argument set to ``0``. -.. cfunction:: int PyRun_AnyFileEx(FILE *fp, const char *filename, int closeit) +.. c:function:: int PyRun_AnyFileEx(FILE *fp, const char *filename, int closeit) - This is a simplified interface to :cfunc:`PyRun_AnyFileExFlags` below, leaving + This is a simplified interface to :c:func:`PyRun_AnyFileExFlags` below, leaving the *flags* argument set to *NULL*. -.. cfunction:: int PyRun_AnyFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags) +.. c:function:: int PyRun_AnyFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags) If *fp* refers to a file associated with an interactive device (console or terminal input or Unix pseudo-terminal), return the value of - :cfunc:`PyRun_InteractiveLoop`, otherwise return the result of - :cfunc:`PyRun_SimpleFile`. If *filename* is *NULL*, this function uses - ``"???"`` as the filename. + :c:func:`PyRun_InteractiveLoop`, otherwise return the result of + :c:func:`PyRun_SimpleFile`. *filename* is decoded from the filesystem + encoding (:func:`sys.getfilesystemencoding`). If *filename* is *NULL*, this + function uses ``"???"`` as the filename. -.. cfunction:: int PyRun_SimpleString(const char *command) +.. c:function:: int PyRun_SimpleString(const char *command) - This is a simplified interface to :cfunc:`PyRun_SimpleStringFlags` below, + This is a simplified interface to :c:func:`PyRun_SimpleStringFlags` below, leaving the *PyCompilerFlags\** argument set to NULL. -.. cfunction:: int PyRun_SimpleStringFlags(const char *command, PyCompilerFlags *flags) +.. c:function:: int PyRun_SimpleStringFlags(const char *command, PyCompilerFlags *flags) Executes the Python source code from *command* in the :mod:`__main__` module according to the *flags* argument. If :mod:`__main__` does not already exist, it @@ -90,103 +89,109 @@ the same library that the Python runtime is using. ``Py_InspectFlag`` is not set. -.. cfunction:: int PyRun_SimpleFile(FILE *fp, const char *filename) +.. c:function:: int PyRun_SimpleFile(FILE *fp, const char *filename) - This is a simplified interface to :cfunc:`PyRun_SimpleFileExFlags` below, + This is a simplified interface to :c:func:`PyRun_SimpleFileExFlags` below, leaving *closeit* set to ``0`` and *flags* set to *NULL*. -.. cfunction:: int PyRun_SimpleFileFlags(FILE *fp, const char *filename, PyCompilerFlags *flags) +.. c:function:: int PyRun_SimpleFileFlags(FILE *fp, const char *filename, PyCompilerFlags *flags) - This is a simplified interface to :cfunc:`PyRun_SimpleFileExFlags` below, + This is a simplified interface to :c:func:`PyRun_SimpleFileExFlags` below, leaving *closeit* set to ``0``. -.. cfunction:: int PyRun_SimpleFileEx(FILE *fp, const char *filename, int closeit) +.. c:function:: int PyRun_SimpleFileEx(FILE *fp, const char *filename, int closeit) - This is a simplified interface to :cfunc:`PyRun_SimpleFileExFlags` below, + This is a simplified interface to :c:func:`PyRun_SimpleFileExFlags` below, leaving *flags* set to *NULL*. -.. cfunction:: int PyRun_SimpleFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags) +.. c:function:: int PyRun_SimpleFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags) - Similar to :cfunc:`PyRun_SimpleStringFlags`, but the Python source code is read - from *fp* instead of an in-memory string. *filename* should be the name of the - file. If *closeit* is true, the file is closed before PyRun_SimpleFileExFlags - returns. + Similar to :c:func:`PyRun_SimpleStringFlags`, but the Python source code is read + from *fp* instead of an in-memory string. *filename* should be the name of + the file, it is decoded from the filesystem encoding + (:func:`sys.getfilesystemencoding`). If *closeit* is true, the file is + closed before PyRun_SimpleFileExFlags returns. -.. cfunction:: int PyRun_InteractiveOne(FILE *fp, const char *filename) +.. c:function:: int PyRun_InteractiveOne(FILE *fp, const char *filename) - This is a simplified interface to :cfunc:`PyRun_InteractiveOneFlags` below, + This is a simplified interface to :c:func:`PyRun_InteractiveOneFlags` below, leaving *flags* set to *NULL*. -.. cfunction:: int PyRun_InteractiveOneFlags(FILE *fp, const char *filename, PyCompilerFlags *flags) +.. c:function:: int PyRun_InteractiveOneFlags(FILE *fp, const char *filename, PyCompilerFlags *flags) Read and execute a single statement from a file associated with an interactive device according to the *flags* argument. The user will be - prompted using ``sys.ps1`` and ``sys.ps2``. Returns ``0`` when the input was + prompted using ``sys.ps1`` and ``sys.ps2``. *filename* is decoded from the + filesystem encoding (:func:`sys.getfilesystemencoding`). + + Returns ``0`` when the input was executed successfully, ``-1`` if there was an exception, or an error code from the :file:`errcode.h` include file distributed as part of Python if there was a parse error. (Note that :file:`errcode.h` is not included by :file:`Python.h`, so must be included specifically if needed.) -.. cfunction:: int PyRun_InteractiveLoop(FILE *fp, const char *filename) +.. c:function:: int PyRun_InteractiveLoop(FILE *fp, const char *filename) - This is a simplified interface to :cfunc:`PyRun_InteractiveLoopFlags` below, + This is a simplified interface to :c:func:`PyRun_InteractiveLoopFlags` below, leaving *flags* set to *NULL*. -.. cfunction:: int PyRun_InteractiveLoopFlags(FILE *fp, const char *filename, PyCompilerFlags *flags) +.. c:function:: int PyRun_InteractiveLoopFlags(FILE *fp, const char *filename, PyCompilerFlags *flags) Read and execute statements from a file associated with an interactive device until EOF is reached. The user will be prompted using ``sys.ps1`` and - ``sys.ps2``. Returns ``0`` at EOF. + ``sys.ps2``. *filename* is decoded from the filesystem encoding + (:func:`sys.getfilesystemencoding`). Returns ``0`` at EOF. -.. cfunction:: struct _node* PyParser_SimpleParseString(const char *str, int start) +.. c:function:: struct _node* PyParser_SimpleParseString(const char *str, int start) This is a simplified interface to - :cfunc:`PyParser_SimpleParseStringFlagsFilename` below, leaving *filename* set + :c:func:`PyParser_SimpleParseStringFlagsFilename` below, leaving *filename* set to *NULL* and *flags* set to ``0``. -.. cfunction:: struct _node* PyParser_SimpleParseStringFlags( const char *str, int start, int flags) +.. c:function:: struct _node* PyParser_SimpleParseStringFlags( const char *str, int start, int flags) This is a simplified interface to - :cfunc:`PyParser_SimpleParseStringFlagsFilename` below, leaving *filename* set + :c:func:`PyParser_SimpleParseStringFlagsFilename` below, leaving *filename* set to *NULL*. -.. cfunction:: struct _node* PyParser_SimpleParseStringFlagsFilename( const char *str, const char *filename, int start, int flags) +.. c:function:: struct _node* PyParser_SimpleParseStringFlagsFilename( const char *str, const char *filename, int start, int flags) Parse Python source code from *str* using the start token *start* according to the *flags* argument. The result can be used to create a code object which can be evaluated efficiently. This is useful if a code fragment must be evaluated - many times. + many times. *filename* is decoded from the filesystem encoding + (:func:`sys.getfilesystemencoding`). -.. cfunction:: struct _node* PyParser_SimpleParseFile(FILE *fp, const char *filename, int start) +.. c:function:: struct _node* PyParser_SimpleParseFile(FILE *fp, const char *filename, int start) - This is a simplified interface to :cfunc:`PyParser_SimpleParseFileFlags` below, + This is a simplified interface to :c:func:`PyParser_SimpleParseFileFlags` below, leaving *flags* set to ``0`` -.. cfunction:: struct _node* PyParser_SimpleParseFileFlags(FILE *fp, const char *filename, int start, int flags) +.. c:function:: struct _node* PyParser_SimpleParseFileFlags(FILE *fp, const char *filename, int start, int flags) - Similar to :cfunc:`PyParser_SimpleParseStringFlagsFilename`, but the Python + Similar to :c:func:`PyParser_SimpleParseStringFlagsFilename`, but the Python source code is read from *fp* instead of an in-memory string. -.. cfunction:: PyObject* PyRun_String(const char *str, int start, PyObject *globals, PyObject *locals) +.. c:function:: PyObject* PyRun_String(const char *str, int start, PyObject *globals, PyObject *locals) - This is a simplified interface to :cfunc:`PyRun_StringFlags` below, leaving + This is a simplified interface to :c:func:`PyRun_StringFlags` below, leaving *flags* set to *NULL*. -.. cfunction:: PyObject* PyRun_StringFlags(const char *str, int start, PyObject *globals, PyObject *locals, PyCompilerFlags *flags) +.. c:function:: PyObject* PyRun_StringFlags(const char *str, int start, PyObject *globals, PyObject *locals, PyCompilerFlags *flags) Execute Python source code from *str* in the context specified by the dictionaries *globals* and *locals* with the compiler flags specified by @@ -197,57 +202,73 @@ the same library that the Python runtime is using. exception was raised. -.. cfunction:: PyObject* PyRun_File(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals) +.. c:function:: PyObject* PyRun_File(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals) - This is a simplified interface to :cfunc:`PyRun_FileExFlags` below, leaving + This is a simplified interface to :c:func:`PyRun_FileExFlags` below, leaving *closeit* set to ``0`` and *flags* set to *NULL*. -.. cfunction:: PyObject* PyRun_FileEx(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, int closeit) +.. c:function:: PyObject* PyRun_FileEx(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, int closeit) - This is a simplified interface to :cfunc:`PyRun_FileExFlags` below, leaving + This is a simplified interface to :c:func:`PyRun_FileExFlags` below, leaving *flags* set to *NULL*. -.. cfunction:: PyObject* PyRun_FileFlags(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, PyCompilerFlags *flags) +.. c:function:: PyObject* PyRun_FileFlags(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, PyCompilerFlags *flags) - This is a simplified interface to :cfunc:`PyRun_FileExFlags` below, leaving + This is a simplified interface to :c:func:`PyRun_FileExFlags` below, leaving *closeit* set to ``0``. -.. cfunction:: PyObject* PyRun_FileExFlags(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, int closeit, PyCompilerFlags *flags) +.. c:function:: PyObject* PyRun_FileExFlags(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, int closeit, PyCompilerFlags *flags) - Similar to :cfunc:`PyRun_StringFlags`, but the Python source code is read from - *fp* instead of an in-memory string. *filename* should be the name of the file. - If *closeit* is true, the file is closed before :cfunc:`PyRun_FileExFlags` + Similar to :c:func:`PyRun_StringFlags`, but the Python source code is read from + *fp* instead of an in-memory string. *filename* should be the name of the file, + it is decoded from the filesystem encoding (:func:`sys.getfilesystemencoding`). + If *closeit* is true, the file is closed before :c:func:`PyRun_FileExFlags` returns. -.. cfunction:: PyObject* Py_CompileString(const char *str, const char *filename, int start) +.. c:function:: PyObject* Py_CompileString(const char *str, const char *filename, int start) - This is a simplified interface to :cfunc:`Py_CompileStringFlags` below, leaving + This is a simplified interface to :c:func:`Py_CompileStringFlags` below, leaving *flags* set to *NULL*. -.. cfunction:: PyObject* Py_CompileStringFlags(const char *str, const char *filename, int start, PyCompilerFlags *flags) +.. c:function:: PyObject* Py_CompileStringFlags(const char *str, const char *filename, int start, PyCompilerFlags *flags) + + This is a simplified interface to :c:func:`Py_CompileStringExFlags` below, with + *optimize* set to ``-1``. + + +.. c:function:: PyObject* Py_CompileStringExFlags(const char *str, const char *filename, int start, PyCompilerFlags *flags, int optimize) Parse and compile the Python source code in *str*, returning the resulting code object. The start token is given by *start*; this can be used to constrain the code which can be compiled and should be :const:`Py_eval_input`, :const:`Py_file_input`, or :const:`Py_single_input`. The filename specified by *filename* is used to construct the code object and may appear in tracebacks or - :exc:`SyntaxError` exception messages. This returns *NULL* if the code cannot - be parsed or compiled. + :exc:`SyntaxError` exception messages, it is decoded from the filesystem + encoding (:func:`sys.getfilesystemencoding`). This returns *NULL* if the + code cannot be parsed or compiled. + + The integer *optimize* specifies the optimization level of the compiler; a + value of ``-1`` selects the optimization level of the interpreter as given by + :option:`-O` options. Explicit levels are ``0`` (no optimization; + ``__debug__`` is true), ``1`` (asserts are removed, ``__debug__`` is false) + or ``2`` (docstrings are removed too). + + .. versionadded:: 3.2 -.. cfunction:: PyObject* PyEval_EvalCode(PyCodeObject *co, PyObject *globals, PyObject *locals) +.. c:function:: PyObject* PyEval_EvalCode(PyObject *co, PyObject *globals, PyObject *locals) - This is a simplified interface to :cfunc:`PyEval_EvalCodeEx`, with just + This is a simplified interface to :c:func:`PyEval_EvalCodeEx`, with just the code object, and the dictionaries of global and local variables. The other arguments are set to *NULL*. -.. cfunction:: PyObject* PyEval_EvalCodeEx(PyCodeObject *co, PyObject *globals, PyObject *locals, PyObject **args, int argcount, PyObject **kws, int kwcount, PyObject **defs, int defcount, PyObject *closure) +.. c:function:: PyObject* PyEval_EvalCodeEx(PyObject *co, PyObject *globals, PyObject *locals, PyObject **args, int argcount, PyObject **kws, int kwcount, PyObject **defs, int defcount, PyObject *closure) Evaluate a precompiled code object, given a particular environment for its evaluation. This environment consists of dictionaries of global and local @@ -255,13 +276,13 @@ the same library that the Python runtime is using. cells. -.. cfunction:: PyObject* PyEval_EvalFrame(PyFrameObject *f) +.. c:function:: PyObject* PyEval_EvalFrame(PyFrameObject *f) Evaluate an execution frame. This is a simplified interface to PyEval_EvalFrameEx, for backward compatibility. -.. cfunction:: PyObject* PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) +.. c:function:: PyObject* PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) This is the main, unvarnished function of Python interpretation. It is literally 2000 lines long. The code object associated with the execution @@ -271,39 +292,39 @@ the same library that the Python runtime is using. :meth:`throw` methods of generator objects. -.. cfunction:: int PyEval_MergeCompilerFlags(PyCompilerFlags *cf) +.. c:function:: int PyEval_MergeCompilerFlags(PyCompilerFlags *cf) This function changes the flags of the current evaluation frame, and returns true on success, false on failure. -.. cvar:: int Py_eval_input +.. c:var:: int Py_eval_input .. index:: single: Py_CompileString() The start symbol from the Python grammar for isolated expressions; for use with - :cfunc:`Py_CompileString`. + :c:func:`Py_CompileString`. -.. cvar:: int Py_file_input +.. c:var:: int Py_file_input .. index:: single: Py_CompileString() The start symbol from the Python grammar for sequences of statements as read - from a file or other source; for use with :cfunc:`Py_CompileString`. This is + from a file or other source; for use with :c:func:`Py_CompileString`. This is the symbol to use when compiling arbitrarily long Python source code. -.. cvar:: int Py_single_input +.. c:var:: int Py_single_input .. index:: single: Py_CompileString() The start symbol from the Python grammar for a single statement; for use with - :cfunc:`Py_CompileString`. This is the symbol used for the interactive + :c:func:`Py_CompileString`. This is the symbol used for the interactive interpreter loop. -.. ctype:: struct PyCompilerFlags +.. c:type:: struct PyCompilerFlags This is the structure used to hold compiler flags. In cases where code is only being compiled, it is passed as ``int flags``, and in cases where code is being @@ -319,7 +340,7 @@ the same library that the Python runtime is using. } -.. cvar:: int CO_FUTURE_DIVISION +.. c:var:: int CO_FUTURE_DIVISION This bit can be set in *flags* to cause division operator ``/`` to be interpreted as "true division" according to :pep:`238`. diff --git a/Doc/c-api/weakref.rst b/Doc/c-api/weakref.rst index 8a36110386..6cb3e33fe8 100644 --- a/Doc/c-api/weakref.rst +++ b/Doc/c-api/weakref.rst @@ -11,22 +11,22 @@ simple reference object, and the second acts as a proxy for the original object as much as it can. -.. cfunction:: int PyWeakref_Check(ob) +.. c:function:: int PyWeakref_Check(ob) Return true if *ob* is either a reference or proxy object. -.. cfunction:: int PyWeakref_CheckRef(ob) +.. c:function:: int PyWeakref_CheckRef(ob) Return true if *ob* is a reference object. -.. cfunction:: int PyWeakref_CheckProxy(ob) +.. c:function:: int PyWeakref_CheckProxy(ob) Return true if *ob* is a proxy object. -.. cfunction:: PyObject* PyWeakref_NewRef(PyObject *ob, PyObject *callback) +.. c:function:: PyObject* PyWeakref_NewRef(PyObject *ob, PyObject *callback) Return a weak reference object for the object *ob*. This will always return a new reference, but is not guaranteed to create a new object; an existing @@ -38,7 +38,7 @@ as much as it can. *NULL*, this will return *NULL* and raise :exc:`TypeError`. -.. cfunction:: PyObject* PyWeakref_NewProxy(PyObject *ob, PyObject *callback) +.. c:function:: PyObject* PyWeakref_NewProxy(PyObject *ob, PyObject *callback) Return a weak reference proxy object for the object *ob*. This will always return a new reference, but is not guaranteed to create a new object; an @@ -50,20 +50,20 @@ as much as it can. ``None``, or *NULL*, this will return *NULL* and raise :exc:`TypeError`. -.. cfunction:: PyObject* PyWeakref_GetObject(PyObject *ref) +.. c:function:: PyObject* PyWeakref_GetObject(PyObject *ref) Return the referenced object from a weak reference, *ref*. If the referent is no longer live, returns :const:`Py_None`. - .. warning:: + .. note:: This function returns a **borrowed reference** to the referenced object. - This means that you should always call :cfunc:`Py_INCREF` on the object + This means that you should always call :c:func:`Py_INCREF` on the object except if you know that it cannot be destroyed while you are still using it. -.. cfunction:: PyObject* PyWeakref_GET_OBJECT(PyObject *ref) +.. c:function:: PyObject* PyWeakref_GET_OBJECT(PyObject *ref) - Similar to :cfunc:`PyWeakref_GetObject`, but implemented as a macro that does no + Similar to :c:func:`PyWeakref_GetObject`, but implemented as a macro that does no error checking. diff --git a/Doc/conf.py b/Doc/conf.py index 5939f7a907..85eb9fa3bb 100644 --- a/Doc/conf.py +++ b/Doc/conf.py @@ -1,4 +1,3 @@ -# -*- coding: utf-8 -*- # # Python documentation build configuration file # @@ -66,6 +65,9 @@ highlight_language = 'python3' # Options for HTML output # ----------------------- +html_theme = 'default' +html_theme_options = {'collapsiblesidebar': True} + # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. html_last_updated_fmt = '%b %d, %Y' @@ -86,7 +88,7 @@ html_additional_pages = { } # Output an OpenSearch description file. -html_use_opensearch = 'http://docs.python.org/3.1' +html_use_opensearch = 'http://docs.python.org/dev/py3k' # Additional static files. html_static_path = ['tools/sphinxext/static'] @@ -115,8 +117,6 @@ latex_documents = [ 'The Python/C API', _stdauthor, 'manual'), ('distutils/index', 'distutils.tex', 'Distributing Python Modules', _stdauthor, 'manual'), - ('documenting/index', 'documenting.tex', - 'Documenting Python', 'Georg Brandl', 'manual'), ('extending/index', 'extending.tex', 'Extending and Embedding Python', _stdauthor, 'manual'), ('install/index', 'install.tex', @@ -154,7 +154,7 @@ latex_preamble = r''' latex_appendices = ['glossary', 'about', 'license', 'copyright'] # Get LaTeX to handle Unicode correctly -latex_elements = {'inputenc': r'\usepackage[utf8x]{inputenc}'} +latex_elements = {'inputenc': r'\usepackage[utf8x]{inputenc}', 'utf8extra': ''} # Options for the coverage checker # -------------------------------- diff --git a/Doc/contents.rst b/Doc/contents.rst index e938fcd93d..c0c6af34d9 100644 --- a/Doc/contents.rst +++ b/Doc/contents.rst @@ -13,7 +13,6 @@ c-api/index.rst distutils/index.rst install/index.rst - documenting/index.rst howto/index.rst faq/index.rst glossary.rst diff --git a/Doc/copyright.rst b/Doc/copyright.rst index 893b541b22..9a245c8b72 100644 --- a/Doc/copyright.rst +++ b/Doc/copyright.rst @@ -4,7 +4,7 @@ Copyright Python and this documentation is: -Copyright © 2001-2010 Python Software Foundation. All rights reserved. +Copyright © 2001-2012 Python Software Foundation. All rights reserved. Copyright © 2000 BeOpen.com. All rights reserved. diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat index dc058d54ee..c7d7bd140a 100644 --- a/Doc/data/refcounts.dat +++ b/Doc/data/refcounts.dat @@ -281,6 +281,12 @@ PyErr_NewException:char*:name:: PyErr_NewException:PyObject*:base:0: PyErr_NewException:PyObject*:dict:0: +PyErr_NewExceptionWithDoc:PyObject*::+1: +PyErr_NewExceptionWithDoc:char*:name:: +PyErr_NewExceptionWithDoc:char*:doc:: +PyErr_NewExceptionWithDoc:PyObject*:base:0: +PyErr_NewExceptionWithDoc:PyObject*:dict:0: + PyErr_NoMemory:PyObject*::null: PyErr_NormalizeException:void::: @@ -370,6 +376,8 @@ PyEval_EvalCode:PyCodeObject*:co:0: PyEval_EvalCode:PyObject*:globals:0: PyEval_EvalCode:PyObject*:locals:0: +PyException_GetTraceback:PyObject*::+1: + PyFile_AsFile:FILE*::: PyFile_AsFile:PyFileObject*:p:0: @@ -493,6 +501,11 @@ PyImport_ExecCodeModule:PyObject*::+1: PyImport_ExecCodeModule:char*:name:: PyImport_ExecCodeModule:PyObject*:co:0: +PyImport_ExecCodeModuleEx:PyObject*::+1: +PyImport_ExecCodeModuleEx:char*:name:: +PyImport_ExecCodeModuleEx:PyObject*:co:0: +PyImport_ExecCodeModuleEx:char*:pathname:: + PyImport_GetMagicNumber:long::: PyImport_GetModuleDict:PyObject*::0: @@ -512,6 +525,13 @@ PyImport_ImportModuleEx:PyObject*:globals:0:??? PyImport_ImportModuleEx:PyObject*:locals:0:??? PyImport_ImportModuleEx:PyObject*:fromlist:0:??? +PyImport_ImportModuleLevel:PyObject*::+1: +PyImport_ImportModuleLevel:char*:name:: +PyImport_ImportModuleLevel:PyObject*:globals:0:??? +PyImport_ImportModuleLevel:PyObject*:locals:0:??? +PyImport_ImportModuleLevel:PyObject*:fromlist:0:??? +PyImport_ImportModuleLevel:int:level:: + PyImport_ReloadModule:PyObject*::+1: PyImport_ReloadModule:PyObject*:m:0: @@ -828,9 +848,6 @@ PyNumber_InPlaceXor:PyObject*::+1: PyNumber_InPlaceXor:PyObject*:v:0: PyNumber_InPlaceXor:PyObject*:w:0: -PyNumber_Int:PyObject*::+1: -PyNumber_Int:PyObject*:o:0: - PyNumber_Invert:PyObject*::+1: PyNumber_Invert:PyObject*:o:0: @@ -1290,6 +1307,9 @@ PyString_AsEncodedString:const char*:errors:: PySys_AddWarnOption:void::: PySys_AddWarnOption:char*:s:: +PySys_AddXOption:void::: +PySys_AddXOption:const wchar_t*:s:: + PySys_GetFile:FILE*::: PySys_GetFile:char*:name:: PySys_GetFile:FILE*:def:: @@ -1297,6 +1317,8 @@ PySys_GetFile:FILE*:def:: PySys_GetObject:PyObject*::0: PySys_GetObject:char*:name:: +PySys_GetXOptions:PyObject*::0: + PySys_SetArgv:int::: PySys_SetArgv:int:argc:: PySys_SetArgv:char**:argv:: diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst index 712fffbe33..e15dc76c83 100644 --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -31,8 +31,9 @@ setup script). Indirectly provides the :class:`distutils.dist.Distribution` and +====================+================================+=============================================================+ | *name* | The name of the package | a string | +--------------------+--------------------------------+-------------------------------------------------------------+ - | *version* | The version number of the | See :mod:`distutils.version` | - | | package | | + | *version* | The version number of the | a string | + | | package; see | | + | | :mod:`distutils.version` | | +--------------------+--------------------------------+-------------------------------------------------------------+ | *description* | A single line describing the | a string | | | package | | @@ -49,14 +50,14 @@ setup script). Indirectly provides the :class:`distutils.dist.Distribution` and | | maintainer, if different from | | | | the author | | +--------------------+--------------------------------+-------------------------------------------------------------+ - | *maintainer_email* | The email address of the | | + | *maintainer_email* | The email address of the | a string | | | current maintainer, if | | | | different from the author | | +--------------------+--------------------------------+-------------------------------------------------------------+ - | *url* | A URL for the package | a URL | + | *url* | A URL for the package | a string | | | (homepage) | | +--------------------+--------------------------------+-------------------------------------------------------------+ - | *download_url* | A URL to download the package | a URL | + | *download_url* | A URL to download the package | a string | +--------------------+--------------------------------+-------------------------------------------------------------+ | *packages* | A list of Python packages that | a list of strings | | | distutils will manipulate | | @@ -68,14 +69,13 @@ setup script). Indirectly provides the :class:`distutils.dist.Distribution` and | | files to be built and | | | | installed | | +--------------------+--------------------------------+-------------------------------------------------------------+ - | *ext_modules* | A list of Python extensions to | A list of instances of | + | *ext_modules* | A list of Python extensions to | a list of instances of | | | be built | :class:`distutils.core.Extension` | +--------------------+--------------------------------+-------------------------------------------------------------+ - | *classifiers* | A list of categories for the | The list of available | - | | package | categorizations is at | - | | | http://pypi.python.org/pypi?:action=list_classifiers. | + | *classifiers* | A list of categories for the | a list of strings; valid classifiers are listed on `PyPI | + | | package | <http://pypi.python.org/pypi?:action=list_classifiers>`_. | +--------------------+--------------------------------+-------------------------------------------------------------+ - | *distclass* | the :class:`Distribution` | A subclass of | + | *distclass* | the :class:`Distribution` | a subclass of | | | class to use | :class:`distutils.core.Distribution` | +--------------------+--------------------------------+-------------------------------------------------------------+ | *script_name* | The name of the setup.py | a string | @@ -85,15 +85,15 @@ setup script). Indirectly provides the :class:`distutils.dist.Distribution` and | *script_args* | Arguments to supply to the | a list of strings | | | setup script | | +--------------------+--------------------------------+-------------------------------------------------------------+ - | *options* | default options for the setup | a string | + | *options* | default options for the setup | a dictionary | | | script | | +--------------------+--------------------------------+-------------------------------------------------------------+ | *license* | The license for the package | a string | +--------------------+--------------------------------+-------------------------------------------------------------+ - | *keywords* | Descriptive meta-data, see | | + | *keywords* | Descriptive meta-data, see | a list of strings or a comma-separated string | | | :pep:`314` | | +--------------------+--------------------------------+-------------------------------------------------------------+ - | *platforms* | | | + | *platforms* | | a list of strings or a comma-separated string | +--------------------+--------------------------------+-------------------------------------------------------------+ | *cmdclass* | A mapping of command names to | a dictionary | | | :class:`Command` subclasses | | @@ -165,13 +165,13 @@ the full reference. +------------------------+--------------------------------+---------------------------+ | argument name | value | type | +========================+================================+===========================+ - | *name* | the full name of the | string | + | *name* | the full name of the | a string | | | extension, including any | | | | packages --- ie. *not* a | | | | filename or pathname, but | | | | Python dotted name | | +------------------------+--------------------------------+---------------------------+ - | *sources* | list of source filenames, | string | + | *sources* | list of source filenames, | a list of strings | | | relative to the distribution | | | | root (where the setup script | | | | lives), in Unix form (slash- | | @@ -184,12 +184,12 @@ the full reference. | | as source for a Python | | | | extension. | | +------------------------+--------------------------------+---------------------------+ - | *include_dirs* | list of directories to search | string | + | *include_dirs* | list of directories to search | a list of strings | | | for C/C++ header files (in | | | | Unix form for portability) | | +------------------------+--------------------------------+---------------------------+ - | *define_macros* | list of macros to define; each | (string, string) tuple or | - | | macro is defined using a | (name, ``None``) | + | *define_macros* | list of macros to define; each | a list of tuples | + | | macro is defined using a | | | | 2-tuple ``(name, value)``, | | | | where *value* is | | | | either the string to define it | | @@ -200,31 +200,31 @@ the full reference. | | on Unix C compiler command | | | | line) | | +------------------------+--------------------------------+---------------------------+ - | *undef_macros* | list of macros to undefine | string | + | *undef_macros* | list of macros to undefine | a list of strings | | | explicitly | | +------------------------+--------------------------------+---------------------------+ - | *library_dirs* | list of directories to search | string | + | *library_dirs* | list of directories to search | a list of strings | | | for C/C++ libraries at link | | | | time | | +------------------------+--------------------------------+---------------------------+ - | *libraries* | list of library names (not | string | + | *libraries* | list of library names (not | a list of strings | | | filenames or paths) to link | | | | against | | +------------------------+--------------------------------+---------------------------+ - | *runtime_library_dirs* | list of directories to search | string | + | *runtime_library_dirs* | list of directories to search | a list of strings | | | for C/C++ libraries at run | | | | time (for shared extensions, | | | | this is when the extension is | | | | loaded) | | +------------------------+--------------------------------+---------------------------+ - | *extra_objects* | list of extra files to link | string | + | *extra_objects* | list of extra files to link | a list of strings | | | with (eg. object files not | | | | implied by 'sources', static | | | | library that must be | | | | explicitly specified, binary | | | | resource files, etc.) | | +------------------------+--------------------------------+---------------------------+ - | *extra_compile_args* | any extra platform- and | string | + | *extra_compile_args* | any extra platform- and | a list of strings | | | compiler-specific information | | | | to use when compiling the | | | | source files in 'sources'. For | | @@ -235,7 +235,7 @@ the full reference. | | for other platforms it could | | | | be anything. | | +------------------------+--------------------------------+---------------------------+ - | *extra_link_args* | any extra platform- and | string | + | *extra_link_args* | any extra platform- and | a list of strings | | | compiler-specific information | | | | to use when linking object | | | | files together to create the | | @@ -244,7 +244,7 @@ the full reference. | | Similar interpretation as for | | | | 'extra_compile_args'. | | +------------------------+--------------------------------+---------------------------+ - | *export_symbols* | list of symbols to be exported | string | + | *export_symbols* | list of symbols to be exported | a list of strings | | | from a shared extension. Not | | | | used on all platforms, and not | | | | generally necessary for Python | | @@ -252,15 +252,20 @@ the full reference. | | export exactly one symbol: | | | | ``init`` + extension_name. | | +------------------------+--------------------------------+---------------------------+ - | *depends* | list of files that the | string | + | *depends* | list of files that the | a list of strings | | | extension depends on | | +------------------------+--------------------------------+---------------------------+ - | *language* | extension language (i.e. | string | + | *language* | extension language (i.e. | a string | | | ``'c'``, ``'c++'``, | | | | ``'objc'``). Will be detected | | | | from the source extensions if | | | | not provided. | | +------------------------+--------------------------------+---------------------------+ + | *optional* | specifies that a build failure | a boolean | + | | in the extension should not | | + | | abort the build process, but | | + | | simply skip the extension. | | + +------------------------+--------------------------------+---------------------------+ .. class:: Distribution @@ -444,7 +449,9 @@ This module provides the following functions. Define a preprocessor macro for all compilations driven by this compiler object. The optional parameter *value* should be a string; if it is not supplied, then the macro will be defined without an explicit value and the exact outcome - depends on the compiler used (XXX true? does ANSI say anything about this?) + depends on the compiler used. + + .. XXX true? does ANSI say anything about this? .. method:: CCompiler.undefine_macro(name) @@ -598,7 +605,9 @@ This module provides the following functions. *output_libname* should be a library name, not a filename; the filename will be inferred from the library name. *output_dir* is the directory where the library - file will be put. XXX defaults to what? + file will be put. + + .. XXX defaults to what? *debug* is a boolean; if true, debugging information will be included in the library (note that on most platforms, it is the compile step where this matters: @@ -718,30 +727,29 @@ This module provides the following functions. Invokes :func:`distutils.util.execute` This method invokes a Python function *func* with the given arguments *args*, after logging and taking into account - the *dry_run* flag. XXX see also. + the *dry_run* flag. .. method:: CCompiler.spawn(cmd) Invokes :func:`distutils.util.spawn`. This invokes an external process to run - the given command. XXX see also. + the given command. .. method:: CCompiler.mkpath(name[, mode=511]) Invokes :func:`distutils.dir_util.mkpath`. This creates a directory and any - missing ancestor directories. XXX see also. + missing ancestor directories. .. method:: CCompiler.move_file(src, dst) - Invokes :meth:`distutils.file_util.move_file`. Renames *src* to *dst*. XXX see - also. + Invokes :meth:`distutils.file_util.move_file`. Renames *src* to *dst*. .. method:: CCompiler.announce(msg[, level=1]) - Write a message using :func:`distutils.log.debug`. XXX see also. + Write a message using :func:`distutils.log.debug`. .. method:: CCompiler.warn(msg) @@ -869,8 +877,6 @@ tarballs or zipfiles. prefix of all files and directories in the archive. *root_dir* and *base_dir* both default to the current directory. Returns the name of the archive file. - .. XXX This should be changed to support bz2 files. - .. function:: make_tarball(base_name, base_dir[, compress='gzip', verbose=0, dry_run=0]) @@ -882,8 +888,6 @@ tarballs or zipfiles. possibly plus the appropriate compression extension (:file:`.gz`, :file:`.bz2` or :file:`.Z`). Return the output filename. - .. XXX This should be replaced with calls to the :mod:`tarfile` module. - .. function:: make_zipfile(base_name, base_dir[, verbose=0, dry_run=0]) @@ -995,8 +999,6 @@ directories. errors are ignored (apart from being reported to ``sys.stdout`` if *verbose* is true). -.. XXX Some of this could be replaced with the shutil module? - :mod:`distutils.file_util` --- Single file operations ===================================================== @@ -1110,8 +1112,6 @@ other utility module. * ``macosx-10.6-intel`` - .. % XXX isn't this also provided by some other non-distutils module? - .. function:: convert_path(pathname) @@ -1199,9 +1199,9 @@ other utility module. .. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None]) Byte-compile a collection of Python source files to either :file:`.pyc` or - :file:`.pyo` files in the same directory. *py_files* is a list of files to - compile; any files that don't end in :file:`.py` are silently skipped. - *optimize* must be one of the following: + :file:`.pyo` files in a :file:`__pycache__` subdirectory (see :pep:`3147`). + *py_files* is a list of files to compile; any files that don't end in + :file:`.py` are silently skipped. *optimize* must be one of the following: * ``0`` - don't optimize (generate :file:`.pyc`) * ``1`` - normal optimization (like ``python -O``) @@ -1226,6 +1226,11 @@ other utility module. is used by the script generated in indirect mode; unless you know what you're doing, leave it set to ``None``. + .. versionchanged:: 3.2.3 + Create ``.pyc`` or ``.pyo`` files with an :func:`import magic tag + <imp.get_tag>` in their name, in a :file:`__pycache__` subdirectory + instead of files without tag in the current directory. + .. function:: rfc822_escape(header) @@ -1311,9 +1316,6 @@ provides the following additional features: the "negative alias" of :option:`--verbose`, then :option:`--quiet` on the command line sets *verbose* to false. -.. XXX Should be replaced with :mod:`optparse`. - - .. function:: fancy_getopt(options, negative_opt, object, args) Wrapper function. *options* is a list of ``(long_option, short_option, @@ -1329,9 +1331,6 @@ provides the following additional features: Wraps *text* to less than *width* wide. - .. XXX Should be replaced with :mod:`textwrap` (which is available in Python - 2.3 and later). - .. class:: FancyGetopt([option_table=None]) @@ -1394,10 +1393,6 @@ filesystem and building lists of files. :synopsis: A simple logging mechanism, 282-style -.. XXX Should be replaced with standard :mod:`logging` module. - - - :mod:`distutils.spawn` --- Spawn a sub-process ============================================== @@ -1736,7 +1731,7 @@ Subclasses of :class:`Command` must define the following methods. Set final values for all the options that this command supports. This is always called as late as possible, ie. after any option assignments from the command-line or from other commands have been done. Thus, this is the place - to to code option dependencies: if *foo* depends on *bar*, then it is safe to + to code option dependencies: if *foo* depends on *bar*, then it is safe to set *foo* from *bar* as long as *foo* still has the same value it was assigned in :meth:`initialize_options`. @@ -1815,7 +1810,7 @@ Subclasses of :class:`Command` must define the following methods. .. module:: distutils.command.bdist_msi :synopsis: Build a binary distribution as a Windows MSI file -.. class:: bdist_msi(Command) +.. class:: bdist_msi Builds a `Windows Installer`_ (.msi) binary package. @@ -1894,9 +1889,9 @@ Subclasses of :class:`Command` must define the following methods. :synopsis: Build the .py/.pyc files of a package -.. class:: build_py(Command) +.. class:: build_py -.. class:: build_py_2to3(build_py) +.. class:: build_py_2to3 Alternative implementation of build_py which also runs the 2to3 conversion library on each .py file that is going to be diff --git a/Doc/distutils/builtdist.rst b/Doc/distutils/builtdist.rst index 2697ba04cb..d1ab7dbce9 100644 --- a/Doc/distutils/builtdist.rst +++ b/Doc/distutils/builtdist.rst @@ -139,13 +139,13 @@ The following sections give details on the individual :command:`bdist_\*` commands. -.. _creating-dumb: +.. .. _creating-dumb: -Creating dumb built distributions -================================= +.. Creating dumb built distributions +.. ================================= .. XXX Need to document absolute vs. prefix-relative packages here, but first - I have to implement it! + I have to implement it! .. _creating-rpms: @@ -425,7 +425,7 @@ built-in functions in the installation script. Which folders are available depends on the exact Windows version, and probably also the configuration. For details refer to Microsoft's documentation of the - :cfunc:`SHGetSpecialFolderPath` function. + :c:func:`SHGetSpecialFolderPath` function. .. function:: create_shortcut(target, description, filename[, arguments[, workdir[, iconpath[, iconindex]]]]) diff --git a/Doc/distutils/commandref.rst b/Doc/distutils/commandref.rst index fbe40de6c2..6a2ac960f1 100644 --- a/Doc/distutils/commandref.rst +++ b/Doc/distutils/commandref.rst @@ -53,7 +53,7 @@ This command installs all (Python) scripts in the distribution. Creating a source distribution: the :command:`sdist` command ============================================================ -**\*\*** fragment moved down from above: needs context! **\*\*** +.. XXX fragment moved down from above: needs context! The manifest template commands are: @@ -90,7 +90,7 @@ character, and ``[range]`` matches any of the characters in *range* (e.g., character" is platform-specific: on Unix it is anything except slash; on Windows anything except backslash or colon. -**\*\*** Windows support not there yet **\*\*** +.. XXX Windows support not there yet .. % \section{Creating a built distribution: the .. % \protect\command{bdist} command family} diff --git a/Doc/distutils/examples.rst b/Doc/distutils/examples.rst index e8d1ec9937..b268486398 100644 --- a/Doc/distutils/examples.rst +++ b/Doc/distutils/examples.rst @@ -258,9 +258,8 @@ Running the ``check`` command will display some warnings:: If you use the reStructuredText syntax in the ``long_description`` field and -`docutils <http://docutils.sourceforge.net/>`_ is installed you can check if -the syntax is fine with the ``check`` command, using the ``restructuredtext`` -option. +`docutils`_ is installed you can check if the syntax is fine with the +``check`` command, using the ``restructuredtext`` option. For example, if the :file:`setup.py` script is changed like this:: @@ -291,3 +290,4 @@ by using the :mod:`docutils` parser:: .. % \section{Putting it all together} +.. _docutils: http://docutils.sourceforge.net diff --git a/Doc/distutils/introduction.rst b/Doc/distutils/introduction.rst index 8dc604d0d0..0ece646c34 100644 --- a/Doc/distutils/introduction.rst +++ b/Doc/distutils/introduction.rst @@ -79,11 +79,17 @@ Some observations: for an example) To create a source distribution for this module, you would create a setup -script, :file:`setup.py`, containing the above code, and run:: +script, :file:`setup.py`, containing the above code, and run this command from a +terminal:: python setup.py sdist -which will create an archive file (e.g., tarball on Unix, ZIP file on Windows) +For Windows, open a command prompt window (:menuselection:`Start --> +Accessories`) and change the command to:: + + setup.py sdist + +:command:`sdist` will create an archive file (e.g., tarball on Unix, ZIP file on Windows) containing your setup script :file:`setup.py`, and your module :file:`foo.py`. The archive file will be named :file:`foo-1.0.tar.gz` (or :file:`.zip`), and will unpack into a directory :file:`foo-1.0`. diff --git a/Doc/distutils/setupscript.rst b/Doc/distutils/setupscript.rst index 9208e36995..80292432a9 100644 --- a/Doc/distutils/setupscript.rst +++ b/Doc/distutils/setupscript.rst @@ -72,7 +72,7 @@ Thus, when you say ``packages = ['foo']`` in your setup script, you are promising that the Distutils will find a file :file:`foo/__init__.py` (which might be spelled differently on your system, but you get the idea) relative to the directory where your setup script lives. If you break this promise, the -Distutils will issue a warning but still process the broken package anyways. +Distutils will issue a warning but still process the broken package anyway. If you use a different convention to lay out your source directory, that's no problem: you just have to supply the :option:`package_dir` option to tell the @@ -254,7 +254,7 @@ code: it's probably better to write C code like :: If you need to include header files from some other Python extension, you can take advantage of the fact that header files are installed in a consistent way -by the Distutils :command:`install_header` command. For example, the Numerical +by the Distutils :command:`install_headers` command. For example, the Numerical Python header files are installed (on a standard Unix installation) to :file:`/usr/local/include/python1.5/Numerical`. (The exact location will differ according to your platform and Python installation.) Since the Python include diff --git a/Doc/distutils/sourcedist.rst b/Doc/distutils/sourcedist.rst index 2dea83d06c..1666436be0 100644 --- a/Doc/distutils/sourcedist.rst +++ b/Doc/distutils/sourcedist.rst @@ -68,10 +68,10 @@ source distribution: :option:`packages` options * all C source files mentioned in the :option:`ext_modules` or - :option:`libraries` options + :option:`libraries` options ( - .. XXX Getting C library sources is currently broken -- no - :meth:`get_source_files` method in :file:`build_clib.py`! + .. XXX getting C library sources currently broken---no + :meth:`get_source_files` method in :file:`build_clib.py`! * scripts identified by the :option:`scripts` option See :ref:`distutils-installing-scripts`. @@ -103,10 +103,20 @@ per line, regular files (or symlinks to them) only. If you do supply your own :file:`MANIFEST`, you must specify everything: the default set of files described above does not apply in this case. -.. versionadded:: 3.1 +.. versionchanged:: 3.1 + An existing generated :file:`MANIFEST` will be regenerated without + :command:`sdist` comparing its modification time to the one of + :file:`MANIFEST.in` or :file:`setup.py`. + +.. versionchanged:: 3.1.3 :file:`MANIFEST` files start with a comment indicating they are generated. Files without this comment are not overwritten or removed. +.. versionchanged:: 3.2.2 + :command:`sdist` will read a :file:`MANIFEST` file if no :file:`MANIFEST.in` + exists, like it used to do. + + The manifest template has one command per line, where each command specifies a set of files to include or exclude from the source distribution. For an example, again we turn to the Distutils' own manifest template:: @@ -185,8 +195,12 @@ Manifest-related options The normal course of operations for the :command:`sdist` command is as follows: -* if the manifest file, :file:`MANIFEST` doesn't exist, read :file:`MANIFEST.in` - and create the manifest +* if the manifest file (:file:`MANIFEST` by default) exists and the first line + does not have a comment indicating it is generated from :file:`MANIFEST.in`, + then it is used as is, unaltered + +* if the manifest file doesn't exist or has been previously automatically + generated, read :file:`MANIFEST.in` and create the manifest * if neither :file:`MANIFEST` nor :file:`MANIFEST.in` exist, create a manifest with just the default file set @@ -204,8 +218,3 @@ distribution:: python setup.py sdist --manifest-only :option:`-o` is a shortcut for :option:`--manifest-only`. - -.. versionchanged:: 3.1 - An existing generated :file:`MANIFEST` will be regenerated without - :command:`sdist` comparing its modification time to the one of - :file:`MANIFEST.in` or :file:`setup.py`. diff --git a/Doc/distutils/uploading.rst b/Doc/distutils/uploading.rst index fd0c50873c..1b3cb589ec 100644 --- a/Doc/distutils/uploading.rst +++ b/Doc/distutils/uploading.rst @@ -67,9 +67,10 @@ In that case, :file:`README.txt` is a regular reStructuredText text file located in the root of the package besides :file:`setup.py`. To prevent registering broken reStructuredText content, you can use the -:program:`rst2html` program that is provided by the :mod:`docutils` package -and check the ``long_description`` from the command line:: +:program:`rst2html` program that is provided by the :mod:`docutils` package and +check the ``long_description`` from the command line:: $ python setup.py --long-description | rst2html.py > output.html -:mod:`docutils` will display a warning if there's something wrong with your syntax. +:mod:`docutils` will display a warning if there's something wrong with your +syntax. diff --git a/Doc/documenting/building.rst b/Doc/documenting/building.rst deleted file mode 100644 index 9ab25196df..0000000000 --- a/Doc/documenting/building.rst +++ /dev/null @@ -1,91 +0,0 @@ -Building the documentation -========================== - -You need to have Python 2.4 or higher installed; the toolset used to build the -docs is written in Python. It is called *Sphinx*, it is not included in this -tree, but maintained separately. Also needed are the docutils, supplying the -base markup that Sphinx uses, Jinja, a templating engine, and optionally -Pygments, a code highlighter. - - -Using make ----------- - -Luckily, a Makefile has been prepared so that on Unix, provided you have -installed Python and Subversion, you can just run :: - - make html - -to check out the necessary toolset in the `tools/` subdirectory and build the -HTML output files. To view the generated HTML, point your favorite browser at -the top-level index `build/html/index.html` after running "make". - -Available make targets are: - - * "html", which builds standalone HTML files for offline viewing. - - * "htmlhelp", which builds HTML files and a HTML Help project file usable to - convert them into a single Compiled HTML (.chm) file -- these are popular - under Microsoft Windows, but very handy on every platform. - - To create the CHM file, you need to run the Microsoft HTML Help Workshop - over the generated project (.hhp) file. - - * "latex", which builds LaTeX source files as input to "pdflatex" to produce - PDF documents. - - * "text", which builds a plain text file for each source file. - - * "linkcheck", which checks all external references to see whether they are - broken, redirected or malformed, and outputs this information to stdout - as well as a plain-text (.txt) file. - - * "changes", which builds an overview over all versionadded/versionchanged/ - deprecated items in the current version. This is meant as a help for the - writer of the "What's New" document. - - * "coverage", which builds a coverage overview for standard library modules - and C API. - - * "pydoc-topics", which builds a Python module containing a dictionary with - plain text documentation for the labels defined in - `tools/sphinxext/pyspecific.py` -- pydoc needs these to show topic and - keyword help. - -A "make update" updates the Subversion checkouts in `tools/`. - - -Without make ------------- - -You'll need to install the Sphinx package, either by checking it out via :: - - svn co http://svn.python.org/projects/external/Sphinx-0.6.5/sphinx tools/sphinx - -or by installing it from PyPI. - -Then, you need to install Docutils, either by checking it out via :: - - svn co http://svn.python.org/projects/external/docutils-0.6/docutils tools/docutils - -or by installing it from http://docutils.sf.net/. - -You also need Jinja2, either by checking it out via :: - - svn co http://svn.python.org/projects/external/Jinja-2.3.1/jinja2 tools/jinja2 - -or by installing it from PyPI. - -You can optionally also install Pygments, either as a checkout via :: - - svn co http://svn.python.org/projects/external/Pygments-1.3.1/pygments tools/pygments - -or from PyPI at http://pypi.python.org/pypi/Pygments. - - -Then, make an output directory, e.g. under `build/`, and run :: - - python tools/sphinx-build.py -b<builder> . build/<outputdirectory> - -where `<builder>` is one of html, text, latex, or htmlhelp (for explanations see -the make targets above). diff --git a/Doc/documenting/fromlatex.rst b/Doc/documenting/fromlatex.rst deleted file mode 100644 index 116524ac55..0000000000 --- a/Doc/documenting/fromlatex.rst +++ /dev/null @@ -1,202 +0,0 @@ -.. highlightlang:: rest - -Differences to the LaTeX markup -=============================== - -Though the markup language is different, most of the concepts and markup types -of the old LaTeX docs have been kept -- environments as reST directives, inline -commands as reST roles and so forth. - -However, there are some differences in the way these work, partly due to the -differences in the markup languages, partly due to improvements in Sphinx. This -section lists these differences, in order to give those familiar with the old -format a quick overview of what they might run into. - -Inline markup -------------- - -These changes have been made to inline markup: - -* **Cross-reference roles** - - Most of the following semantic roles existed previously as inline commands, - but didn't do anything except formatting the content as code. Now, they - cross-reference to known targets (some names have also been shortened): - - | *mod* (previously *refmodule* or *module*) - | *func* (previously *function*) - | *data* (new) - | *const* - | *class* - | *meth* (previously *method*) - | *attr* (previously *member*) - | *exc* (previously *exception*) - | *cdata* - | *cfunc* (previously *cfunction*) - | *cmacro* (previously *csimplemacro*) - | *ctype* - - Also different is the handling of *func* and *meth*: while previously - parentheses were added to the callable name (like ``\func{str()}``), they are - now appended by the build system -- appending them in the source will result - in double parentheses. This also means that ``:func:`str(object)``` will not - work as expected -- use ````str(object)```` instead! - -* **Inline commands implemented as directives** - - These were inline commands in LaTeX, but are now directives in reST: - - | *deprecated* - | *versionadded* - | *versionchanged* - - These are used like so:: - - .. deprecated:: 2.5 - Reason of deprecation. - - Also, no period is appended to the text for *versionadded* and - *versionchanged*. - - | *note* - | *warning* - - These are used like so:: - - .. note:: - - Content of note. - -* **Otherwise changed commands** - - The *samp* command previously formatted code and added quotation marks around - it. The *samp* role, however, features a new highlighting system just like - *file* does: - - ``:samp:`open({filename}, {mode})``` results in :samp:`open({filename}, {mode})` - -* **Dropped commands** - - These were commands in LaTeX, but are not available as roles: - - | *bfcode* - | *character* (use :samp:`\`\`'c'\`\``) - | *citetitle* (use ```Title <URL>`_``) - | *code* (use ````code````) - | *email* (just write the address in body text) - | *filenq* - | *filevar* (use the ``{...}`` highlighting feature of *file*) - | *programopt*, *longprogramopt* (use *option*) - | *ulink* (use ```Title <URL>`_``) - | *url* (just write the URL in body text) - | *var* (use ``*var*``) - | *infinity*, *plusminus* (use the Unicode character) - | *shortversion*, *version* (use the ``|version|`` and ``|release|`` substitutions) - | *emph*, *strong* (use the reST markup) - -* **Backslash escaping** - - In reST, a backslash must be escaped in normal text, and in the content of - roles. However, in code literals and literal blocks, it must not be escaped. - Example: ``:file:`C:\\Temp\\my.tmp``` vs. ````open("C:\Temp\my.tmp")````. - - -Information units ------------------ - -Information units (*...desc* environments) have been made reST directives. -These changes to information units should be noted: - -* **New names** - - "desc" has been removed from every name. Additionally, these directives have - new names: - - | *cfunction* (previously *cfuncdesc*) - | *cmacro* (previously *csimplemacrodesc*) - | *exception* (previously *excdesc*) - | *function* (previously *funcdesc*) - | *attribute* (previously *memberdesc*) - - The *classdesc\** and *excclassdesc* environments have been dropped, the - *class* and *exception* directives support classes documented with and without - constructor arguments. - -* **Multiple objects** - - The equivalent of the *...line* commands is:: - - .. function:: do_foo(bar) - do_bar(baz) - - Description of the functions. - - IOW, just give one signatures per line, at the same indentation level. - -* **Arguments** - - There is no *optional* command. Just give function signatures like they - should appear in the output:: - - .. function:: open(filename[, mode[, buffering]]) - - Description. - - Note: markup in the signature is not supported. - -* **Indexing** - - The *...descni* environments have been dropped. To mark an information unit - as unsuitable for index entry generation, use the *noindex* option like so:: - - .. function:: foo_* - :noindex: - - Description. - -* **New information units** - - There are new generic information units: One is called "describe" and can be - used to document things that are not covered by the other units:: - - .. describe:: a == b - - The equals operator. - - The others are:: - - .. cmdoption:: -O - - Describes a command-line option. - - .. envvar:: PYTHONINSPECT - - Describes an environment variable. - - -Structure ---------- - -The LaTeX docs were split in several toplevel manuals. Now, all files are part -of the same documentation tree, as indicated by the *toctree* directives in the -sources (though individual output formats may choose to split them up into parts -again). Every *toctree* directive embeds other files as subdocuments of the -current file (this structure is not necessarily mirrored in the filesystem -layout). The toplevel file is :file:`contents.rst`. - -However, most of the old directory structure has been kept, with the -directories renamed as follows: - -* :file:`api` -> :file:`c-api` -* :file:`dist` -> :file:`distutils`, with the single TeX file split up -* :file:`doc` -> :file:`documenting` -* :file:`ext` -> :file:`extending` -* :file:`inst` -> :file:`installing` -* :file:`lib` -> :file:`library` -* :file:`mac` -> merged into :file:`library`, with :file:`mac/using.tex` - moved to :file:`using/mac.rst` -* :file:`ref` -> :file:`reference` -* :file:`tut` -> :file:`tutorial`, with the single TeX file split up - - -.. XXX more (index-generating, production lists, ...) diff --git a/Doc/documenting/index.rst b/Doc/documenting/index.rst deleted file mode 100644 index 2c186a7bbb..0000000000 --- a/Doc/documenting/index.rst +++ /dev/null @@ -1,38 +0,0 @@ -.. _documenting-index: - -###################### - Documenting Python -###################### - - -The Python language has a substantial body of documentation, much of it -contributed by various authors. The markup used for the Python documentation is -`reStructuredText`_, developed by the `docutils`_ project, amended by custom -directives and using a toolset named `Sphinx`_ to postprocess the HTML output. - -This document describes the style guide for our documentation as well as the -custom reStructuredText markup introduced by Sphinx to support Python -documentation and how it should be used. - -.. _reStructuredText: http://docutils.sf.net/rst.html -.. _docutils: http://docutils.sf.net/ -.. _Sphinx: http://sphinx.pocoo.org/ - -.. note:: - - If you're interested in contributing to Python's documentation, there's no - need to write reStructuredText if you're not so inclined; plain text - contributions are more than welcome as well. Send an e-mail to - docs@python.org or open an issue on the :ref:`tracker <reporting-bugs>`. - - -.. toctree:: - :numbered: - :maxdepth: 1 - - intro.rst - style.rst - rest.rst - markup.rst - fromlatex.rst - building.rst diff --git a/Doc/documenting/intro.rst b/Doc/documenting/intro.rst deleted file mode 100644 index e02ad7de22..0000000000 --- a/Doc/documenting/intro.rst +++ /dev/null @@ -1,29 +0,0 @@ -Introduction -============ - -Python's documentation has long been considered to be good for a free -programming language. There are a number of reasons for this, the most -important being the early commitment of Python's creator, Guido van Rossum, to -providing documentation on the language and its libraries, and the continuing -involvement of the user community in providing assistance for creating and -maintaining documentation. - -The involvement of the community takes many forms, from authoring to bug reports -to just plain complaining when the documentation could be more complete or -easier to use. - -This document is aimed at authors and potential authors of documentation for -Python. More specifically, it is for people contributing to the standard -documentation and developing additional documents using the same tools as the -standard documents. This guide will be less useful for authors using the Python -documentation tools for topics other than Python, and less useful still for -authors not using the tools at all. - -If your interest is in contributing to the Python documentation, but you don't -have the time or inclination to learn reStructuredText and the markup structures -documented here, there's a welcoming place for you among the Python contributors -as well. Any time you feel that you can clarify existing documentation or -provide documentation that's missing, the existing documentation team will -gladly work with you to integrate your text, dealing with the markup for you. -Please don't let the material in this document stand between the documentation -and your desire to help out!
\ No newline at end of file diff --git a/Doc/documenting/markup.rst b/Doc/documenting/markup.rst deleted file mode 100644 index a2b91238a9..0000000000 --- a/Doc/documenting/markup.rst +++ /dev/null @@ -1,861 +0,0 @@ -.. highlightlang:: rest - -Additional Markup Constructs -============================ - -Sphinx adds a lot of new directives and interpreted text roles to standard reST -markup. This section contains the reference material for these facilities. -Documentation for "standard" reST constructs is not included here, though -they are used in the Python documentation. - -.. note:: - - This is just an overview of Sphinx' extended markup capabilities; full - coverage can be found in `its own documentation - <http://sphinx.pocoo.org/contents.html>`_. - - -Meta-information markup ------------------------ - -.. describe:: sectionauthor - - Identifies the author of the current section. The argument should include - the author's name such that it can be used for presentation (though it isn't) - and email address. The domain name portion of the address should be lower - case. Example:: - - .. sectionauthor:: Guido van Rossum <guido@python.org> - - Currently, this markup isn't reflected in the output in any way, but it helps - keep track of contributions. - - -Module-specific markup ----------------------- - -The markup described in this section is used to provide information about a -module being documented. Each module should be documented in its own file. -Normally this markup appears after the title heading of that file; a typical -file might start like this:: - - :mod:`parrot` -- Dead parrot access - =================================== - - .. module:: parrot - :platform: Unix, Windows - :synopsis: Analyze and reanimate dead parrots. - .. moduleauthor:: Eric Cleese <eric@python.invalid> - .. moduleauthor:: John Idle <john@python.invalid> - -As you can see, the module-specific markup consists of two directives, the -``module`` directive and the ``moduleauthor`` directive. - -.. describe:: module - - This directive marks the beginning of the description of a module, package, - or submodule. The name should be fully qualified (i.e. including the - package name for submodules). - - The ``platform`` option, if present, is a comma-separated list of the - platforms on which the module is available (if it is available on all - platforms, the option should be omitted). The keys are short identifiers; - examples that are in use include "IRIX", "Mac", "Windows", and "Unix". It is - important to use a key which has already been used when applicable. - - The ``synopsis`` option should consist of one sentence describing the - module's purpose -- it is currently only used in the Global Module Index. - - The ``deprecated`` option can be given (with no value) to mark a module as - deprecated; it will be designated as such in various locations then. - -.. describe:: moduleauthor - - The ``moduleauthor`` directive, which can appear multiple times, names the - authors of the module code, just like ``sectionauthor`` names the author(s) - of a piece of documentation. It too does not result in any output currently. - -.. note:: - - It is important to make the section title of a module-describing file - meaningful since that value will be inserted in the table-of-contents trees - in overview files. - - -Information units ------------------ - -There are a number of directives used to describe specific features provided by -modules. Each directive requires one or more signatures to provide basic -information about what is being described, and the content should be the -description. The basic version makes entries in the general index; if no index -entry is desired, you can give the directive option flag ``:noindex:``. The -following example shows all of the features of this directive type:: - - .. function:: spam(eggs) - ham(eggs) - :noindex: - - Spam or ham the foo. - -The signatures of object methods or data attributes should always include the -type name (``.. method:: FileInput.input(...)``), even if it is obvious from the -context which type they belong to; this is to enable consistent -cross-references. If you describe methods belonging to an abstract protocol, -such as "context managers", include a (pseudo-)type name too to make the -index entries more informative. - -The directives are: - -.. describe:: cfunction - - Describes a C function. The signature should be given as in C, e.g.:: - - .. cfunction:: PyObject* PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems) - - This is also used to describe function-like preprocessor macros. The names - of the arguments should be given so they may be used in the description. - - Note that you don't have to backslash-escape asterisks in the signature, - as it is not parsed by the reST inliner. - -.. describe:: cmember - - Describes a C struct member. Example signature:: - - .. cmember:: PyObject* PyTypeObject.tp_bases - - The text of the description should include the range of values allowed, how - the value should be interpreted, and whether the value can be changed. - References to structure members in text should use the ``member`` role. - -.. describe:: cmacro - - Describes a "simple" C macro. Simple macros are macros which are used - for code expansion, but which do not take arguments so cannot be described as - functions. This is not to be used for simple constant definitions. Examples - of its use in the Python documentation include :cmacro:`PyObject_HEAD` and - :cmacro:`Py_BEGIN_ALLOW_THREADS`. - -.. describe:: ctype - - Describes a C type. The signature should just be the type name. - -.. describe:: cvar - - Describes a global C variable. The signature should include the type, such - as:: - - .. cvar:: PyObject* PyClass_Type - -.. describe:: data - - Describes global data in a module, including both variables and values used - as "defined constants." Class and object attributes are not documented - using this directive. - -.. describe:: exception - - Describes an exception class. The signature can, but need not include - parentheses with constructor arguments. - -.. describe:: function - - Describes a module-level function. The signature should include the - parameters, enclosing optional parameters in brackets. Default values can be - given if it enhances clarity. For example:: - - .. function:: repeat([repeat=3[, number=1000000]]) - - Object methods are not documented using this directive. Bound object methods - placed in the module namespace as part of the public interface of the module - are documented using this, as they are equivalent to normal functions for - most purposes. - - The description should include information about the parameters required and - how they are used (especially whether mutable objects passed as parameters - are modified), side effects, and possible exceptions. A small example may be - provided. - -.. describe:: class - - Describes a class. The signature can include parentheses with parameters - which will be shown as the constructor arguments. - -.. describe:: attribute - - Describes an object data attribute. The description should include - information about the type of the data to be expected and whether it may be - changed directly. This directive should be nested in a class directive, - like in this example:: - - .. class:: Spam - - Description of the class. - - .. data:: ham - - Description of the attribute. - - If is also possible to document an attribute outside of a class directive, - for example if the documentation for different attributes and methods is - split in multiple sections. The class name should then be included - explicitly:: - - .. data:: Spam.eggs - -.. describe:: method - - Describes an object method. The parameters should not include the ``self`` - parameter. The description should include similar information to that - described for ``function``. This directive should be nested in a class - directive, like in the example above. - -.. describe:: opcode - - Describes a Python :term:`bytecode` instruction. - -.. describe:: cmdoption - - Describes a Python command line option or switch. Option argument names - should be enclosed in angle brackets. Example:: - - .. cmdoption:: -m <module> - - Run a module as a script. - -.. describe:: envvar - - Describes an environment variable that Python uses or defines. - - -There is also a generic version of these directives: - -.. describe:: describe - - This directive produces the same formatting as the specific ones explained - above but does not create index entries or cross-referencing targets. It is - used, for example, to describe the directives in this document. Example:: - - .. describe:: opcode - - Describes a Python bytecode instruction. - - -Showing code examples ---------------------- - -Examples of Python source code or interactive sessions are represented using -standard reST literal blocks. They are started by a ``::`` at the end of the -preceding paragraph and delimited by indentation. - -Representing an interactive session requires including the prompts and output -along with the Python code. No special markup is required for interactive -sessions. After the last line of input or output presented, there should not be -an "unused" primary prompt; this is an example of what *not* to do:: - - >>> 1 + 1 - 2 - >>> - -Syntax highlighting is handled in a smart way: - -* There is a "highlighting language" for each source file. Per default, - this is ``'python'`` as the majority of files will have to highlight Python - snippets. - -* Within Python highlighting mode, interactive sessions are recognized - automatically and highlighted appropriately. - -* The highlighting language can be changed using the ``highlightlang`` - directive, used as follows:: - - .. highlightlang:: c - - This language is used until the next ``highlightlang`` directive is - encountered. - -* The values normally used for the highlighting language are: - - * ``python`` (the default) - * ``c`` - * ``rest`` - * ``none`` (no highlighting) - -* If highlighting with the current language fails, the block is not highlighted - in any way. - -Longer displays of verbatim text may be included by storing the example text in -an external file containing only plain text. The file may be included using the -``literalinclude`` directive. [1]_ For example, to include the Python source file -:file:`example.py`, use:: - - .. literalinclude:: example.py - -The file name is relative to the current file's path. Documentation-specific -include files should be placed in the ``Doc/includes`` subdirectory. - - -Inline markup -------------- - -As said before, Sphinx uses interpreted text roles to insert semantic markup in -documents. - -Names of local variables, such as function/method arguments, are an exception, -they should be marked simply with ``*var*``. - -For all other roles, you have to write ``:rolename:`content```. - -There are some additional facilities that make cross-referencing roles more -versatile: - -* You may supply an explicit title and reference target, like in reST direct - hyperlinks: ``:role:`title <target>``` will refer to *target*, but the link - text will be *title*. - -* If you prefix the content with ``!``, no reference/hyperlink will be created. - -* For the Python object roles, if you prefix the content with ``~``, the link - text will only be the last component of the target. For example, - ``:meth:`~Queue.Queue.get``` will refer to ``Queue.Queue.get`` but only - display ``get`` as the link text. - - In HTML output, the link's ``title`` attribute (that is e.g. shown as a - tool-tip on mouse-hover) will always be the full target name. - -The following roles refer to objects in modules and are possibly hyperlinked if -a matching identifier is found: - -.. describe:: mod - - The name of a module; a dotted name may be used. This should also be used for - package names. - -.. describe:: func - - The name of a Python function; dotted names may be used. The role text - should not include trailing parentheses to enhance readability. The - parentheses are stripped when searching for identifiers. - -.. describe:: data - - The name of a module-level variable or constant. - -.. describe:: const - - The name of a "defined" constant. This may be a C-language ``#define`` - or a Python variable that is not intended to be changed. - -.. describe:: class - - A class name; a dotted name may be used. - -.. describe:: meth - - The name of a method of an object. The role text should include the type - name and the method name. A dotted name may be used. - -.. describe:: attr - - The name of a data attribute of an object. - -.. describe:: exc - - The name of an exception. A dotted name may be used. - -The name enclosed in this markup can include a module name and/or a class name. -For example, ``:func:`filter``` could refer to a function named ``filter`` in -the current module, or the built-in function of that name. In contrast, -``:func:`foo.filter``` clearly refers to the ``filter`` function in the ``foo`` -module. - -Normally, names in these roles are searched first without any further -qualification, then with the current module name prepended, then with the -current module and class name (if any) prepended. If you prefix the name with a -dot, this order is reversed. For example, in the documentation of the -:mod:`codecs` module, ``:func:`open``` always refers to the built-in function, -while ``:func:`.open``` refers to :func:`codecs.open`. - -A similar heuristic is used to determine whether the name is an attribute of -the currently documented class. - -The following roles create cross-references to C-language constructs if they -are defined in the API documentation: - -.. describe:: cdata - - The name of a C-language variable. - -.. describe:: cfunc - - The name of a C-language function. Should include trailing parentheses. - -.. describe:: cmacro - - The name of a "simple" C macro, as defined above. - -.. describe:: ctype - - The name of a C-language type. - - -The following role does possibly create a cross-reference, but does not refer -to objects: - -.. describe:: token - - The name of a grammar token (used in the reference manual to create links - between production displays). - - -The following role creates a cross-reference to the term in the glossary: - -.. describe:: term - - Reference to a term in the glossary. The glossary is created using the - ``glossary`` directive containing a definition list with terms and - definitions. It does not have to be in the same file as the ``term`` - markup, in fact, by default the Python docs have one global glossary - in the ``glossary.rst`` file. - - If you use a term that's not explained in a glossary, you'll get a warning - during build. - ---------- - -The following roles don't do anything special except formatting the text -in a different style: - -.. describe:: command - - The name of an OS-level command, such as ``rm``. - -.. describe:: dfn - - Mark the defining instance of a term in the text. (No index entries are - generated.) - -.. describe:: envvar - - An environment variable. Index entries are generated. - -.. describe:: file - - The name of a file or directory. Within the contents, you can use curly - braces to indicate a "variable" part, for example:: - - ... is installed in :file:`/usr/lib/python2.{x}/site-packages` ... - - In the built documentation, the ``x`` will be displayed differently to - indicate that it is to be replaced by the Python minor version. - -.. describe:: guilabel - - Labels presented as part of an interactive user interface should be marked - using ``guilabel``. This includes labels from text-based interfaces such as - those created using :mod:`curses` or other text-based libraries. Any label - used in the interface should be marked with this role, including button - labels, window titles, field names, menu and menu selection names, and even - values in selection lists. - -.. describe:: kbd - - Mark a sequence of keystrokes. What form the key sequence takes may depend - on platform- or application-specific conventions. When there are no relevant - conventions, the names of modifier keys should be spelled out, to improve - accessibility for new users and non-native speakers. For example, an - *xemacs* key sequence may be marked like ``:kbd:`C-x C-f```, but without - reference to a specific application or platform, the same sequence should be - marked as ``:kbd:`Control-x Control-f```. - -.. describe:: keyword - - The name of a keyword in Python. - -.. describe:: mailheader - - The name of an RFC 822-style mail header. This markup does not imply that - the header is being used in an email message, but can be used to refer to any - header of the same "style." This is also used for headers defined by the - various MIME specifications. The header name should be entered in the same - way it would normally be found in practice, with the camel-casing conventions - being preferred where there is more than one common usage. For example: - ``:mailheader:`Content-Type```. - -.. describe:: makevar - - The name of a :command:`make` variable. - -.. describe:: manpage - - A reference to a Unix manual page including the section, - e.g. ``:manpage:`ls(1)```. - -.. describe:: menuselection - - Menu selections should be marked using the ``menuselection`` role. This is - used to mark a complete sequence of menu selections, including selecting - submenus and choosing a specific operation, or any subsequence of such a - sequence. The names of individual selections should be separated by - ``-->``. - - For example, to mark the selection "Start > Programs", use this markup:: - - :menuselection:`Start --> Programs` - - When including a selection that includes some trailing indicator, such as the - ellipsis some operating systems use to indicate that the command opens a - dialog, the indicator should be omitted from the selection name. - -.. describe:: mimetype - - The name of a MIME type, or a component of a MIME type (the major or minor - portion, taken alone). - -.. describe:: newsgroup - - The name of a Usenet newsgroup. - -.. describe:: option - - A command-line option of Python. The leading hyphen(s) must be included. - If a matching ``cmdoption`` directive exists, it is linked to. For options - of other programs or scripts, use simple ````code```` markup. - -.. describe:: program - - The name of an executable program. This may differ from the file name for - the executable for some platforms. In particular, the ``.exe`` (or other) - extension should be omitted for Windows programs. - -.. describe:: regexp - - A regular expression. Quotes should not be included. - -.. describe:: samp - - A piece of literal text, such as code. Within the contents, you can use - curly braces to indicate a "variable" part, as in ``:file:``. - - If you don't need the "variable part" indication, use the standard - ````code```` instead. - -.. describe:: var - - A Python or C variable or parameter name. - - -The following roles generate external links: - -.. describe:: pep - - A reference to a Python Enhancement Proposal. This generates appropriate - index entries. The text "PEP *number*\ " is generated; in the HTML output, - this text is a hyperlink to an online copy of the specified PEP. - -.. describe:: rfc - - A reference to an Internet Request for Comments. This generates appropriate - index entries. The text "RFC *number*\ " is generated; in the HTML output, - this text is a hyperlink to an online copy of the specified RFC. - - -Note that there are no special roles for including hyperlinks as you can use -the standard reST markup for that purpose. - - -.. _doc-ref-role: - -Cross-linking markup --------------------- - -To support cross-referencing to arbitrary sections in the documentation, the -standard reST labels are "abused" a bit: Every label must precede a section -title; and every label name must be unique throughout the entire documentation -source. - -You can then reference to these sections using the ``:ref:`label-name``` role. - -Example:: - - .. _my-reference-label: - - Section to cross-reference - -------------------------- - - This is the text of the section. - - It refers to the section itself, see :ref:`my-reference-label`. - -The ``:ref:`` invocation is replaced with the section title. - - -Paragraph-level markup ----------------------- - -These directives create short paragraphs and can be used inside information -units as well as normal text: - -.. describe:: note - - An especially important bit of information about an API that a user should be - aware of when using whatever bit of API the note pertains to. The content of - the directive should be written in complete sentences and include all - appropriate punctuation. - - Example:: - - .. note:: - - This function is not suitable for sending spam e-mails. - -.. describe:: warning - - An important bit of information about an API that a user should be aware of - when using whatever bit of API the warning pertains to. The content of the - directive should be written in complete sentences and include all appropriate - punctuation. In the interest of not scaring users away from pages filled - with warnings, this directive should only be chosen over ``note`` for - information regarding the possibility of crashes, data loss, or security - implications. - -.. describe:: versionadded - - This directive documents the version of Python which added the described - feature to the library or C API. When this applies to an entire module, it - should be placed at the top of the module section before any prose. - - The first argument must be given and is the version in question; you can add - a second argument consisting of a *brief* explanation of the change. - - Example:: - - .. versionadded:: 3.1 - The *spam* parameter. - - Note that there must be no blank line between the directive head and the - explanation; this is to make these blocks visually continuous in the markup. - -.. describe:: versionchanged - - Similar to ``versionadded``, but describes when and what changed in the named - feature in some way (new parameters, changed side effects, etc.). - --------------- - -.. describe:: impl-detail - - This directive is used to mark CPython-specific information. Use either with - a block content or a single sentence as an argument, i.e. either :: - - .. impl-detail:: - - This describes some implementation detail. - - More explanation. - - or :: - - .. impl-detail:: This shortly mentions an implementation detail. - - "\ **CPython implementation detail:**\ " is automatically prepended to the - content. - -.. describe:: seealso - - Many sections include a list of references to module documentation or - external documents. These lists are created using the ``seealso`` directive. - - The ``seealso`` directive is typically placed in a section just before any - sub-sections. For the HTML output, it is shown boxed off from the main flow - of the text. - - The content of the ``seealso`` directive should be a reST definition list. - Example:: - - .. seealso:: - - Module :mod:`zipfile` - Documentation of the :mod:`zipfile` standard module. - - `GNU tar manual, Basic Tar Format <http://link>`_ - Documentation for tar archive files, including GNU tar extensions. - -.. describe:: rubric - - This directive creates a paragraph heading that is not used to create a - table of contents node. It is currently used for the "Footnotes" caption. - -.. describe:: centered - - This directive creates a centered boldfaced paragraph. Use it as follows:: - - .. centered:: - - Paragraph contents. - - -Table-of-contents markup ------------------------- - -Since reST does not have facilities to interconnect several documents, or split -documents into multiple output files, Sphinx uses a custom directive to add -relations between the single files the documentation is made of, as well as -tables of contents. The ``toctree`` directive is the central element. - -.. describe:: toctree - - This directive inserts a "TOC tree" at the current location, using the - individual TOCs (including "sub-TOC trees") of the files given in the - directive body. A numeric ``maxdepth`` option may be given to indicate the - depth of the tree; by default, all levels are included. - - Consider this example (taken from the library reference index):: - - .. toctree:: - :maxdepth: 2 - - intro - strings - datatypes - numeric - (many more files listed here) - - This accomplishes two things: - - * Tables of contents from all those files are inserted, with a maximum depth - of two, that means one nested heading. ``toctree`` directives in those - files are also taken into account. - * Sphinx knows that the relative order of the files ``intro``, - ``strings`` and so forth, and it knows that they are children of the - shown file, the library index. From this information it generates "next - chapter", "previous chapter" and "parent chapter" links. - - In the end, all files included in the build process must occur in one - ``toctree`` directive; Sphinx will emit a warning if it finds a file that is - not included, because that means that this file will not be reachable through - standard navigation. - - The special file ``contents.rst`` at the root of the source directory is the - "root" of the TOC tree hierarchy; from it the "Contents" page is generated. - - -Index-generating markup ------------------------ - -Sphinx automatically creates index entries from all information units (like -functions, classes or attributes) like discussed before. - -However, there is also an explicit directive available, to make the index more -comprehensive and enable index entries in documents where information is not -mainly contained in information units, such as the language reference. - -The directive is ``index`` and contains one or more index entries. Each entry -consists of a type and a value, separated by a colon. - -For example:: - - .. index:: - single: execution; context - module: __main__ - module: sys - triple: module; search; path - -This directive contains five entries, which will be converted to entries in the -generated index which link to the exact location of the index statement (or, in -case of offline media, the corresponding page number). - -The possible entry types are: - -single - Creates a single index entry. Can be made a subentry by separating the - subentry text with a semicolon (this notation is also used below to describe - what entries are created). -pair - ``pair: loop; statement`` is a shortcut that creates two index entries, - namely ``loop; statement`` and ``statement; loop``. -triple - Likewise, ``triple: module; search; path`` is a shortcut that creates three - index entries, which are ``module; search path``, ``search; path, module`` and - ``path; module search``. -module, keyword, operator, object, exception, statement, builtin - These all create two index entries. For example, ``module: hashlib`` creates - the entries ``module; hashlib`` and ``hashlib; module``. - -For index directives containing only "single" entries, there is a shorthand -notation:: - - .. index:: BNF, grammar, syntax, notation - -This creates four index entries. - - -Grammar production displays ---------------------------- - -Special markup is available for displaying the productions of a formal grammar. -The markup is simple and does not attempt to model all aspects of BNF (or any -derived forms), but provides enough to allow context-free grammars to be -displayed in a way that causes uses of a symbol to be rendered as hyperlinks to -the definition of the symbol. There is this directive: - -.. describe:: productionlist - - This directive is used to enclose a group of productions. Each production is - given on a single line and consists of a name, separated by a colon from the - following definition. If the definition spans multiple lines, each - continuation line must begin with a colon placed at the same column as in the - first line. - - Blank lines are not allowed within ``productionlist`` directive arguments. - - The definition can contain token names which are marked as interpreted text - (e.g. ``unaryneg ::= "-" `integer```) -- this generates cross-references - to the productions of these tokens. - - Note that no further reST parsing is done in the production, so that you - don't have to escape ``*`` or ``|`` characters. - - -.. XXX describe optional first parameter - -The following is an example taken from the Python Reference Manual:: - - .. productionlist:: - try_stmt: try1_stmt | try2_stmt - try1_stmt: "try" ":" `suite` - : ("except" [`expression` ["," `target`]] ":" `suite`)+ - : ["else" ":" `suite`] - : ["finally" ":" `suite`] - try2_stmt: "try" ":" `suite` - : "finally" ":" `suite` - - -Substitutions -------------- - -The documentation system provides three substitutions that are defined by default. -They are set in the build configuration file :file:`conf.py`. - -.. describe:: |release| - - Replaced by the Python release the documentation refers to. This is the full - version string including alpha/beta/release candidate tags, e.g. ``2.5.2b3``. - -.. describe:: |version| - - Replaced by the Python version the documentation refers to. This consists - only of the major and minor version parts, e.g. ``2.5``, even for version - 2.5.1. - -.. describe:: |today| - - Replaced by either today's date, or the date set in the build configuration - file. Normally has the format ``April 14, 2007``. - - -.. rubric:: Footnotes - -.. [1] There is a standard ``.. include`` directive, but it raises errors if the - file is not found. This one only emits a warning. diff --git a/Doc/documenting/rest.rst b/Doc/documenting/rest.rst deleted file mode 100644 index 9b6b89b8fa..0000000000 --- a/Doc/documenting/rest.rst +++ /dev/null @@ -1,243 +0,0 @@ -.. highlightlang:: rest - -reStructuredText Primer -======================= - -This section is a brief introduction to reStructuredText (reST) concepts and -syntax, intended to provide authors with enough information to author documents -productively. Since reST was designed to be a simple, unobtrusive markup -language, this will not take too long. - -.. seealso:: - - The authoritative `reStructuredText User - Documentation <http://docutils.sourceforge.net/rst.html>`_. - - -Paragraphs ----------- - -The paragraph is the most basic block in a reST document. Paragraphs are simply -chunks of text separated by one or more blank lines. As in Python, indentation -is significant in reST, so all lines of the same paragraph must be left-aligned -to the same level of indentation. - - -Inline markup -------------- - -The standard reST inline markup is quite simple: use - -* one asterisk: ``*text*`` for emphasis (italics), -* two asterisks: ``**text**`` for strong emphasis (boldface), and -* backquotes: ````text```` for code samples. - -If asterisks or backquotes appear in running text and could be confused with -inline markup delimiters, they have to be escaped with a backslash. - -Be aware of some restrictions of this markup: - -* it may not be nested, -* content may not start or end with whitespace: ``* text*`` is wrong, -* it must be separated from surrounding text by non-word characters. Use a - backslash escaped space to work around that: ``thisis\ *one*\ word``. - -These restrictions may be lifted in future versions of the docutils. - -reST also allows for custom "interpreted text roles"', which signify that the -enclosed text should be interpreted in a specific way. Sphinx uses this to -provide semantic markup and cross-referencing of identifiers, as described in -the appropriate section. The general syntax is ``:rolename:`content```. - - -Lists and Quotes ----------------- - -List markup is natural: just place an asterisk at the start of a paragraph and -indent properly. The same goes for numbered lists; they can also be -autonumbered using a ``#`` sign:: - - * This is a bulleted list. - * It has two items, the second - item uses two lines. - - 1. This is a numbered list. - 2. It has two items too. - - #. This is a numbered list. - #. It has two items too. - - -Nested lists are possible, but be aware that they must be separated from the -parent list items by blank lines:: - - * this is - * a list - - * with a nested list - * and some subitems - - * and here the parent list continues - -Definition lists are created as follows:: - - term (up to a line of text) - Definition of the term, which must be indented - - and can even consist of multiple paragraphs - - next term - Description. - - -Paragraphs are quoted by just indenting them more than the surrounding -paragraphs. - - -Source Code ------------ - -Literal code blocks are introduced by ending a paragraph with the special marker -``::``. The literal block must be indented:: - - This is a normal text paragraph. The next paragraph is a code sample:: - - It is not processed in any way, except - that the indentation is removed. - - It can span multiple lines. - - This is a normal text paragraph again. - -The handling of the ``::`` marker is smart: - -* If it occurs as a paragraph of its own, that paragraph is completely left - out of the document. -* If it is preceded by whitespace, the marker is removed. -* If it is preceded by non-whitespace, the marker is replaced by a single - colon. - -That way, the second sentence in the above example's first paragraph would be -rendered as "The next paragraph is a code sample:". - - -Hyperlinks ----------- - -External links -^^^^^^^^^^^^^^ - -Use ```Link text <http://target>`_`` for inline web links. If the link text -should be the web address, you don't need special markup at all, the parser -finds links and mail addresses in ordinary text. - -Internal links -^^^^^^^^^^^^^^ - -Internal linking is done via a special reST role, see the section on specific -markup, :ref:`doc-ref-role`. - - -Sections --------- - -Section headers are created by underlining (and optionally overlining) the -section title with a punctuation character, at least as long as the text:: - - ================= - This is a heading - ================= - -Normally, there are no heading levels assigned to certain characters as the -structure is determined from the succession of headings. However, for the -Python documentation, we use this convention: - -* ``#`` with overline, for parts -* ``*`` with overline, for chapters -* ``=``, for sections -* ``-``, for subsections -* ``^``, for subsubsections -* ``"``, for paragraphs - - -Explicit Markup ---------------- - -"Explicit markup" is used in reST for most constructs that need special -handling, such as footnotes, specially-highlighted paragraphs, comments, and -generic directives. - -An explicit markup block begins with a line starting with ``..`` followed by -whitespace and is terminated by the next paragraph at the same level of -indentation. (There needs to be a blank line between explicit markup and normal -paragraphs. This may all sound a bit complicated, but it is intuitive enough -when you write it.) - - -Directives ----------- - -A directive is a generic block of explicit markup. Besides roles, it is one of -the extension mechanisms of reST, and Sphinx makes heavy use of it. - -Basically, a directive consists of a name, arguments, options and content. (Keep -this terminology in mind, it is used in the next chapter describing custom -directives.) Looking at this example, :: - - .. function:: foo(x) - foo(y, z) - :bar: no - - Return a line of text input from the user. - -``function`` is the directive name. It is given two arguments here, the -remainder of the first line and the second line, as well as one option ``bar`` -(as you can see, options are given in the lines immediately following the -arguments and indicated by the colons). - -The directive content follows after a blank line and is indented relative to the -directive start. - - -Footnotes ---------- - -For footnotes, use ``[#]_`` to mark the footnote location, and add the footnote -body at the bottom of the document after a "Footnotes" rubric heading, like so:: - - Lorem ipsum [#]_ dolor sit amet ... [#]_ - - .. rubric:: Footnotes - - .. [#] Text of the first footnote. - .. [#] Text of the second footnote. - -You can also explicitly number the footnotes for better context. - - -Comments --------- - -Every explicit markup block which isn't a valid markup construct (like the -footnotes above) is regarded as a comment. - - -Source encoding ---------------- - -Since the easiest way to include special characters like em dashes or copyright -signs in reST is to directly write them as Unicode characters, one has to -specify an encoding: - -All Python documentation source files must be in UTF-8 encoding, and the HTML -documents written from them will be in that encoding as well. - - -Gotchas -------- - -There are some problems one commonly runs into while authoring reST documents: - -* **Separation of inline markup:** As said above, inline markup spans must be - separated from the surrounding text by non-word characters, you have to use - an escaped space to get around that. diff --git a/Doc/documenting/style.rst b/Doc/documenting/style.rst deleted file mode 100644 index c3dded9b77..0000000000 --- a/Doc/documenting/style.rst +++ /dev/null @@ -1,70 +0,0 @@ -.. highlightlang:: rest - -Style Guide -=========== - -The Python documentation should follow the `Apple Publications Style Guide`_ -wherever possible. This particular style guide was selected mostly because it -seems reasonable and is easy to get online. - -Topics which are not covered in Apple's style guide will be discussed in -this document. - -All reST files use an indentation of 3 spaces. The maximum line length is 80 -characters for normal text, but tables, deeply indented code samples and long -links may extend beyond that. - -Make generous use of blank lines where applicable; they help grouping things -together. - -A sentence-ending period may be followed by one or two spaces; while reST -ignores the second space, it is customarily put in by some users, for example -to aid Emacs' auto-fill mode. - -Footnotes are generally discouraged, though they may be used when they are the -best way to present specific information. When a footnote reference is added at -the end of the sentence, it should follow the sentence-ending punctuation. The -reST markup should appear something like this:: - - This sentence has a footnote reference. [#]_ This is the next sentence. - -Footnotes should be gathered at the end of a file, or if the file is very long, -at the end of a section. The docutils will automatically create backlinks to -the footnote reference. - -Footnotes may appear in the middle of sentences where appropriate. - -Many special names are used in the Python documentation, including the names of -operating systems, programming languages, standards bodies, and the like. Most -of these entities are not assigned any special markup, but the preferred -spellings are given here to aid authors in maintaining the consistency of -presentation in the Python documentation. - -Other terms and words deserve special mention as well; these conventions should -be used to ensure consistency throughout the documentation: - -CPU - For "central processing unit." Many style guides say this should be spelled - out on the first use (and if you must use it, do so!). For the Python - documentation, this abbreviation should be avoided since there's no - reasonable way to predict which occurrence will be the first seen by the - reader. It is better to use the word "processor" instead. - -POSIX - The name assigned to a particular group of standards. This is always - uppercase. - -Python - The name of our favorite programming language is always capitalized. - -Unicode - The name of a character set and matching encoding. This is always written - capitalized. - -Unix - The name of the operating system developed at AT&T Bell Labs in the early - 1970s. - - -.. _Apple Publications Style Guide: http://developer.apple.com/mac/library/documentation/UserExperience/Conceptual/APStyleGuide/APSG_2009.pdf - diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst index 2ea12532b0..3143c990e5 100644 --- a/Doc/extending/embedding.rst +++ b/Doc/extending/embedding.rst @@ -25,19 +25,16 @@ the Python interpreter to run some Python code. So if you are embedding Python, you are providing your own main program. One of the things this main program has to do is initialize the Python interpreter. At -the very least, you have to call the function :cfunc:`Py_Initialize`. There are +the very least, you have to call the function :c:func:`Py_Initialize`. There are optional calls to pass command line arguments to Python. Then later you can call the interpreter from any part of the application. There are several different ways to call the interpreter: you can pass a string -containing Python statements to :cfunc:`PyRun_SimpleString`, or you can pass a +containing Python statements to :c:func:`PyRun_SimpleString`, or you can pass a stdio file pointer and a file name (for identification in error messages only) -to :cfunc:`PyRun_SimpleFile`. You can also call the lower-level operations +to :c:func:`PyRun_SimpleFile`. You can also call the lower-level operations described in the previous chapters to construct and use Python objects. -A simple demo of embedding Python can be found in the directory -:file:`Demo/embed/` of the source distribution. - .. seealso:: @@ -69,12 +66,12 @@ perform some operation on a file. :: } The above code first initializes the Python interpreter with -:cfunc:`Py_Initialize`, followed by the execution of a hard-coded Python script -that print the date and time. Afterwards, the :cfunc:`Py_Finalize` call shuts +:c:func:`Py_Initialize`, followed by the execution of a hard-coded Python script +that print the date and time. Afterwards, the :c:func:`Py_Finalize` call shuts the interpreter down, followed by the end of the program. In a real program, you may want to get the Python script from another source, perhaps a text-editor routine, a file, or a database. Getting the Python code from a file can better -be done by using the :cfunc:`PyRun_SimpleFile` function, which saves you the +be done by using the :c:func:`PyRun_SimpleFile` function, which saves you the trouble of allocating memory space and loading the file contents. @@ -136,8 +133,9 @@ The code to run a function defined in a Python script is: This code loads a Python script using ``argv[1]``, and calls the function named in ``argv[2]``. Its integer arguments are the other values of the ``argv`` -array. If you compile and link this program (let's call the finished executable -:program:`call`), and use it to execute a Python script, such as:: +array. If you :ref:`compile and link <compiling>` this program (let's call +the finished executable :program:`call`), and use it to execute a Python +script, such as:: def multiply(a,b): print("Will compute", a, "times", b) @@ -162,8 +160,8 @@ interesting part with respect to embedding Python starts with :: pModule = PyImport_Import(pName); After initializing the interpreter, the script is loaded using -:cfunc:`PyImport_Import`. This routine needs a Python string as its argument, -which is constructed using the :cfunc:`PyString_FromString` data conversion +:c:func:`PyImport_Import`. This routine needs a Python string as its argument, +which is constructed using the :c:func:`PyString_FromString` data conversion routine. :: pFunc = PyObject_GetAttrString(pModule, argv[2]); @@ -175,7 +173,7 @@ routine. :: Py_XDECREF(pFunc); Once the script is loaded, the name we're looking for is retrieved using -:cfunc:`PyObject_GetAttrString`. If the name exists, and the object returned is +:c:func:`PyObject_GetAttrString`. If the name exists, and the object returned is callable, you can safely assume that it is a function. The program then proceeds by constructing a tuple of arguments as normal. The call to the Python function is then made with:: @@ -229,8 +227,8 @@ Python extension. For example:: return PyModule_Create(&EmbModule); } -Insert the above code just above the :cfunc:`main` function. Also, insert the -following two statements before the call to :cfunc:`Py_Initialize`:: +Insert the above code just above the :c:func:`main` function. Also, insert the +following two statements before the call to :c:func:`Py_Initialize`:: numargs = argc; PyImport_AppendInittab("emb", &PyInit_emb); @@ -260,37 +258,53 @@ write the main program in C++, and use the C++ compiler to compile and link your program. There is no need to recompile Python itself using C++. -.. _link-reqs: - -Linking Requirements -==================== - -While the :program:`configure` script shipped with the Python sources will -correctly build Python to export the symbols needed by dynamically linked -extensions, this is not automatically inherited by applications which embed the -Python library statically, at least on Unix. This is an issue when the -application is linked to the static runtime library (:file:`libpython.a`) and -needs to load dynamic extensions (implemented as :file:`.so` files). +.. _compiling: -The problem is that some entry points are defined by the Python runtime solely -for extension modules to use. If the embedding application does not use any of -these entry points, some linkers will not include those entries in the symbol -table of the finished executable. Some additional options are needed to inform -the linker not to remove these symbols. - -Determining the right options to use for any given platform can be quite -difficult, but fortunately the Python configuration already has those values. -To retrieve them from an installed Python interpreter, start an interactive -interpreter and have a short session like this:: +Compiling and Linking under Unix-like systems +============================================= - >>> import distutils.sysconfig - >>> distutils.sysconfig.get_config_var('LINKFORSHARED') +It is not necessarily trivial to find the right flags to pass to your +compiler (and linker) in order to embed the Python interpreter into your +application, particularly because Python needs to load library modules +implemented as C dynamic extensions (:file:`.so` files) linked against +it. + +To find out the required compiler and linker flags, you can execute the +:file:`python{X.Y}-config` script which is generated as part of the +installation process (a :file:`python3-config` script may also be +available). This script has several options, of which the following will +be directly useful to you: + +* ``pythonX.Y-config --cflags`` will give you the recommended flags when + compiling:: + + $ /opt/bin/python3.2-config --cflags + -I/opt/include/python3.2m -I/opt/include/python3.2m -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes + +* ``pythonX.Y-config --ldflags`` will give you the recommended flags when + linking:: + + $ /opt/bin/python3.2-config --ldflags + -I/opt/lib/python3.2/config-3.2m -lpthread -ldl -lutil -lm -lpython3.2m -Xlinker -export-dynamic + +.. note:: + To avoid confusion between several Python installations (and especially + between the system Python and your own compiled Python), it is recommended + that you use the absolute path to :file:`python{X.Y}-config`, as in the above + example. + +If this procedure doesn't work for you (it is not guaranteed to work for +all Unix-like platforms; however, we welcome :ref:`bug reports <reporting-bugs>`) +you will have to read your system's documentation about dynamic linking and/or +examine Python's :file:`Makefile` (use :func:`sysconfig.get_makefile_filename` +to find its location) and compilation +options. In this case, the :mod:`sysconfig` module is a useful tool to +programmatically extract the configuration values that you will want to +combine together:: + + >>> import sysconfig + >>> sysconfig.get_config_var('LINKFORSHARED') '-Xlinker -export-dynamic' -.. index:: module: distutils.sysconfig - -The contents of the string presented will be the options that should be used. -If the string is empty, there's no need to add any additional options. The -:const:`LINKFORSHARED` definition corresponds to the variable of the same name -in Python's top-level :file:`Makefile`. +.. XXX similar documentation for Windows missing diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst index 68dc9d662b..c4ced1af8b 100644 --- a/Doc/extending/extending.rst +++ b/Doc/extending/extending.rst @@ -20,6 +20,13 @@ source file by including the header ``"Python.h"``. The compilation of an extension module depends on its intended use as well as on your system setup; details are given in later chapters. +Do note that if your use case is calling C library functions or system calls, +you should consider using the :mod:`ctypes` module rather than writing custom +C code. Not only does :mod:`ctypes` let you write Python code to interface +with C code, but it is more portable between implementations of Python than +writing and compiling an extension module which typically ties you to CPython. + + .. _extending-simpleexample: @@ -28,7 +35,7 @@ A Simple Example Let's create an extension module called ``spam`` (the favorite food of Monty Python fans...) and let's say we want to create a Python interface to the C -library function :cfunc:`system`. [#]_ This function takes a null-terminated +library function :c:func:`system`. [#]_ This function takes a null-terminated character string as argument and returns an integer. We want this function to be callable from Python as follows:: @@ -58,8 +65,8 @@ All user-visible symbols defined by :file:`Python.h` have a prefix of ``Py`` or since they are used extensively by the Python interpreter, ``"Python.h"`` includes a few standard header files: ``<stdio.h>``, ``<string.h>``, ``<errno.h>``, and ``<stdlib.h>``. If the latter header file does not exist on -your system, it declares the functions :cfunc:`malloc`, :cfunc:`free` and -:cfunc:`realloc` directly. +your system, it declares the functions :c:func:`malloc`, :c:func:`free` and +:c:func:`realloc` directly. The next thing we add to our module file is the C function that will be called when the Python expression ``spam.system(string)`` is evaluated (we'll see @@ -89,12 +96,12 @@ The *args* argument will be a pointer to a Python tuple object containing the arguments. Each item of the tuple corresponds to an argument in the call's argument list. The arguments are Python objects --- in order to do anything with them in our C function we have to convert them to C values. The function -:cfunc:`PyArg_ParseTuple` in the Python API checks the argument types and +:c:func:`PyArg_ParseTuple` in the Python API checks the argument types and converts them to C values. It uses a template string to determine the required types of the arguments as well as the types of the C variables into which to store the converted values. More about this later. -:cfunc:`PyArg_ParseTuple` returns true (nonzero) if all arguments have the right +:c:func:`PyArg_ParseTuple` returns true (nonzero) if all arguments have the right type and its components have been stored in the variables whose addresses are passed. It returns false (zero) if an invalid argument list was passed. In the latter case it also raises an appropriate exception so the calling function can @@ -119,77 +126,77 @@ to know about them to understand how errors are passed around. The Python API defines a number of functions to set various types of exceptions. -The most common one is :cfunc:`PyErr_SetString`. Its arguments are an exception +The most common one is :c:func:`PyErr_SetString`. Its arguments are an exception object and a C string. The exception object is usually a predefined object like -:cdata:`PyExc_ZeroDivisionError`. The C string indicates the cause of the error +:c:data:`PyExc_ZeroDivisionError`. The C string indicates the cause of the error and is converted to a Python string object and stored as the "associated value" of the exception. -Another useful function is :cfunc:`PyErr_SetFromErrno`, which only takes an +Another useful function is :c:func:`PyErr_SetFromErrno`, which only takes an exception argument and constructs the associated value by inspection of the -global variable :cdata:`errno`. The most general function is -:cfunc:`PyErr_SetObject`, which takes two object arguments, the exception and -its associated value. You don't need to :cfunc:`Py_INCREF` the objects passed +global variable :c:data:`errno`. The most general function is +:c:func:`PyErr_SetObject`, which takes two object arguments, the exception and +its associated value. You don't need to :c:func:`Py_INCREF` the objects passed to any of these functions. You can test non-destructively whether an exception has been set with -:cfunc:`PyErr_Occurred`. This returns the current exception object, or *NULL* +:c:func:`PyErr_Occurred`. This returns the current exception object, or *NULL* if no exception has occurred. You normally don't need to call -:cfunc:`PyErr_Occurred` to see whether an error occurred in a function call, +:c:func:`PyErr_Occurred` to see whether an error occurred in a function call, since you should be able to tell from the return value. When a function *f* that calls another function *g* detects that the latter fails, *f* should itself return an error value (usually *NULL* or ``-1``). It -should *not* call one of the :cfunc:`PyErr_\*` functions --- one has already +should *not* call one of the :c:func:`PyErr_\*` functions --- one has already been called by *g*. *f*'s caller is then supposed to also return an error -indication to *its* caller, again *without* calling :cfunc:`PyErr_\*`, and so on +indication to *its* caller, again *without* calling :c:func:`PyErr_\*`, and so on --- the most detailed cause of the error was already reported by the function that first detected it. Once the error reaches the Python interpreter's main loop, this aborts the currently executing Python code and tries to find an exception handler specified by the Python programmer. (There are situations where a module can actually give a more detailed error -message by calling another :cfunc:`PyErr_\*` function, and in such cases it is +message by calling another :c:func:`PyErr_\*` function, and in such cases it is fine to do so. As a general rule, however, this is not necessary, and can cause information about the cause of the error to be lost: most operations can fail for a variety of reasons.) To ignore an exception set by a function call that failed, the exception -condition must be cleared explicitly by calling :cfunc:`PyErr_Clear`. The only -time C code should call :cfunc:`PyErr_Clear` is if it doesn't want to pass the +condition must be cleared explicitly by calling :c:func:`PyErr_Clear`. The only +time C code should call :c:func:`PyErr_Clear` is if it doesn't want to pass the error on to the interpreter but wants to handle it completely by itself (possibly by trying something else, or pretending nothing went wrong). -Every failing :cfunc:`malloc` call must be turned into an exception --- the -direct caller of :cfunc:`malloc` (or :cfunc:`realloc`) must call -:cfunc:`PyErr_NoMemory` and return a failure indicator itself. All the -object-creating functions (for example, :cfunc:`PyLong_FromLong`) already do -this, so this note is only relevant to those who call :cfunc:`malloc` directly. +Every failing :c:func:`malloc` call must be turned into an exception --- the +direct caller of :c:func:`malloc` (or :c:func:`realloc`) must call +:c:func:`PyErr_NoMemory` and return a failure indicator itself. All the +object-creating functions (for example, :c:func:`PyLong_FromLong`) already do +this, so this note is only relevant to those who call :c:func:`malloc` directly. -Also note that, with the important exception of :cfunc:`PyArg_ParseTuple` and +Also note that, with the important exception of :c:func:`PyArg_ParseTuple` and friends, functions that return an integer status usually return a positive value or zero for success and ``-1`` for failure, like Unix system calls. -Finally, be careful to clean up garbage (by making :cfunc:`Py_XDECREF` or -:cfunc:`Py_DECREF` calls for objects you have already created) when you return +Finally, be careful to clean up garbage (by making :c:func:`Py_XDECREF` or +:c:func:`Py_DECREF` calls for objects you have already created) when you return an error indicator! The choice of which exception to raise is entirely yours. There are predeclared C objects corresponding to all built-in Python exceptions, such as -:cdata:`PyExc_ZeroDivisionError`, which you can use directly. Of course, you -should choose exceptions wisely --- don't use :cdata:`PyExc_TypeError` to mean -that a file couldn't be opened (that should probably be :cdata:`PyExc_IOError`). -If something's wrong with the argument list, the :cfunc:`PyArg_ParseTuple` -function usually raises :cdata:`PyExc_TypeError`. If you have an argument whose +:c:data:`PyExc_ZeroDivisionError`, which you can use directly. Of course, you +should choose exceptions wisely --- don't use :c:data:`PyExc_TypeError` to mean +that a file couldn't be opened (that should probably be :c:data:`PyExc_IOError`). +If something's wrong with the argument list, the :c:func:`PyArg_ParseTuple` +function usually raises :c:data:`PyExc_TypeError`. If you have an argument whose value must be in a particular range or must satisfy other conditions, -:cdata:`PyExc_ValueError` is appropriate. +:c:data:`PyExc_ValueError` is appropriate. You can also define a new exception that is unique to your module. For this, you usually declare a static object variable at the beginning of your file:: static PyObject *SpamError; -and initialize it in your module's initialization function (:cfunc:`PyInit_spam`) +and initialize it in your module's initialization function (:c:func:`PyInit_spam`) with an exception object (leaving out the error checking for now):: PyMODINIT_FUNC @@ -208,14 +215,14 @@ with an exception object (leaving out the error checking for now):: } Note that the Python name for the exception object is :exc:`spam.error`. The -:cfunc:`PyErr_NewException` function may create a class with the base class +:c:func:`PyErr_NewException` function may create a class with the base class being :exc:`Exception` (unless another class is passed in instead of *NULL*), described in :ref:`bltin-exceptions`. -Note also that the :cdata:`SpamError` variable retains a reference to the newly +Note also that the :c:data:`SpamError` variable retains a reference to the newly created exception class; this is intentional! Since the exception could be removed from the module by external code, an owned reference to the class is -needed to ensure that it will not be discarded, causing :cdata:`SpamError` to +needed to ensure that it will not be discarded, causing :c:data:`SpamError` to become a dangling pointer. Should it become a dangling pointer, C code which raises the exception could cause a core dump or other unintended side effects. @@ -223,7 +230,7 @@ We discuss the use of ``PyMODINIT_FUNC`` as a function return type later in this sample. The :exc:`spam.error` exception can be raised in your extension module using a -call to :cfunc:`PyErr_SetString` as shown below:: +call to :c:func:`PyErr_SetString` as shown below:: static PyObject * spam_system(PyObject *self, PyObject *args) @@ -255,19 +262,19 @@ statement:: It returns *NULL* (the error indicator for functions returning object pointers) if an error is detected in the argument list, relying on the exception set by -:cfunc:`PyArg_ParseTuple`. Otherwise the string value of the argument has been -copied to the local variable :cdata:`command`. This is a pointer assignment and +:c:func:`PyArg_ParseTuple`. Otherwise the string value of the argument has been +copied to the local variable :c:data:`command`. This is a pointer assignment and you are not supposed to modify the string to which it points (so in Standard C, -the variable :cdata:`command` should properly be declared as ``const char +the variable :c:data:`command` should properly be declared as ``const char *command``). -The next statement is a call to the Unix function :cfunc:`system`, passing it -the string we just got from :cfunc:`PyArg_ParseTuple`:: +The next statement is a call to the Unix function :c:func:`system`, passing it +the string we just got from :c:func:`PyArg_ParseTuple`:: sts = system(command); -Our :func:`spam.system` function must return the value of :cdata:`sts` as a -Python object. This is done using the function :cfunc:`PyLong_FromLong`. :: +Our :func:`spam.system` function must return the value of :c:data:`sts` as a +Python object. This is done using the function :c:func:`PyLong_FromLong`. :: return PyLong_FromLong(sts); @@ -275,14 +282,14 @@ In this case, it will return an integer object. (Yes, even integers are objects on the heap in Python!) If you have a C function that returns no useful argument (a function returning -:ctype:`void`), the corresponding Python function must return ``None``. You -need this idiom to do so (which is implemented by the :cmacro:`Py_RETURN_NONE` +:c:type:`void`), the corresponding Python function must return ``None``. You +need this idiom to do so (which is implemented by the :c:macro:`Py_RETURN_NONE` macro):: Py_INCREF(Py_None); return Py_None; -:cdata:`Py_None` is the C name for the special Python object ``None``. It is a +:c:data:`Py_None` is the C name for the special Python object ``None``. It is a genuine Python object rather than a *NULL* pointer, which means "error" in most contexts, as we have seen. @@ -292,7 +299,7 @@ contexts, as we have seen. The Module's Method Table and Initialization Function ===================================================== -I promised to show how :cfunc:`spam_system` is called from Python programs. +I promised to show how :c:func:`spam_system` is called from Python programs. First, we need to list its name and address in a "method table":: static PyMethodDef SpamMethods[] = { @@ -306,16 +313,16 @@ First, we need to list its name and address in a "method table":: Note the third entry (``METH_VARARGS``). This is a flag telling the interpreter the calling convention to be used for the C function. It should normally always be ``METH_VARARGS`` or ``METH_VARARGS | METH_KEYWORDS``; a value of ``0`` means -that an obsolete variant of :cfunc:`PyArg_ParseTuple` is used. +that an obsolete variant of :c:func:`PyArg_ParseTuple` is used. When using only ``METH_VARARGS``, the function should expect the Python-level parameters to be passed in as a tuple acceptable for parsing via -:cfunc:`PyArg_ParseTuple`; more information on this function is provided below. +:c:func:`PyArg_ParseTuple`; more information on this function is provided below. The :const:`METH_KEYWORDS` bit may be set in the third field if keyword arguments should be passed to the function. In this case, the C function should accept a third ``PyObject \*`` parameter which will be a dictionary of keywords. -Use :cfunc:`PyArg_ParseTupleAndKeywords` to parse the arguments to such a +Use :c:func:`PyArg_ParseTupleAndKeywords` to parse the arguments to such a function. The method table must be referenced in the module definition structure:: @@ -331,7 +338,7 @@ The method table must be referenced in the module definition structure:: This structure, in turn, must be passed to the interpreter in the module's initialization function. The initialization function must be named -:cfunc:`PyInit_name`, where *name* is the name of the module, and should be the +:c:func:`PyInit_name`, where *name* is the name of the module, and should be the only non-\ ``static`` item defined in the module file:: PyMODINIT_FUNC @@ -345,25 +352,25 @@ declares any special linkage declarations required by the platform, and for C++ declares the function as ``extern "C"``. When the Python program imports module :mod:`spam` for the first time, -:cfunc:`PyInit_spam` is called. (See below for comments about embedding Python.) -It calls :cfunc:`PyModule_Create`, which returns a module object, and +:c:func:`PyInit_spam` is called. (See below for comments about embedding Python.) +It calls :c:func:`PyModule_Create`, which returns a module object, and inserts built-in function objects into the newly created module based upon the -table (an array of :ctype:`PyMethodDef` structures) found in the module definition. -:cfunc:`PyModule_Create` returns a pointer to the module object +table (an array of :c:type:`PyMethodDef` structures) found in the module definition. +:c:func:`PyModule_Create` returns a pointer to the module object that it creates. It may abort with a fatal error for certain errors, or return *NULL* if the module could not be initialized satisfactorily. The init function must return the module object to its caller, so that it then gets inserted into ``sys.modules``. -When embedding Python, the :cfunc:`PyInit_spam` function is not called -automatically unless there's an entry in the :cdata:`PyImport_Inittab` table. -To add the module to the initialization table, use :cfunc:`PyImport_AppendInittab`, +When embedding Python, the :c:func:`PyInit_spam` function is not called +automatically unless there's an entry in the :c:data:`PyImport_Inittab` table. +To add the module to the initialization table, use :c:func:`PyImport_AppendInittab`, optionally followed by an import of the module:: int main(int argc, char *argv[]) { - /* Add a builtin module, before Py_Initialize */ + /* Add a built-in module, before Py_Initialize */ PyImport_AppendInittab("spam", PyInit_spam); /* Pass argv[0] to the Python interpreter */ @@ -383,19 +390,14 @@ source distribution. .. note:: Removing entries from ``sys.modules`` or importing compiled modules into - multiple interpreters within a process (or following a :cfunc:`fork` without an - intervening :cfunc:`exec`) can create problems for some extension modules. + multiple interpreters within a process (or following a :c:func:`fork` without an + intervening :c:func:`exec`) can create problems for some extension modules. Extension module authors should exercise caution when initializing internal data structures. A more substantial example module is included in the Python source distribution as :file:`Modules/xxmodule.c`. This file may be used as a template or simply -read as an example. The :program:`modulator.py` script included in the source -distribution or Windows install provides a simple graphical user interface for -declaring the functions and objects which a module should implement, and can -generate a template which can be filled in. The script lives in the -:file:`Tools/modulator/` directory; see the :file:`README` file there for more -information. +read as an example. .. _compilation: @@ -453,7 +455,7 @@ look at the implementation of the :option:`-c` command line option in Calling a Python function is easy. First, the Python program must somehow pass you the Python function object. You should provide a function (or some other interface) to do this. When this function is called, save a pointer to the -Python function object (be careful to :cfunc:`Py_INCREF` it!) in a global +Python function object (be careful to :c:func:`Py_INCREF` it!) in a global variable --- or wherever you see fit. For example, the following function might be part of a module definition:: @@ -482,10 +484,10 @@ be part of a module definition:: This function must be registered with the interpreter using the :const:`METH_VARARGS` flag; this is described in section :ref:`methodtable`. The -:cfunc:`PyArg_ParseTuple` function and its arguments are documented in section +:c:func:`PyArg_ParseTuple` function and its arguments are documented in section :ref:`parsetuple`. -The macros :cfunc:`Py_XINCREF` and :cfunc:`Py_XDECREF` increment/decrement the +The macros :c:func:`Py_XINCREF` and :c:func:`Py_XDECREF` increment/decrement the reference count of an object and are safe in the presence of *NULL* pointers (but note that *temp* will not be *NULL* in this context). More info on them in section :ref:`refcounts`. @@ -493,12 +495,12 @@ in section :ref:`refcounts`. .. index:: single: PyObject_CallObject() Later, when it is time to call the function, you call the C function -:cfunc:`PyObject_CallObject`. This function has two arguments, both pointers to +:c:func:`PyObject_CallObject`. This function has two arguments, both pointers to arbitrary Python objects: the Python function, and the argument list. The argument list must always be a tuple object, whose length is the number of arguments. To call the Python function with no arguments, pass in NULL, or an empty tuple; to call it with one argument, pass a singleton tuple. -:cfunc:`Py_BuildValue` returns a tuple when its format string consists of zero +:c:func:`Py_BuildValue` returns a tuple when its format string consists of zero or more format codes between parentheses. For example:: int arg; @@ -512,25 +514,25 @@ or more format codes between parentheses. For example:: result = PyObject_CallObject(my_callback, arglist); Py_DECREF(arglist); -:cfunc:`PyObject_CallObject` returns a Python object pointer: this is the return -value of the Python function. :cfunc:`PyObject_CallObject` is +:c:func:`PyObject_CallObject` returns a Python object pointer: this is the return +value of the Python function. :c:func:`PyObject_CallObject` is "reference-count-neutral" with respect to its arguments. In the example a new -tuple was created to serve as the argument list, which is :cfunc:`Py_DECREF`\ +tuple was created to serve as the argument list, which is :c:func:`Py_DECREF`\ -ed immediately after the call. -The return value of :cfunc:`PyObject_CallObject` is "new": either it is a brand +The return value of :c:func:`PyObject_CallObject` is "new": either it is a brand new object, or it is an existing object whose reference count has been incremented. So, unless you want to save it in a global variable, you should -somehow :cfunc:`Py_DECREF` the result, even (especially!) if you are not +somehow :c:func:`Py_DECREF` the result, even (especially!) if you are not interested in its value. Before you do this, however, it is important to check that the return value isn't *NULL*. If it is, the Python function terminated by raising an exception. -If the C code that called :cfunc:`PyObject_CallObject` is called from Python, it +If the C code that called :c:func:`PyObject_CallObject` is called from Python, it should now return an error indication to its Python caller, so the interpreter can print a stack trace, or the calling Python code can handle the exception. If this is not possible or desirable, the exception should be cleared by calling -:cfunc:`PyErr_Clear`. For example:: +:c:func:`PyErr_Clear`. For example:: if (result == NULL) return NULL; /* Pass error back */ @@ -538,12 +540,12 @@ If this is not possible or desirable, the exception should be cleared by calling Py_DECREF(result); Depending on the desired interface to the Python callback function, you may also -have to provide an argument list to :cfunc:`PyObject_CallObject`. In some cases +have to provide an argument list to :c:func:`PyObject_CallObject`. In some cases the argument list is also provided by the Python program, through the same interface that specified the callback function. It can then be saved and used in the same manner as the function object. In other cases, you may have to construct a new tuple to pass as the argument list. The simplest way to do this -is to call :cfunc:`Py_BuildValue`. For example, if you want to pass an integral +is to call :c:func:`Py_BuildValue`. For example, if you want to pass an integral event code, you might use the following code:: PyObject *arglist; @@ -558,11 +560,11 @@ event code, you might use the following code:: Note the placement of ``Py_DECREF(arglist)`` immediately after the call, before the error check! Also note that strictly speaking this code is not complete: -:cfunc:`Py_BuildValue` may run out of memory, and this should be checked. +:c:func:`Py_BuildValue` may run out of memory, and this should be checked. You may also call a function with keyword arguments by using -:cfunc:`PyObject_Call`, which supports arguments and keyword arguments. As in -the above example, we use :cfunc:`Py_BuildValue` to construct the dictionary. :: +:c:func:`PyObject_Call`, which supports arguments and keyword arguments. As in +the above example, we use :c:func:`Py_BuildValue` to construct the dictionary. :: PyObject *dict; ... @@ -582,7 +584,7 @@ Extracting Parameters in Extension Functions .. index:: single: PyArg_ParseTuple() -The :cfunc:`PyArg_ParseTuple` function is declared as follows:: +The :c:func:`PyArg_ParseTuple` function is declared as follows:: int PyArg_ParseTuple(PyObject *arg, char *format, ...); @@ -592,7 +594,7 @@ whose syntax is explained in :ref:`arg-parsing` in the Python/C API Reference Manual. The remaining arguments must be addresses of variables whose type is determined by the format string. -Note that while :cfunc:`PyArg_ParseTuple` checks that the Python arguments have +Note that while :c:func:`PyArg_ParseTuple` checks that the Python arguments have the required types, it cannot check the validity of the addresses of C variables passed to the call: if you make mistakes there, your code will probably crash or at least overwrite random bits in memory. So be careful! @@ -674,17 +676,17 @@ Keyword Parameters for Extension Functions .. index:: single: PyArg_ParseTupleAndKeywords() -The :cfunc:`PyArg_ParseTupleAndKeywords` function is declared as follows:: +The :c:func:`PyArg_ParseTupleAndKeywords` function is declared as follows:: int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict, char *format, char *kwlist[], ...); The *arg* and *format* parameters are identical to those of the -:cfunc:`PyArg_ParseTuple` function. The *kwdict* parameter is the dictionary of +:c:func:`PyArg_ParseTuple` function. The *kwdict* parameter is the dictionary of keywords received as the third parameter from the Python runtime. The *kwlist* parameter is a *NULL*-terminated list of strings which identify the parameters; the names are matched with the type information from *format* from left to -right. On success, :cfunc:`PyArg_ParseTupleAndKeywords` returns true, otherwise +right. On success, :c:func:`PyArg_ParseTupleAndKeywords` returns true, otherwise it returns false and raises an appropriate exception. .. note:: @@ -748,19 +750,19 @@ Philbrick (philbrick@hks.com):: Building Arbitrary Values ========================= -This function is the counterpart to :cfunc:`PyArg_ParseTuple`. It is declared +This function is the counterpart to :c:func:`PyArg_ParseTuple`. It is declared as follows:: PyObject *Py_BuildValue(char *format, ...); It recognizes a set of format units similar to the ones recognized by -:cfunc:`PyArg_ParseTuple`, but the arguments (which are input to the function, +:c:func:`PyArg_ParseTuple`, but the arguments (which are input to the function, not output) must not be pointers, just values. It returns a new Python object, suitable for returning from a C function called from Python. -One difference with :cfunc:`PyArg_ParseTuple`: while the latter requires its +One difference with :c:func:`PyArg_ParseTuple`: while the latter requires its first argument to be a tuple (since Python argument lists are always represented -as tuples internally), :cfunc:`Py_BuildValue` does not always build a tuple. It +as tuples internally), :c:func:`Py_BuildValue` does not always build a tuple. It builds a tuple only if its format string contains two or more format units. If the format string is empty, it returns ``None``; if it contains exactly one format unit, it returns whatever object is described by that format unit. To @@ -794,18 +796,18 @@ Reference Counts In languages like C or C++, the programmer is responsible for dynamic allocation and deallocation of memory on the heap. In C, this is done using the functions -:cfunc:`malloc` and :cfunc:`free`. In C++, the operators ``new`` and +:c:func:`malloc` and :c:func:`free`. In C++, the operators ``new`` and ``delete`` are used with essentially the same meaning and we'll restrict the following discussion to the C case. -Every block of memory allocated with :cfunc:`malloc` should eventually be -returned to the pool of available memory by exactly one call to :cfunc:`free`. -It is important to call :cfunc:`free` at the right time. If a block's address -is forgotten but :cfunc:`free` is not called for it, the memory it occupies +Every block of memory allocated with :c:func:`malloc` should eventually be +returned to the pool of available memory by exactly one call to :c:func:`free`. +It is important to call :c:func:`free` at the right time. If a block's address +is forgotten but :c:func:`free` is not called for it, the memory it occupies cannot be reused until the program terminates. This is called a :dfn:`memory -leak`. On the other hand, if a program calls :cfunc:`free` for a block and then +leak`. On the other hand, if a program calls :c:func:`free` for a block and then continues to use the block, it creates a conflict with re-use of the block -through another :cfunc:`malloc` call. This is called :dfn:`using freed memory`. +through another :c:func:`malloc` call. This is called :dfn:`using freed memory`. It has the same bad consequences as referencing uninitialized data --- core dumps, wrong results, mysterious crashes. @@ -822,7 +824,7 @@ long-running process that uses the leaking function frequently. Therefore, it's important to prevent leaks from happening by having a coding convention or strategy that minimizes this kind of errors. -Since Python makes heavy use of :cfunc:`malloc` and :cfunc:`free`, it needs a +Since Python makes heavy use of :c:func:`malloc` and :c:func:`free`, it needs a strategy to avoid memory leaks as well as the use of freed memory. The chosen method is called :dfn:`reference counting`. The principle is simple: every object contains a counter, which is incremented when a reference to the object @@ -834,11 +836,11 @@ An alternative strategy is called :dfn:`automatic garbage collection`. (Sometimes, reference counting is also referred to as a garbage collection strategy, hence my use of "automatic" to distinguish the two.) The big advantage of automatic garbage collection is that the user doesn't need to call -:cfunc:`free` explicitly. (Another claimed advantage is an improvement in speed +:c:func:`free` explicitly. (Another claimed advantage is an improvement in speed or memory usage --- this is no hard fact however.) The disadvantage is that for C, there is no truly portable automatic garbage collector, while reference -counting can be implemented portably (as long as the functions :cfunc:`malloc` -and :cfunc:`free` are available --- which the C Standard guarantees). Maybe some +counting can be implemented portably (as long as the functions :c:func:`malloc` +and :c:func:`free` are available --- which the C Standard guarantees). Maybe some day a sufficiently portable automatic garbage collector will be available for C. Until then, we'll have to live with reference counts. @@ -873,9 +875,9 @@ Reference Counting in Python ---------------------------- There are two macros, ``Py_INCREF(x)`` and ``Py_DECREF(x)``, which handle the -incrementing and decrementing of the reference count. :cfunc:`Py_DECREF` also +incrementing and decrementing of the reference count. :c:func:`Py_DECREF` also frees the object when the count reaches zero. For flexibility, it doesn't call -:cfunc:`free` directly --- rather, it makes a call through a function pointer in +:c:func:`free` directly --- rather, it makes a call through a function pointer in the object's :dfn:`type object`. For this purpose (and others), every object also contains a pointer to its type object. @@ -883,13 +885,13 @@ The big question now remains: when to use ``Py_INCREF(x)`` and ``Py_DECREF(x)``? Let's first introduce some terms. Nobody "owns" an object; however, you can :dfn:`own a reference` to an object. An object's reference count is now defined as the number of owned references to it. The owner of a reference is -responsible for calling :cfunc:`Py_DECREF` when the reference is no longer +responsible for calling :c:func:`Py_DECREF` when the reference is no longer needed. Ownership of a reference can be transferred. There are three ways to -dispose of an owned reference: pass it on, store it, or call :cfunc:`Py_DECREF`. +dispose of an owned reference: pass it on, store it, or call :c:func:`Py_DECREF`. Forgetting to dispose of an owned reference creates a memory leak. It is also possible to :dfn:`borrow` [#]_ a reference to an object. The -borrower of a reference should not call :cfunc:`Py_DECREF`. The borrower must +borrower of a reference should not call :c:func:`Py_DECREF`. The borrower must not hold on to the object longer than the owner from which it was borrowed. Using a borrowed reference after the owner has disposed of it risks using freed memory and should be avoided completely. [#]_ @@ -903,7 +905,7 @@ reference can be used after the owner from which it was borrowed has in fact disposed of it. A borrowed reference can be changed into an owned reference by calling -:cfunc:`Py_INCREF`. This does not affect the status of the owner from which the +:c:func:`Py_INCREF`. This does not affect the status of the owner from which the reference was borrowed --- it creates a new owned reference, and gives full owner responsibilities (the new owner must dispose of the reference properly, as well as the previous owner). @@ -920,36 +922,36 @@ reference or not. Most functions that return a reference to an object pass on ownership with the reference. In particular, all functions whose function it is to create a new -object, such as :cfunc:`PyLong_FromLong` and :cfunc:`Py_BuildValue`, pass +object, such as :c:func:`PyLong_FromLong` and :c:func:`Py_BuildValue`, pass ownership to the receiver. Even if the object is not actually new, you still receive ownership of a new reference to that object. For instance, -:cfunc:`PyLong_FromLong` maintains a cache of popular values and can return a +:c:func:`PyLong_FromLong` maintains a cache of popular values and can return a reference to a cached item. Many functions that extract objects from other objects also transfer ownership -with the reference, for instance :cfunc:`PyObject_GetAttrString`. The picture +with the reference, for instance :c:func:`PyObject_GetAttrString`. The picture is less clear, here, however, since a few common routines are exceptions: -:cfunc:`PyTuple_GetItem`, :cfunc:`PyList_GetItem`, :cfunc:`PyDict_GetItem`, and -:cfunc:`PyDict_GetItemString` all return references that you borrow from the +:c:func:`PyTuple_GetItem`, :c:func:`PyList_GetItem`, :c:func:`PyDict_GetItem`, and +:c:func:`PyDict_GetItemString` all return references that you borrow from the tuple, list or dictionary. -The function :cfunc:`PyImport_AddModule` also returns a borrowed reference, even +The function :c:func:`PyImport_AddModule` also returns a borrowed reference, even though it may actually create the object it returns: this is possible because an owned reference to the object is stored in ``sys.modules``. When you pass an object reference into another function, in general, the function borrows the reference from you --- if it needs to store it, it will use -:cfunc:`Py_INCREF` to become an independent owner. There are exactly two -important exceptions to this rule: :cfunc:`PyTuple_SetItem` and -:cfunc:`PyList_SetItem`. These functions take over ownership of the item passed -to them --- even if they fail! (Note that :cfunc:`PyDict_SetItem` and friends +:c:func:`Py_INCREF` to become an independent owner. There are exactly two +important exceptions to this rule: :c:func:`PyTuple_SetItem` and +:c:func:`PyList_SetItem`. These functions take over ownership of the item passed +to them --- even if they fail! (Note that :c:func:`PyDict_SetItem` and friends don't take over ownership --- they are "normal.") When a C function is called from Python, it borrows references to its arguments from the caller. The caller owns a reference to the object, so the borrowed reference's lifetime is guaranteed until the function returns. Only when such a borrowed reference must be stored or passed on, it must be turned into an owned -reference by calling :cfunc:`Py_INCREF`. +reference by calling :c:func:`Py_INCREF`. The object reference returned from a C function that is called from Python must be an owned reference --- ownership is transferred from the function to its @@ -965,7 +967,7 @@ There are a few situations where seemingly harmless use of a borrowed reference can lead to problems. These all have to do with implicit invocations of the interpreter, which can cause the owner of a reference to dispose of it. -The first and most important case to know about is using :cfunc:`Py_DECREF` on +The first and most important case to know about is using :c:func:`Py_DECREF` on an unrelated object while borrowing a reference to a list item. For instance:: void @@ -981,7 +983,7 @@ This function first borrows a reference to ``list[0]``, then replaces ``list[1]`` with the value ``0``, and finally prints the borrowed reference. Looks harmless, right? But it's not! -Let's follow the control flow into :cfunc:`PyList_SetItem`. The list owns +Let's follow the control flow into :c:func:`PyList_SetItem`. The list owns references to all its items, so when item 1 is replaced, it has to dispose of the original item 1. Now let's suppose the original item 1 was an instance of a user-defined class, and let's further suppose that the class defined a @@ -990,8 +992,8 @@ disposing of it will call its :meth:`__del__` method. Since it is written in Python, the :meth:`__del__` method can execute arbitrary Python code. Could it perhaps do something to invalidate the reference to -``item`` in :cfunc:`bug`? You bet! Assuming that the list passed into -:cfunc:`bug` is accessible to the :meth:`__del__` method, it could execute a +``item`` in :c:func:`bug`? You bet! Assuming that the list passed into +:c:func:`bug` is accessible to the :meth:`__del__` method, it could execute a statement to the effect of ``del list[0]``, and assuming this was the last reference to that object, it would free the memory associated with it, thereby invalidating ``item``. @@ -1018,8 +1020,8 @@ The second case of problems with a borrowed reference is a variant involving threads. Normally, multiple threads in the Python interpreter can't get in each other's way, because there is a global lock protecting Python's entire object space. However, it is possible to temporarily release this lock using the macro -:cmacro:`Py_BEGIN_ALLOW_THREADS`, and to re-acquire it using -:cmacro:`Py_END_ALLOW_THREADS`. This is common around blocking I/O calls, to +:c:macro:`Py_BEGIN_ALLOW_THREADS`, and to re-acquire it using +:c:macro:`Py_END_ALLOW_THREADS`. This is common around blocking I/O calls, to let other threads use the processor while waiting for the I/O to complete. Obviously, the following function has the same problem as the previous one:: @@ -1048,11 +1050,11 @@ function --- if each function were to test for *NULL*, there would be a lot of redundant tests and the code would run more slowly. It is better to test for *NULL* only at the "source:" when a pointer that may be -*NULL* is received, for example, from :cfunc:`malloc` or from a function that +*NULL* is received, for example, from :c:func:`malloc` or from a function that may raise an exception. -The macros :cfunc:`Py_INCREF` and :cfunc:`Py_DECREF` do not check for *NULL* -pointers --- however, their variants :cfunc:`Py_XINCREF` and :cfunc:`Py_XDECREF` +The macros :c:func:`Py_INCREF` and :c:func:`Py_DECREF` do not check for *NULL* +pointers --- however, their variants :c:func:`Py_XINCREF` and :c:func:`Py_XDECREF` do. The macros for checking for a particular object type (``Pytype_Check()``) don't @@ -1126,7 +1128,7 @@ other extension modules must be exported in a different way. Python provides a special mechanism to pass C-level information (pointers) from one extension module to another one: Capsules. A Capsule is a Python data type -which stores a pointer (:ctype:`void \*`). Capsules can only be created and +which stores a pointer (:c:type:`void \*`). Capsules can only be created and accessed via their C API, but they can be passed around like any other Python object. In particular, they can be assigned to a name in an extension module's namespace. Other extension modules can then import this module, retrieve the @@ -1139,8 +1141,8 @@ various tasks of storing and retrieving the pointers can be distributed in different ways between the module providing the code and the client modules. Whichever method you choose, it's important to name your Capsules properly. -The function :cfunc:`PyCapsule_New` takes a name parameter -(:ctype:`const char \*`); you're permitted to pass in a *NULL* name, but +The function :c:func:`PyCapsule_New` takes a name parameter +(:c:type:`const char \*`); you're permitted to pass in a *NULL* name, but we strongly encourage you to specify a name. Properly named Capsules provide a degree of runtime type-safety; there is no feasible way to tell one unnamed Capsule from another. @@ -1150,7 +1152,7 @@ this convention:: modulename.attributename -The convenience function :cfunc:`PyCapsule_Import` makes it easy to +The convenience function :c:func:`PyCapsule_Import` makes it easy to load a C API provided via a Capsule, but only if the Capsule's name matches this convention. This behavior gives C API users a high degree of certainty that the Capsule they load contains the correct C API. @@ -1158,19 +1160,19 @@ of certainty that the Capsule they load contains the correct C API. The following example demonstrates an approach that puts most of the burden on the writer of the exporting module, which is appropriate for commonly used library modules. It stores all C API pointers (just one in the example!) in an -array of :ctype:`void` pointers which becomes the value of a Capsule. The header +array of :c:type:`void` pointers which becomes the value of a Capsule. The header file corresponding to the module provides a macro that takes care of importing the module and retrieving its C API pointers; client modules only have to call this macro before accessing the C API. The exporting module is a modification of the :mod:`spam` module from section :ref:`extending-simpleexample`. The function :func:`spam.system` does not call -the C library function :cfunc:`system` directly, but a function -:cfunc:`PySpam_System`, which would of course do something more complicated in +the C library function :c:func:`system` directly, but a function +:c:func:`PySpam_System`, which would of course do something more complicated in reality (such as adding "spam" to every command). This function -:cfunc:`PySpam_System` is also exported to other extension modules. +:c:func:`PySpam_System` is also exported to other extension modules. -The function :cfunc:`PySpam_System` is a plain C function, declared +The function :c:func:`PySpam_System` is a plain C function, declared ``static`` like everything else:: static int @@ -1179,7 +1181,7 @@ The function :cfunc:`PySpam_System` is a plain C function, declared return system(command); } -The function :cfunc:`spam_system` is modified in a trivial way:: +The function :c:func:`spam_system` is modified in a trivial way:: static PyObject * spam_system(PyObject *self, PyObject *args) @@ -1283,8 +1285,8 @@ like this:: #endif /* !defined(Py_SPAMMODULE_H) */ All that a client module must do in order to have access to the function -:cfunc:`PySpam_System` is to call the function (or rather macro) -:cfunc:`import_spam` in its initialization function:: +:c:func:`PySpam_System` is to call the function (or rather macro) +:c:func:`import_spam` in its initialization function:: PyMODINIT_FUNC PyInit_client(void) diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst index d48efc94f1..2ba01bc53b 100644 --- a/Doc/extending/newtypes.rst +++ b/Doc/extending/newtypes.rst @@ -26,12 +26,11 @@ The Basics ========== The Python runtime sees all Python objects as variables of type -:ctype:`PyObject\*`. A :ctype:`PyObject` is not a very magnificent object - it +:c:type:`PyObject\*`. A :c:type:`PyObject` is not a very magnificent object - it just contains the refcount and a pointer to the object's "type object". This is where the action is; the type object determines which (C) functions get called when, for instance, an attribute gets looked up on an object or it is multiplied -by another object. These C functions are called "type methods" to distinguish -them from things like ``[].append`` (which we call "object methods"). +by another object. These C functions are called "type methods". So, if you want to define a new object type, you need to create a new type object. @@ -95,7 +94,7 @@ Moving on, we come to the crunch --- the type object. :: "Noddy objects", /* tp_doc */ }; -Now if you go and look up the definition of :ctype:`PyTypeObject` in +Now if you go and look up the definition of :c:type:`PyTypeObject` in :file:`object.h` you'll see that it has many more fields that the definition above. The remaining fields will be filled with zeros by the C compiler, and it's common practice to not specify them explicitly unless you need them. @@ -111,7 +110,7 @@ This line is a bit of a wart; what we'd like to write is:: as the type of a type object is "type", but this isn't strictly conforming C and some compilers complain. Fortunately, this member will be filled in for us by -:cfunc:`PyType_Ready`. :: +:c:func:`PyType_Ready`. :: "noddy.Noddy", /* tp_name */ @@ -130,7 +129,7 @@ the type is :class:`Noddy`, so we set the type name to :class:`noddy.Noddy`. :: sizeof(noddy_NoddyObject), /* tp_basicsize */ This is so that Python knows how much memory to allocate when you call -:cfunc:`PyObject_New`. +:c:func:`PyObject_New`. .. note:: @@ -170,12 +169,12 @@ the module. We'll expand this example later to have more interesting behavior. For now, all we want to be able to do is to create new :class:`Noddy` objects. To enable object creation, we have to provide a :attr:`tp_new` implementation. In this case, we can just use the default implementation provided by the API -function :cfunc:`PyType_GenericNew`. We'd like to just assign this to the +function :c:func:`PyType_GenericNew`. We'd like to just assign this to the :attr:`tp_new` slot, but we can't, for portability sake, On some platforms or compilers, we can't statically initialize a structure member with a function defined in another C module, so, instead, we'll assign the :attr:`tp_new` slot in the module initialization function just before calling -:cfunc:`PyType_Ready`:: +:c:func:`PyType_Ready`:: noddy_NoddyType.tp_new = PyType_GenericNew; if (PyType_Ready(&noddy_NoddyType) < 0) @@ -185,7 +184,7 @@ All the other type methods are *NULL*, so we'll go over them later --- that's for a later section! Everything else in the file should be familiar, except for some code in -:cfunc:`PyInit_noddy`:: +:c:func:`PyInit_noddy`:: if (PyType_Ready(&noddy_NoddyType) < 0) return; @@ -273,7 +272,7 @@ which is assigned to the :attr:`tp_dealloc` member:: (destructor)Noddy_dealloc, /*tp_dealloc*/ This method decrements the reference counts of the two Python attributes. We use -:cfunc:`Py_XDECREF` here because the :attr:`first` and :attr:`last` members +:c:func:`Py_XDECREF` here because the :attr:`first` and :attr:`last` members could be *NULL*. It then calls the :attr:`tp_free` member of the object's type to free the object's memory. Note that the object's type might not be :class:`NoddyType`, because the object may be an instance of a subclass. @@ -319,8 +318,8 @@ the :meth:`__new__` method. One reason to implement a new method is to assure the initial values of instance variables. In this case, we use the new method to make sure that the initial values of the members :attr:`first` and :attr:`last` are not *NULL*. If we didn't care whether the initial values were -*NULL*, we could have used :cfunc:`PyType_GenericNew` as our new method, as we -did before. :cfunc:`PyType_GenericNew` initializes all of the instance variable +*NULL*, we could have used :c:func:`PyType_GenericNew` as our new method, as we +did before. :c:func:`PyType_GenericNew` initializes all of the instance variable members to *NULL*. The new method is a static method that is passed the type being instantiated and @@ -330,7 +329,7 @@ often ignore the arguments, leaving the argument handling to initializer methods. Note that if the type supports subclassing, the type passed may not be the type being defined. The new method calls the tp_alloc slot to allocate memory. We don't fill the :attr:`tp_alloc` slot ourselves. Rather -:cfunc:`PyType_Ready` fills it for us by inheriting it from our base class, +:c:func:`PyType_Ready` fills it for us by inheriting it from our base class, which is :class:`object` by default. Most types use the default allocation. .. note:: @@ -515,8 +514,8 @@ object being created or used, so all we need to do is to add the Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/ -We rename :cfunc:`PyInit_noddy` to :cfunc:`PyInit_noddy2` and update the module -name in the :ctype:`PyModuleDef` struct. +We rename :c:func:`PyInit_noddy` to :c:func:`PyInit_noddy2` and update the module +name in the :c:type:`PyModuleDef` struct. Finally, we update our :file:`setup.py` file to build the new module:: @@ -582,7 +581,7 @@ closure. The new value may be *NULL*, in which case the attribute is being deleted. In our setter, we raise an error if the attribute is deleted or if the attribute value is not a string. -We create an array of :ctype:`PyGetSetDef` structures:: +We create an array of :c:type:`PyGetSetDef` structures:: static PyGetSetDef Noddy_getseters[] = { {"first", @@ -602,7 +601,7 @@ and register it in the :attr:`tp_getset` slot:: to register our attribute getters and setters. -The last item in a :ctype:`PyGetSetDef` structure is the closure mentioned +The last item in a :c:type:`PyGetSetDef` structure is the closure mentioned above. In this case, we aren't using the closure, so we just pass *NULL*. We also remove the member definitions for these attributes:: @@ -647,8 +646,8 @@ be passed:: With these changes, we can assure that the :attr:`first` and :attr:`last` members are never *NULL* so we can remove checks for *NULL* values in almost all -cases. This means that most of the :cfunc:`Py_XDECREF` calls can be converted to -:cfunc:`Py_DECREF` calls. The only place we can't change these calls is in the +cases. This means that most of the :c:func:`Py_XDECREF` calls can be converted to +:c:func:`Py_DECREF` calls. The only place we can't change these calls is in the deallocator, where there is the possibility that the initialization of these members failed in the constructor. @@ -713,13 +712,13 @@ cycles:: } For each subobject that can participate in cycles, we need to call the -:cfunc:`visit` function, which is passed to the traversal method. The -:cfunc:`visit` function takes as arguments the subobject and the extra argument +:c:func:`visit` function, which is passed to the traversal method. The +:c:func:`visit` function takes as arguments the subobject and the extra argument *arg* passed to the traversal method. It returns an integer value that must be returned if it is non-zero. -Python provides a :cfunc:`Py_VISIT` macro that automates calling visit -functions. With :cfunc:`Py_VISIT`, :cfunc:`Noddy_traverse` can be simplified:: +Python provides a :c:func:`Py_VISIT` macro that automates calling visit +functions. With :c:func:`Py_VISIT`, :c:func:`Noddy_traverse` can be simplified:: static int Noddy_traverse(Noddy *self, visitproc visit, void *arg) @@ -732,7 +731,7 @@ functions. With :cfunc:`Py_VISIT`, :cfunc:`Noddy_traverse` can be simplified:: .. note:: Note that the :attr:`tp_traverse` implementation must name its arguments exactly - *visit* and *arg* in order to use :cfunc:`Py_VISIT`. This is to encourage + *visit* and *arg* in order to use :c:func:`Py_VISIT`. This is to encourage uniformity across these boring implementations. We also need to provide a method for clearing any subobjects that can @@ -762,18 +761,18 @@ to use it:: Py_TYPE(self)->tp_free((PyObject*)self); } -Notice the use of a temporary variable in :cfunc:`Noddy_clear`. We use the +Notice the use of a temporary variable in :c:func:`Noddy_clear`. We use the temporary variable so that we can set each member to *NULL* before decrementing its reference count. We do this because, as was discussed earlier, if the reference count drops to zero, we might cause code to run that calls back into the object. In addition, because we now support garbage collection, we also have to worry about code being run that triggers garbage collection. If garbage collection is run, our :attr:`tp_traverse` handler could get called. We can't -take a chance of having :cfunc:`Noddy_traverse` called when a member's reference +take a chance of having :c:func:`Noddy_traverse` called when a member's reference count has dropped to zero and its value hasn't been set to *NULL*. -Python provides a :cfunc:`Py_CLEAR` that automates the careful decrementing of -reference counts. With :cfunc:`Py_CLEAR`, the :cfunc:`Noddy_clear` function can +Python provides a :c:func:`Py_CLEAR` that automates the careful decrementing of +reference counts. With :c:func:`Py_CLEAR`, the :c:func:`Noddy_clear` function can be simplified:: static int @@ -829,7 +828,7 @@ previous sections. We will break down the main differences between them. :: The primary difference for derived type objects is that the base type's object structure must be the first value. The base type will already include the -:cfunc:`PyObject_HEAD` at the beginning of its structure. +:c:func:`PyObject_HEAD` at the beginning of its structure. When a Python object is a :class:`Shoddy` instance, its *PyObject\** pointer can be safely cast to both *PyListObject\** and *Shoddy\**. :: @@ -851,10 +850,10 @@ This pattern is important when writing a type with custom :attr:`new` and memory for the object with :attr:`tp_alloc`, that will be handled by the base class when calling its :attr:`tp_new`. -When filling out the :cfunc:`PyTypeObject` for the :class:`Shoddy` type, you see -a slot for :cfunc:`tp_base`. Due to cross platform compiler issues, you can't -fill that field directly with the :cfunc:`PyList_Type`; it can be done later in -the module's :cfunc:`init` function. :: +When filling out the :c:func:`PyTypeObject` for the :class:`Shoddy` type, you see +a slot for :c:func:`tp_base`. Due to cross platform compiler issues, you can't +fill that field directly with the :c:func:`PyList_Type`; it can be done later in +the module's :c:func:`init` function. :: PyMODINIT_FUNC PyInit_shoddy(void) @@ -874,12 +873,12 @@ the module's :cfunc:`init` function. :: return m; } -Before calling :cfunc:`PyType_Ready`, the type structure must have the +Before calling :c:func:`PyType_Ready`, the type structure must have the :attr:`tp_base` slot filled in. When we are deriving a new type, it is not -necessary to fill out the :attr:`tp_alloc` slot with :cfunc:`PyType_GenericNew` +necessary to fill out the :attr:`tp_alloc` slot with :c:func:`PyType_GenericNew` -- the allocate function from the base type will be inherited. -After that, calling :cfunc:`PyType_Ready` and adding the type object to the +After that, calling :c:func:`PyType_Ready` and adding the type object to the module is the same as with the basic :class:`Noddy` examples. @@ -891,7 +890,7 @@ Type Methods This section aims to give a quick fly-by on the various type methods you can implement and what they do. -Here is the definition of :ctype:`PyTypeObject`, with some fields only used in +Here is the definition of :c:type:`PyTypeObject`, with some fields only used in debug builds omitted: .. literalinclude:: ../includes/typestruct.h @@ -969,8 +968,8 @@ which a deallocator performs which may cause additional Python code to be executed may detect that an exception has been set. This can lead to misleading errors from the interpreter. The proper way to protect against this is to save a pending exception before performing the unsafe action, and restoring it when -done. This can be done using the :cfunc:`PyErr_Fetch` and -:cfunc:`PyErr_Restore` functions:: +done. This can be done using the :c:func:`PyErr_Fetch` and +:c:func:`PyErr_Restore` functions:: static void my_dealloc(PyObject *obj) @@ -1060,8 +1059,8 @@ a special case, for which the new value passed to the handler is *NULL*. Python supports two pairs of attribute handlers; a type that supports attributes only needs to implement the functions for one pair. The difference is that one -pair takes the name of the attribute as a :ctype:`char\*`, while the other -accepts a :ctype:`PyObject\*`. Each type can use whichever pair makes more +pair takes the name of the attribute as a :c:type:`char\*`, while the other +accepts a :c:type:`PyObject\*`. Each type can use whichever pair makes more sense for the implementation's convenience. :: getattrfunc tp_getattr; /* char * version */ @@ -1072,7 +1071,7 @@ sense for the implementation's convenience. :: If accessing attributes of an object is always a simple operation (this will be explained shortly), there are generic implementations which can be used to -provide the :ctype:`PyObject\*` version of the attribute management functions. +provide the :c:type:`PyObject\*` version of the attribute management functions. The actual need for type-specific attribute handlers almost completely disappeared starting with Python 2.2, though there are many examples which have not been updated to use some of the new generic mechanism that is available. @@ -1086,7 +1085,7 @@ Generic Attribute Management Most extension types only use *simple* attributes. So, what makes the attributes simple? There are only a couple of conditions that must be met: -#. The name of the attributes must be known when :cfunc:`PyType_Ready` is +#. The name of the attributes must be known when :c:func:`PyType_Ready` is called. #. No special processing is needed to record that an attribute was looked up or @@ -1095,7 +1094,7 @@ attributes simple? There are only a couple of conditions that must be met: Note that this list does not place any restrictions on the values of the attributes, when the values are computed, or how relevant data is stored. -When :cfunc:`PyType_Ready` is called, it uses three tables referenced by the +When :c:func:`PyType_Ready` is called, it uses three tables referenced by the type object to create :term:`descriptor`\s which are placed in the dictionary of the type object. Each descriptor controls access to one attribute of the instance object. Each of the tables is optional; if all three are *NULL*, instances of @@ -1110,7 +1109,7 @@ The tables are declared as three fields of the type object:: struct PyGetSetDef *tp_getset; If :attr:`tp_methods` is not *NULL*, it must refer to an array of -:ctype:`PyMethodDef` structures. Each entry in the table is an instance of this +:c:type:`PyMethodDef` structures. Each entry in the table is an instance of this structure:: typedef struct PyMethodDef { @@ -1192,9 +1191,9 @@ of *NULL* is required. Type-specific Attribute Management ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -For simplicity, only the :ctype:`char\*` version will be demonstrated here; the -type of the name parameter is the only difference between the :ctype:`char\*` -and :ctype:`PyObject\*` flavors of the interface. This example effectively does +For simplicity, only the :c:type:`char\*` version will be demonstrated here; the +type of the name parameter is the only difference between the :c:type:`char\*` +and :c:type:`PyObject\*` flavors of the interface. This example effectively does the same thing as the generic example above, but does not use the generic support added in Python 2.2. It explains how the handler functions are called, so that if you do need to extend their functionality, you'll understand @@ -1242,8 +1241,8 @@ Object Comparison The :attr:`tp_richcompare` handler is called when comparisons are needed. It is analogous to the :ref:`rich comparison methods <richcmpfuncs>`, like -:meth:`__lt__`, and also called by :cfunc:`PyObject_RichCompare` and -:cfunc:`PyObject_RichCompareBool`. +:meth:`__lt__`, and also called by :c:func:`PyObject_RichCompare` and +:c:func:`PyObject_RichCompareBool`. This function is called with two Python objects and the operator as arguments, where the operator is one of ``Py_EQ``, ``Py_NE``, ``Py_LE``, ``Py_GT``, @@ -1306,8 +1305,8 @@ to indicate the presence of a slot, but a slot may still be unfilled.) :: If you wish your object to be able to act like a number, a sequence, or a mapping object, then you place the address of a structure that implements the C -type :ctype:`PyNumberMethods`, :ctype:`PySequenceMethods`, or -:ctype:`PyMappingMethods`, respectively. It is up to you to fill in this +type :c:type:`PyNumberMethods`, :c:type:`PySequenceMethods`, or +:c:type:`PyMappingMethods`, respectively. It is up to you to fill in this structure with appropriate values. You can find examples of the use of each of these in the :file:`Objects` directory of the Python source distribution. :: @@ -1339,11 +1338,11 @@ This function takes three arguments: the call is ``obj1('hello')``, then *arg1* is ``obj1``. #. *arg2* is a tuple containing the arguments to the call. You can use - :cfunc:`PyArg_ParseTuple` to extract the arguments. + :c:func:`PyArg_ParseTuple` to extract the arguments. #. *arg3* is a dictionary of keyword arguments that were passed. If this is non-*NULL* and you support keyword arguments, use - :cfunc:`PyArg_ParseTupleAndKeywords` to extract the arguments. If you do not + :c:func:`PyArg_ParseTupleAndKeywords` to extract the arguments. If you do not want to support keyword arguments and this is non-*NULL*, raise a :exc:`TypeError` with a message saying that keyword arguments are not supported. @@ -1417,7 +1416,7 @@ to participate in the weak reference mechanism without incurring the overhead on those objects which do not benefit by weak referencing (such as numbers). For an object to be weakly referencable, the extension must include a -:ctype:`PyObject\*` field in the instance structure for the use of the weak +:c:type:`PyObject\*` field in the instance structure for the use of the weak reference mechanism; it must be initialized to *NULL* by the object's constructor. It must also set the :attr:`tp_weaklistoffset` field of the corresponding type object to the offset of the field. For example, the instance @@ -1493,7 +1492,7 @@ the function you want (for example, ``tp_richcompare``). You will find examples of the function you want to implement. When you need to verify that an object is an instance of the type you are -implementing, use the :cfunc:`PyObject_TypeCheck` function. A sample of its use +implementing, use the :c:func:`PyObject_TypeCheck` function. A sample of its use might be something like the following:: if (! PyObject_TypeCheck(some_object, &MyType)) { diff --git a/Doc/extending/windows.rst b/Doc/extending/windows.rst index d1d0cf7ee9..3fd5e576de 100644 --- a/Doc/extending/windows.rst +++ b/Doc/extending/windows.rst @@ -98,8 +98,8 @@ described here are distributed with the Python sources in the it. Copy your C sources into it. Note that the module source file name does not necessarily have to match the module name, but the name of the initialization function should match the module name --- you can only import a - module :mod:`spam` if its initialization function is called :cfunc:`initspam`, - and it should call :cfunc:`Py_InitModule` with the string ``"spam"`` as its + module :mod:`spam` if its initialization function is called :c:func:`initspam`, + and it should call :c:func:`Py_InitModule` with the string ``"spam"`` as its first argument (use the minimal :file:`example.c` in this directory as a guide). By convention, it lives in a file called :file:`spam.c` or :file:`spammodule.c`. The output file should be called :file:`spam.pyd` (in Release mode) or @@ -259,7 +259,7 @@ use these commands:: The first command created three files: :file:`spam.obj`, :file:`spam.dll` and :file:`spam.lib`. :file:`Spam.dll` does not contain any Python functions (such -as :cfunc:`PyArg_ParseTuple`), but it does know how to find the Python code +as :c:func:`PyArg_ParseTuple`), but it does know how to find the Python code thanks to :file:`pythonXY.lib`. The second command created :file:`ni.dll` (and :file:`.obj` and :file:`.lib`), diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst index 3889774ee5..e45aaaacb6 100644 --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -380,11 +380,24 @@ is exactly the same type of object that a lambda form yields) is assigned! Can Python be compiled to machine code, C or some other language? ----------------------------------------------------------------- -Not easily. Python's high level data types, dynamic typing of objects and +Practical answer: + +`Cython <http://cython.org/>`_ and `Pyrex <http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/>`_ +compile a modified version of Python with optional annotations into C +extensions. `Weave <http://www.scipy.org/Weave>`_ makes it easy to +intermingle Python and C code in various ways to increase performance. +`Nuitka <http://www.nuitka.net/>`_ is an up-and-coming compiler of Python +into C++ code, aiming to support the full Python language. + +Theoretical answer: + + .. XXX not sure what to make of this + +Not trivially. Python's high level data types, dynamic typing of objects and run-time invocation of the interpreter (using :func:`eval` or :func:`exec`) -together mean that a "compiled" Python program would probably consist mostly of -calls into the Python run-time system, even for seemingly simple operations like -``x+1``. +together mean that a naïvely "compiled" Python program would probably consist +mostly of calls into the Python run-time system, even for seemingly simple +operations like ``x+1``. Several projects described in the Python newsgroup or at past `Python conferences <http://python.org/community/workshops/>`_ have shown that this @@ -395,101 +408,64 @@ speedups of 1000x are feasible for small demo programs. See the proceedings from the `1997 Python conference <http://python.org/workshops/1997-10/proceedings/>`_ for more information.) -Internally, Python source code is always translated into a bytecode -representation, and this bytecode is then executed by the Python virtual -machine. In order to avoid the overhead of repeatedly parsing and translating -modules that rarely change, this byte code is written into a file whose name -ends in ".pyc" whenever a module is parsed. When the corresponding .py file is -changed, it is parsed and translated again and the .pyc file is rewritten. - -There is no performance difference once the .pyc file has been loaded, as the -bytecode read from the .pyc file is exactly the same as the bytecode created by -direct translation. The only difference is that loading code from a .pyc file -is faster than parsing and translating a .py file, so the presence of -precompiled .pyc files improves the start-up time of Python scripts. If -desired, the Lib/compileall.py module can be used to create valid .pyc files for -a given set of modules. - -Note that the main script executed by Python, even if its filename ends in .py, -is not compiled to a .pyc file. It is compiled to bytecode, but the bytecode is -not saved to a file. Usually main scripts are quite short, so this doesn't cost -much speed. - -.. XXX check which of these projects are still alive - -There are also several programs which make it easier to intermingle Python and C -code in various ways to increase performance. See, for example, `Psyco -<http://psyco.sourceforge.net/>`_, `Pyrex -<http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/>`_, `PyInline -<http://pyinline.sourceforge.net/>`_, `Py2Cmod -<http://sourceforge.net/projects/py2cmod/>`_, and `Weave -<http://www.scipy.org/Weave>`_. - How does Python manage memory? ------------------------------ The details of Python memory management depend on the implementation. The -standard C implementation of Python uses reference counting to detect -inaccessible objects, and another mechanism to collect reference cycles, +standard implementation of Python, :term:`CPython`, uses reference counting to +detect inaccessible objects, and another mechanism to collect reference cycles, periodically executing a cycle detection algorithm which looks for inaccessible cycles and deletes the objects involved. The :mod:`gc` module provides functions to perform a garbage collection, obtain debugging statistics, and tune the collector's parameters. -Jython relies on the Java runtime so the JVM's garbage collector is used. This -difference can cause some subtle porting problems if your Python code depends on -the behavior of the reference counting implementation. +Other implementations (such as `Jython <http://www.jython.org>`_ or +`PyPy <http://www.pypy.org>`_), however, can rely on a different mechanism +such as a full-blown garbage collector. This difference can cause some +subtle porting problems if your Python code depends on the behavior of the +reference counting implementation. -.. XXX relevant for Python 3? +In some Python implementations, the following code (which is fine in CPython) +will probably run out of file descriptors:: - Sometimes objects get stuck in traceback temporarily and hence are not - deallocated when you might expect. Clear the traceback with:: + for file in very_long_list_of_files: + f = open(file) + c = f.read(1) + +Indeed, using CPython's reference counting and destructor scheme, each new +assignment to *f* closes the previous file. With a traditional GC, however, +those file objects will only get collected (and closed) at varying and possibly +long intervals. - import sys - sys.last_traceback = None +If you want to write code that will work with any Python implementation, +you should explicitly close the file or use the :keyword:`with` statement; +this will work regardless of memory management scheme:: - Tracebacks are used for reporting errors, implementing debuggers and related - things. They contain a portion of the program state extracted during the - handling of an exception (usually the most recent exception). + for file in very_long_list_of_files: + with open(file) as f: + c = f.read(1) -In the absence of circularities, Python programs do not need to manage memory -explicitly. -Why doesn't Python use a more traditional garbage collection scheme? For one -thing, this is not a C standard feature and hence it's not portable. (Yes, we -know about the Boehm GC library. It has bits of assembler code for *most* -common platforms, not for all of them, and although it is mostly transparent, it -isn't completely transparent; patches are required to get Python to work with -it.) +Why doesn't CPython use a more traditional garbage collection scheme? +--------------------------------------------------------------------- + +For one thing, this is not a C standard feature and hence it's not portable. +(Yes, we know about the Boehm GC library. It has bits of assembler code for +*most* common platforms, not for all of them, and although it is mostly +transparent, it isn't completely transparent; patches are required to get +Python to work with it.) Traditional GC also becomes a problem when Python is embedded into other applications. While in a standalone Python it's fine to replace the standard malloc() and free() with versions provided by the GC library, an application embedding Python may want to have its *own* substitute for malloc() and free(), -and may not want Python's. Right now, Python works with anything that +and may not want Python's. Right now, CPython works with anything that implements malloc() and free() properly. -In Jython, the following code (which is fine in CPython) will probably run out -of file descriptors long before it runs out of memory:: - - for file in very_long_list_of_files: - f = open(file) - c = f.read(1) - -Using the current reference counting and destructor scheme, each new assignment -to f closes the previous file. Using GC, this is not guaranteed. If you want -to write code that will work with any Python implementation, you should -explicitly close the file or use the :keyword:`with` statement; this will work -regardless of GC:: - for file in very_long_list_of_files: - with open(file) as f: - c = f.read(1) - - -Why isn't all memory freed when Python exits? ---------------------------------------------- +Why isn't all memory freed when CPython exits? +---------------------------------------------- Objects referenced from the global namespaces of Python modules are not always deallocated when Python exits. This may happen if there are circular @@ -649,9 +625,10 @@ order to remind you of that fact, it does not return the sorted list. This way, you won't be fooled into accidentally overwriting a list when you need a sorted copy but also need to keep the unsorted version around. -In Python 2.4 a new builtin -- :func:`sorted` -- has been added. This function -creates a new list from a provided iterable, sorts it and returns it. For -example, here's how to iterate over the keys of a dictionary in sorted order:: +If you want to return a new list, use the built-in :func:`sorted` function +instead. This function creates a new list from a provided iterable, sorts +it and returns it. For example, here's how to iterate over the keys of a +dictionary in sorted order:: for key in sorted(mydict): ... # do whatever with mydict[key]... @@ -668,7 +645,7 @@ construction of large programs. Python 2.6 adds an :mod:`abc` module that lets you define Abstract Base Classes (ABCs). You can then use :func:`isinstance` and :func:`issubclass` to check whether an instance or a class implements a particular ABC. The -:mod:`collections` modules defines a set of useful ABCs such as +:mod:`collections` module defines a set of useful ABCs such as :class:`Iterable`, :class:`Container`, and :class:`MutableMapping`. For Python, many of the advantages of interface specifications can be obtained diff --git a/Doc/faq/extending.rst b/Doc/faq/extending.rst index 6527ff189b..7c684a0929 100644 --- a/Doc/faq/extending.rst +++ b/Doc/faq/extending.rst @@ -37,18 +37,13 @@ Writing C is hard; are there any alternatives? There are a number of alternatives to writing your own C extensions, depending on what you're trying to do. -.. XXX make sure these all work; mention Cython +.. XXX make sure these all work -If you need more speed, `Psyco <http://psyco.sourceforge.net/>`_ generates x86 -assembly code from Python bytecode. You can use Psyco to compile the most -time-critical functions in your code, and gain a significant improvement with -very little effort, as long as you're running on a machine with an -x86-compatible processor. - -`Pyrex <http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/>`_ is a compiler -that accepts a slightly modified form of Python and generates the corresponding -C code. Pyrex makes it possible to write an extension without having to learn -Python's C API. +`Cython <http://cython.org>`_ and its relative `Pyrex +<http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/>`_ are compilers +that accept a slightly modified form of Python and generate the corresponding +C code. Cython and Pyrex make it possible to write an extension without having +to learn Python's C API. If you need to interface to some C or C++ library for which no Python extension currently exists, you can try wrapping the library's data types and functions @@ -56,67 +51,63 @@ with a tool such as `SWIG <http://www.swig.org>`_. `SIP <http://www.riverbankcomputing.co.uk/software/sip/>`__, `CXX <http://cxx.sourceforge.net/>`_ `Boost <http://www.boost.org/libs/python/doc/index.html>`_, or `Weave -<http://www.scipy.org/Weave>`_ are also alternatives for wrapping C++ libraries. +<http://www.scipy.org/Weave>`_ are also alternatives for wrapping +C++ libraries. How can I execute arbitrary Python statements from C? ----------------------------------------------------- -The highest-level function to do this is :cfunc:`PyRun_SimpleString` which takes +The highest-level function to do this is :c:func:`PyRun_SimpleString` which takes a single string argument to be executed in the context of the module ``__main__`` and returns 0 for success and -1 when an exception occurred (including ``SyntaxError``). If you want more control, use -:cfunc:`PyRun_String`; see the source for :cfunc:`PyRun_SimpleString` in +:c:func:`PyRun_String`; see the source for :c:func:`PyRun_SimpleString` in ``Python/pythonrun.c``. How can I evaluate an arbitrary Python expression from C? --------------------------------------------------------- -Call the function :cfunc:`PyRun_String` from the previous question with the -start symbol :cdata:`Py_eval_input`; it parses an expression, evaluates it and +Call the function :c:func:`PyRun_String` from the previous question with the +start symbol :c:data:`Py_eval_input`; it parses an expression, evaluates it and returns its value. How do I extract C values from a Python object? ----------------------------------------------- -That depends on the object's type. If it's a tuple, :cfunc:`PyTuple_Size` -returns its length and :cfunc:`PyTuple_GetItem` returns the item at a specified -index. Lists have similar functions, :cfunc:`PyListSize` and -:cfunc:`PyList_GetItem`. +That depends on the object's type. If it's a tuple, :c:func:`PyTuple_Size` +returns its length and :c:func:`PyTuple_GetItem` returns the item at a specified +index. Lists have similar functions, :c:func:`PyListSize` and +:c:func:`PyList_GetItem`. -For strings, :cfunc:`PyString_Size` returns its length and -:cfunc:`PyString_AsString` a pointer to its value. Note that Python strings may -contain null bytes so C's :cfunc:`strlen` should not be used. +For strings, :c:func:`PyString_Size` returns its length and +:c:func:`PyString_AsString` a pointer to its value. Note that Python strings may +contain null bytes so C's :c:func:`strlen` should not be used. To test the type of an object, first make sure it isn't *NULL*, and then use -:cfunc:`PyString_Check`, :cfunc:`PyTuple_Check`, :cfunc:`PyList_Check`, etc. +:c:func:`PyString_Check`, :c:func:`PyTuple_Check`, :c:func:`PyList_Check`, etc. There is also a high-level API to Python objects which is provided by the so-called 'abstract' interface -- read ``Include/abstract.h`` for further details. It allows interfacing with any kind of Python sequence using calls -like :cfunc:`PySequence_Length`, :cfunc:`PySequence_GetItem`, etc.) as well as +like :c:func:`PySequence_Length`, :c:func:`PySequence_GetItem`, etc.) as well as many other useful protocols. How do I use Py_BuildValue() to create a tuple of arbitrary length? ------------------------------------------------------------------- -You can't. Use ``t = PyTuple_New(n)`` instead, and fill it with objects using -``PyTuple_SetItem(t, i, o)`` -- note that this "eats" a reference count of -``o``, so you have to :cfunc:`Py_INCREF` it. Lists have similar functions -``PyList_New(n)`` and ``PyList_SetItem(l, i, o)``. Note that you *must* set all -the tuple items to some value before you pass the tuple to Python code -- -``PyTuple_New(n)`` initializes them to NULL, which isn't a valid Python value. +You can't. Use :c:func:`PyTuple_Pack` instead. How do I call an object's method from C? ---------------------------------------- -The :cfunc:`PyObject_CallMethod` function can be used to call an arbitrary +The :c:func:`PyObject_CallMethod` function can be used to call an arbitrary method of an object. The parameters are the object, the name of the method to -call, a format string like that used with :cfunc:`Py_BuildValue`, and the +call, a format string like that used with :c:func:`Py_BuildValue`, and the argument values:: PyObject * @@ -124,7 +115,7 @@ argument values:: char *arg_format, ...); This works for any object that has methods -- whether built-in or user-defined. -You are responsible for eventually :cfunc:`Py_DECREF`\ 'ing the return value. +You are responsible for eventually :c:func:`Py_DECREF`\ 'ing the return value. To call, e.g., a file object's "seek" method with arguments 10, 0 (assuming the file object pointer is "f"):: @@ -137,7 +128,7 @@ file object pointer is "f"):: Py_DECREF(res); } -Note that since :cfunc:`PyObject_CallObject` *always* wants a tuple for the +Note that since :c:func:`PyObject_CallObject` *always* wants a tuple for the argument list, to call a function without arguments, pass "()" for the format, and to call a function with one argument, surround the argument in parentheses, e.g. "(i)". @@ -151,21 +142,30 @@ this object to :data:`sys.stdout` and :data:`sys.stderr`. Call print_error, or just allow the standard traceback mechanism to work. Then, the output will go wherever your ``write()`` method sends it. -The easiest way to do this is to use the StringIO class in the standard library. +The easiest way to do this is to use the :class:`io.StringIO` class:: + + >>> import io, sys + >>> sys.stdout = io.StringIO() + >>> print('foo') + >>> print('hello world!') + >>> sys.stderr.write(sys.stdout.getvalue()) + foo + hello world! -Sample code and use for catching stdout: +A custom object to do the same would look like this:: - >>> class StdoutCatcher: + >>> import io, sys + >>> class StdoutCatcher(io.TextIOBase): ... def __init__(self): - ... self.data = '' + ... self.data = [] ... def write(self, stuff): - ... self.data = self.data + stuff + ... self.data.append(stuff) ... >>> import sys >>> sys.stdout = StdoutCatcher() >>> print('foo') >>> print('hello world!') - >>> sys.stderr.write(sys.stdout.data) + >>> sys.stderr.write(''.join(sys.stdout.data)) foo hello world! @@ -188,7 +188,7 @@ module) as follows:: attr = PyObject_GetAttrString(module, "<attrname>"); -Calling :cfunc:`PyObject_SetAttrString` to assign to variables in the module +Calling :c:func:`PyObject_SetAttrString` to assign to variables in the module also works. @@ -201,11 +201,7 @@ begin by reading :ref:`the "Extending and Embedding" document whole lot of difference between C and C++ -- so the strategy of building a new Python type around a C structure (pointer) type will also work for C++ objects. -For C++ libraries, you can look at `SIP -<http://www.riverbankcomputing.co.uk/sip/>`_, `CXX -<http://cxx.sourceforge.net/>`_, `Boost -<http://www.boost.org/libs/python/doc/index.html>`_, `Weave -<http://www.scipy.org/Weave>`_ or `SWIG <http://www.swig.org>`_ +For C++ libraries, see :ref:`c-wrapper-software`. I added a module using the Setup file and the make fails; why? @@ -273,16 +269,16 @@ the input is invalid. In Python you can use the :mod:`codeop` module, which approximates the parser's behavior sufficiently. IDLE uses this, for example. -The easiest way to do it in C is to call :cfunc:`PyRun_InteractiveLoop` (perhaps +The easiest way to do it in C is to call :c:func:`PyRun_InteractiveLoop` (perhaps in a separate thread) and let the Python interpreter handle the input for -you. You can also set the :cfunc:`PyOS_ReadlineFunctionPointer` to point at your +you. You can also set the :c:func:`PyOS_ReadlineFunctionPointer` to point at your custom input function. See ``Modules/readline.c`` and ``Parser/myreadline.c`` for more hints. However sometimes you have to run the embedded Python interpreter in the same thread as your rest application and you can't allow the -:cfunc:`PyRun_InteractiveLoop` to stop while waiting for user input. The one -solution then is to call :cfunc:`PyParser_ParseString` and test for ``e.error`` +:c:func:`PyRun_InteractiveLoop` to stop while waiting for user input. The one +solution then is to call :c:func:`PyParser_ParseString` and test for ``e.error`` equal to ``E_EOF``, which means the input is incomplete). Here's a sample code fragment, untested, inspired by code from Alex Farber:: @@ -313,8 +309,8 @@ fragment, untested, inspired by code from Alex Farber:: } Another solution is trying to compile the received string with -:cfunc:`Py_CompileString`. If it compiles without errors, try to execute the -returned code object by calling :cfunc:`PyEval_EvalCode`. Otherwise save the +:c:func:`Py_CompileString`. If it compiles without errors, try to execute the +returned code object by calling :c:func:`PyEval_EvalCode`. Otherwise save the input for later. If the compilation fails, find out if it's an error or just more input is required - by extracting the message string from the exception tuple and comparing it to the string "unexpected EOF while parsing". Here is a @@ -382,7 +378,7 @@ complete example using the GNU readline library (you may want to ignore if (ps1 == prompt || /* ">>> " or */ '\n' == code[i + j - 1]) /* "... " and double '\n' */ { /* so execute it */ - dum = PyEval_EvalCode ((PyCodeObject *)src, glb, loc); + dum = PyEval_EvalCode (src, glb, loc); Py_XDECREF (dum); Py_XDECREF (src); free (code); @@ -443,7 +439,7 @@ extension module using g++ (e.g., ``g++ -shared -o mymodule.so mymodule.o``). Can I create an object class with some methods implemented in C and others in Python (e.g. through inheritance)? ---------------------------------------------------------------------------------------------------------------- -In Python 2.2, you can inherit from builtin classes such as :class:`int`, +In Python 2.2, you can inherit from built-in classes such as :class:`int`, :class:`list`, :class:`dict`, etc. The Boost Python Library (BPL, http://www.boost.org/libs/python/doc/index.html) @@ -466,8 +462,8 @@ This can easily occur when using pre-built extension packages. RedHat Linux 7.x, in particular, provided a "python2" binary that is compiled with 4-byte Unicode. This only causes the link failure if the extension uses any of the ``PyUnicode_*()`` functions. It is also a problem if an extension uses any of -the Unicode-related format specifiers for :cfunc:`Py_BuildValue` (or similar) or -parameter specifications for :cfunc:`PyArg_ParseTuple`. +the Unicode-related format specifiers for :c:func:`Py_BuildValue` (or similar) or +parameter specifications for :c:func:`PyArg_ParseTuple`. You can check the size of the Unicode character a Python interpreter is using by checking the value of sys.maxunicode: diff --git a/Doc/faq/general.rst b/Doc/faq/general.rst index 111e3123db..9f26dc9f8a 100644 --- a/Doc/faq/general.rst +++ b/Doc/faq/general.rst @@ -19,7 +19,7 @@ It has interfaces to many system calls and libraries, as well as to various window systems, and is extensible in C or C++. It is also usable as an extension language for applications that need a programmable interface. Finally, Python is portable: it runs on many Unix variants, on the Mac, and on -PCs under MS-DOS, Windows, Windows NT, and OS/2. +Windows 2000 and later. To find out more, start with :ref:`tutorial-index`. The `Beginner's Guide to Python <http://wiki.python.org/moin/BeginnersGuide>`_ links to other @@ -157,16 +157,14 @@ How do I obtain a copy of the Python source? The latest Python source distribution is always available from python.org, at http://www.python.org/download/. The latest development sources can be obtained -via anonymous Subversion at http://svn.python.org/projects/python/trunk. +via anonymous Mercurial access at http://hg.python.org/cpython. The source distribution is a gzipped tar file containing the complete C source, Sphinx-formatted documentation, Python library modules, example programs, and several useful pieces of freely distributable software. The source will compile and run out of the box on most UNIX platforms. -.. XXX update link once the dev faq is relocated - -Consult the `Developer FAQ <http://www.python.org/dev/faq/>`__ for more +Consult the `Developer FAQ <http://docs.python.org/devguide/faq>`__ for more information on getting the source code and compiling it. @@ -221,10 +219,8 @@ releases are announced on the comp.lang.python and comp.lang.python.announce newsgroups and on the Python home page at http://www.python.org/; an RSS feed of news is available. -.. XXX update link once the dev faq is relocated - You can also access the development version of Python through Subversion. See -http://www.python.org/dev/faq/ for details. +http://docs.python.org/devguide/faq for details. How do I submit bug reports and patches for Python? @@ -239,10 +235,8 @@ you updates as we act on your bug. If you had previously used SourceForge to report bugs to Python, you can obtain your Roundup password through Roundup's `password reset procedure <http://bugs.python.org/user?@template=forgotten>`_. -.. XXX adapt link to dev guide - For more information on how Python is developed, consult `the Python Developer's -Guide <http://python.org/dev/>`_. +Guide <http://docs.python.org/devguide/>`_. Are there any published articles about Python that I can reference? @@ -475,38 +469,3 @@ http://www.python.org/editors/ for a full list of Python editing environments. If you want to discuss Python's use in education, you may be interested in joining `the edu-sig mailing list <http://python.org/community/sigs/current/edu-sig>`_. - - -Upgrading Python -================ - -What is this bsddb185 module my application keeps complaining about? --------------------------------------------------------------------- - -.. XXX remove this question? - -Starting with Python2.3, the distribution includes the `PyBSDDB package -<http://pybsddb.sf.net/>` as a replacement for the old bsddb module. It -includes functions which provide backward compatibility at the API level, but -requires a newer version of the underlying `Berkeley DB -<http://www.sleepycat.com>`_ library. Files created with the older bsddb module -can't be opened directly using the new module. - -Using your old version of Python and a pair of scripts which are part of Python -2.3 (db2pickle.py and pickle2db.py, in the Tools/scripts directory) you can -convert your old database files to the new format. Using your old Python -version, run the db2pickle.py script to convert it to a pickle, e.g.:: - - python2.2 <pathto>/db2pickley.py database.db database.pck - -Rename your database file:: - - mv database.db olddatabase.db - -Now convert the pickle file to a new format database:: - - python <pathto>/pickle2db.py database.db database.pck - -The precise commands you use will vary depending on the particulars of your -installation. For full details about operation of these two scripts check the -doc string at the start of each one. diff --git a/Doc/faq/gui.rst b/Doc/faq/gui.rst index 03177c9586..f697cd3c51 100644 --- a/Doc/faq/gui.rst +++ b/Doc/faq/gui.rst @@ -15,7 +15,9 @@ General GUI Questions What platform-independent GUI toolkits exist for Python? ======================================================== -Depending on what platform(s) you are aiming at, there are several. +Depending on what platform(s) you are aiming at, there are several. Some +of them haven't been ported to Python 3 yet. At least `Tkinter`_ and `Qt`_ +are known to be Python 3-compatible. .. XXX check links @@ -23,12 +25,14 @@ Tkinter ------- Standard builds of Python include an object-oriented interface to the Tcl/Tk -widget set, called Tkinter. This is probably the easiest to install and use. -For more info about Tk, including pointers to the source, see the Tcl/Tk home -page at http://www.tcl.tk. Tcl/Tk is fully portable to the MacOS, Windows, and -Unix platforms. - -wxWindows +widget set, called :ref:`tkinter <Tkinter>`. This is probably the easiest to +install (since it comes included with most +`binary distributions <http://www.python.org/download/>`_ of Python) and use. +For more info about Tk, including pointers to the source, see the +`Tcl/Tk home page <http://www.tcl.tk>`_. Tcl/Tk is fully portable to the +MacOS, Windows, and Unix platforms. + +wxWidgets --------- wxWidgets (http://www.wxwidgets.org) is a free, portable GUI class @@ -47,22 +51,29 @@ Both wxWidgets and wxPython are free, open source, software with permissive licences that allow their use in commercial products as well as in freeware or shareware. + Qt --- -There are bindings available for the Qt toolkit (`PyQt -<http://www.riverbankcomputing.co.uk/software/pyqt/>`_) and for KDE (PyKDE). If -you're writing open source software, you don't need to pay for PyQt, but if you -want to write proprietary applications, you must buy a PyQt license from -`Riverbank Computing <http://www.riverbankcomputing.co.uk>`_ and (up to Qt 4.4; -Qt 4.5 upwards is licensed under the LGPL license) a Qt license from `Trolltech -<http://www.trolltech.com>`_. +There are bindings available for the Qt toolkit (using either `PyQt +<http://www.riverbankcomputing.co.uk/software/pyqt/>`_ or `PySide +<http://www.pyside.org/>`_) and for KDE (`PyKDE <http://www.riverbankcomputing.co.uk/software/pykde/intro>`__). +PyQt is currently more mature than PySide, but you must buy a PyQt license from +`Riverbank Computing <http://www.riverbankcomputing.co.uk/software/pyqt/license>`_ +if you want to write proprietary applications. PySide is free for all applications. + +Qt 4.5 upwards is licensed under the LGPL license; also, commercial licenses +are available from `Nokia <http://qt.nokia.com/>`_. Gtk+ ---- -PyGtk bindings for the `Gtk+ toolkit <http://www.gtk.org>`_ have been -implemented by by James Henstridge; see ftp://ftp.gtk.org/pub/gtk/python/. +The `GObject introspection bindings <https://live.gnome.org/PyGObject>`_ +for Python allow you to write GTK+ 3 applications. There is also a +`Python GTK+ 3 Tutorial <http://python-gtk-3-tutorial.readthedocs.org/en/latest/>`_. + +The older PyGtk bindings for the `Gtk+ 2 toolkit <http://www.gtk.org>`_ have +been implemented by James Henstridge; see <http://www.pygtk.org>. FLTK ---- @@ -91,14 +102,15 @@ What platform-specific GUI toolkits exist for Python? `The Mac port <http://python.org/download/mac>`_ by Jack Jansen has a rich and ever-growing set of modules that support the native Mac toolbox calls. The port -includes support for MacOS9 and MacOS X's Carbon libraries. By installing the -`PyObjc Objective-C bridge <http://pyobjc.sourceforge.net>`_, Python programs -can use MacOS X's Cocoa libraries. See the documentation that comes with the Mac -port. +supports MacOS X's Carbon libraries. + +By installing the `PyObjc Objective-C bridge +<http://pyobjc.sourceforge.net>`_, Python programs can use MacOS X's +Cocoa libraries. See the documentation that comes with the Mac port. :ref:`Pythonwin <windows-faq>` by Mark Hammond includes an interface to the -Microsoft Foundation Classes and a Python programming environment using it -that's written mostly in Python. +Microsoft Foundation Classes and a Python programming environment +that's written mostly in Python using the MFC classes. Tkinter questions @@ -111,23 +123,26 @@ Freeze is a tool to create stand-alone applications. When freezing Tkinter applications, the applications will not be truly stand-alone, as the application will still need the Tcl and Tk libraries. -One solution is to ship the application with the tcl and tk libraries, and point +One solution is to ship the application with the Tcl and Tk libraries, and point to them at run-time using the :envvar:`TCL_LIBRARY` and :envvar:`TK_LIBRARY` environment variables. To get truly stand-alone applications, the Tcl scripts that form the library have to be integrated into the application as well. One tool supporting that is SAM (stand-alone modules), which is part of the Tix distribution -(http://tix.mne.com). Build Tix with SAM enabled, perform the appropriate call -to Tclsam_init etc inside Python's Modules/tkappinit.c, and link with libtclsam -and libtksam (you might include the Tix libraries as well). +(http://tix.sourceforge.net/). + +Build Tix with SAM enabled, perform the appropriate call to +:c:func:`Tclsam_init`, etc. inside Python's +:file:`Modules/tkappinit.c`, and link with libtclsam and libtksam (you +might include the Tix libraries as well). Can I have Tk events handled while waiting for I/O? --------------------------------------------------- Yes, and you don't even need threads! But you'll have to restructure your I/O -code a bit. Tk has the equivalent of Xt's XtAddInput() call, which allows you +code a bit. Tk has the equivalent of Xt's :c:func:`XtAddInput()` call, which allows you to register a callback function which will be called from the Tk mainloop when I/O is possible on a file descriptor. Here's what you need:: diff --git a/Doc/faq/installed.rst b/Doc/faq/installed.rst index 390c85abe4..efec9bf791 100644 --- a/Doc/faq/installed.rst +++ b/Doc/faq/installed.rst @@ -24,14 +24,14 @@ there are several possible ways it could have gotten there. it; you'll have to figure out who's been using the machine and might have installed it. * A third-party application installed on the machine might have been written in - Python and included a Python installation. For a home computer, the most - common such application is `PySol <http://pysolfc.sourceforge.net/>`_, a - solitaire game that includes over 1000 different games and variations. + Python and included a Python installation. There are many such applications, + from GUI programs to network servers and administrative scripts. * Some Windows machines also have Python installed. At this writing we're aware of computers from Hewlett-Packard and Compaq that include Python. Apparently some of HP/Compaq's administrative tools are written in Python. -* All Apple computers running Mac OS X have Python installed; it's included in - the base installation. +* Many Unix-compatible operating systems, such as Mac OS X and some Linux + distributions, have Python installed by default; it's included in the base + installation. Can I delete Python? diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst index 180bd3916f..a079cb160c 100644 --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -25,10 +25,10 @@ your topic of interest will usually find something helpful. Where is the math.py (socket.py, regex.py, etc.) source file? ------------------------------------------------------------- -If you can't find a source file for a module it may be a builtin or dynamically -loaded module implemented in C, C++ or other compiled language. In this case -you may not have the source file or it may be something like mathmodule.c, -somewhere in a C source directory (not on the Python Path). +If you can't find a source file for a module it may be a built-in or +dynamically loaded module implemented in C, C++ or other compiled language. +In this case you may not have the source file or it may be something like +mathmodule.c, somewhere in a C source directory (not on the Python Path). There are (at least) three kinds of modules in Python: @@ -285,11 +285,15 @@ queue as there are threads. How do I parcel out work among a bunch of worker threads? --------------------------------------------------------- -Use the :mod:`queue` module to create a queue containing a list of jobs. The -:class:`~queue.Queue` class maintains a list of objects with ``.put(obj)`` to -add an item to the queue and ``.get()`` to return an item. The class will take -care of the locking necessary to ensure that each job is handed out exactly -once. +The easiest way is to use the new :mod:`concurrent.futures` module, +especially the :mod:`~concurrent.futures.ThreadPoolExecutor` class. + +Or, if you want fine control over the dispatching algorithm, you can write +your own logic manually. Use the :mod:`queue` module to create a queue +containing a list of jobs. The :class:`~queue.Queue` class maintains a +list of objects with ``.put(obj)`` to add an item to the queue and ``.get()`` +to return an item. The class will take care of the locking necessary to +ensure that each job is handed out exactly once. Here's a trivial example:: @@ -352,7 +356,7 @@ provides a featureful interface. What kinds of global value mutation are thread-safe? ---------------------------------------------------- -A global interpreter lock (GIL) is used internally to ensure that only one +A :term:`global interpreter lock` (GIL) is used internally to ensure that only one thread runs in the Python VM at a time. In general, Python offers to switch among threads only between bytecode instructions; how frequently it switches can be set via :func:`sys.setswitchinterval`. Each bytecode instruction and @@ -361,7 +365,7 @@ therefore atomic from the point of view of a Python program. In theory, this means an exact accounting requires an exact understanding of the PVM bytecode implementation. In practice, it means that operations on shared -variables of builtin data types (ints, lists, dicts, etc) that "look atomic" +variables of built-in data types (ints, lists, dicts, etc) that "look atomic" really are. For example, the following operations are all atomic (L, L1, L2 are lists, D, @@ -395,32 +399,34 @@ lists. When in doubt, use a mutex! Can't we get rid of the Global Interpreter Lock? ------------------------------------------------ -.. XXX mention multiprocessing .. XXX link to dbeazley's talk about GIL? -The Global Interpreter Lock (GIL) is often seen as a hindrance to Python's +The :term:`global interpreter lock` (GIL) is often seen as a hindrance to Python's deployment on high-end multiprocessor server machines, because a multi-threaded Python program effectively only uses one CPU, due to the insistence that (almost) all Python code can only run while the GIL is held. Back in the days of Python 1.5, Greg Stein actually implemented a comprehensive patch set (the "free threading" patches) that removed the GIL and replaced it -with fine-grained locking. Unfortunately, even on Windows (where locks are very -efficient) this ran ordinary Python code about twice as slow as the interpreter -using the GIL. On Linux the performance loss was even worse because pthread -locks aren't as efficient. - -Since then, the idea of getting rid of the GIL has occasionally come up but -nobody has found a way to deal with the expected slowdown, and users who don't -use threads would not be happy if their code ran at half at the speed. Greg's -free threading patch set has not been kept up-to-date for later Python versions. +with fine-grained locking. Adam Olsen recently did a similar experiment +in his `python-safethread <http://code.google.com/p/python-safethread/>`_ +project. Unfortunately, both experiments exhibited a sharp drop in single-thread +performance (at least 30% slower), due to the amount of fine-grained locking +necessary to compensate for the removal of the GIL. This doesn't mean that you can't make good use of Python on multi-CPU machines! You just have to be creative with dividing the work up between multiple -*processes* rather than multiple *threads*. Judicious use of C extensions will -also help; if you use a C extension to perform a time-consuming task, the -extension can release the GIL while the thread of execution is in the C code and -allow other threads to get some work done. +*processes* rather than multiple *threads*. The +:class:`~concurrent.futures.ProcessPoolExecutor` class in the new +:mod:`concurrent.futures` module provides an easy way of doing so; the +:mod:`multiprocessing` module provides a lower-level API in case you want +more control over dispatching of tasks. + +Judicious use of C extensions will also help; if you use a C extension to +perform a time-consuming task, the extension can release the GIL while the +thread of execution is in the C code and allow other threads to get some work +done. Some standard library modules such as :mod:`zlib` and :mod:`hashlib` +already do this. It has been suggested that the GIL should be a per-interpreter-state lock rather than truly global; interpreters then wouldn't be able to share objects. @@ -511,9 +517,9 @@ I can't seem to use os.read() on a pipe created with os.popen(); why? :func:`os.read` is a low-level function which takes a file descriptor, a small integer representing the opened file. :func:`os.popen` creates a high-level -file object, the same type returned by the builtin :func:`open` function. Thus, -to read n bytes from a pipe p created with :func:`os.popen`, you need to use -``p.read(n)``. +file object, the same type returned by the built-in :func:`open` function. +Thus, to read n bytes from a pipe p created with :func:`os.popen`, you need to +use ``p.read(n)``. .. XXX update to use subprocess. See the :ref:`subprocess-replacements` section. @@ -751,7 +757,8 @@ some sample code:: How do I avoid blocking in the connect() method of a socket? ------------------------------------------------------------ -The select module is commonly used to help with asynchronous I/O on sockets. +The :mod:`select` module is commonly used to help with asynchronous I/O on +sockets. To prevent the TCP connect from blocking, you can set the socket to non-blocking mode. Then when you do the ``connect()``, you will either connect immediately @@ -765,6 +772,12 @@ just return the errno value. To poll, you can call ``connect_ex()`` again later -- ``0`` or ``errno.EISCONN`` indicate that you're connected -- or you can pass this socket to select to check if it's writable. +.. note:: + The :mod:`asyncore` module presents a framework-like approach to the problem + of writing non-blocking networking code. + The third-party `Twisted <http://twistedmatrix.com/>`_ library is + a popular and feature-rich alternative. + Databases ========= @@ -801,52 +814,6 @@ than a third of a second. This often beats doing something more complex and general such as using gdbm with pickle/shelve. -If my program crashes with a bsddb (or anydbm) database open, it gets corrupted. How come? ------------------------------------------------------------------------------------------- - -.. XXX move this FAQ entry elsewhere? - -.. note:: - - The bsddb module is now available as a standalone package `pybsddb - <http://www.jcea.es/programacion/pybsddb.htm>`_. - -Databases opened for write access with the bsddb module (and often by the anydbm -module, since it will preferentially use bsddb) must explicitly be closed using -the ``.close()`` method of the database. The underlying library caches database -contents which need to be converted to on-disk form and written. - -If you have initialized a new bsddb database but not written anything to it -before the program crashes, you will often wind up with a zero-length file and -encounter an exception the next time the file is opened. - - -I tried to open Berkeley DB file, but bsddb produces bsddb.error: (22, 'Invalid argument'). Help! How can I restore my data? ----------------------------------------------------------------------------------------------------------------------------- - -.. XXX move this FAQ entry elsewhere? - -.. note:: - - The bsddb module is now available as a standalone package `pybsddb - <http://www.jcea.es/programacion/pybsddb.htm>`_. - -Don't panic! Your data is probably intact. The most frequent cause for the error -is that you tried to open an earlier Berkeley DB file with a later version of -the Berkeley DB library. - -Many Linux systems now have all three versions of Berkeley DB available. If you -are migrating from version 1 to a newer version use db_dump185 to dump a plain -text version of the database. If you are migrating from version 2 to version 3 -use db2_dump to create a plain text version of the database. In either case, -use db_load to create a new native database for the latest version installed on -your computer. If you have version 3 of Berkeley DB installed, you should be -able to use db2_load to create a native version 2 database. - -You should move away from Berkeley DB version 1 files because the hash file code -contains known bugs that can corrupt your data. - - Mathematics and Numerics ======================== diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 7226e703a1..5de36764c5 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -115,166 +115,6 @@ Yes. The coding style required for standard library modules is documented as :pep:`8`. -My program is too slow. How do I speed it up? ---------------------------------------------- - -That's a tough one, in general. There are many tricks to speed up Python code; -consider rewriting parts in C as a last resort. - -In some cases it's possible to automatically translate Python to C or x86 -assembly language, meaning that you don't have to modify your code to gain -increased speed. - -.. XXX seems to have overlap with other questions! - -`Pyrex <http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/>`_ can compile a -slightly modified version of Python code into a C extension, and can be used on -many different platforms. - -`Psyco <http://psyco.sourceforge.net>`_ is a just-in-time compiler that -translates Python code into x86 assembly language. If you can use it, Psyco can -provide dramatic speedups for critical functions. - -The rest of this answer will discuss various tricks for squeezing a bit more -speed out of Python code. *Never* apply any optimization tricks unless you know -you need them, after profiling has indicated that a particular function is the -heavily executed hot spot in the code. Optimizations almost always make the -code less clear, and you shouldn't pay the costs of reduced clarity (increased -development time, greater likelihood of bugs) unless the resulting performance -benefit is worth it. - -There is a page on the wiki devoted to `performance tips -<http://wiki.python.org/moin/PythonSpeed/PerformanceTips>`_. - -Guido van Rossum has written up an anecdote related to optimization at -http://www.python.org/doc/essays/list2str.html. - -One thing to notice is that function and (especially) method calls are rather -expensive; if you have designed a purely OO interface with lots of tiny -functions that don't do much more than get or set an instance variable or call -another method, you might consider using a more direct way such as directly -accessing instance variables. Also see the standard module :mod:`profile` which -makes it possible to find out where your program is spending most of its time -(if you have some patience -- the profiling itself can slow your program down by -an order of magnitude). - -Remember that many standard optimization heuristics you may know from other -programming experience may well apply to Python. For example it may be faster -to send output to output devices using larger writes rather than smaller ones in -order to reduce the overhead of kernel system calls. Thus CGI scripts that -write all output in "one shot" may be faster than those that write lots of small -pieces of output. - -Also, be sure to use Python's core features where appropriate. For example, -slicing allows programs to chop up lists and other sequence objects in a single -tick of the interpreter's mainloop using highly optimized C implementations. -Thus to get the same effect as:: - - L2 = [] - for i in range[3]: - L2.append(L1[i]) - -it is much shorter and far faster to use :: - - L2 = list(L1[:3]) # "list" is redundant if L1 is a list. - -Note that the functionally-oriented builtins such as :func:`map`, :func:`zip`, -and friends can be a convenient accelerator for loops that perform a single -task. For example to pair the elements of two lists together:: - - >>> list(zip([1, 2, 3], [4, 5, 6])) - [(1, 4), (2, 5), (3, 6)] - -or to compute a number of sines:: - - >>> list(map(math.sin, (1, 2, 3, 4))) - [0.841470984808, 0.909297426826, 0.14112000806, -0.756802495308] - -The operation completes very quickly in such cases. - -Other examples include the ``join()`` and ``split()`` :ref:`methods -of string objects <string-methods>`. - -For example if s1..s7 are large (10K+) strings then -``"".join([s1,s2,s3,s4,s5,s6,s7])`` may be far faster than the more obvious -``s1+s2+s3+s4+s5+s6+s7``, since the "summation" will compute many -subexpressions, whereas ``join()`` does all the copying in one pass. For -manipulating strings, use the ``replace()`` and the ``format()`` :ref:`methods -on string objects <string-methods>`. Use regular expressions only when you're -not dealing with constant string patterns. - -Be sure to use the :meth:`list.sort` builtin method to do sorting, and see the -`sorting mini-HOWTO <http://wiki.python.org/moin/HowTo/Sorting>`_ for examples -of moderately advanced usage. :meth:`list.sort` beats other techniques for -sorting in all but the most extreme circumstances. - -Another common trick is to "push loops into functions or methods." For example -suppose you have a program that runs slowly and you use the profiler to -determine that a Python function ``ff()`` is being called lots of times. If you -notice that ``ff()``:: - - def ff(x): - ... # do something with x computing result... - return result - -tends to be called in loops like:: - - list = map(ff, oldlist) - -or:: - - for x in sequence: - value = ff(x) - ... # do something with value... - -then you can often eliminate function call overhead by rewriting ``ff()`` to:: - - def ffseq(seq): - resultseq = [] - for x in seq: - ... # do something with x computing result... - resultseq.append(result) - return resultseq - -and rewrite the two examples to ``list = ffseq(oldlist)`` and to:: - - for value in ffseq(sequence): - ... # do something with value... - -Single calls to ``ff(x)`` translate to ``ffseq([x])[0]`` with little penalty. -Of course this technique is not always appropriate and there are other variants -which you can figure out. - -You can gain some performance by explicitly storing the results of a function or -method lookup into a local variable. A loop like:: - - for key in token: - dict[key] = dict.get(key, 0) + 1 - -resolves ``dict.get`` every iteration. If the method isn't going to change, a -slightly faster implementation is:: - - dict_get = dict.get # look up the method once - for key in token: - dict[key] = dict_get(key, 0) + 1 - -Default arguments can be used to determine values once, at compile time instead -of at run time. This can only be done for functions or objects which will not -be changed during program execution, such as replacing :: - - def degree_sin(deg): - return math.sin(deg * math.pi / 180.0) - -with :: - - def degree_sin(deg, factor=math.pi/180.0, sin=math.sin): - return sin(deg * factor) - -Because this trick uses default arguments for terms which should not be changed, -it should only be used when you are not concerned with presenting a possibly -confusing API to your users. - - Core Language ============= @@ -361,7 +201,7 @@ Though a bit surprising at first, a moment's consideration explains this. On one hand, requiring :keyword:`global` for assigned variables provides a bar against unintended side-effects. On the other hand, if ``global`` was required for all global references, you'd be using ``global`` all the time. You'd have -to declare as global every reference to a builtin function or to a component of +to declare as global every reference to a built-in function or to a component of an imported module. This clutter would defeat the usefulness of the ``global`` declaration for identifying side-effects. @@ -472,15 +312,6 @@ calling another function by using ``*`` and ``**``:: ... g(x, *args, **kwargs) -In the unlikely case that you care about Python versions older than 2.0, use -:func:`apply`:: - - def f(x, *args, **kwargs): - ... - kwargs['width'] = '14.3c' - ... - apply(g, (x,)+args, kwargs) - How do I write a function with output parameters (call by reference)? --------------------------------------------------------------------- @@ -695,61 +526,21 @@ are not truly operators but syntactic delimiters in assignment statements. Is there an equivalent of C's "?:" ternary operator? ---------------------------------------------------- -Yes, this feature was added in Python 2.5. The syntax would be as follows:: +Yes, there is. The syntax is as follows:: [on_true] if [expression] else [on_false] x, y = 50, 25 - small = x if x < y else y -For versions previous to 2.5 the answer would be 'No'. - -.. XXX remove rest? +Before this syntax was introduced in Python 2.5, a common idiom was to use +logical operators:: -In many cases you can mimic ``a ? b : c`` with ``a and b or c``, but there's a -flaw: if *b* is zero (or empty, or ``None`` -- anything that tests false) then -*c* will be selected instead. In many cases you can prove by looking at the -code that this can't happen (e.g. because *b* is a constant or has a type that -can never be false), but in general this can be a problem. + [expression] and [on_true] or [on_false] -Tim Peters (who wishes it was Steve Majewski) suggested the following solution: -``(a and [b] or [c])[0]``. Because ``[b]`` is a singleton list it is never -false, so the wrong path is never taken; then applying ``[0]`` to the whole -thing gets the *b* or *c* that you really wanted. Ugly, but it gets you there -in the rare cases where it is really inconvenient to rewrite your code using -'if'. - -The best course is usually to write a simple ``if...else`` statement. Another -solution is to implement the ``?:`` operator as a function:: - - def q(cond, on_true, on_false): - if cond: - if not isfunction(on_true): - return on_true - else: - return on_true() - else: - if not isfunction(on_false): - return on_false - else: - return on_false() - -In most cases you'll pass b and c directly: ``q(a, b, c)``. To avoid evaluating -b or c when they shouldn't be, encapsulate them within a lambda function, e.g.: -``q(a, lambda: b, lambda: c)``. - -It has been asked *why* Python has no if-then-else expression. There are -several answers: many languages do just fine without one; it can easily lead to -less readable code; no sufficiently "Pythonic" syntax has been discovered; a -search of the standard library found remarkably few places where using an -if-then-else expression would make the code more understandable. - -In 2002, :pep:`308` was written proposing several possible syntaxes and the -community was asked to vote on the issue. The vote was inconclusive. Most -people liked one of the syntaxes, but also hated other syntaxes; many votes -implied that people preferred no ternary operator rather than having a syntax -they hated. +However, this idiom is unsafe, as it can give wrong results when *on_true* +has a false boolean value. Therefore, it is always better to use +the ``... if ... else ...`` form. Is it possible to write obfuscated one-liners in Python? @@ -868,15 +659,21 @@ the :ref:`string-formatting` section, e.g. ``"{:04d}".format(144)`` yields How do I modify a string in place? ---------------------------------- -You can't, because strings are immutable. If you need an object with this -ability, try converting the string to a list or use the array module:: +You can't, because strings are immutable. In most situations, you should +simply construct a new string from the various parts you want to assemble +it from. However, if you need an object with the ability to modify in-place +unicode data, try using a :class:`io.StringIO` object or the :mod:`array` +module:: >>> s = "Hello, world" - >>> a = list(s) - >>> print(a) - ['H', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd'] - >>> a[7:] = list("there!") - >>> ''.join(a) + >>> sio = io.StringIO(s) + >>> sio.getvalue() + 'Hello, world' + >>> sio.seek(7) + 7 + >>> sio.write("there!") + 6 + >>> sio.getvalue() 'Hello, there!' >>> import array @@ -951,11 +748,11 @@ There are various techniques. Is there an equivalent to Perl's chomp() for removing trailing newlines from strings? ------------------------------------------------------------------------------------- -Starting with Python 2.2, you can use ``S.rstrip("\r\n")`` to remove all -occurrences of any line terminator from the end of the string ``S`` without -removing other trailing whitespace. If the string ``S`` represents more than -one line, with several empty lines at the end, the line terminators for all the -blank lines will be removed:: +You can use ``S.rstrip("\r\n")`` to remove all occurrences of any line +terminator from the end of the string ``S`` without removing other trailing +whitespace. If the string ``S`` represents more than one line, with several +empty lines at the end, the line terminators for all the blank lines will +be removed:: >>> lines = ("line 1 \r\n" ... "\r\n" @@ -966,15 +763,6 @@ blank lines will be removed:: Since this is typically only desired when reading text one line at a time, using ``S.rstrip()`` this way works well. -For older versions of Python, there are two partial substitutes: - -- If you want to remove all trailing whitespace, use the ``rstrip()`` method of - string objects. This removes all trailing whitespace, not just a single - newline. - -- Otherwise, if there is only one line in the string ``S``, use - ``S.splitlines()[0]``. - Is there a scanf() or sscanf() equivalent? ------------------------------------------ @@ -987,8 +775,8 @@ and then convert decimal strings to numeric values using :func:`int` or :func:`float`. ``split()`` supports an optional "sep" parameter which is useful if the line uses something other than whitespace as a separator. -For more complicated input parsing, regular expressions more powerful than C's -:cfunc:`sscanf` and better suited for the task. +For more complicated input parsing, regular expressions are more powerful +than C's :c:func:`sscanf` and better suited for the task. What does 'UnicodeDecodeError' or 'UnicodeEncodeError' error mean? @@ -997,6 +785,94 @@ What does 'UnicodeDecodeError' or 'UnicodeEncodeError' error mean? See the :ref:`unicode-howto`. +Performance +=========== + +My program is too slow. How do I speed it up? +--------------------------------------------- + +That's a tough one, in general. First, here are a list of things to +remember before diving further: + +* Performance characteristics vary accross Python implementations. This FAQ + focusses on :term:`CPython`. +* Behaviour can vary accross operating systems, especially when talking about + I/O or multi-threading. +* You should always find the hot spots in your program *before* attempting to + optimize any code (see the :mod:`profile` module). +* Writing benchmark scripts will allow you to iterate quickly when searching + for improvements (see the :mod:`timeit` module). +* It is highly recommended to have good code coverage (through unit testing + or any other technique) before potentially introducing regressions hidden + in sophisticated optimizations. + +That being said, there are many tricks to speed up Python code. Here are +some general principles which go a long way towards reaching acceptable +performance levels: + +* Making your algorithms faster (or changing to faster ones) can yield + much larger benefits than trying to sprinkle micro-optimization tricks + all over your code. + +* Use the right data structures. Study documentation for the :ref:`bltin-types` + and the :mod:`collections` module. + +* When the standard library provides a primitive for doing something, it is + likely (although not guaranteed) to be faster than any alternative you + may come up with. This is doubly true for primitives written in C, such + as builtins and some extension types. For example, be sure to use + either the :meth:`list.sort` built-in method or the related :func:`sorted` + function to do sorting (and see the + `sorting mini-HOWTO <http://wiki.python.org/moin/HowTo/Sorting>`_ for examples + of moderately advanced usage). + +* Abstractions tend to create indirections and force the interpreter to work + more. If the levels of indirection outweigh the amount of useful work + done, your program will be slower. You should avoid excessive abstraction, + especially under the form of tiny functions or methods (which are also often + detrimental to readability). + +If you have reached the limit of what pure Python can allow, there are tools +to take you further away. For example, `Cython <http://cython.org>`_ can +compile a slightly modified version of Python code into a C extension, and +can be used on many different platforms. Cython can take advantage of +compilation (and optional type annotations) to make your code significantly +faster than when interpreted. If you are confident in your C programming +skills, you can also :ref:`write a C extension module <extending-index>` +yourself. + +.. seealso:: + The wiki page devoted to `performance tips + <http://wiki.python.org/moin/PythonSpeed/PerformanceTips>`_. + +.. _efficient_string_concatenation: + +What is the most efficient way to concatenate many strings together? +-------------------------------------------------------------------- + +:class:`str` and :class:`bytes` objects are immutable, therefore concatenating +many strings together is inefficient as each concatenation creates a new +object. In the general case, the total runtime cost is quadratic in the +total string length. + +To accumulate many :class:`str` objects, the recommended idiom is to place +them into a list and call :meth:`str.join` at the end:: + + chunks = [] + for s in my_strings: + chunks.append(s) + result = ''.join(chunks) + +(another reasonably efficient idiom is to use :class:`io.StringIO`) + +To accumulate many :class:`bytes` objects, the recommended idiom is to extend +a :class:`bytearray` object using in-place concatenation (the ``+=`` operator):: + + result = bytearray() + for b in my_bytes_objects: + result += b + + Sequences (Tuples/Lists) ======================== @@ -1033,7 +909,7 @@ trailing newline from a string. How do I iterate over a sequence in reverse order? -------------------------------------------------- -Use the :func:`reversed` builtin function, which is new in Python 2.4:: +Use the :func:`reversed` built-in function, which is new in Python 2.4:: for x in reversed(sequence): ... # do something with x... @@ -1066,15 +942,8 @@ list, deleting duplicates as you go:: else: last = mylist[i] -If all elements of the list may be used as dictionary keys (i.e. they are all -hashable) this is often faster :: - - d = {} - for x in mylist: - d[x] = 1 - mylist = list(d.keys()) - -In Python 2.5 and later, the following is possible instead:: +If all elements of the list may be used as set keys (i.e. they are all +:term:`hashable`) this is often faster :: mylist = list(set(mylist)) @@ -1444,15 +1313,7 @@ not:: C.count = 314 -Static methods are possible since Python 2.2:: - - class C: - def static(arg1, arg2, arg3): - # No 'self' parameter! - ... - static = staticmethod(static) - -With Python 2.4's decorators, this can also be written as :: +Static methods are possible:: class C: @staticmethod diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst index 7cd16b0f11..68a1b5c153 100644 --- a/Doc/faq/windows.rst +++ b/Doc/faq/windows.rst @@ -445,7 +445,7 @@ present, and ``getch()`` which gets one character without echoing it. How do I emulate os.kill() in Windows? -------------------------------------- -To terminate a process, you can use ctypes:: +Prior to Python 2.7 and 3.2, to terminate a process, you can use :mod:`ctypes`:: import ctypes @@ -455,6 +455,11 @@ To terminate a process, you can use ctypes:: handle = kernel32.OpenProcess(1, 0, pid) return (0 != kernel32.TerminateProcess(handle, 0)) +In 2.7 and 3.2, :func:`os.kill` is implemented similar to the above function, +with the additional feature of being able to send CTRL+C and CTRL+BREAK +to console subprocesses which are designed to handle those signals. See +:func:`os.kill` for further details. + Why does os.path.isdir() fail on NT shared directories? ------------------------------------------------------- @@ -536,12 +541,12 @@ assumed by the Python interpreter it won't work. The Python 1.5.* DLLs (``python15.dll``) are all compiled with MS VC++ 5.0 and with multithreading-DLL options (``/MD``). -If you can't change compilers or flags, try using :cfunc:`Py_RunSimpleString`. +If you can't change compilers or flags, try using :c:func:`Py_RunSimpleString`. A trick to get it to run an arbitrary file is to construct a call to -:func:`execfile` with the name of your file as argument. +:func:`exec` and :func:`open` with the name of your file as argument. Also note that you can not mix-and-match Debug and Release versions. If you -wish to use the Debug Multithreaded DLL, then your module *must* have an "_d" +wish to use the Debug Multithreaded DLL, then your module *must* have ``_d`` appended to the base name. diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 6a4daa54b6..3b211aedef 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -27,12 +27,17 @@ Glossary :ref:`2to3-reference`. abstract base class - Abstract Base Classes (abbreviated ABCs) complement :term:`duck-typing` by + Abstract base classes complement :term:`duck-typing` by providing a way to define interfaces when other techniques like - :func:`hasattr` would be clumsy. Python comes with many built-in ABCs for + :func:`hasattr` would be clumsy or subtly wrong (for example with + :ref:`magic methods <special-lookup>`). ABCs introduce virtual + subclasses, which are classes that don't inherit from a class but are + still recognized by :func:`isinstance` and :func:`issubclass`; see the + :mod:`abc` module documentation. Python comes with many built-in ABCs for data structures (in the :mod:`collections` module), numbers (in the - :mod:`numbers` module), and streams (in the :mod:`io` module). You can - create your own ABC with the :mod:`abc` module. + :mod:`numbers` module), streams (in the :mod:`io` module), import finders + and loaders (in the :mod:`importlib.abc` module). You can create your own + ABCs with the :mod:`abc` module. argument A value passed to a function or method, assigned to a named local @@ -57,11 +62,14 @@ Glossary bytecode Python source code is compiled into bytecode, the internal representation - of a Python program in the interpreter. The bytecode is also cached in - ``.pyc`` and ``.pyo`` files so that executing the same file is faster the - second time (recompilation from source to bytecode can be avoided). This - "intermediate language" is said to run on a :term:`virtual machine` - that executes the machine code corresponding to each bytecode. + of a Python program in the CPython interpreter. The bytecode is also + cached in ``.pyc`` and ``.pyo`` files so that executing the same file is + faster the second time (recompilation from source to bytecode can be + avoided). This "intermediate language" is said to run on a + :term:`virtual machine` that executes the machine code corresponding to + each bytecode. Do note that bytecodes are not expected to work between + different Python virtual machines, nor to be stable between Python + releases. A list of bytecode instructions can be found in the documentation for :ref:`the dis module <bytecodes>`. @@ -158,8 +166,8 @@ Glossary well-designed code improves its flexibility by allowing polymorphic substitution. Duck-typing avoids tests using :func:`type` or :func:`isinstance`. (Note, however, that duck-typing can be complemented - with :term:`abstract base class`\ es.) Instead, it typically employs - :func:`hasattr` tests or :term:`EAFP` programming. + with :term:`abstract base classes <abstract base class>`.) Instead, it + typically employs :func:`hasattr` tests or :term:`EAFP` programming. EAFP Easier to ask for forgiveness than permission. This common Python coding @@ -234,6 +242,8 @@ Glossary performs garbage collection via reference counting and a cyclic garbage collector that is able to detect and break reference cycles. + .. index:: single: generator + generator A function which returns an iterator. It looks like a normal function except that it contains :keyword:`yield` statements for producing a series @@ -247,7 +257,7 @@ Glossary .. index:: single: generator expression generator expression - An expression that returns a generator. It looks like a normal expression + An expression that returns an iterator. It looks like a normal expression followed by a :keyword:`for` expression defining a loop variable, range, and an optional :keyword:`if` expression. The combined expression generates values for an enclosing function:: @@ -327,7 +337,7 @@ Glossary slowly. See also :term:`interactive`. iterable - A container object capable of returning its members one at a + An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as :class:`list`, :class:`str`, and :class:`tuple`) and some non-sequence types like :class:`dict` and :class:`file` and objects of any classes you @@ -397,6 +407,12 @@ Glossary the :term:`EAFP` approach and is characterized by the presence of many :keyword:`if` statements. + In a multi-threaded environment, the LBYL approach can risk introducing a + race condition between "the looking" and "the leaping". For example, the + code, ``if key in mapping: return mapping[key]`` can fail if another + thread removes *key* from *mapping* after the test, but before the lookup. + This issue can be solved with locks or by using the EAFP approach. + list A built-in Python :term:`sequence`. Despite its name it is more akin to an array in other languages than to a linked list since access to @@ -417,9 +433,12 @@ Glossary :class:`importlib.abc.Loader` for an :term:`abstract base class`. mapping - A container object (such as :class:`dict`) which supports arbitrary key - lookups using the special method :meth:`__getitem__`. Mappings also - support :meth:`__len__`, :meth:`__iter__`, and :meth:`__contains__`. + A container object that supports arbitrary key lookups and implements the + methods specified in the :class:`~collections.Mapping` or + :class:`~collections.MutableMapping` + :ref:`abstract base classes <collections-abstract-base-classes>`. Examples + include :class:`dict`, :class:`collections.defaultdict`, + :class:`collections.OrderedDict` and :class:`collections.Counter`. metaclass The class of a class. Class definitions create a class name, a class @@ -440,6 +459,14 @@ Glossary its first :term:`argument` (which is usually called ``self``). See :term:`function` and :term:`nested scope`. + method resolution order + Method Resolution Order is the order in which base classes are searched + for a member during lookup. See `The Python 2.3 Method Resolution Order + <http://www.python.org/download/releases/2.3/mro/>`_. + + MRO + See :term:`method resolution order`. + mutable Mutable objects can change their value but keep their :func:`id`. See also :term:`immutable`. @@ -465,7 +492,7 @@ Glossary :func:`builtins.open` and :func:`os.open` are distinguished by their namespaces. Namespaces also aid readability and maintainability by making it clear which module implements a function. For instance, writing - :func:`random.seed` or :func:`itertools.izip` makes it clear that those + :func:`random.seed` or :func:`itertools.islice` makes it clear that those functions are implemented by the :mod:`random` and :mod:`itertools` modules, respectively. diff --git a/Doc/howto/cporting.rst b/Doc/howto/cporting.rst index d20e4a6a94..71844969b6 100644 --- a/Doc/howto/cporting.rst +++ b/Doc/howto/cporting.rst @@ -22,7 +22,7 @@ Conditional compilation ======================= The easiest way to compile only some code for 3.0 is to check if -:cmacro:`PY_MAJOR_VERSION` is greater than or equal to 3. :: +:c:macro:`PY_MAJOR_VERSION` is greater than or equal to 3. :: #if PY_MAJOR_VERSION >= 3 #define IS_PY3K @@ -47,12 +47,12 @@ Python 3.0's :func:`str` (``PyString_*`` functions in C) type is equivalent to 2.x's :func:`unicode` (``PyUnicode_*``). The old 8-bit string type has become :func:`bytes`. Python 2.6 and later provide a compatibility header, :file:`bytesobject.h`, mapping ``PyBytes`` names to ``PyString`` ones. For best -compatibility with 3.0, :ctype:`PyUnicode` should be used for textual data and -:ctype:`PyBytes` for binary data. It's also important to remember that -:ctype:`PyBytes` and :ctype:`PyUnicode` in 3.0 are not interchangeable like -:ctype:`PyString` and :ctype:`PyUnicode` are in 2.x. The following example -shows best practices with regards to :ctype:`PyUnicode`, :ctype:`PyString`, -and :ctype:`PyBytes`. :: +compatibility with 3.0, :c:type:`PyUnicode` should be used for textual data and +:c:type:`PyBytes` for binary data. It's also important to remember that +:c:type:`PyBytes` and :c:type:`PyUnicode` in 3.0 are not interchangeable like +:c:type:`PyString` and :c:type:`PyUnicode` are in 2.x. The following example +shows best practices with regards to :c:type:`PyUnicode`, :c:type:`PyString`, +and :c:type:`PyBytes`. :: #include "stdlib.h" #include "Python.h" @@ -120,7 +120,7 @@ can also be used in some cases. :: Module initialization and state =============================== -Python 3.0 has a revamped extension module initialization system. (See PEP +Python 3.0 has a revamped extension module initialization system. (See :pep:`3121`.) Instead of storing module state in globals, they should be stored in an interpreter specific structure. Creating modules that act correctly in both 2.x and 3.0 is tricky. The following simple example demonstrates how. :: diff --git a/Doc/howto/descriptor.rst b/Doc/howto/descriptor.rst index cdb6a8ec3d..1616f67e12 100644 --- a/Doc/howto/descriptor.rst +++ b/Doc/howto/descriptor.rst @@ -42,7 +42,7 @@ classes (a class is new style if it inherits from :class:`object` or Descriptors are a powerful, general purpose protocol. They are the mechanism behind properties, methods, static methods, class methods, and :func:`super()`. -They are used used throughout Python itself to implement the new style classes +They are used throughout Python itself to implement the new style classes introduced in version 2.2. Descriptors simplify the underlying C-code and offer a flexible set of new tools for everyday Python programs. @@ -97,7 +97,7 @@ transforms ``b.x`` into ``type(b).__dict__['x'].__get__(b, type(b))``. The implementation works through a precedence chain that gives data descriptors priority over instance variables, instance variables priority over non-data descriptors, and assigns lowest priority to :meth:`__getattr__` if provided. The -full C implementation can be found in :cfunc:`PyObject_GenericGetAttr()` in +full C implementation can be found in :c:func:`PyObject_GenericGetAttr()` in `Objects/object.c <http://svn.python.org/view/python/trunk/Objects/object.c?view=markup>`_\. For classes, the machinery is in :meth:`type.__getattribute__` which transforms @@ -131,7 +131,7 @@ search using :meth:`object.__getattribute__`. Note, in Python 2.2, ``super(B, obj).m()`` would only invoke :meth:`__get__` if ``m`` was a data descriptor. In Python 2.3, non-data descriptors also get invoked unless an old-style class is involved. The implementation details are -in :cfunc:`super_getattro()` in +in :c:func:`super_getattro()` in `Objects/typeobject.c <http://svn.python.org/view/python/trunk/Objects/typeobject.c?view=markup>`_ and a pure Python equivalent can be found in `Guido's Tutorial`_. @@ -297,7 +297,7 @@ Running the interpreter shows how the function descriptor works in practice:: The output suggests that bound and unbound methods are two different types. While they could have been implemented that way, the actual C implementation of -:ctype:`PyMethod_Type` in +:c:type:`PyMethod_Type` in `Objects/classobject.c <http://svn.python.org/view/python/trunk/Objects/classobject.c?view=markup>`_ is a single object with two different representations depending on whether the :attr:`im_self` field is set or is *NULL* (the C equivalent of *None*). diff --git a/Doc/howto/doanddont.rst b/Doc/howto/doanddont.rst deleted file mode 100644 index 365a6209d4..0000000000 --- a/Doc/howto/doanddont.rst +++ /dev/null @@ -1,290 +0,0 @@ -************************************ - Idioms and Anti-Idioms in Python -************************************ - -:Author: Moshe Zadka - -This document is placed in the public domain. - - -.. topic:: Abstract - - This document can be considered a companion to the tutorial. It shows how to use - Python, and even more importantly, how *not* to use Python. - - -Language Constructs You Should Not Use -====================================== - -While Python has relatively few gotchas compared to other languages, it still -has some constructs which are only useful in corner cases, or are plain -dangerous. - - -from module import \* ---------------------- - - -Inside Function Definitions -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -``from module import *`` is *invalid* inside function definitions. While many -versions of Python do not check for the invalidity, it does not make it more -valid, no more than having a smart lawyer makes a man innocent. Do not use it -like that ever. Even in versions where it was accepted, it made the function -execution slower, because the compiler could not be certain which names are -local and which are global. In Python 2.1 this construct causes warnings, and -sometimes even errors. - - -At Module Level -^^^^^^^^^^^^^^^ - -While it is valid to use ``from module import *`` at module level it is usually -a bad idea. For one, this loses an important property Python otherwise has --- -you can know where each toplevel name is defined by a simple "search" function -in your favourite editor. You also open yourself to trouble in the future, if -some module grows additional functions or classes. - -One of the most awful question asked on the newsgroup is why this code:: - - f = open("www") - f.read() - -does not work. Of course, it works just fine (assuming you have a file called -"www".) But it does not work if somewhere in the module, the statement ``from os -import *`` is present. The :mod:`os` module has a function called :func:`open` -which returns an integer. While it is very useful, shadowing builtins is one of -its least useful properties. - -Remember, you can never know for sure what names a module exports, so either -take what you need --- ``from module import name1, name2``, or keep them in the -module and access on a per-need basis --- ``import module; print(module.name)``. - - -When It Is Just Fine -^^^^^^^^^^^^^^^^^^^^ - -There are situations in which ``from module import *`` is just fine: - -* The interactive prompt. For example, ``from math import *`` makes Python an - amazing scientific calculator. - -* When extending a module in C with a module in Python. - -* When the module advertises itself as ``from import *`` safe. - - -from module import name1, name2 -------------------------------- - -This is a "don't" which is much weaker than the previous "don't"s but is still -something you should not do if you don't have good reasons to do that. The -reason it is usually bad idea is because you suddenly have an object which lives -in two separate namespaces. When the binding in one namespace changes, the -binding in the other will not, so there will be a discrepancy between them. This -happens when, for example, one module is reloaded, or changes the definition of -a function at runtime. - -Bad example:: - - # foo.py - a = 1 - - # bar.py - from foo import a - if something(): - a = 2 # danger: foo.a != a - -Good example:: - - # foo.py - a = 1 - - # bar.py - import foo - if something(): - foo.a = 2 - - -except: -------- - -Python has the ``except:`` clause, which catches all exceptions. Since *every* -error in Python raises an exception, using ``except:`` can make many -programming errors look like runtime problems, which hinders the debugging -process. - -The following code shows a great example of why this is bad:: - - try: - foo = opne("file") # misspelled "open" - except: - sys.exit("could not open file!") - -The second line triggers a :exc:`NameError`, which is caught by the except -clause. The program will exit, and the error message the program prints will -make you think the problem is the readability of ``"file"`` when in fact -the real error has nothing to do with ``"file"``. - -A better way to write the above is :: - - try: - foo = opne("file") - except IOError: - sys.exit("could not open file") - -When this is run, Python will produce a traceback showing the :exc:`NameError`, -and it will be immediately apparent what needs to be fixed. - -.. index:: bare except, except; bare - -Because ``except:`` catches *all* exceptions, including :exc:`SystemExit`, -:exc:`KeyboardInterrupt`, and :exc:`GeneratorExit` (which is not an error and -should not normally be caught by user code), using a bare ``except:`` is almost -never a good idea. In situations where you need to catch all "normal" errors, -such as in a framework that runs callbacks, you can catch the base class for -all normal exceptions, :exc:`Exception`. - - -Exceptions -========== - -Exceptions are a useful feature of Python. You should learn to raise them -whenever something unexpected occurs, and catch them only where you can do -something about them. - -The following is a very popular anti-idiom :: - - def get_status(file): - if not os.path.exists(file): - print("file not found") - sys.exit(1) - return open(file).readline() - -Consider the case where the file gets deleted between the time the call to -:func:`os.path.exists` is made and the time :func:`open` is called. In that -case the last line will raise an :exc:`IOError`. The same thing would happen -if *file* exists but has no read permission. Since testing this on a normal -machine on existent and non-existent files makes it seem bugless, the test -results will seem fine, and the code will get shipped. Later an unhandled -:exc:`IOError` (or perhaps some other :exc:`EnvironmentError`) escapes to the -user, who gets to watch the ugly traceback. - -Here is a somewhat better way to do it. :: - - def get_status(file): - try: - return open(file).readline() - except EnvironmentError as err: - print("Unable to open file: {}".format(err)) - sys.exit(1) - -In this version, *either* the file gets opened and the line is read (so it -works even on flaky NFS or SMB connections), or an error message is printed -that provides all the available information on why the open failed, and the -application is aborted. - -However, even this version of :func:`get_status` makes too many assumptions --- -that it will only be used in a short running script, and not, say, in a long -running server. Sure, the caller could do something like :: - - try: - status = get_status(log) - except SystemExit: - status = None - -But there is a better way. You should try to use as few ``except`` clauses in -your code as you can --- the ones you do use will usually be inside calls which -should always succeed, or a catch-all in a main function. - -So, an even better version of :func:`get_status()` is probably :: - - def get_status(file): - return open(file).readline() - -The caller can deal with the exception if it wants (for example, if it tries -several files in a loop), or just let the exception filter upwards to *its* -caller. - -But the last version still has a serious problem --- due to implementation -details in CPython, the file would not be closed when an exception is raised -until the exception handler finishes; and, worse, in other implementations -(e.g., Jython) it might not be closed at all regardless of whether or not -an exception is raised. - -The best version of this function uses the ``open()`` call as a context -manager, which will ensure that the file gets closed as soon as the -function returns:: - - def get_status(file): - with open(file) as fp: - return fp.readline() - - -Using the Batteries -=================== - -Every so often, people seem to be writing stuff in the Python library again, -usually poorly. While the occasional module has a poor interface, it is usually -much better to use the rich standard library and data types that come with -Python than inventing your own. - -A useful module very few people know about is :mod:`os.path`. It always has the -correct path arithmetic for your operating system, and will usually be much -better than whatever you come up with yourself. - -Compare:: - - # ugh! - return dir+"/"+file - # better - return os.path.join(dir, file) - -More useful functions in :mod:`os.path`: :func:`basename`, :func:`dirname` and -:func:`splitext`. - -There are also many useful built-in functions people seem not to be aware of -for some reason: :func:`min` and :func:`max` can find the minimum/maximum of -any sequence with comparable semantics, for example, yet many people write -their own :func:`max`/:func:`min`. Another highly useful function is -:func:`functools.reduce` which can be used to repeatly apply a binary -operation to a sequence, reducing it to a single value. For example, compute -a factorial with a series of multiply operations:: - - >>> n = 4 - >>> import operator, functools - >>> functools.reduce(operator.mul, range(1, n+1)) - 24 - -When it comes to parsing numbers, note that :func:`float`, :func:`int` and -:func:`long` all accept string arguments and will reject ill-formed strings -by raising an :exc:`ValueError`. - - -Using Backslash to Continue Statements -====================================== - -Since Python treats a newline as a statement terminator, and since statements -are often more than is comfortable to put in one line, many people do:: - - if foo.bar()['first'][0] == baz.quux(1, 2)[5:9] and \ - calculate_number(10, 20) != forbulate(500, 360): - pass - -You should realize that this is dangerous: a stray space after the ``\`` would -make this line wrong, and stray spaces are notoriously hard to see in editors. -In this case, at least it would be a syntax error, but if the code was:: - - value = foo.bar()['first'][0]*baz.quux(1, 2)[5:9] \ - + calculate_number(10, 20)*forbulate(500, 360) - -then it would just be subtly wrong. - -It is usually much better to use the implicit continuation inside parenthesis: - -This version is bulletproof:: - - value = (foo.bar()['first'][0]*baz.quux(1, 2)[5:9] - + calculate_number(10, 20)*forbulate(500, 360)) - diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index bfd2c96397..8934d5e7f4 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -1010,135 +1010,6 @@ Some of the functions in this module are: Consult the operator module's documentation for a complete list. - -The functional module ---------------------- - -Collin Winter's `functional module <http://oakwinter.com/code/functional/>`__ -provides a number of more advanced tools for functional programming. It also -reimplements several Python built-ins, trying to make them more intuitive to -those used to functional programming in other languages. - -This section contains an introduction to some of the most important functions in -``functional``; full documentation can be found at `the project's website -<http://oakwinter.com/code/functional/documentation/>`__. - -``compose(outer, inner, unpack=False)`` - -The ``compose()`` function implements function composition. In other words, it -returns a wrapper around the ``outer`` and ``inner`` callables, such that the -return value from ``inner`` is fed directly to ``outer``. That is, :: - - >>> def add(a, b): - ... return a + b - ... - >>> def double(a): - ... return 2 * a - ... - >>> compose(double, add)(5, 6) - 22 - -is equivalent to :: - - >>> double(add(5, 6)) - 22 - -The ``unpack`` keyword is provided to work around the fact that Python functions -are not always `fully curried <http://en.wikipedia.org/wiki/Currying>`__. By -default, it is expected that the ``inner`` function will return a single object -and that the ``outer`` function will take a single argument. Setting the -``unpack`` argument causes ``compose`` to expect a tuple from ``inner`` which -will be expanded before being passed to ``outer``. Put simply, :: - - compose(f, g)(5, 6) - -is equivalent to:: - - f(g(5, 6)) - -while :: - - compose(f, g, unpack=True)(5, 6) - -is equivalent to:: - - f(*g(5, 6)) - -Even though ``compose()`` only accepts two functions, it's trivial to build up a -version that will compose any number of functions. We'll use -:func:`functools.reduce`, ``compose()`` and ``partial()`` (the last of which is -provided by both ``functional`` and ``functools``). :: - - from functional import compose, partial - import functools - - - multi_compose = partial(functools.reduce, compose) - - -We can also use ``map()``, ``compose()`` and ``partial()`` to craft a version of -``"".join(...)`` that converts its arguments to string:: - - from functional import compose, partial - - join = compose("".join, partial(map, str)) - - -``flip(func)`` - -``flip()`` wraps the callable in ``func`` and causes it to receive its -non-keyword arguments in reverse order. :: - - >>> def triple(a, b, c): - ... return (a, b, c) - ... - >>> triple(5, 6, 7) - (5, 6, 7) - >>> - >>> flipped_triple = flip(triple) - >>> flipped_triple(5, 6, 7) - (7, 6, 5) - -``foldl(func, start, iterable)`` - -``foldl()`` takes a binary function, a starting value (usually some kind of -'zero'), and an iterable. The function is applied to the starting value and the -first element of the list, then the result of that and the second element of the -list, then the result of that and the third element of the list, and so on. - -This means that a call such as:: - - foldl(f, 0, [1, 2, 3]) - -is equivalent to:: - - f(f(f(0, 1), 2), 3) - - -``foldl()`` is roughly equivalent to the following recursive function:: - - def foldl(func, start, seq): - if len(seq) == 0: - return start - - return foldl(func, func(start, seq[0]), seq[1:]) - -Speaking of equivalence, the above ``foldl`` call can be expressed in terms of -the built-in :func:`functools.reduce` like so:: - - import functools - functools.reduce(f, [1, 2, 3], 0) - - -We can use ``foldl()``, ``operator.concat()`` and ``partial()`` to write a -cleaner, more aesthetically-pleasing version of Python's ``"".join(...)`` -idiom:: - - from functional import foldl, partial from operator import concat - - join = partial(foldl, concat, "") - - Small functions and the lambda expression ========================================= @@ -1280,9 +1151,9 @@ Text Processing". Mertz also wrote a 3-part series of articles on functional programming for IBM's DeveloperWorks site; see -`part 1 <http://www-128.ibm.com/developerworks/library/l-prog.html>`__, -`part 2 <http://www-128.ibm.com/developerworks/library/l-prog2.html>`__, and -`part 3 <http://www-128.ibm.com/developerworks/linux/library/l-prog3.html>`__, +`part 1 <http://www.ibm.com/developerworks/linux/library/l-prog/index.html>`__, +`part 2 <http://www.ibm.com/developerworks/linux/library/l-prog2/index.html>`__, and +`part 3 <http://www.ibm.com/developerworks/linux/library/l-prog3/index.html>`__, Python documentation diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst index 417ae0047e..11fe108f73 100644 --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -14,11 +14,13 @@ Currently, the HOWTOs are: :maxdepth: 1 advocacy.rst + pyporting.rst cporting.rst curses.rst descriptor.rst - doanddont.rst functional.rst + logging.rst + logging-cookbook.rst regex.rst sockets.rst sorting.rst diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst new file mode 100644 index 0000000000..7dc80215d8 --- /dev/null +++ b/Doc/howto/logging-cookbook.rst @@ -0,0 +1,1104 @@ +.. _logging-cookbook: + +================ +Logging Cookbook +================ + +:Author: Vinay Sajip <vinay_sajip at red-dove dot com> + +This page contains a number of recipes related to logging, which have been found +useful in the past. + +.. currentmodule:: logging + +Using logging in multiple modules +--------------------------------- + +Multiple calls to ``logging.getLogger('someLogger')`` return a reference to the +same logger object. This is true not only within the same module, but also +across modules as long as it is in the same Python interpreter process. It is +true for references to the same object; additionally, application code can +define and configure a parent logger in one module and create (but not +configure) a child logger in a separate module, and all logger calls to the +child will pass up to the parent. Here is a main module:: + + import logging + import auxiliary_module + + # create logger with 'spam_application' + logger = logging.getLogger('spam_application') + logger.setLevel(logging.DEBUG) + # create file handler which logs even debug messages + fh = logging.FileHandler('spam.log') + fh.setLevel(logging.DEBUG) + # create console handler with a higher log level + ch = logging.StreamHandler() + ch.setLevel(logging.ERROR) + # create formatter and add it to the handlers + formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') + fh.setFormatter(formatter) + ch.setFormatter(formatter) + # add the handlers to the logger + logger.addHandler(fh) + logger.addHandler(ch) + + logger.info('creating an instance of auxiliary_module.Auxiliary') + a = auxiliary_module.Auxiliary() + logger.info('created an instance of auxiliary_module.Auxiliary') + logger.info('calling auxiliary_module.Auxiliary.do_something') + a.do_something() + logger.info('finished auxiliary_module.Auxiliary.do_something') + logger.info('calling auxiliary_module.some_function()') + auxiliary_module.some_function() + logger.info('done with auxiliary_module.some_function()') + +Here is the auxiliary module:: + + import logging + + # create logger + module_logger = logging.getLogger('spam_application.auxiliary') + + class Auxiliary: + def __init__(self): + self.logger = logging.getLogger('spam_application.auxiliary.Auxiliary') + self.logger.info('creating an instance of Auxiliary') + def do_something(self): + self.logger.info('doing something') + a = 1 + 1 + self.logger.info('done doing something') + + def some_function(): + module_logger.info('received a call to "some_function"') + +The output looks like this:: + + 2005-03-23 23:47:11,663 - spam_application - INFO - + creating an instance of auxiliary_module.Auxiliary + 2005-03-23 23:47:11,665 - spam_application.auxiliary.Auxiliary - INFO - + creating an instance of Auxiliary + 2005-03-23 23:47:11,665 - spam_application - INFO - + created an instance of auxiliary_module.Auxiliary + 2005-03-23 23:47:11,668 - spam_application - INFO - + calling auxiliary_module.Auxiliary.do_something + 2005-03-23 23:47:11,668 - spam_application.auxiliary.Auxiliary - INFO - + doing something + 2005-03-23 23:47:11,669 - spam_application.auxiliary.Auxiliary - INFO - + done doing something + 2005-03-23 23:47:11,670 - spam_application - INFO - + finished auxiliary_module.Auxiliary.do_something + 2005-03-23 23:47:11,671 - spam_application - INFO - + calling auxiliary_module.some_function() + 2005-03-23 23:47:11,672 - spam_application.auxiliary - INFO - + received a call to 'some_function' + 2005-03-23 23:47:11,673 - spam_application - INFO - + done with auxiliary_module.some_function() + +Multiple handlers and formatters +-------------------------------- + +Loggers are plain Python objects. The :func:`addHandler` method has no minimum +or maximum quota for the number of handlers you may add. Sometimes it will be +beneficial for an application to log all messages of all severities to a text +file while simultaneously logging errors or above to the console. To set this +up, simply configure the appropriate handlers. The logging calls in the +application code will remain unchanged. Here is a slight modification to the +previous simple module-based configuration example:: + + import logging + + logger = logging.getLogger('simple_example') + logger.setLevel(logging.DEBUG) + # create file handler which logs even debug messages + fh = logging.FileHandler('spam.log') + fh.setLevel(logging.DEBUG) + # create console handler with a higher log level + ch = logging.StreamHandler() + ch.setLevel(logging.ERROR) + # create formatter and add it to the handlers + formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') + ch.setFormatter(formatter) + fh.setFormatter(formatter) + # add the handlers to logger + logger.addHandler(ch) + logger.addHandler(fh) + + # 'application' code + logger.debug('debug message') + logger.info('info message') + logger.warn('warn message') + logger.error('error message') + logger.critical('critical message') + +Notice that the 'application' code does not care about multiple handlers. All +that changed was the addition and configuration of a new handler named *fh*. + +The ability to create new handlers with higher- or lower-severity filters can be +very helpful when writing and testing an application. Instead of using many +``print`` statements for debugging, use ``logger.debug``: Unlike the print +statements, which you will have to delete or comment out later, the logger.debug +statements can remain intact in the source code and remain dormant until you +need them again. At that time, the only change that needs to happen is to +modify the severity level of the logger and/or handler to debug. + +.. _multiple-destinations: + +Logging to multiple destinations +-------------------------------- + +Let's say you want to log to console and file with different message formats and +in differing circumstances. Say you want to log messages with levels of DEBUG +and higher to file, and those messages at level INFO and higher to the console. +Let's also assume that the file should contain timestamps, but the console +messages should not. Here's how you can achieve this:: + + import logging + + # set up logging to file - see previous section for more details + logging.basicConfig(level=logging.DEBUG, + format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', + datefmt='%m-%d %H:%M', + filename='/temp/myapp.log', + filemode='w') + # define a Handler which writes INFO messages or higher to the sys.stderr + console = logging.StreamHandler() + console.setLevel(logging.INFO) + # set a format which is simpler for console use + formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s') + # tell the handler to use this format + console.setFormatter(formatter) + # add the handler to the root logger + logging.getLogger('').addHandler(console) + + # Now, we can log to the root logger, or any other logger. First the root... + logging.info('Jackdaws love my big sphinx of quartz.') + + # Now, define a couple of other loggers which might represent areas in your + # application: + + logger1 = logging.getLogger('myapp.area1') + logger2 = logging.getLogger('myapp.area2') + + logger1.debug('Quick zephyrs blow, vexing daft Jim.') + logger1.info('How quickly daft jumping zebras vex.') + logger2.warning('Jail zesty vixen who grabbed pay from quack.') + logger2.error('The five boxing wizards jump quickly.') + +When you run this, on the console you will see :: + + root : INFO Jackdaws love my big sphinx of quartz. + myapp.area1 : INFO How quickly daft jumping zebras vex. + myapp.area2 : WARNING Jail zesty vixen who grabbed pay from quack. + myapp.area2 : ERROR The five boxing wizards jump quickly. + +and in the file you will see something like :: + + 10-22 22:19 root INFO Jackdaws love my big sphinx of quartz. + 10-22 22:19 myapp.area1 DEBUG Quick zephyrs blow, vexing daft Jim. + 10-22 22:19 myapp.area1 INFO How quickly daft jumping zebras vex. + 10-22 22:19 myapp.area2 WARNING Jail zesty vixen who grabbed pay from quack. + 10-22 22:19 myapp.area2 ERROR The five boxing wizards jump quickly. + +As you can see, the DEBUG message only shows up in the file. The other messages +are sent to both destinations. + +This example uses console and file handlers, but you can use any number and +combination of handlers you choose. + + +Configuration server example +---------------------------- + +Here is an example of a module using the logging configuration server:: + + import logging + import logging.config + import time + import os + + # read initial config file + logging.config.fileConfig('logging.conf') + + # create and start listener on port 9999 + t = logging.config.listen(9999) + t.start() + + logger = logging.getLogger('simpleExample') + + try: + # loop through logging calls to see the difference + # new configurations make, until Ctrl+C is pressed + while True: + logger.debug('debug message') + logger.info('info message') + logger.warn('warn message') + logger.error('error message') + logger.critical('critical message') + time.sleep(5) + except KeyboardInterrupt: + # cleanup + logging.config.stopListening() + t.join() + +And here is a script that takes a filename and sends that file to the server, +properly preceded with the binary-encoded length, as the new logging +configuration:: + + #!/usr/bin/env python + import socket, sys, struct + + with open(sys.argv[1], 'rb') as f: + data_to_send = f.read() + + HOST = 'localhost' + PORT = 9999 + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + print('connecting...') + s.connect((HOST, PORT)) + print('sending config...') + s.send(struct.pack('>L', len(data_to_send))) + s.send(data_to_send) + s.close() + print('complete') + + +Dealing with handlers that block +-------------------------------- + +.. currentmodule:: logging.handlers + +Sometimes you have to get your logging handlers to do their work without +blocking the thread you’re logging from. This is common in Web applications, +though of course it also occurs in other scenarios. + +A common culprit which demonstrates sluggish behaviour is the +:class:`SMTPHandler`: sending emails can take a long time, for a +number of reasons outside the developer’s control (for example, a poorly +performing mail or network infrastructure). But almost any network-based +handler can block: Even a :class:`SocketHandler` operation may do a +DNS query under the hood which is too slow (and this query can be deep in the +socket library code, below the Python layer, and outside your control). + +One solution is to use a two-part approach. For the first part, attach only a +:class:`QueueHandler` to those loggers which are accessed from +performance-critical threads. They simply write to their queue, which can be +sized to a large enough capacity or initialized with no upper bound to their +size. The write to the queue will typically be accepted quickly, though you +will probably need to catch the :exc:`queue.Full` exception as a precaution +in your code. If you are a library developer who has performance-critical +threads in their code, be sure to document this (together with a suggestion to +attach only ``QueueHandlers`` to your loggers) for the benefit of other +developers who will use your code. + +The second part of the solution is :class:`QueueListener`, which has been +designed as the counterpart to :class:`QueueHandler`. A +:class:`QueueListener` is very simple: it’s passed a queue and some handlers, +and it fires up an internal thread which listens to its queue for LogRecords +sent from ``QueueHandlers`` (or any other source of ``LogRecords``, for that +matter). The ``LogRecords`` are removed from the queue and passed to the +handlers for processing. + +The advantage of having a separate :class:`QueueListener` class is that you +can use the same instance to service multiple ``QueueHandlers``. This is more +resource-friendly than, say, having threaded versions of the existing handler +classes, which would eat up one thread per handler for no particular benefit. + +An example of using these two classes follows (imports omitted):: + + que = queue.Queue(-1) # no limit on size + queue_handler = QueueHandler(que) + handler = logging.StreamHandler() + listener = QueueListener(que, handler) + root = logging.getLogger() + root.addHandler(queue_handler) + formatter = logging.Formatter('%(threadName)s: %(message)s') + handler.setFormatter(formatter) + listener.start() + # The log output will display the thread which generated + # the event (the main thread) rather than the internal + # thread which monitors the internal queue. This is what + # you want to happen. + root.warning('Look out!') + listener.stop() + +which, when run, will produce:: + + MainThread: Look out! + + +.. _network-logging: + +Sending and receiving logging events across a network +----------------------------------------------------- + +Let's say you want to send logging events across a network, and handle them at +the receiving end. A simple way of doing this is attaching a +:class:`SocketHandler` instance to the root logger at the sending end:: + + import logging, logging.handlers + + rootLogger = logging.getLogger('') + rootLogger.setLevel(logging.DEBUG) + socketHandler = logging.handlers.SocketHandler('localhost', + logging.handlers.DEFAULT_TCP_LOGGING_PORT) + # don't bother with a formatter, since a socket handler sends the event as + # an unformatted pickle + rootLogger.addHandler(socketHandler) + + # Now, we can log to the root logger, or any other logger. First the root... + logging.info('Jackdaws love my big sphinx of quartz.') + + # Now, define a couple of other loggers which might represent areas in your + # application: + + logger1 = logging.getLogger('myapp.area1') + logger2 = logging.getLogger('myapp.area2') + + logger1.debug('Quick zephyrs blow, vexing daft Jim.') + logger1.info('How quickly daft jumping zebras vex.') + logger2.warning('Jail zesty vixen who grabbed pay from quack.') + logger2.error('The five boxing wizards jump quickly.') + +At the receiving end, you can set up a receiver using the :mod:`socketserver` +module. Here is a basic working example:: + + import pickle + import logging + import logging.handlers + import socketserver + import struct + + + class LogRecordStreamHandler(socketserver.StreamRequestHandler): + """Handler for a streaming logging request. + + This basically logs the record using whatever logging policy is + configured locally. + """ + + def handle(self): + """ + Handle multiple requests - each expected to be a 4-byte length, + followed by the LogRecord in pickle format. Logs the record + according to whatever policy is configured locally. + """ + while True: + chunk = self.connection.recv(4) + if len(chunk) < 4: + break + slen = struct.unpack('>L', chunk)[0] + chunk = self.connection.recv(slen) + while len(chunk) < slen: + chunk = chunk + self.connection.recv(slen - len(chunk)) + obj = self.unPickle(chunk) + record = logging.makeLogRecord(obj) + self.handleLogRecord(record) + + def unPickle(self, data): + return pickle.loads(data) + + def handleLogRecord(self, record): + # if a name is specified, we use the named logger rather than the one + # implied by the record. + if self.server.logname is not None: + name = self.server.logname + else: + name = record.name + logger = logging.getLogger(name) + # N.B. EVERY record gets logged. This is because Logger.handle + # is normally called AFTER logger-level filtering. If you want + # to do filtering, do it at the client end to save wasting + # cycles and network bandwidth! + logger.handle(record) + + class LogRecordSocketReceiver(socketserver.ThreadingTCPServer): + """ + Simple TCP socket-based logging receiver suitable for testing. + """ + + allow_reuse_address = 1 + + def __init__(self, host='localhost', + port=logging.handlers.DEFAULT_TCP_LOGGING_PORT, + handler=LogRecordStreamHandler): + socketserver.ThreadingTCPServer.__init__(self, (host, port), handler) + self.abort = 0 + self.timeout = 1 + self.logname = None + + def serve_until_stopped(self): + import select + abort = 0 + while not abort: + rd, wr, ex = select.select([self.socket.fileno()], + [], [], + self.timeout) + if rd: + self.handle_request() + abort = self.abort + + def main(): + logging.basicConfig( + format='%(relativeCreated)5d %(name)-15s %(levelname)-8s %(message)s') + tcpserver = LogRecordSocketReceiver() + print('About to start TCP server...') + tcpserver.serve_until_stopped() + + if __name__ == '__main__': + main() + +First run the server, and then the client. On the client side, nothing is +printed on the console; on the server side, you should see something like:: + + About to start TCP server... + 59 root INFO Jackdaws love my big sphinx of quartz. + 59 myapp.area1 DEBUG Quick zephyrs blow, vexing daft Jim. + 69 myapp.area1 INFO How quickly daft jumping zebras vex. + 69 myapp.area2 WARNING Jail zesty vixen who grabbed pay from quack. + 69 myapp.area2 ERROR The five boxing wizards jump quickly. + +Note that there are some security issues with pickle in some scenarios. If +these affect you, you can use an alternative serialization scheme by overriding +the :meth:`makePickle` method and implementing your alternative there, as +well as adapting the above script to use your alternative serialization. + + +.. _context-info: + +Adding contextual information to your logging output +---------------------------------------------------- + +Sometimes you want logging output to contain contextual information in +addition to the parameters passed to the logging call. For example, in a +networked application, it may be desirable to log client-specific information +in the log (e.g. remote client's username, or IP address). Although you could +use the *extra* parameter to achieve this, it's not always convenient to pass +the information in this way. While it might be tempting to create +:class:`Logger` instances on a per-connection basis, this is not a good idea +because these instances are not garbage collected. While this is not a problem +in practice, when the number of :class:`Logger` instances is dependent on the +level of granularity you want to use in logging an application, it could +be hard to manage if the number of :class:`Logger` instances becomes +effectively unbounded. + + +Using LoggerAdapters to impart contextual information +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An easy way in which you can pass contextual information to be output along +with logging event information is to use the :class:`LoggerAdapter` class. +This class is designed to look like a :class:`Logger`, so that you can call +:meth:`debug`, :meth:`info`, :meth:`warning`, :meth:`error`, +:meth:`exception`, :meth:`critical` and :meth:`log`. These methods have the +same signatures as their counterparts in :class:`Logger`, so you can use the +two types of instances interchangeably. + +When you create an instance of :class:`LoggerAdapter`, you pass it a +:class:`Logger` instance and a dict-like object which contains your contextual +information. When you call one of the logging methods on an instance of +:class:`LoggerAdapter`, it delegates the call to the underlying instance of +:class:`Logger` passed to its constructor, and arranges to pass the contextual +information in the delegated call. Here's a snippet from the code of +:class:`LoggerAdapter`:: + + def debug(self, msg, *args, **kwargs): + """ + Delegate a debug call to the underlying logger, after adding + contextual information from this adapter instance. + """ + msg, kwargs = self.process(msg, kwargs) + self.logger.debug(msg, *args, **kwargs) + +The :meth:`process` method of :class:`LoggerAdapter` is where the contextual +information is added to the logging output. It's passed the message and +keyword arguments of the logging call, and it passes back (potentially) +modified versions of these to use in the call to the underlying logger. The +default implementation of this method leaves the message alone, but inserts +an 'extra' key in the keyword argument whose value is the dict-like object +passed to the constructor. Of course, if you had passed an 'extra' keyword +argument in the call to the adapter, it will be silently overwritten. + +The advantage of using 'extra' is that the values in the dict-like object are +merged into the :class:`LogRecord` instance's __dict__, allowing you to use +customized strings with your :class:`Formatter` instances which know about +the keys of the dict-like object. If you need a different method, e.g. if you +want to prepend or append the contextual information to the message string, +you just need to subclass :class:`LoggerAdapter` and override :meth:`process` +to do what you need. Here's an example script which uses this class, which +also illustrates what dict-like behaviour is needed from an arbitrary +'dict-like' object for use in the constructor:: + + import logging + + class ConnInfo: + """ + An example class which shows how an arbitrary class can be used as + the 'extra' context information repository passed to a LoggerAdapter. + """ + + def __getitem__(self, name): + """ + To allow this instance to look like a dict. + """ + from random import choice + if name == 'ip': + result = choice(['127.0.0.1', '192.168.0.1']) + elif name == 'user': + result = choice(['jim', 'fred', 'sheila']) + else: + result = self.__dict__.get(name, '?') + return result + + def __iter__(self): + """ + To allow iteration over keys, which will be merged into + the LogRecord dict before formatting and output. + """ + keys = ['ip', 'user'] + keys.extend(self.__dict__.keys()) + return keys.__iter__() + + if __name__ == '__main__': + from random import choice + levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL) + a1 = logging.LoggerAdapter(logging.getLogger('a.b.c'), + { 'ip' : '123.231.231.123', 'user' : 'sheila' }) + logging.basicConfig(level=logging.DEBUG, + format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s') + a1.debug('A debug message') + a1.info('An info message with %s', 'some parameters') + a2 = logging.LoggerAdapter(logging.getLogger('d.e.f'), ConnInfo()) + for x in range(10): + lvl = choice(levels) + lvlname = logging.getLevelName(lvl) + a2.log(lvl, 'A message at %s level with %d %s', lvlname, 2, 'parameters') + +When this script is run, the output should look something like this:: + + 2008-01-18 14:49:54,023 a.b.c DEBUG IP: 123.231.231.123 User: sheila A debug message + 2008-01-18 14:49:54,023 a.b.c INFO IP: 123.231.231.123 User: sheila An info message with some parameters + 2008-01-18 14:49:54,023 d.e.f CRITICAL IP: 192.168.0.1 User: jim A message at CRITICAL level with 2 parameters + 2008-01-18 14:49:54,033 d.e.f INFO IP: 192.168.0.1 User: jim A message at INFO level with 2 parameters + 2008-01-18 14:49:54,033 d.e.f WARNING IP: 192.168.0.1 User: sheila A message at WARNING level with 2 parameters + 2008-01-18 14:49:54,033 d.e.f ERROR IP: 127.0.0.1 User: fred A message at ERROR level with 2 parameters + 2008-01-18 14:49:54,033 d.e.f ERROR IP: 127.0.0.1 User: sheila A message at ERROR level with 2 parameters + 2008-01-18 14:49:54,033 d.e.f WARNING IP: 192.168.0.1 User: sheila A message at WARNING level with 2 parameters + 2008-01-18 14:49:54,033 d.e.f WARNING IP: 192.168.0.1 User: jim A message at WARNING level with 2 parameters + 2008-01-18 14:49:54,033 d.e.f INFO IP: 192.168.0.1 User: fred A message at INFO level with 2 parameters + 2008-01-18 14:49:54,033 d.e.f WARNING IP: 192.168.0.1 User: sheila A message at WARNING level with 2 parameters + 2008-01-18 14:49:54,033 d.e.f WARNING IP: 127.0.0.1 User: jim A message at WARNING level with 2 parameters + + +.. _filters-contextual: + +Using Filters to impart contextual information +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You can also add contextual information to log output using a user-defined +:class:`Filter`. ``Filter`` instances are allowed to modify the ``LogRecords`` +passed to them, including adding additional attributes which can then be output +using a suitable format string, or if needed a custom :class:`Formatter`. + +For example in a web application, the request being processed (or at least, +the interesting parts of it) can be stored in a threadlocal +(:class:`threading.local`) variable, and then accessed from a ``Filter`` to +add, say, information from the request - say, the remote IP address and remote +user's username - to the ``LogRecord``, using the attribute names 'ip' and +'user' as in the ``LoggerAdapter`` example above. In that case, the same format +string can be used to get similar output to that shown above. Here's an example +script:: + + import logging + from random import choice + + class ContextFilter(logging.Filter): + """ + This is a filter which injects contextual information into the log. + + Rather than use actual contextual information, we just use random + data in this demo. + """ + + USERS = ['jim', 'fred', 'sheila'] + IPS = ['123.231.231.123', '127.0.0.1', '192.168.0.1'] + + def filter(self, record): + + record.ip = choice(ContextFilter.IPS) + record.user = choice(ContextFilter.USERS) + return True + + if __name__ == '__main__': + levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL) + logging.basicConfig(level=logging.DEBUG, + format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s') + a1 = logging.getLogger('a.b.c') + a2 = logging.getLogger('d.e.f') + + f = ContextFilter() + a1.addFilter(f) + a2.addFilter(f) + a1.debug('A debug message') + a1.info('An info message with %s', 'some parameters') + for x in range(10): + lvl = choice(levels) + lvlname = logging.getLevelName(lvl) + a2.log(lvl, 'A message at %s level with %d %s', lvlname, 2, 'parameters') + +which, when run, produces something like:: + + 2010-09-06 22:38:15,292 a.b.c DEBUG IP: 123.231.231.123 User: fred A debug message + 2010-09-06 22:38:15,300 a.b.c INFO IP: 192.168.0.1 User: sheila An info message with some parameters + 2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL level with 2 parameters + 2010-09-06 22:38:15,300 d.e.f ERROR IP: 127.0.0.1 User: jim A message at ERROR level with 2 parameters + 2010-09-06 22:38:15,300 d.e.f DEBUG IP: 127.0.0.1 User: sheila A message at DEBUG level with 2 parameters + 2010-09-06 22:38:15,300 d.e.f ERROR IP: 123.231.231.123 User: fred A message at ERROR level with 2 parameters + 2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 192.168.0.1 User: jim A message at CRITICAL level with 2 parameters + 2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL level with 2 parameters + 2010-09-06 22:38:15,300 d.e.f DEBUG IP: 192.168.0.1 User: jim A message at DEBUG level with 2 parameters + 2010-09-06 22:38:15,301 d.e.f ERROR IP: 127.0.0.1 User: sheila A message at ERROR level with 2 parameters + 2010-09-06 22:38:15,301 d.e.f DEBUG IP: 123.231.231.123 User: fred A message at DEBUG level with 2 parameters + 2010-09-06 22:38:15,301 d.e.f INFO IP: 123.231.231.123 User: fred A message at INFO level with 2 parameters + + +.. _multiple-processes: + +Logging to a single file from multiple processes +------------------------------------------------ + +Although logging is thread-safe, and logging to a single file from multiple +threads in a single process *is* supported, logging to a single file from +*multiple processes* is *not* supported, because there is no standard way to +serialize access to a single file across multiple processes in Python. If you +need to log to a single file from multiple processes, one way of doing this is +to have all the processes log to a :class:`SocketHandler`, and have a separate +process which implements a socket server which reads from the socket and logs +to file. (If you prefer, you can dedicate one thread in one of the existing +processes to perform this function.) :ref:`This section <network-logging>` +documents this approach in more detail and includes a working socket receiver +which can be used as a starting point for you to adapt in your own +applications. + +If you are using a recent version of Python which includes the +:mod:`multiprocessing` module, you could write your own handler which uses the +:class:`Lock` class from this module to serialize access to the file from +your processes. The existing :class:`FileHandler` and subclasses do not make +use of :mod:`multiprocessing` at present, though they may do so in the future. +Note that at present, the :mod:`multiprocessing` module does not provide +working lock functionality on all platforms (see +http://bugs.python.org/issue3770). + +.. currentmodule:: logging.handlers + +Alternatively, you can use a ``Queue`` and a :class:`QueueHandler` to send +all logging events to one of the processes in your multi-process application. +The following example script demonstrates how you can do this; in the example +a separate listener process listens for events sent by other processes and logs +them according to its own logging configuration. Although the example only +demonstrates one way of doing it (for example, you may want to use a listener +thread rather than a separate listener process -- the implementation would be +analogous) it does allow for completely different logging configurations for +the listener and the other processes in your application, and can be used as +the basis for code meeting your own specific requirements:: + + # You'll need these imports in your own code + import logging + import logging.handlers + import multiprocessing + + # Next two import lines for this demo only + from random import choice, random + import time + + # + # Because you'll want to define the logging configurations for listener and workers, the + # listener and worker process functions take a configurer parameter which is a callable + # for configuring logging for that process. These functions are also passed the queue, + # which they use for communication. + # + # In practice, you can configure the listener however you want, but note that in this + # simple example, the listener does not apply level or filter logic to received records. + # In practice, you would probably want to do this logic in the worker processes, to avoid + # sending events which would be filtered out between processes. + # + # The size of the rotated files is made small so you can see the results easily. + def listener_configurer(): + root = logging.getLogger() + h = logging.handlers.RotatingFileHandler('mptest.log', 'a', 300, 10) + f = logging.Formatter('%(asctime)s %(processName)-10s %(name)s %(levelname)-8s %(message)s') + h.setFormatter(f) + root.addHandler(h) + + # This is the listener process top-level loop: wait for logging events + # (LogRecords)on the queue and handle them, quit when you get a None for a + # LogRecord. + def listener_process(queue, configurer): + configurer() + while True: + try: + record = queue.get() + if record is None: # We send this as a sentinel to tell the listener to quit. + break + logger = logging.getLogger(record.name) + logger.handle(record) # No level or filter logic applied - just do it! + except (KeyboardInterrupt, SystemExit): + raise + except: + import sys, traceback + print >> sys.stderr, 'Whoops! Problem:' + traceback.print_exc(file=sys.stderr) + + # Arrays used for random selections in this demo + + LEVELS = [logging.DEBUG, logging.INFO, logging.WARNING, + logging.ERROR, logging.CRITICAL] + + LOGGERS = ['a.b.c', 'd.e.f'] + + MESSAGES = [ + 'Random message #1', + 'Random message #2', + 'Random message #3', + ] + + # The worker configuration is done at the start of the worker process run. + # Note that on Windows you can't rely on fork semantics, so each process + # will run the logging configuration code when it starts. + def worker_configurer(queue): + h = logging.handlers.QueueHandler(queue) # Just the one handler needed + root = logging.getLogger() + root.addHandler(h) + root.setLevel(logging.DEBUG) # send all messages, for demo; no other level or filter logic applied. + + # This is the worker process top-level loop, which just logs ten events with + # random intervening delays before terminating. + # The print messages are just so you know it's doing something! + def worker_process(queue, configurer): + configurer(queue) + name = multiprocessing.current_process().name + print('Worker started: %s' % name) + for i in range(10): + time.sleep(random()) + logger = logging.getLogger(choice(LOGGERS)) + level = choice(LEVELS) + message = choice(MESSAGES) + logger.log(level, message) + print('Worker finished: %s' % name) + + # Here's where the demo gets orchestrated. Create the queue, create and start + # the listener, create ten workers and start them, wait for them to finish, + # then send a None to the queue to tell the listener to finish. + def main(): + queue = multiprocessing.Queue(-1) + listener = multiprocessing.Process(target=listener_process, + args=(queue, listener_configurer)) + listener.start() + workers = [] + for i in range(10): + worker = multiprocessing.Process(target=worker_process, + args=(queue, worker_configurer)) + workers.append(worker) + worker.start() + for w in workers: + w.join() + queue.put_nowait(None) + listener.join() + + if __name__ == '__main__': + main() + +A variant of the above script keeps the logging in the main process, in a +separate thread:: + + import logging + import logging.config + import logging.handlers + from multiprocessing import Process, Queue + import random + import threading + import time + + def logger_thread(q): + while True: + record = q.get() + if record is None: + break + logger = logging.getLogger(record.name) + logger.handle(record) + + + def worker_process(q): + qh = logging.handlers.QueueHandler(q) + root = logging.getLogger() + root.setLevel(logging.DEBUG) + root.addHandler(qh) + levels = [logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, + logging.CRITICAL] + loggers = ['foo', 'foo.bar', 'foo.bar.baz', + 'spam', 'spam.ham', 'spam.ham.eggs'] + for i in range(100): + lvl = random.choice(levels) + logger = logging.getLogger(random.choice(loggers)) + logger.log(lvl, 'Message no. %d', i) + + if __name__ == '__main__': + q = Queue() + d = { + 'version': 1, + 'formatters': { + 'detailed': { + 'class': 'logging.Formatter', + 'format': '%(asctime)s %(name)-15s %(levelname)-8s %(processName)-10s %(message)s' + } + }, + 'handlers': { + 'console': { + 'class': 'logging.StreamHandler', + 'level': 'INFO', + }, + 'file': { + 'class': 'logging.FileHandler', + 'filename': 'mplog.log', + 'mode': 'w', + 'formatter': 'detailed', + }, + 'foofile': { + 'class': 'logging.FileHandler', + 'filename': 'mplog-foo.log', + 'mode': 'w', + 'formatter': 'detailed', + }, + 'errors': { + 'class': 'logging.FileHandler', + 'filename': 'mplog-errors.log', + 'mode': 'w', + 'level': 'ERROR', + 'formatter': 'detailed', + }, + }, + 'loggers': { + 'foo': { + 'handlers' : ['foofile'] + } + }, + 'root': { + 'level': 'DEBUG', + 'handlers': ['console', 'file', 'errors'] + }, + } + workers = [] + for i in range(5): + wp = Process(target=worker_process, name='worker %d' % (i + 1), args=(q,)) + workers.append(wp) + wp.start() + logging.config.dictConfig(d) + lp = threading.Thread(target=logger_thread, args=(q,)) + lp.start() + # At this point, the main process could do some useful work of its own + # Once it's done that, it can wait for the workers to terminate... + for wp in workers: + wp.join() + # And now tell the logging thread to finish up, too + q.put(None) + lp.join() + +This variant shows how you can e.g. apply configuration for particular loggers +- e.g. the ``foo`` logger has a special handler which stores all events in the +``foo`` subsystem in a file ``mplog-foo.log``. This will be used by the logging +machinery in the main process (even though the logging events are generated in +the worker processes) to direct the messages to the appropriate destinations. + +Using file rotation +------------------- + +.. sectionauthor:: Doug Hellmann, Vinay Sajip (changes) +.. (see <http://blog.doughellmann.com/2007/05/pymotw-logging.html>) + +Sometimes you want to let a log file grow to a certain size, then open a new +file and log to that. You may want to keep a certain number of these files, and +when that many files have been created, rotate the files so that the number of +files and the size of the files both remain bounded. For this usage pattern, the +logging package provides a :class:`RotatingFileHandler`:: + + import glob + import logging + import logging.handlers + + LOG_FILENAME = 'logging_rotatingfile_example.out' + + # Set up a specific logger with our desired output level + my_logger = logging.getLogger('MyLogger') + my_logger.setLevel(logging.DEBUG) + + # Add the log message handler to the logger + handler = logging.handlers.RotatingFileHandler( + LOG_FILENAME, maxBytes=20, backupCount=5) + + my_logger.addHandler(handler) + + # Log some messages + for i in range(20): + my_logger.debug('i = %d' % i) + + # See what files are created + logfiles = glob.glob('%s*' % LOG_FILENAME) + + for filename in logfiles: + print(filename) + +The result should be 6 separate files, each with part of the log history for the +application:: + + logging_rotatingfile_example.out + logging_rotatingfile_example.out.1 + logging_rotatingfile_example.out.2 + logging_rotatingfile_example.out.3 + logging_rotatingfile_example.out.4 + logging_rotatingfile_example.out.5 + +The most current file is always :file:`logging_rotatingfile_example.out`, +and each time it reaches the size limit it is renamed with the suffix +``.1``. Each of the existing backup files is renamed to increment the suffix +(``.1`` becomes ``.2``, etc.) and the ``.6`` file is erased. + +Obviously this example sets the log length much too small as an extreme +example. You would want to set *maxBytes* to an appropriate value. + +.. _zeromq-handlers: + +Subclassing QueueHandler - a ZeroMQ example +------------------------------------------- + +You can use a :class:`QueueHandler` subclass to send messages to other kinds +of queues, for example a ZeroMQ 'publish' socket. In the example below,the +socket is created separately and passed to the handler (as its 'queue'):: + + import zmq # using pyzmq, the Python binding for ZeroMQ + import json # for serializing records portably + + ctx = zmq.Context() + sock = zmq.Socket(ctx, zmq.PUB) # or zmq.PUSH, or other suitable value + sock.bind('tcp://*:5556') # or wherever + + class ZeroMQSocketHandler(QueueHandler): + def enqueue(self, record): + data = json.dumps(record.__dict__) + self.queue.send(data) + + handler = ZeroMQSocketHandler(sock) + + +Of course there are other ways of organizing this, for example passing in the +data needed by the handler to create the socket:: + + class ZeroMQSocketHandler(QueueHandler): + def __init__(self, uri, socktype=zmq.PUB, ctx=None): + self.ctx = ctx or zmq.Context() + socket = zmq.Socket(self.ctx, socktype) + socket.bind(uri) + QueueHandler.__init__(self, socket) + + def enqueue(self, record): + data = json.dumps(record.__dict__) + self.queue.send(data) + + def close(self): + self.queue.close() + + +Subclassing QueueListener - a ZeroMQ example +-------------------------------------------- + +You can also subclass :class:`QueueListener` to get messages from other kinds +of queues, for example a ZeroMQ 'subscribe' socket. Here's an example:: + + class ZeroMQSocketListener(QueueListener): + def __init__(self, uri, *handlers, **kwargs): + self.ctx = kwargs.get('ctx') or zmq.Context() + socket = zmq.Socket(self.ctx, zmq.SUB) + socket.setsockopt(zmq.SUBSCRIBE, '') # subscribe to everything + socket.connect(uri) + + def dequeue(self): + msg = self.queue.recv() + return logging.makeLogRecord(json.loads(msg)) + + +.. seealso:: + + Module :mod:`logging` + API reference for the logging module. + + Module :mod:`logging.config` + Configuration API for the logging module. + + Module :mod:`logging.handlers` + Useful handlers included with the logging module. + + :ref:`A basic logging tutorial <logging-basic-tutorial>` + + :ref:`A more advanced logging tutorial <logging-advanced-tutorial>` + + +An example dictionary-based configuration +----------------------------------------- + +Below is an example of a logging configuration dictionary - it's taken from +the `documentation on the Django project <https://docs.djangoproject.com/en/1.3/topics/logging/#configuring-logging>`_. +This dictionary is passed to :func:`~logging.config.dictConfig` to put the configuration into effect:: + + LOGGING = { + 'version': 1, + 'disable_existing_loggers': True, + 'formatters': { + 'verbose': { + 'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s' + }, + 'simple': { + 'format': '%(levelname)s %(message)s' + }, + }, + 'filters': { + 'special': { + '()': 'project.logging.SpecialFilter', + 'foo': 'bar', + } + }, + 'handlers': { + 'null': { + 'level':'DEBUG', + 'class':'django.utils.log.NullHandler', + }, + 'console':{ + 'level':'DEBUG', + 'class':'logging.StreamHandler', + 'formatter': 'simple' + }, + 'mail_admins': { + 'level': 'ERROR', + 'class': 'django.utils.log.AdminEmailHandler', + 'filters': ['special'] + } + }, + 'loggers': { + 'django': { + 'handlers':['null'], + 'propagate': True, + 'level':'INFO', + }, + 'django.request': { + 'handlers': ['mail_admins'], + 'level': 'ERROR', + 'propagate': False, + }, + 'myproject.custom': { + 'handlers': ['console', 'mail_admins'], + 'level': 'INFO', + 'filters': ['special'] + } + } + } + +For more information about this configuration, you can see the `relevant +section <https://docs.djangoproject.com/en/1.3/topics/logging/#configuring-logging>`_ +of the Django documentation. diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst new file mode 100644 index 0000000000..2c9514a58d --- /dev/null +++ b/Doc/howto/logging.rst @@ -0,0 +1,1026 @@ +============= +Logging HOWTO +============= + +:Author: Vinay Sajip <vinay_sajip at red-dove dot com> + +.. _logging-basic-tutorial: + +.. currentmodule:: logging + +Basic Logging Tutorial +---------------------- + +Logging is a means of tracking events that happen when some software runs. The +software's developer adds logging calls to their code to indicate that certain +events have occurred. An event is described by a descriptive message which can +optionally contain variable data (i.e. data that is potentially different for +each occurrence of the event). Events also have an importance which the +developer ascribes to the event; the importance can also be called the *level* +or *severity*. + +When to use logging +^^^^^^^^^^^^^^^^^^^ + +Logging provides a set of convenience functions for simple logging usage. These +are :func:`debug`, :func:`info`, :func:`warning`, :func:`error` and +:func:`critical`. To determine when to use logging, see the table below, which +states, for each of a set of common tasks, the best tool to use for it. + ++-------------------------------------+--------------------------------------+ +| Task you want to perform | The best tool for the task | ++=====================================+======================================+ +| Display console output for ordinary | :func:`print` | +| usage of a command line script or | | +| program | | ++-------------------------------------+--------------------------------------+ +| Report events that occur during | :func:`logging.info` (or | +| normal operation of a program (e.g. | :func:`logging.debug` for very | +| for status monitoring or fault | detailed output for diagnostic | +| investigation) | purposes) | ++-------------------------------------+--------------------------------------+ +| Issue a warning regarding a | :func:`warnings.warn` in library | +| particular runtime event | code if the issue is avoidable and | +| | the client application should be | +| | modified to eliminate the warning | +| | | +| | :func:`logging.warning` if there is | +| | nothing the client application can do| +| | about the situation, but the event | +| | should still be noted | ++-------------------------------------+--------------------------------------+ +| Report an error regarding a | Raise an exception | +| particular runtime event | | ++-------------------------------------+--------------------------------------+ +| Report suppression of an error | :func:`logging.error`, | +| without raising an exception (e.g. | :func:`logging.exception` or | +| error handler in a long-running | :func:`logging.critical` as | +| server process) | appropriate for the specific error | +| | and application domain | ++-------------------------------------+--------------------------------------+ + +The logging functions are named after the level or severity of the events +they are used to track. The standard levels and their applicability are +described below (in increasing order of severity): + ++--------------+---------------------------------------------+ +| Level | When it's used | ++==============+=============================================+ +| ``DEBUG`` | Detailed information, typically of interest | +| | only when diagnosing problems. | ++--------------+---------------------------------------------+ +| ``INFO`` | Confirmation that things are working as | +| | expected. | ++--------------+---------------------------------------------+ +| ``WARNING`` | An indication that something unexpected | +| | happened, or indicative of some problem in | +| | the near future (e.g. 'disk space low'). | +| | The software is still working as expected. | ++--------------+---------------------------------------------+ +| ``ERROR`` | Due to a more serious problem, the software | +| | has not been able to perform some function. | ++--------------+---------------------------------------------+ +| ``CRITICAL`` | A serious error, indicating that the program| +| | itself may be unable to continue running. | ++--------------+---------------------------------------------+ + +The default level is ``WARNING``, which means that only events of this level +and above will be tracked, unless the logging package is configured to do +otherwise. + +Events that are tracked can be handled in different ways. The simplest way of +handling tracked events is to print them to the console. Another common way +is to write them to a disk file. + + +.. _howto-minimal-example: + +A simple example +^^^^^^^^^^^^^^^^ + +A very simple example is:: + + import logging + logging.warning('Watch out!') # will print a message to the console + logging.info('I told you so') # will not print anything + +If you type these lines into a script and run it, you'll see:: + + WARNING:root:Watch out! + +printed out on the console. The ``INFO`` message doesn't appear because the +default level is ``WARNING``. The printed message includes the indication of +the level and the description of the event provided in the logging call, i.e. +'Watch out!'. Don't worry about the 'root' part for now: it will be explained +later. The actual output can be formatted quite flexibly if you need that; +formatting options will also be explained later. + + +Logging to a file +^^^^^^^^^^^^^^^^^ + +A very common situation is that of recording logging events in a file, so let's +look at that next:: + + import logging + logging.basicConfig(filename='example.log',level=logging.DEBUG) + logging.debug('This message should go to the log file') + logging.info('So should this') + logging.warning('And this, too') + +And now if we open the file and look at what we have, we should find the log +messages:: + + DEBUG:root:This message should go to the log file + INFO:root:So should this + WARNING:root:And this, too + +This example also shows how you can set the logging level which acts as the +threshold for tracking. In this case, because we set the threshold to +``DEBUG``, all of the messages were printed. + +If you want to set the logging level from a command-line option such as:: + + --log=INFO + +and you have the value of the parameter passed for ``--log`` in some variable +*loglevel*, you can use:: + + getattr(logging, loglevel.upper()) + +to get the value which you'll pass to :func:`basicConfig` via the *level* +argument. You may want to error check any user input value, perhaps as in the +following example:: + + # assuming loglevel is bound to the string value obtained from the + # command line argument. Convert to upper case to allow the user to + # specify --log=DEBUG or --log=debug + numeric_level = getattr(logging, loglevel.upper(), None) + if not isinstance(numeric_level, int): + raise ValueError('Invalid log level: %s' % loglevel) + logging.basicConfig(level=numeric_level, ...) + +The call to :func:`basicConfig` should come *before* any calls to :func:`debug`, +:func:`info` etc. As it's intended as a one-off simple configuration facility, +only the first call will actually do anything: subsequent calls are effectively +no-ops. + +If you run the above script several times, the messages from successive runs +are appended to the file *example.log*. If you want each run to start afresh, +not remembering the messages from earlier runs, you can specify the *filemode* +argument, by changing the call in the above example to:: + + logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG) + +The output will be the same as before, but the log file is no longer appended +to, so the messages from earlier runs are lost. + + +Logging from multiple modules +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If your program consists of multiple modules, here's an example of how you +could organize logging in it:: + + # myapp.py + import logging + import mylib + + def main(): + logging.basicConfig(filename='myapp.log', level=logging.INFO) + logging.info('Started') + mylib.do_something() + logging.info('Finished') + + if __name__ == '__main__': + main() + +:: + + # mylib.py + import logging + + def do_something(): + logging.info('Doing something') + +If you run *myapp.py*, you should see this in *myapp.log*:: + + INFO:root:Started + INFO:root:Doing something + INFO:root:Finished + +which is hopefully what you were expecting to see. You can generalize this to +multiple modules, using the pattern in *mylib.py*. Note that for this simple +usage pattern, you won't know, by looking in the log file, *where* in your +application your messages came from, apart from looking at the event +description. If you want to track the location of your messages, you'll need +to refer to the documentation beyond the tutorial level -- see +:ref:`logging-advanced-tutorial`. + + +Logging variable data +^^^^^^^^^^^^^^^^^^^^^ + +To log variable data, use a format string for the event description message and +append the variable data as arguments. For example:: + + import logging + logging.warning('%s before you %s', 'Look', 'leap!') + +will display:: + + WARNING:root:Look before you leap! + +As you can see, merging of variable data into the event description message +uses the old, %-style of string formatting. This is for backwards +compatibility: the logging package pre-dates newer formatting options such as +:meth:`str.format` and :class:`string.Template`. These newer formatting +options *are* supported, but exploring them is outside the scope of this +tutorial. + + +Changing the format of displayed messages +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To change the format which is used to display messages, you need to +specify the format you want to use:: + + import logging + logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG) + logging.debug('This message should appear on the console') + logging.info('So should this') + logging.warning('And this, too') + +which would print:: + + DEBUG:This message should appear on the console + INFO:So should this + WARNING:And this, too + +Notice that the 'root' which appeared in earlier examples has disappeared. For +a full set of things that can appear in format strings, you can refer to the +documentation for :ref:`logrecord-attributes`, but for simple usage, you just +need the *levelname* (severity), *message* (event description, including +variable data) and perhaps to display when the event occurred. This is +described in the next section. + + +Displaying the date/time in messages +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To display the date and time of an event, you would place '%(asctime)s' in +your format string:: + + import logging + logging.basicConfig(format='%(asctime)s %(message)s') + logging.warning('is when this event was logged.') + +which should print something like this:: + + 2010-12-12 11:41:42,612 is when this event was logged. + +The default format for date/time display (shown above) is ISO8601. If you need +more control over the formatting of the date/time, provide a *datefmt* +argument to ``basicConfig``, as in this example:: + + import logging + logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p') + logging.warning('is when this event was logged.') + +which would display something like this:: + + 12/12/2010 11:46:36 AM is when this event was logged. + +The format of the *datefmt* argument is the same as supported by +:func:`time.strftime`. + + +Next Steps +^^^^^^^^^^ + +That concludes the basic tutorial. It should be enough to get you up and +running with logging. There's a lot more that the logging package offers, but +to get the best out of it, you'll need to invest a little more of your time in +reading the following sections. If you're ready for that, grab some of your +favourite beverage and carry on. + +If your logging needs are simple, then use the above examples to incorporate +logging into your own scripts, and if you run into problems or don't +understand something, please post a question on the comp.lang.python Usenet +group (available at http://groups.google.com/group/comp.lang.python) and you +should receive help before too long. + +Still here? You can carry on reading the next few sections, which provide a +slightly more advanced/in-depth tutorial than the basic one above. After that, +you can take a look at the :ref:`logging-cookbook`. + +.. _logging-advanced-tutorial: + + +Advanced Logging Tutorial +------------------------- + +The logging library takes a modular approach and offers several categories +of components: loggers, handlers, filters, and formatters. + +* Loggers expose the interface that application code directly uses. +* Handlers send the log records (created by loggers) to the appropriate + destination. +* Filters provide a finer grained facility for determining which log records + to output. +* Formatters specify the layout of log records in the final output. + +Logging is performed by calling methods on instances of the :class:`Logger` +class (hereafter called :dfn:`loggers`). Each instance has a name, and they are +conceptually arranged in a namespace hierarchy using dots (periods) as +separators. For example, a logger named 'scan' is the parent of loggers +'scan.text', 'scan.html' and 'scan.pdf'. Logger names can be anything you want, +and indicate the area of an application in which a logged message originates. + +A good convention to use when naming loggers is to use a module-level logger, +in each module which uses logging, named as follows:: + + logger = logging.getLogger(__name__) + +This means that logger names track the package/module hierarchy, and it's +intuitively obvious where events are logged just from the logger name. + +The root of the hierarchy of loggers is called the root logger. That's the +logger used by the functions :func:`debug`, :func:`info`, :func:`warning`, +:func:`error` and :func:`critical`, which just call the same-named method of +the root logger. The functions and the methods have the same signatures. The +root logger's name is printed as 'root' in the logged output. + +It is, of course, possible to log messages to different destinations. Support +is included in the package for writing log messages to files, HTTP GET/POST +locations, email via SMTP, generic sockets, queues, or OS-specific logging +mechanisms such as syslog or the Windows NT event log. Destinations are served +by :dfn:`handler` classes. You can create your own log destination class if +you have special requirements not met by any of the built-in handler classes. + +By default, no destination is set for any logging messages. You can specify +a destination (such as console or file) by using :func:`basicConfig` as in the +tutorial examples. If you call the functions :func:`debug`, :func:`info`, +:func:`warning`, :func:`error` and :func:`critical`, they will check to see +if no destination is set; and if one is not set, they will set a destination +of the console (``sys.stderr``) and a default format for the displayed +message before delegating to the root logger to do the actual message output. + +The default format set by :func:`basicConfig` for messages is:: + + severity:logger name:message + +You can change this by passing a format string to :func:`basicConfig` with the +*format* keyword argument. For all options regarding how a format string is +constructed, see :ref:`formatter-objects`. + + +Loggers +^^^^^^^ + +:class:`Logger` objects have a threefold job. First, they expose several +methods to application code so that applications can log messages at runtime. +Second, logger objects determine which log messages to act upon based upon +severity (the default filtering facility) or filter objects. Third, logger +objects pass along relevant log messages to all interested log handlers. + +The most widely used methods on logger objects fall into two categories: +configuration and message sending. + +These are the most common configuration methods: + +* :meth:`Logger.setLevel` specifies the lowest-severity log message a logger + will handle, where debug is the lowest built-in severity level and critical + is the highest built-in severity. For example, if the severity level is + INFO, the logger will handle only INFO, WARNING, ERROR, and CRITICAL messages + and will ignore DEBUG messages. + +* :meth:`Logger.addHandler` and :meth:`Logger.removeHandler` add and remove + handler objects from the logger object. Handlers are covered in more detail + in :ref:`handler-basic`. + +* :meth:`Logger.addFilter` and :meth:`Logger.removeFilter` add and remove filter + objects from the logger object. Filters are covered in more detail in + :ref:`filter`. + +You don't need to always call these methods on every logger you create. See the +last two paragraphs in this section. + +With the logger object configured, the following methods create log messages: + +* :meth:`Logger.debug`, :meth:`Logger.info`, :meth:`Logger.warning`, + :meth:`Logger.error`, and :meth:`Logger.critical` all create log records with + a message and a level that corresponds to their respective method names. The + message is actually a format string, which may contain the standard string + substitution syntax of ``%s``, ``%d``, ``%f``, and so on. The + rest of their arguments is a list of objects that correspond with the + substitution fields in the message. With regard to ``**kwargs``, the + logging methods care only about a keyword of ``exc_info`` and use it to + determine whether to log exception information. + +* :meth:`Logger.exception` creates a log message similar to + :meth:`Logger.error`. The difference is that :meth:`Logger.exception` dumps a + stack trace along with it. Call this method only from an exception handler. + +* :meth:`Logger.log` takes a log level as an explicit argument. This is a + little more verbose for logging messages than using the log level convenience + methods listed above, but this is how to log at custom log levels. + +:func:`getLogger` returns a reference to a logger instance with the specified +name if it is provided, or ``root`` if not. The names are period-separated +hierarchical structures. Multiple calls to :func:`getLogger` with the same name +will return a reference to the same logger object. Loggers that are further +down in the hierarchical list are children of loggers higher up in the list. +For example, given a logger with a name of ``foo``, loggers with names of +``foo.bar``, ``foo.bar.baz``, and ``foo.bam`` are all descendants of ``foo``. + +Loggers have a concept of *effective level*. If a level is not explicitly set +on a logger, the level of its parent is used instead as its effective level. +If the parent has no explicit level set, *its* parent is examined, and so on - +all ancestors are searched until an explicitly set level is found. The root +logger always has an explicit level set (``WARNING`` by default). When deciding +whether to process an event, the effective level of the logger is used to +determine whether the event is passed to the logger's handlers. + +Child loggers propagate messages up to the handlers associated with their +ancestor loggers. Because of this, it is unnecessary to define and configure +handlers for all the loggers an application uses. It is sufficient to +configure handlers for a top-level logger and create child loggers as needed. +(You can, however, turn off propagation by setting the *propagate* +attribute of a logger to *False*.) + + +.. _handler-basic: + +Handlers +^^^^^^^^ + +:class:`~logging.Handler` objects are responsible for dispatching the +appropriate log messages (based on the log messages' severity) to the handler's +specified destination. Logger objects can add zero or more handler objects to +themselves with an :func:`addHandler` method. As an example scenario, an +application may want to send all log messages to a log file, all log messages +of error or higher to stdout, and all messages of critical to an email address. +This scenario requires three individual handlers where each handler is +responsible for sending messages of a specific severity to a specific location. + +The standard library includes quite a few handler types (see +:ref:`useful-handlers`); the tutorials use mainly :class:`StreamHandler` and +:class:`FileHandler` in its examples. + +There are very few methods in a handler for application developers to concern +themselves with. The only handler methods that seem relevant for application +developers who are using the built-in handler objects (that is, not creating +custom handlers) are the following configuration methods: + +* The :meth:`Handler.setLevel` method, just as in logger objects, specifies the + lowest severity that will be dispatched to the appropriate destination. Why + are there two :func:`setLevel` methods? The level set in the logger + determines which severity of messages it will pass to its handlers. The level + set in each handler determines which messages that handler will send on. + +* :func:`setFormatter` selects a Formatter object for this handler to use. + +* :func:`addFilter` and :func:`removeFilter` respectively configure and + deconfigure filter objects on handlers. + +Application code should not directly instantiate and use instances of +:class:`Handler`. Instead, the :class:`Handler` class is a base class that +defines the interface that all handlers should have and establishes some +default behavior that child classes can use (or override). + + +Formatters +^^^^^^^^^^ + +Formatter objects configure the final order, structure, and contents of the log +message. Unlike the base :class:`logging.Handler` class, application code may +instantiate formatter classes, although you could likely subclass the formatter +if your application needs special behavior. The constructor takes three +optional arguments -- a message format string, a date format string and a style +indicator. + +.. method:: logging.Formatter.__init__(fmt=None, datefmt=None, style='%') + +If there is no message format string, the default is to use the +raw message. If there is no date format string, the default date format is:: + + %Y-%m-%d %H:%M:%S + +with the milliseconds tacked on at the end. The ``style`` is one of `%`, '{' +or '$'. If one of these is not specified, then '%' will be used. + +If the ``style`` is '%', the message format string uses +``%(<dictionary key>)s`` styled string substitution; the possible keys are +documented in :ref:`logrecord-attributes`. If the style is '{', the message +format string is assumed to be compatible with :meth:`str.format` (using +keyword arguments), while if the style is '$' then the message format string +should conform to what is expected by :meth:`string.Template.substitute`. + +.. versionchanged:: 3.2 + Added the ``style`` parameter. + +The following message format string will log the time in a human-readable +format, the severity of the message, and the contents of the message, in that +order:: + + '%(asctime)s - %(levelname)s - %(message)s' + +Formatters use a user-configurable function to convert the creation time of a +record to a tuple. By default, :func:`time.localtime` is used; to change this +for a particular formatter instance, set the ``converter`` attribute of the +instance to a function with the same signature as :func:`time.localtime` or +:func:`time.gmtime`. To change it for all formatters, for example if you want +all logging times to be shown in GMT, set the ``converter`` attribute in the +Formatter class (to ``time.gmtime`` for GMT display). + + +Configuring Logging +^^^^^^^^^^^^^^^^^^^ + +.. currentmodule:: logging.config + +Programmers can configure logging in three ways: + +1. Creating loggers, handlers, and formatters explicitly using Python + code that calls the configuration methods listed above. +2. Creating a logging config file and reading it using the :func:`fileConfig` + function. +3. Creating a dictionary of configuration information and passing it + to the :func:`dictConfig` function. + +For the reference documentation on the last two options, see +:ref:`logging-config-api`. The following example configures a very simple +logger, a console handler, and a simple formatter using Python code:: + + import logging + + # create logger + logger = logging.getLogger('simple_example') + logger.setLevel(logging.DEBUG) + + # create console handler and set level to debug + ch = logging.StreamHandler() + ch.setLevel(logging.DEBUG) + + # create formatter + formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') + + # add formatter to ch + ch.setFormatter(formatter) + + # add ch to logger + logger.addHandler(ch) + + # 'application' code + logger.debug('debug message') + logger.info('info message') + logger.warn('warn message') + logger.error('error message') + logger.critical('critical message') + +Running this module from the command line produces the following output:: + + $ python simple_logging_module.py + 2005-03-19 15:10:26,618 - simple_example - DEBUG - debug message + 2005-03-19 15:10:26,620 - simple_example - INFO - info message + 2005-03-19 15:10:26,695 - simple_example - WARNING - warn message + 2005-03-19 15:10:26,697 - simple_example - ERROR - error message + 2005-03-19 15:10:26,773 - simple_example - CRITICAL - critical message + +The following Python module creates a logger, handler, and formatter nearly +identical to those in the example listed above, with the only difference being +the names of the objects:: + + import logging + import logging.config + + logging.config.fileConfig('logging.conf') + + # create logger + logger = logging.getLogger('simpleExample') + + # 'application' code + logger.debug('debug message') + logger.info('info message') + logger.warn('warn message') + logger.error('error message') + logger.critical('critical message') + +Here is the logging.conf file:: + + [loggers] + keys=root,simpleExample + + [handlers] + keys=consoleHandler + + [formatters] + keys=simpleFormatter + + [logger_root] + level=DEBUG + handlers=consoleHandler + + [logger_simpleExample] + level=DEBUG + handlers=consoleHandler + qualname=simpleExample + propagate=0 + + [handler_consoleHandler] + class=StreamHandler + level=DEBUG + formatter=simpleFormatter + args=(sys.stdout,) + + [formatter_simpleFormatter] + format=%(asctime)s - %(name)s - %(levelname)s - %(message)s + datefmt= + +The output is nearly identical to that of the non-config-file-based example:: + + $ python simple_logging_config.py + 2005-03-19 15:38:55,977 - simpleExample - DEBUG - debug message + 2005-03-19 15:38:55,979 - simpleExample - INFO - info message + 2005-03-19 15:38:56,054 - simpleExample - WARNING - warn message + 2005-03-19 15:38:56,055 - simpleExample - ERROR - error message + 2005-03-19 15:38:56,130 - simpleExample - CRITICAL - critical message + +You can see that the config file approach has a few advantages over the Python +code approach, mainly separation of configuration and code and the ability of +noncoders to easily modify the logging properties. + +.. currentmodule:: logging + +Note that the class names referenced in config files need to be either relative +to the logging module, or absolute values which can be resolved using normal +import mechanisms. Thus, you could use either +:class:`~logging.handlers.WatchedFileHandler` (relative to the logging module) or +``mypackage.mymodule.MyHandler`` (for a class defined in package ``mypackage`` +and module ``mymodule``, where ``mypackage`` is available on the Python import +path). + +In Python 3.2, a new means of configuring logging has been introduced, using +dictionaries to hold configuration information. This provides a superset of the +functionality of the config-file-based approach outlined above, and is the +recommended configuration method for new applications and deployments. Because +a Python dictionary is used to hold configuration information, and since you +can populate that dictionary using different means, you have more options for +configuration. For example, you can use a configuration file in JSON format, +or, if you have access to YAML processing functionality, a file in YAML +format, to populate the configuration dictionary. Or, of course, you can +construct the dictionary in Python code, receive it in pickled form over a +socket, or use whatever approach makes sense for your application. + +Here's an example of the same configuration as above, in YAML format for +the new dictionary-based approach:: + + version: 1 + formatters: + simple: + format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s' + handlers: + console: + class: logging.StreamHandler + level: DEBUG + formatter: simple + stream: ext://sys.stdout + loggers: + simpleExample: + level: DEBUG + handlers: [console] + propagate: no + root: + level: DEBUG + handlers: [console] + +For more information about logging using a dictionary, see +:ref:`logging-config-api`. + +What happens if no configuration is provided +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If no logging configuration is provided, it is possible to have a situation +where a logging event needs to be output, but no handlers can be found to +output the event. The behaviour of the logging package in these +circumstances is dependent on the Python version. + +For versions of Python prior to 3.2, the behaviour is as follows: + +* If *logging.raiseExceptions* is *False* (production mode), the event is + silently dropped. + +* If *logging.raiseExceptions* is *True* (development mode), a message + 'No handlers could be found for logger X.Y.Z' is printed once. + +In Python 3.2 and later, the behaviour is as follows: + +* The event is output using a 'handler of last resort', stored in + ``logging.lastResort``. This internal handler is not associated with any + logger, and acts like a :class:`~logging.StreamHandler` which writes the + event description message to the current value of ``sys.stderr`` (therefore + respecting any redirections which may be in effect). No formatting is + done on the message - just the bare event description message is printed. + The handler's level is set to ``WARNING``, so all events at this and + greater severities will be output. + +To obtain the pre-3.2 behaviour, ``logging.lastResort`` can be set to *None*. + +.. _library-config: + +Configuring Logging for a Library +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When developing a library which uses logging, you should take care to +document how the library uses logging - for example, the names of loggers +used. Some consideration also needs to be given to its logging configuration. +If the using application does not use logging, and library code makes logging +calls, then (as described in the previous section) events of severity +``WARNING`` and greater will be printed to ``sys.stderr``. This is regarded as +the best default behaviour. + +If for some reason you *don't* want these messages printed in the absence of +any logging configuration, you can attach a do-nothing handler to the top-level +logger for your library. This avoids the message being printed, since a handler +will be always be found for the library's events: it just doesn't produce any +output. If the library user configures logging for application use, presumably +that configuration will add some handlers, and if levels are suitably +configured then logging calls made in library code will send output to those +handlers, as normal. + +A do-nothing handler is included in the logging package: +:class:`~logging.NullHandler` (since Python 3.1). An instance of this handler +could be added to the top-level logger of the logging namespace used by the +library (*if* you want to prevent your library's logged events being output to +``sys.stderr`` in the absence of logging configuration). If all logging by a +library *foo* is done using loggers with names matching 'foo.x', 'foo.x.y', +etc. then the code:: + + import logging + logging.getLogger('foo').addHandler(logging.NullHandler()) + +should have the desired effect. If an organisation produces a number of +libraries, then the logger name specified can be 'orgname.foo' rather than +just 'foo'. + +**PLEASE NOTE:** It is strongly advised that you *do not add any handlers other +than* :class:`~logging.NullHandler` *to your library's loggers*. This is +because the configuration of handlers is the prerogative of the application +developer who uses your library. The application developer knows their target +audience and what handlers are most appropriate for their application: if you +add handlers 'under the hood', you might well interfere with their ability to +carry out unit tests and deliver logs which suit their requirements. + + +Logging Levels +-------------- + +The numeric values of logging levels are given in the following table. These are +primarily of interest if you want to define your own levels, and need them to +have specific values relative to the predefined levels. If you define a level +with the same numeric value, it overwrites the predefined value; the predefined +name is lost. + ++--------------+---------------+ +| Level | Numeric value | ++==============+===============+ +| ``CRITICAL`` | 50 | ++--------------+---------------+ +| ``ERROR`` | 40 | ++--------------+---------------+ +| ``WARNING`` | 30 | ++--------------+---------------+ +| ``INFO`` | 20 | ++--------------+---------------+ +| ``DEBUG`` | 10 | ++--------------+---------------+ +| ``NOTSET`` | 0 | ++--------------+---------------+ + +Levels can also be associated with loggers, being set either by the developer or +through loading a saved logging configuration. When a logging method is called +on a logger, the logger compares its own level with the level associated with +the method call. If the logger's level is higher than the method call's, no +logging message is actually generated. This is the basic mechanism controlling +the verbosity of logging output. + +Logging messages are encoded as instances of the :class:`~logging.LogRecord` +class. When a logger decides to actually log an event, a +:class:`~logging.LogRecord` instance is created from the logging message. + +Logging messages are subjected to a dispatch mechanism through the use of +:dfn:`handlers`, which are instances of subclasses of the :class:`Handler` +class. Handlers are responsible for ensuring that a logged message (in the form +of a :class:`LogRecord`) ends up in a particular location (or set of locations) +which is useful for the target audience for that message (such as end users, +support desk staff, system administrators, developers). Handlers are passed +:class:`LogRecord` instances intended for particular destinations. Each logger +can have zero, one or more handlers associated with it (via the +:meth:`~Logger.addHandler` method of :class:`Logger`). In addition to any +handlers directly associated with a logger, *all handlers associated with all +ancestors of the logger* are called to dispatch the message (unless the +*propagate* flag for a logger is set to a false value, at which point the +passing to ancestor handlers stops). + +Just as for loggers, handlers can have levels associated with them. A handler's +level acts as a filter in the same way as a logger's level does. If a handler +decides to actually dispatch an event, the :meth:`~Handler.emit` method is used +to send the message to its destination. Most user-defined subclasses of +:class:`Handler` will need to override this :meth:`~Handler.emit`. + +.. _custom-levels: + +Custom Levels +^^^^^^^^^^^^^ + +Defining your own levels is possible, but should not be necessary, as the +existing levels have been chosen on the basis of practical experience. +However, if you are convinced that you need custom levels, great care should +be exercised when doing this, and it is possibly *a very bad idea to define +custom levels if you are developing a library*. That's because if multiple +library authors all define their own custom levels, there is a chance that +the logging output from such multiple libraries used together will be +difficult for the using developer to control and/or interpret, because a +given numeric value might mean different things for different libraries. + +.. _useful-handlers: + +Useful Handlers +--------------- + +In addition to the base :class:`Handler` class, many useful subclasses are +provided: + +#. :class:`StreamHandler` instances send messages to streams (file-like + objects). + +#. :class:`FileHandler` instances send messages to disk files. + +#. :class:`~handlers.BaseRotatingHandler` is the base class for handlers that + rotate log files at a certain point. It is not meant to be instantiated + directly. Instead, use :class:`~handlers.RotatingFileHandler` or + :class:`~handlers.TimedRotatingFileHandler`. + +#. :class:`~handlers.RotatingFileHandler` instances send messages to disk + files, with support for maximum log file sizes and log file rotation. + +#. :class:`~handlers.TimedRotatingFileHandler` instances send messages to + disk files, rotating the log file at certain timed intervals. + +#. :class:`~handlers.SocketHandler` instances send messages to TCP/IP + sockets. + +#. :class:`~handlers.DatagramHandler` instances send messages to UDP + sockets. + +#. :class:`~handlers.SMTPHandler` instances send messages to a designated + email address. + +#. :class:`~handlers.SysLogHandler` instances send messages to a Unix + syslog daemon, possibly on a remote machine. + +#. :class:`~handlers.NTEventLogHandler` instances send messages to a + Windows NT/2000/XP event log. + +#. :class:`~handlers.MemoryHandler` instances send messages to a buffer + in memory, which is flushed whenever specific criteria are met. + +#. :class:`~handlers.HTTPHandler` instances send messages to an HTTP + server using either ``GET`` or ``POST`` semantics. + +#. :class:`~handlers.WatchedFileHandler` instances watch the file they are + logging to. If the file changes, it is closed and reopened using the file + name. This handler is only useful on Unix-like systems; Windows does not + support the underlying mechanism used. + +#. :class:`~handlers.QueueHandler` instances send messages to a queue, such as + those implemented in the :mod:`queue` or :mod:`multiprocessing` modules. + +#. :class:`NullHandler` instances do nothing with error messages. They are used + by library developers who want to use logging, but want to avoid the 'No + handlers could be found for logger XXX' message which can be displayed if + the library user has not configured logging. See :ref:`library-config` for + more information. + +.. versionadded:: 3.1 + The :class:`NullHandler` class. + +.. versionadded:: 3.2 + The :class:`~handlers.QueueHandler` class. + +The :class:`NullHandler`, :class:`StreamHandler` and :class:`FileHandler` +classes are defined in the core logging package. The other handlers are +defined in a sub- module, :mod:`logging.handlers`. (There is also another +sub-module, :mod:`logging.config`, for configuration functionality.) + +Logged messages are formatted for presentation through instances of the +:class:`Formatter` class. They are initialized with a format string suitable for +use with the % operator and a dictionary. + +For formatting multiple messages in a batch, instances of +:class:`BufferingFormatter` can be used. In addition to the format string (which +is applied to each message in the batch), there is provision for header and +trailer format strings. + +When filtering based on logger level and/or handler level is not enough, +instances of :class:`Filter` can be added to both :class:`Logger` and +:class:`Handler` instances (through their :meth:`addFilter` method). Before +deciding to process a message further, both loggers and handlers consult all +their filters for permission. If any filter returns a false value, the message +is not processed further. + +The basic :class:`Filter` functionality allows filtering by specific logger +name. If this feature is used, messages sent to the named logger and its +children are allowed through the filter, and all others dropped. + + +.. _logging-exceptions: + +Exceptions raised during logging +-------------------------------- + +The logging package is designed to swallow exceptions which occur while logging +in production. This is so that errors which occur while handling logging events +- such as logging misconfiguration, network or other similar errors - do not +cause the application using logging to terminate prematurely. + +:class:`SystemExit` and :class:`KeyboardInterrupt` exceptions are never +swallowed. Other exceptions which occur during the :meth:`emit` method of a +:class:`Handler` subclass are passed to its :meth:`handleError` method. + +The default implementation of :meth:`handleError` in :class:`Handler` checks +to see if a module-level variable, :data:`raiseExceptions`, is set. If set, a +traceback is printed to :data:`sys.stderr`. If not set, the exception is swallowed. + +**Note:** The default value of :data:`raiseExceptions` is ``True``. This is because +during development, you typically want to be notified of any exceptions that +occur. It's advised that you set :data:`raiseExceptions` to ``False`` for production +usage. + +.. currentmodule:: logging + +.. _arbitrary-object-messages: + +Using arbitrary objects as messages +----------------------------------- + +In the preceding sections and examples, it has been assumed that the message +passed when logging the event is a string. However, this is not the only +possibility. You can pass an arbitrary object as a message, and its +:meth:`__str__` method will be called when the logging system needs to convert +it to a string representation. In fact, if you want to, you can avoid +computing a string representation altogether - for example, the +:class:`SocketHandler` emits an event by pickling it and sending it over the +wire. + + +Optimization +------------ + +Formatting of message arguments is deferred until it cannot be avoided. +However, computing the arguments passed to the logging method can also be +expensive, and you may want to avoid doing it if the logger will just throw +away your event. To decide what to do, you can call the :meth:`isEnabledFor` +method which takes a level argument and returns true if the event would be +created by the Logger for that level of call. You can write code like this:: + + if logger.isEnabledFor(logging.DEBUG): + logger.debug('Message with %s, %s', expensive_func1(), + expensive_func2()) + +so that if the logger's threshold is set above ``DEBUG``, the calls to +:func:`expensive_func1` and :func:`expensive_func2` are never made. + +There are other optimizations which can be made for specific applications which +need more precise control over what logging information is collected. Here's a +list of things you can do to avoid processing during logging which you don't +need: + ++-----------------------------------------------+----------------------------------------+ +| What you don't want to collect | How to avoid collecting it | ++===============================================+========================================+ +| Information about where calls were made from. | Set ``logging._srcfile`` to ``None``. | ++-----------------------------------------------+----------------------------------------+ +| Threading information. | Set ``logging.logThreads`` to ``0``. | ++-----------------------------------------------+----------------------------------------+ +| Process information. | Set ``logging.logProcesses`` to ``0``. | ++-----------------------------------------------+----------------------------------------+ + +Also note that the core logging module only includes the basic handlers. If +you don't import :mod:`logging.handlers` and :mod:`logging.config`, they won't +take up any memory. + +.. seealso:: + + Module :mod:`logging` + API reference for the logging module. + + Module :mod:`logging.config` + Configuration API for the logging module. + + Module :mod:`logging.handlers` + Useful handlers included with the logging module. + + :ref:`A logging cookbook <logging-cookbook>` + diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst new file mode 100644 index 0000000000..df0d299f4a --- /dev/null +++ b/Doc/howto/pyporting.rst @@ -0,0 +1,715 @@ +.. _pyporting-howto: + +********************************* +Porting Python 2 Code to Python 3 +********************************* + +:author: Brett Cannon + +.. topic:: Abstract + + With Python 3 being the future of Python while Python 2 is still in active + use, it is good to have your project available for both major releases of + Python. This guide is meant to help you choose which strategy works best + for your project to support both Python 2 & 3 along with how to execute + that strategy. + + If you are looking to port an extension module instead of pure Python code, + please see :ref:`cporting-howto`. + + +Choosing a Strategy +=================== + +When a project makes the decision that it's time to support both Python 2 & 3, +a decision needs to be made as to how to go about accomplishing that goal. +The chosen strategy will depend on how large the project's existing +codebase is and how much divergence you want from your Python 2 codebase from +your Python 3 one (e.g., starting a new version with Python 3). + +If your project is brand-new or does not have a large codebase, then you may +want to consider writing/porting :ref:`all of your code for Python 3 +and use 3to2 <use_3to2>` to port your code for Python 2. + +If you would prefer to maintain a codebase which is semantically **and** +syntactically compatible with Python 2 & 3 simultaneously, you can write +:ref:`use_same_source`. While this tends to lead to somewhat non-idiomatic +code, it does mean you keep a rapid development process for you, the developer. + +Finally, you do have the option of :ref:`using 2to3 <use_2to3>` to translate +Python 2 code into Python 3 code (with some manual help). This can take the +form of branching your code and using 2to3 to start a Python 3 branch. You can +also have users perform the translation as installation time automatically so +that you only have to maintain a Python 2 codebase. + +Regardless of which approach you choose, porting is not as hard or +time-consuming as you might initially think. You can also tackle the problem +piece-meal as a good portion of porting is simply updating your code to follow +current best practices in a Python 2/3 compatible way. + + +Universal Bits of Advice +------------------------ + +Regardless of what strategy you pick, there are a few things you should +consider. + +One is make sure you have a robust test suite. You need to make sure everything +continues to work, just like when you support a new minor version of Python. +This means making sure your test suite is thorough and is ported properly +between Python 2 & 3. You will also most likely want to use something like tox_ +to automate testing between both a Python 2 and Python 3 VM. + +Two, once your project has Python 3 support, make sure to add the proper +classifier on the Cheeseshop_ (PyPI_). To have your project listed as Python 3 +compatible it must have the +`Python 3 classifier <http://pypi.python.org/pypi?:action=browse&c=533>`_ +(from +http://techspot.zzzeek.org/2011/01/24/zzzeek-s-guide-to-python-3-porting/):: + + setup( + name='Your Library', + version='1.0', + classifiers=[ + # make sure to use :: Python *and* :: Python :: 3 so + # that pypi can list the package on the python 3 page + 'Programming Language :: Python', + 'Programming Language :: Python :: 3' + ], + packages=['yourlibrary'], + # make sure to add custom_fixers to the MANIFEST.in + include_package_data=True, + # ... + ) + + +Doing so will cause your project to show up in the +`Python 3 packages list +<http://pypi.python.org/pypi?:action=browse&c=533&show=all>`_. You will know +you set the classifier properly as visiting your project page on the Cheeseshop +will show a Python 3 logo in the upper-left corner of the page. + +Three, the six_ project provides a library which helps iron out differences +between Python 2 & 3. If you find there is a sticky point that is a continual +point of contention in your translation or maintenance of code, consider using +a source-compatible solution relying on six. If you have to create your own +Python 2/3 compatible solution, you can use ``sys.version_info[0] >= 3`` as a +guard. + +Four, read all the approaches. Just because some bit of advice applies to one +approach more than another doesn't mean that some advice doesn't apply to other +strategies. + +Five, drop support for older Python versions if possible. `Python 2.5`_ +introduced a lot of useful syntax and libraries which have become idiomatic +in Python 3. `Python 2.6`_ introduced future statements which makes +compatibility much easier if you are going from Python 2 to 3. +`Python 2.7`_ continues the trend in the stdlib. So choose the newest version +of Python which you believe can be your minimum support version +and work from there. + + +.. _tox: http://codespeak.net/tox/ +.. _Cheeseshop: +.. _PyPI: http://pypi.python.org/ +.. _six: http://packages.python.org/six +.. _Python 2.7: http://www.python.org/2.7.x +.. _Python 2.6: http://www.python.org/2.6.x +.. _Python 2.5: http://www.python.org/2.5.x +.. _Python 2.4: http://www.python.org/2.4.x +.. _Python 2.3: http://www.python.org/2.3.x +.. _Python 2.2: http://www.python.org/2.2.x + + +.. _use_3to2: + +Python 3 and 3to2 +================= + +If you are starting a new project or your codebase is small enough, you may +want to consider writing your code for Python 3 and backporting to Python 2 +using 3to2_. Thanks to Python 3 being more strict about things than Python 2 +(e.g., bytes vs. strings), the source translation can be easier and more +straightforward than from Python 2 to 3. Plus it gives you more direct +experience developing in Python 3 which, since it is the future of Python, is a +good thing long-term. + +A drawback of this approach is that 3to2 is a third-party project. This means +that the Python core developers (and thus this guide) can make no promises +about how well 3to2 works at any time. There is nothing to suggest, though, +that 3to2 is not a high-quality project. + + +.. _3to2: https://bitbucket.org/amentajo/lib3to2/overview + + +.. _use_2to3: + +Python 2 and 2to3 +================= + +Included with Python since 2.6, the 2to3_ tool (and :mod:`lib2to3` module) +helps with porting Python 2 to Python 3 by performing various source +translations. This is a perfect solution for projects which wish to branch +their Python 3 code from their Python 2 codebase and maintain them as +independent codebases. You can even begin preparing to use this approach +today by writing future-compatible Python code which works cleanly in +Python 2 in conjunction with 2to3; all steps outlined below will work +with Python 2 code up to the point when the actual use of 2to3 occurs. + +Use of 2to3 as an on-demand translation step at install time is also possible, +preventing the need to maintain a separate Python 3 codebase, but this approach +does come with some drawbacks. While users will only have to pay the +translation cost once at installation, you as a developer will need to pay the +cost regularly during development. If your codebase is sufficiently large +enough then the translation step ends up acting like a compilation step, +robbing you of the rapid development process you are used to with Python. +Obviously the time required to translate a project will vary, so do an +experimental translation just to see how long it takes to evaluate whether you +prefer this approach compared to using :ref:`use_same_source` or simply keeping +a separate Python 3 codebase. + +Below are the typical steps taken by a project which uses a 2to3-based approach +to supporting Python 2 & 3. + + +Support Python 2.7 +------------------ + +As a first step, make sure that your project is compatible with `Python 2.7`_. +This is just good to do as Python 2.7 is the last release of Python 2 and thus +will be used for a rather long time. It also allows for use of the ``-3`` flag +to Python to help discover places in your code which 2to3 cannot handle but are +known to cause issues. + +Try to Support `Python 2.6`_ and Newer Only +------------------------------------------- + +While not possible for all projects, if you can support `Python 2.6`_ and newer +**only**, your life will be much easier. Various future statements, stdlib +additions, etc. exist only in Python 2.6 and later which greatly assist in +porting to Python 3. But if you project must keep support for `Python 2.5`_ (or +even `Python 2.4`_) then it is still possible to port to Python 3. + +Below are the benefits you gain if you only have to support Python 2.6 and +newer. Some of these options are personal choice while others are +**strongly** recommended (the ones that are more for personal choice are +labeled as such). If you continue to support older versions of Python then you +at least need to watch out for situations that these solutions fix. + + +``from __future__ import print_function`` +''''''''''''''''''''''''''''''''''''''''' + +This is a personal choice. 2to3 handles the translation from the print +statement to the print function rather well so this is an optional step. This +future statement does help, though, with getting used to typing +``print('Hello, World')`` instead of ``print 'Hello, World'``. + + +``from __future__ import unicode_literals`` +''''''''''''''''''''''''''''''''''''''''''' + +Another personal choice. You can always mark what you want to be a (unicode) +string with a ``u`` prefix to get the same effect. But regardless of whether +you use this future statement or not, you **must** make sure you know exactly +which Python 2 strings you want to be bytes, and which are to be strings. This +means you should, **at minimum** mark all strings that are meant to be text +strings with a ``u`` prefix if you do not use this future statement. + + +Bytes literals +'''''''''''''' + +This is a **very** important one. The ability to prefix Python 2 strings that +are meant to contain bytes with a ``b`` prefix help to very clearly delineate +what is and is not a Python 3 string. When you run 2to3 on code, all Python 2 +strings become Python 3 strings **unless** they are prefixed with ``b``. + +There are some differences between byte literals in Python 2 and those in +Python 3 thanks to the bytes type just being an alias to ``str`` in Python 2. +Probably the biggest "gotcha" is that indexing results in different values. In +Python 2, the value of ``b'py'[1]`` is ``'y'``, while in Python 3 it's ``121``. +You can avoid this disparity by always slicing at the size of a single element: +``b'py'[1:2]`` is ``'y'`` in Python 2 and ``b'y'`` in Python 3 (i.e., close +enough). + +You cannot concatenate bytes and strings in Python 3. But since in Python +2 has bytes aliased to ``str``, it will succeed: ``b'a' + u'b'`` works in +Python 2, but ``b'a' + 'b'`` in Python 3 is a :exc:`TypeError`. A similar issue +also comes about when doing comparisons between bytes and strings. + + +Supporting `Python 2.5`_ and Newer Only +--------------------------------------- + +If you are supporting `Python 2.5`_ and newer there are still some features of +Python that you can utilize. + + +``from __future__ import absolute_import`` +'''''''''''''''''''''''''''''''''''''''''' + +Implicit relative imports (e.g., importing ``spam.bacon`` from within +``spam.eggs`` with the statement ``import bacon``) does not work in Python 3. +This future statement moves away from that and allows the use of explicit +relative imports (e.g., ``from . import bacon``). + +In `Python 2.5`_ you must use +the __future__ statement to get to use explicit relative imports and prevent +implicit ones. In `Python 2.6`_ explicit relative imports are available without +the statement, but you still want the __future__ statement to prevent implicit +relative imports. In `Python 2.7`_ the __future__ statement is not needed. In +other words, unless you are only supporting Python 2.7 or a version earlier +than Python 2.5, use the __future__ statement. + + + +Handle Common "Gotchas" +----------------------- + +There are a few things that just consistently come up as sticking points for +people which 2to3 cannot handle automatically or can easily be done in Python 2 +to help modernize your code. + + +``from __future__ import division`` +''''''''''''''''''''''''''''''''''' + +While the exact same outcome can be had by using the ``-Qnew`` argument to +Python, using this future statement lifts the requirement that your users use +the flag to get the expected behavior of division in Python 3 +(e.g., ``1/2 == 0.5; 1//2 == 0``). + + + +Specify when opening a file as binary +''''''''''''''''''''''''''''''''''''' + +Unless you have been working on Windows, there is a chance you have not always +bothered to add the ``b`` mode when opening a binary file (e.g., ``rb`` for +binary reading). Under Python 3, binary files and text files are clearly +distinct and mutually incompatible; see the :mod:`io` module for details. +Therefore, you **must** make a decision of whether a file will be used for +binary access (allowing to read and/or write bytes data) or text access +(allowing to read and/or write unicode data). + +Text files +'''''''''' + +Text files created using ``open()`` under Python 2 return byte strings, +while under Python 3 they return unicode strings. Depending on your porting +strategy, this can be an issue. + +If you want text files to return unicode strings in Python 2, you have two +possibilities: + +* Under Python 2.6 and higher, use :func:`io.open`. Since :func:`io.open` + is essentially the same function in both Python 2 and Python 3, it will + help iron out any issues that might arise. + +* If pre-2.6 compatibility is needed, then you should use :func:`codecs.open` + instead. This will make sure that you get back unicode strings in Python 2. + +Subclass ``object`` +''''''''''''''''''' + +New-style classes have been around since `Python 2.2`_. You need to make sure +you are subclassing from ``object`` to avoid odd edge cases involving method +resolution order, etc. This continues to be totally valid in Python 3 (although +unneeded as all classes implicitly inherit from ``object``). + + +Deal With the Bytes/String Dichotomy +'''''''''''''''''''''''''''''''''''' + +One of the biggest issues people have when porting code to Python 3 is handling +the bytes/string dichotomy. Because Python 2 allowed the ``str`` type to hold +textual data, people have over the years been rather loose in their delineation +of what ``str`` instances held text compared to bytes. In Python 3 you cannot +be so care-free anymore and need to properly handle the difference. The key +handling this issue to make sure that **every** string literal in your +Python 2 code is either syntactically of functionally marked as either bytes or +text data. After this is done you then need to make sure your APIs are designed +to either handle a specific type or made to be properly polymorphic. + + +Mark Up Python 2 String Literals +******************************** + +First thing you must do is designate every single string literal in Python 2 +as either textual or bytes data. If you are only supporting Python 2.6 or +newer, this can be accomplished by marking bytes literals with a ``b`` prefix +and then designating textual data with a ``u`` prefix or using the +``unicode_literals`` future statement. + +If your project supports versions of Python pre-dating 2.6, then you should use +the six_ project and its ``b()`` function to denote bytes literals. For text +literals you can either use six's ``u()`` function or use a ``u`` prefix. + + +Decide what APIs Will Accept +**************************** + +In Python 2 it was very easy to accidentally create an API that accepted both +bytes and textual data. But in Python 3, thanks to the more strict handling of +disparate types, this loose usage of bytes and text together tends to fail. + +Take the dict ``{b'a': 'bytes', u'a': 'text'}`` in Python 2.6. It creates the +dict ``{u'a': 'text'}`` since ``b'a' == u'a'``. But in Python 3 the equivalent +dict creates ``{b'a': 'bytes', 'a': 'text'}``, i.e., no lost data. Similar +issues can crop up when transitioning Python 2 code to Python 3. + +This means you need to choose what an API is going to accept and create and +consistently stick to that API in both Python 2 and 3. + + +Bytes / Unicode Comparison +************************** + +In Python 3, mixing bytes and unicode is forbidden in most situations; it +will raise a :class:`TypeError` where Python 2 would have attempted an implicit +coercion between types. However, there is one case where it doesn't and +it can be very misleading:: + + >>> b"" == "" + False + +This is because an equality comparison is required by the language to always +succeed (and return ``False`` for incompatible types). However, this also +means that code incorrectly ported to Python 3 can display buggy behaviour +if such comparisons are silently executed. To detect such situations, +Python 3 has a ``-b`` flag that will display a warning:: + + $ python3 -b + >>> b"" == "" + __main__:1: BytesWarning: Comparison between bytes and string + False + +To turn the warning into an exception, use the ``-bb`` flag instead:: + + $ python3 -bb + >>> b"" == "" + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + BytesWarning: Comparison between bytes and string + + +Indexing bytes objects +'''''''''''''''''''''' + +Another potentially surprising change is the indexing behaviour of bytes +objects in Python 3:: + + >>> b"xyz"[0] + 120 + +Indeed, Python 3 bytes objects (as well as :class:`bytearray` objects) +are sequences of integers. But code converted from Python 2 will often +assume that indexing a bytestring produces another bytestring, not an +integer. To reconcile both behaviours, use slicing:: + + >>> b"xyz"[0:1] + b'x' + >>> n = 1 + >>> b"xyz"[n:n+1] + b'y' + +The only remaining gotcha is that an out-of-bounds slice returns an empty +bytes object instead of raising ``IndexError``: + + >>> b"xyz"[3] + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + IndexError: index out of range + >>> b"xyz"[3:4] + b'' + + +``__str__()``/``__unicode__()`` +''''''''''''''''''''''''''''''' + +In Python 2, objects can specify both a string and unicode representation of +themselves. In Python 3, though, there is only a string representation. This +becomes an issue as people can inadvertently do things in their ``__str__()`` +methods which have unpredictable results (e.g., infinite recursion if you +happen to use the ``unicode(self).encode('utf8')`` idiom as the body of your +``__str__()`` method). + +There are two ways to solve this issue. One is to use a custom 2to3 fixer. The +blog post at http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/ +specifies how to do this. That will allow 2to3 to change all instances of ``def +__unicode(self): ...`` to ``def __str__(self): ...``. This does require you +define your ``__str__()`` method in Python 2 before your ``__unicode__()`` +method. + +The other option is to use a mixin class. This allows you to only define a +``__unicode__()`` method for your class and let the mixin derive +``__str__()`` for you (code from +http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/):: + + import sys + + class UnicodeMixin(object): + + """Mixin class to handle defining the proper __str__/__unicode__ + methods in Python 2 or 3.""" + + if sys.version_info[0] >= 3: # Python 3 + def __str__(self): + return self.__unicode__() + else: # Python 2 + def __str__(self): + return self.__unicode__().encode('utf8') + + + class Spam(UnicodeMixin): + + def __unicode__(self): + return u'spam-spam-bacon-spam' # 2to3 will remove the 'u' prefix + + +Don't Index on Exceptions +''''''''''''''''''''''''' + +In Python 2, the following worked:: + + >>> exc = Exception(1, 2, 3) + >>> exc.args[1] + 2 + >>> exc[1] # Python 2 only! + 2 + +But in Python 3, indexing directly on an exception is an error. You need to +make sure to only index on the :attr:`BaseException.args` attribute which is a +sequence containing all arguments passed to the :meth:`__init__` method. + +Even better is to use the documented attributes the exception provides. + +Don't use ``__getslice__`` & Friends +'''''''''''''''''''''''''''''''''''' + +Been deprecated for a while, but Python 3 finally drops support for +``__getslice__()``, etc. Move completely over to :meth:`__getitem__` and +friends. + + +Updating doctests +''''''''''''''''' + +2to3_ will attempt to generate fixes for doctests that it comes across. It's +not perfect, though. If you wrote a monolithic set of doctests (e.g., a single +docstring containing all of your doctests), you should at least consider +breaking the doctests up into smaller pieces to make it more manageable to fix. +Otherwise it might very well be worth your time and effort to port your tests +to :mod:`unittest`. + + +Update `map` for imbalanced input sequences +''''''''''''''''''''''''''''''''''''''''''' + +With Python 2, `map` would pad input sequences of unequal length with +`None` values, returning a sequence as long as the longest input sequence. + +With Python 3, if the input sequences to `map` are of unequal length, `map` +will stop at the termination of the shortest of the sequences. For full +compatibility with `map` from Python 2.x, also wrap the sequences in +:func:`itertools.zip_longest`, e.g. ``map(func, *sequences)`` becomes +``list(map(func, itertools.zip_longest(*sequences)))``. + +Eliminate ``-3`` Warnings +------------------------- + +When you run your application's test suite, run it using the ``-3`` flag passed +to Python. This will cause various warnings to be raised during execution about +things that 2to3 cannot handle automatically (e.g., modules that have been +removed). Try to eliminate those warnings to make your code even more portable +to Python 3. + + +Run 2to3 +-------- + +Once you have made your Python 2 code future-compatible with Python 3, it's +time to use 2to3_ to actually port your code. + + +Manually +'''''''' + +To manually convert source code using 2to3_, you use the ``2to3`` script that +is installed with Python 2.6 and later.:: + + 2to3 <directory or file to convert> + +This will cause 2to3 to write out a diff with all of the fixers applied for the +converted source code. If you would like 2to3 to go ahead and apply the changes +you can pass it the ``-w`` flag:: + + 2to3 -w <stuff to convert> + +There are other flags available to control exactly which fixers are applied, +etc. + + +During Installation +''''''''''''''''''' + +When a user installs your project for Python 3, you can have either +:mod:`distutils` or Distribute_ run 2to3_ on your behalf. +For distutils, use the following idiom:: + + try: # Python 3 + from distutils.command.build_py import build_py_2to3 as build_py + except ImportError: # Python 2 + from distutils.command.build_py import build_py + + setup(cmdclass = {'build_py': build_py}, + # ... + ) + +For Distribute:: + + setup(use_2to3=True, + # ... + ) + +This will allow you to not have to distribute a separate Python 3 version of +your project. It does require, though, that when you perform development that +you at least build your project and use the built Python 3 source for testing. + + +Verify & Test +------------- + +At this point you should (hopefully) have your project converted in such a way +that it works in Python 3. Verify it by running your unit tests and making sure +nothing has gone awry. If you miss something then figure out how to fix it in +Python 3, backport to your Python 2 code, and run your code through 2to3 again +to verify the fix transforms properly. + + +.. _2to3: http://docs.python.org/py3k/library/2to3.html +.. _Distribute: http://packages.python.org/distribute/ + + +.. _use_same_source: + +Python 2/3 Compatible Source +============================ + +While it may seem counter-intuitive, you can write Python code which is +source-compatible between Python 2 & 3. It does lead to code that is not +entirely idiomatic Python (e.g., having to extract the currently raised +exception from ``sys.exc_info()[1]``), but it can be run under Python 2 +**and** Python 3 without using 2to3_ as a translation step (although the tool +should be used to help find potential portability problems). This allows you to +continue to have a rapid development process regardless of whether you are +developing under Python 2 or Python 3. Whether this approach or using +:ref:`use_2to3` works best for you will be a per-project decision. + +To get a complete idea of what issues you will need to deal with, see the +`What's New in Python 3.0`_. Others have reorganized the data in other formats +such as http://docs.pythonsprints.com/python3_porting/py-porting.html . + +The following are some steps to take to try to support both Python 2 & 3 from +the same source code. + + +.. _What's New in Python 3.0: http://docs.python.org/release/3.0/whatsnew/3.0.html + + +Follow The Steps for Using 2to3_ +-------------------------------- + +All of the steps outlined in how to +:ref:`port Python 2 code with 2to3 <use_2to3>` apply +to creating a Python 2/3 codebase. This includes trying only support Python 2.6 +or newer (the :mod:`__future__` statements work in Python 3 without issue), +eliminating warnings that are triggered by ``-3``, etc. + +You should even consider running 2to3_ over your code (without committing the +changes). This will let you know where potential pain points are within your +code so that you can fix them properly before they become an issue. + + +Use six_ +-------- + +The six_ project contains many things to help you write portable Python code. +You should make sure to read its documentation from beginning to end and use +any and all features it provides. That way you will minimize any mistakes you +might make in writing cross-version code. + + +Capturing the Currently Raised Exception +---------------------------------------- + +One change between Python 2 and 3 that will require changing how you code (if +you support `Python 2.5`_ and earlier) is +accessing the currently raised exception. In Python 2.5 and earlier the syntax +to access the current exception is:: + + try: + raise Exception() + except Exception, exc: + # Current exception is 'exc' + pass + +This syntax changed in Python 3 (and backported to `Python 2.6`_ and later) +to:: + + try: + raise Exception() + except Exception as exc: + # Current exception is 'exc' + # In Python 3, 'exc' is restricted to the block; Python 2.6 will "leak" + pass + +Because of this syntax change you must change to capturing the current +exception to:: + + try: + raise Exception() + except Exception: + import sys + exc = sys.exc_info()[1] + # Current exception is 'exc' + pass + +You can get more information about the raised exception from +:func:`sys.exc_info` than simply the current exception instance, but you most +likely don't need it. + +.. note:: + In Python 3, the traceback is attached to the exception instance + through the ``__traceback__`` attribute. If the instance is saved in + a local variable that persists outside of the ``except`` block, the + traceback will create a reference cycle with the current frame and its + dictionary of local variables. This will delay reclaiming dead + resources until the next cyclic :term:`garbage collection` pass. + + In Python 2, this problem only occurs if you save the traceback itself + (e.g. the third element of the tuple returned by :func:`sys.exc_info`) + in a variable. + + +Other Resources +=============== + +The authors of the following blog posts, wiki pages, and books deserve special +thanks for making public their tips for porting Python 2 code to Python 3 (and +thus helping provide information for this document): + +* http://python3porting.com/ +* http://docs.pythonsprints.com/python3_porting/py-porting.html +* http://techspot.zzzeek.org/2011/01/24/zzzeek-s-guide-to-python-3-porting/ +* http://dabeaz.blogspot.com/2011/01/porting-py65-and-my-superboard-to.html +* http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/ +* http://lucumr.pocoo.org/2010/2/11/porting-to-python-3-a-guide/ +* http://wiki.python.org/moin/PortingPythonToPy3k + +If you feel there is something missing from this document that should be added, +please email the python-porting_ mailing list. + +.. _python-porting: http://mail.python.org/mailman/listinfo/python-porting diff --git a/Doc/howto/sockets.rst b/Doc/howto/sockets.rst index 04e9b98b2c..a4ae9c0b1b 100644 --- a/Doc/howto/sockets.rst +++ b/Doc/howto/sockets.rst @@ -1,3 +1,5 @@ +.. _socket-howto: + **************************** Socket Programming HOWTO **************************** @@ -60,11 +62,10 @@ Creating a Socket Roughly speaking, when you clicked on the link that brought you to this page, your browser did something like the following:: - #create an INET, STREAMing socket + # create an INET, STREAMing socket s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - #now connect to the web server on port 80 - # - the normal http port - s.connect(("www.mcmillan-inc.com", 80)) + # now connect to the web server on port 80 - the normal http port + s.connect(("www.python.org", 80)) When the ``connect`` completes, the socket ``s`` can be used to send in a request for the text of the page. The same socket will read the @@ -75,13 +76,11 @@ exchanges). What happens in the web server is a bit more complex. First, the web server creates a "server socket":: - #create an INET, STREAMing socket - serversocket = socket.socket( - socket.AF_INET, socket.SOCK_STREAM) - #bind the socket to a public host, - # and a well-known port + # create an INET, STREAMing socket + serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + # bind the socket to a public host, and a well-known port serversocket.bind((socket.gethostname(), 80)) - #become a server socket + # become a server socket serversocket.listen(5) A couple things to notice: we used ``socket.gethostname()`` so that the socket @@ -101,10 +100,10 @@ Now that we have a "server" socket, listening on port 80, we can enter the mainloop of the web server:: while True: - #accept connections from outside + # accept connections from outside (clientsocket, address) = serversocket.accept() - #now do something with the clientsocket - #in this case, we'll pretend this is a threaded server + # now do something with the clientsocket + # in this case, we'll pretend this is a threaded server ct = client_thread(clientsocket) ct.run() @@ -126,12 +125,13 @@ IPC --- If you need fast IPC between two processes on one machine, you should look into -whatever form of shared memory the platform offers. A simple protocol based -around shared memory and locks or semaphores is by far the fastest technique. +pipes or shared memory. If you do decide to use AF_INET sockets, bind the +"server" socket to ``'localhost'``. On most platforms, this will take a +shortcut around a couple of layers of network code and be quite a bit faster. -If you do decide to use sockets, bind the "server" socket to ``'localhost'``. On -most platforms, this will take a shortcut around a couple of layers of network -code and be quite a bit faster. +.. seealso:: + The :mod:`multiprocessing` integrates cross-platform IPC into a higher-level + API. Using a Socket @@ -300,7 +300,7 @@ When Sockets Die Probably the worst thing about using blocking sockets is what happens when the other side comes down hard (without doing a ``close``). Your socket is likely to -hang. SOCKSTREAM is a reliable protocol, and it will wait a long, long time +hang. TCP is a reliable protocol, and it will wait a long, long time before giving up on a connection. If you're using threads, the entire thread is essentially dead. There's not much you can do about it. As long as you aren't doing something dumb, like holding a lock while doing a blocking read, the @@ -395,19 +395,13 @@ Performance There's no question that the fastest sockets code uses non-blocking sockets and select to multiplex them. You can put together something that will saturate a -LAN connection without putting any strain on the CPU. The trouble is that an app -written this way can't do much of anything else - it needs to be ready to -shuffle bytes around at all times. - -Assuming that your app is actually supposed to do something more than that, -threading is the optimal solution, (and using non-blocking sockets will be -faster than using blocking sockets). Unfortunately, threading support in Unixes -varies both in API and quality. So the normal Unix solution is to fork a -subprocess to deal with each connection. The overhead for this is significant -(and don't do this on Windows - the overhead of process creation is enormous -there). It also means that unless each subprocess is completely independent, -you'll need to use another form of IPC, say a pipe, or shared memory and -semaphores, to communicate between the parent and child processes. +LAN connection without putting any strain on the CPU. + +The trouble is that an app written this way can't do much of anything else - +it needs to be ready to shuffle bytes around at all times. Assuming that your +app is actually supposed to do something more than that, threading is the +optimal solution, (and using non-blocking sockets will be faster than using +blocking sockets). Finally, remember that even though blocking sockets are somewhat slower than non-blocking, in many cases they are the "right" solution. After all, if your diff --git a/Doc/howto/sorting.rst b/Doc/howto/sorting.rst index 351713f2cf..d9c70e2d8a 100644 --- a/Doc/howto/sorting.rst +++ b/Doc/howto/sorting.rst @@ -23,7 +23,7 @@ returns a new sorted list:: >>> sorted([5, 2, 3, 1, 4]) [1, 2, 3, 4, 5] -You can also use the :meth:`list.sort` method of a list. It modifies the list +You can also use the :meth:`list.sort` method. It modifies the list in-place (and returns *None* to avoid confusion). Usually it's less convenient than :func:`sorted` - but if you don't need the original list, it's slightly more efficient. @@ -87,9 +87,9 @@ Operator Module Functions ========================= The key-function patterns shown above are very common, so Python provides -convenience functions to make accessor functions easier and faster. The operator -module has :func:`operator.itemgetter`, :func:`operator.attrgetter`, and -an :func:`operator.methodcaller` function. +convenience functions to make accessor functions easier and faster. The +:mod:`operator` module has :func:`~operator.itemgetter`, +:func:`~operator.attrgetter`, and an :func:`~operator.methodcaller` function. Using those functions, the above examples become simpler and faster: @@ -247,6 +247,8 @@ To convert to a key function, just wrap the old comparison function: >>> sorted([5, 2, 4, 1, 3], key=cmp_to_key(reverse_numeric)) [5, 4, 3, 2, 1] +In Python 3.2, the :func:`functools.cmp_to_key` function was added to the +:mod:`functools` module in the standard library. Odd and Ends ============ @@ -254,7 +256,7 @@ Odd and Ends * For locale aware sorting, use :func:`locale.strxfrm` for a key function or :func:`locale.strcoll` for a comparison function. -* The *reverse* parameter still maintains sort stability (i.e. records with +* The *reverse* parameter still maintains sort stability (so that records with equal keys retain the original order). Interestingly, that effect can be simulated without the parameter by using the builtin :func:`reversed` function twice: diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst index 13efa7610f..045fd339fd 100644 --- a/Doc/howto/unicode.rst +++ b/Doc/howto/unicode.rst @@ -4,13 +4,11 @@ Unicode HOWTO ***************** -:Release: 1.11 +:Release: 1.12 -This HOWTO discusses Python 2.x's support for Unicode, and explains +This HOWTO discusses Python support for Unicode, and explains various problems that people commonly encounter when trying to work -with Unicode. (This HOWTO has not yet been updated to cover the 3.x -versions of Python.) - +with Unicode. Introduction to Unicode ======================= @@ -44,14 +42,14 @@ In the 1980s, almost all personal computers were 8-bit, meaning that bytes could hold values ranging from 0 to 255. ASCII codes only went up to 127, so some machines assigned values between 128 and 255 to accented characters. Different machines had different codes, however, which led to problems exchanging files. -Eventually various commonly used sets of values for the 128-255 range emerged. +Eventually various commonly used sets of values for the 128--255 range emerged. Some were true standards, defined by the International Standards Organization, and some were **de facto** conventions that were invented by one company or another and managed to catch on. 255 characters aren't very many. For example, you can't fit both the accented characters used in Western Europe and the Cyrillic alphabet used for Russian -into the 128-255 range because there are more than 127 such characters. +into the 128--255 range because there are more than 127 such characters. You could write files using different codes (all your Russian files in a coding system called KOI8, all your French files in a different coding system called @@ -64,8 +62,8 @@ bits means you have 2^16 = 65,536 distinct values available, making it possible to represent many different characters from many different alphabets; an initial goal was to have Unicode contain the alphabets for every single human language. It turns out that even 16 bits isn't enough to meet that goal, and the modern -Unicode specification uses a wider range of codes, 0-1,114,111 (0x10ffff in -base-16). +Unicode specification uses a wider range of codes, 0 through 1,114,111 (0x10ffff +in base 16). There's a related ISO standard, ISO 10646. Unicode and ISO 10646 were originally separate efforts, but the specifications were merged with the 1.1 @@ -90,7 +88,7 @@ meanings. The Unicode standard describes how characters are represented by **code points**. A code point is an integer value, usually denoted in base 16. In the standard, a code point is written using the notation U+12ca to mean the -character with value 0x12ca (4810 decimal). The Unicode standard contains a lot +character with value 0x12ca (4,810 decimal). The Unicode standard contains a lot of tables listing characters and their corresponding code points:: 0061 'a'; LATIN SMALL LETTER A @@ -117,10 +115,10 @@ Encodings --------- To summarize the previous section: a Unicode string is a sequence of code -points, which are numbers from 0 to 0x10ffff. This sequence needs to be -represented as a set of bytes (meaning, values from 0-255) in memory. The rules -for translating a Unicode string into a sequence of bytes are called an -**encoding**. +points, which are numbers from 0 through 0x10ffff (1,114,111 decimal). This +sequence needs to be represented as a set of bytes (meaning, values +from 0 through 255) in memory. The rules for translating a Unicode string +into a sequence of bytes are called an **encoding**. The first encoding you might think of is an array of 32-bit integers. In this representation, the string "Python" would look like this:: @@ -164,7 +162,7 @@ encoding, for example, are simple; for each code point: case.) Latin-1, also known as ISO-8859-1, is a similar encoding. Unicode code points -0-255 are identical to the Latin-1 values, so converting to this encoding simply +0--255 are identical to the Latin-1 values, so converting to this encoding simply requires converting code points to byte values; if a code point larger than 255 is encountered, the string can't be encoded into Latin-1. @@ -226,8 +224,8 @@ Wikipedia entries are often helpful; see the entries for "character encoding" <http://en.wikipedia.org/wiki/UTF-8>, for example. -Python 2.x's Unicode Support -============================ +Python's Unicode Support +======================== Now that you've learned the rudiments of Unicode, we can look at Python's Unicode features. @@ -284,10 +282,10 @@ that contains the corresponding code point. The reverse operation is the built-in :func:`ord` function that takes a one-character Unicode string and returns the code point value:: - >>> chr(40960) - '\ua000' - >>> ord('\ua000') - 40960 + >>> chr(57344) + '\ue000' + >>> ord('\ue000') + 57344 Converting to Bytes ------------------- @@ -329,7 +327,8 @@ Unicode Literals in Python Source Code In Python source code, specific Unicode code points can be written using the ``\u`` escape sequence, which is followed by four hex digits giving the code -point. The ``\U`` escape sequence is similar, but expects 8 hex digits, not 4:: +point. The ``\U`` escape sequence is similar, but expects eight hex digits, +not four:: >>> s = "a\xac\u1234\u20ac\U00008000" ^^^^ two-digit hex escape @@ -409,7 +408,7 @@ These are grouped into categories such as "Letter", "Number", "Punctuation", or from the above output, ``'Ll'`` means 'Letter, lowercase', ``'No'`` means "Number, other", ``'Mn'`` is "Mark, nonspacing", and ``'So'`` is "Symbol, other". See -<http://unicode.org/Public/5.1.0/ucd/UCD.html#General_Category_Values> for a +<http://www.unicode.org/reports/tr44/#General_Category_Values> for a list of category codes. References @@ -468,18 +467,17 @@ like those in string objects' :meth:`encode` and :meth:`decode` methods. Reading Unicode from a file is therefore simple:: - f = open('unicode.rst', encoding='utf-8') - for line in f: - print(repr(line)) + with open('unicode.rst', encoding='utf-8') as f: + for line in f: + print(repr(line)) It's also possible to open files in update mode, allowing both reading and writing:: - f = open('test', encoding='utf-8', mode='w+') - f.write('\u4500 blah blah blah\n') - f.seek(0) - print(repr(f.readline()[:1])) - f.close() + with open('test', encoding='utf-8', mode='w+') as f: + f.write('\u4500 blah blah blah\n') + f.seek(0) + print(repr(f.readline()[:1])) The Unicode character U+FEFF is used as a byte-order mark (BOM), and is often written as the first character of a file in order to assist with autodetection @@ -516,14 +514,13 @@ usually just provide the Unicode string as the filename, and it will be automatically converted to the right encoding for you:: filename = 'filename\u4500abc' - f = open(filename, 'w') - f.write('blah\n') - f.close() + with open(filename, 'w') as f: + f.write('blah\n') Functions in the :mod:`os` module such as :func:`os.stat` will also accept Unicode filenames. -:func:`os.listdir`, which returns filenames, raises an issue: should it return +Function :func:`os.listdir`, which returns filenames, raises an issue: should it return the Unicode version of filenames, or should it return byte strings containing the encoded versions? :func:`os.listdir` will do both, depending on whether you provided the directory path as a byte string or a Unicode string. If you pass a @@ -555,7 +552,6 @@ should only be used on systems where undecodable file names can be present, i.e. Unix systems. - Tips for Writing Unicode-aware Programs --------------------------------------- @@ -572,39 +568,15 @@ strings, you will find your program vulnerable to bugs wherever you combine the two different kinds of strings. There is no automatic encoding or decoding if you do e.g. ``str + bytes``, a :exc:`TypeError` is raised for this expression. -It's easy to miss such problems if you only test your software with data that -doesn't contain any accents; everything will seem to work, but there's actually -a bug in your program waiting for the first user who attempts to use characters -> 127. A second tip, therefore, is: - - Include characters > 127 and, even better, characters > 255 in your test - data. - When using data coming from a web browser or some other untrusted source, a common technique is to check for illegal characters in a string before using the string in a generated command line or storing it in a database. If you're doing -this, be careful to check the string once it's in the form that will be used or -stored; it's possible for encodings to be used to disguise characters. This is -especially true if the input data also specifies the encoding; many encodings -leave the commonly checked-for characters alone, but Python includes some -encodings such as ``'base64'`` that modify every single character. - -For example, let's say you have a content management system that takes a Unicode -filename, and you want to disallow paths with a '/' character. You might write -this code:: - - def read_file(filename, encoding): - if '/' in filename: - raise ValueError("'/' not allowed in filenames") - unicode_name = filename.decode(encoding) - f = open(unicode_name, 'r') - # ... return contents of file ... - -However, if an attacker could specify the ``'base64'`` encoding, they could pass -``'L2V0Yy9wYXNzd2Q='``, which is the base-64 encoded form of the string -``'/etc/passwd'``, to read a system file. The above code looks for ``'/'`` -characters in the encoded form and misses the dangerous character in the -resulting decoded form. +this, be careful to check the decoded string, not the encoded bytes data; +some encodings may have interesting properties, such as not being bijective +or not being fully ASCII-compatible. This is especially true if the input +data also specifies the encoding, since the attacker can then choose a +clever way to hide malicious text in the encoded bytestream. + References ---------- @@ -613,27 +585,30 @@ The PDF slides for Marc-André Lemburg's presentation "Writing Unicode-aware Applications in Python" are available at <http://downloads.egenix.com/python/LSM2005-Developing-Unicode-aware-applications-in-Python.pdf> and discuss questions of character encodings as well as how to internationalize -and localize an application. +and localize an application. These slides cover Python 2.x only. -Revision History and Acknowledgements -===================================== +Acknowledgements +================ Thanks to the following people who have noted errors or offered suggestions on this article: Nicholas Bastin, Marius Gedminas, Kent Johnson, Ken Krugler, Marc-André Lemburg, Martin von Löwis, Chad Whitacre. -Version 1.0: posted August 5 2005. +.. comment + Revision History + + Version 1.0: posted August 5 2005. -Version 1.01: posted August 7 2005. Corrects factual and markup errors; adds -several links. + Version 1.01: posted August 7 2005. Corrects factual and markup errors; adds + several links. -Version 1.02: posted August 16 2005. Corrects factual errors. + Version 1.02: posted August 16 2005. Corrects factual errors. -Version 1.1: Feb-Nov 2008. Updates the document with respect to Python 3 changes. + Version 1.1: Feb-Nov 2008. Updates the document with respect to Python 3 changes. -Version 1.11: posted June 20 2010. Notes that Python 3.x is not covered, -and that the HOWTO only covers 2.x. + Version 1.11: posted June 20 2010. Notes that Python 3.x is not covered, + and that the HOWTO only covers 2.x. .. comment Describe Python 3.x support (new section? new document?) .. comment Additional topic: building Python w/ UCS2 or UCS4 support diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst index 110b6de3b6..76286bdc27 100644 --- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@ -140,7 +140,7 @@ This is done as follows:: name=Somebody+Here&language=Python&location=Northampton >>> url = 'http://www.example.com/example.cgi' >>> full_url = url + '?' + url_values - >>> data = urllib.request.open(full_url) + >>> data = urllib.request.urlopen(full_url) Notice that the full URL is created by adding a ``?`` to the URL, followed by the encoded values. diff --git a/Doc/howto/webservers.rst b/Doc/howto/webservers.rst index caf0ad6667..72ccd1f690 100644 --- a/Doc/howto/webservers.rst +++ b/Doc/howto/webservers.rst @@ -264,7 +264,7 @@ used for the deployment of WSGI applications. * `FastCGI, SCGI, and Apache: Background and Future <http://www.vmunix.com/mark/blog/archives/2006/01/02/fastcgi-scgi-and-apache-background-and-future/>`_ - is a discussion on why the concept of FastCGI and SCGI is better that that + is a discussion on why the concept of FastCGI and SCGI is better than that of mod_python. @@ -274,7 +274,7 @@ Setting up FastCGI Each web server requires a specific module. * Apache has both `mod_fastcgi <http://www.fastcgi.com/drupal/>`_ and `mod_fcgid - <http://fastcgi.coremail.cn/>`_. ``mod_fastcgi`` is the original one, but it + <http://httpd.apache.org/mod_fcgid/>`_. ``mod_fastcgi`` is the original one, but it has some licensing issues, which is why it is sometimes considered non-free. ``mod_fcgid`` is a smaller, compatible alternative. One of these modules needs to be loaded by Apache. @@ -293,7 +293,7 @@ following WSGI-application:: # -*- coding: UTF-8 -*- import sys, os - from cgi import escape + from html import escape from flup.server.fcgi import WSGIServer def app(environ, start_response): @@ -365,7 +365,7 @@ testing. A really great WSGI feature is middleware. Middleware is a layer around your program which can add various functionality to it. There is quite a bit of -`middleware <http://wsgi.org/wsgi/Middleware_and_Utilities>`_ already +`middleware <http://www.wsgi.org/en/latest/libraries.html>`_ already available. For example, instead of writing your own session management (HTTP is a stateless protocol, so to associate multiple HTTP requests with a single user your application must create and manage such state via a session), you can @@ -396,9 +396,9 @@ compared with other web technologies. .. seealso:: - A good overview of WSGI-related code can be found in the `WSGI wiki - <http://wsgi.org/wsgi>`_, which contains an extensive list of `WSGI servers - <http://wsgi.org/wsgi/Servers>`_ which can be used by *any* application + A good overview of WSGI-related code can be found in the `WSGI homepage + <http://www.wsgi.org/en/latest/index.html>`_, which contains an extensive list of `WSGI servers + <http://www.wsgi.org/en/latest/servers.html>`_ which can be used by *any* application supporting WSGI. You might be interested in some WSGI-supporting modules already contained in diff --git a/Doc/includes/dbpickle.py b/Doc/includes/dbpickle.py index c021eac6c3..b88ee87d87 100644 --- a/Doc/includes/dbpickle.py +++ b/Doc/includes/dbpickle.py @@ -47,7 +47,8 @@ class DBUnpickler(pickle.Unpickler): def main(): - import io, pprint + import io + import pprint # Initialize and populate our database. conn = sqlite3.connect(":memory:") diff --git a/Doc/includes/email-alternative.py b/Doc/includes/email-alternative.py index 82e3ffa3b3..33c430ab95 100644 --- a/Doc/includes/email-alternative.py +++ b/Doc/includes/email-alternative.py @@ -1,4 +1,4 @@ -#! /usr/bin/python +#!/usr/bin/env python3 import smtplib diff --git a/Doc/includes/email-dir.py b/Doc/includes/email-dir.py index 035442bc3a..cc5529e8fe 100644 --- a/Doc/includes/email-dir.py +++ b/Doc/includes/email-dir.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 """Send the contents of a directory as a MIME message.""" diff --git a/Doc/includes/email-headers.py b/Doc/includes/email-headers.py new file mode 100644 index 0000000000..a53317dbd8 --- /dev/null +++ b/Doc/includes/email-headers.py @@ -0,0 +1,17 @@ +# Import the email modules we'll need +from email.parser import Parser + +# If the e-mail headers are in a file, uncomment this line: +#headers = Parser().parse(open(messagefile, 'r')) + +# Or for parsing headers in a string, use: +headers = Parser().parsestr('From: <user@example.com>\n' + 'To: <someone_else@example.com>\n' + 'Subject: Test message\n' + '\n' + 'Body would go here\n') + +# Now the header items can be accessed as a dictionary: +print('To: %s' % headers['to']) +print('From: %s' % headers['from']) +print('Subject: %s' % headers['subject']) diff --git a/Doc/includes/email-mime.py b/Doc/includes/email-mime.py index 7b1c028fad..a90edc1373 100644 --- a/Doc/includes/email-mime.py +++ b/Doc/includes/email-mime.py @@ -27,5 +27,5 @@ for file in pngfiles: # Send the email via our own SMTP server. s = smtplib.SMTP('localhost') -s.sendmail(me, family, msg.as_string()) +s.send_message(msg) s.quit() diff --git a/Doc/includes/email-simple.py b/Doc/includes/email-simple.py index 29bd0782d6..077568d563 100644 --- a/Doc/includes/email-simple.py +++ b/Doc/includes/email-simple.py @@ -17,8 +17,7 @@ msg['Subject'] = 'The contents of %s' % textfile msg['From'] = me msg['To'] = you -# Send the message via our own SMTP server, but don't include the -# envelope header. +# Send the message via our own SMTP server. s = smtplib.SMTP('localhost') -s.sendmail(me, [you], msg.as_string()) +s.send_message(msg) s.quit() diff --git a/Doc/includes/email-unpack.py b/Doc/includes/email-unpack.py index a8f712d26f..3653543566 100644 --- a/Doc/includes/email-unpack.py +++ b/Doc/includes/email-unpack.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 """Unpack a MIME message into a directory of files.""" diff --git a/Doc/includes/minidom-example.py b/Doc/includes/minidom-example.py index 88048c0c6a..5ee7682c19 100644 --- a/Doc/includes/minidom-example.py +++ b/Doc/includes/minidom-example.py @@ -19,11 +19,11 @@ document = """\ dom = xml.dom.minidom.parseString(document) def getText(nodelist): - rc = "" + rc = [] for node in nodelist: if node.nodeType == node.TEXT_NODE: - rc = rc + node.data - return rc + rc.append(node.data) + return ''.join(rc) def handleSlideshow(slideshow): print("<html>") diff --git a/Doc/includes/mp_benchmarks.py b/Doc/includes/mp_benchmarks.py index 72d4426c25..acdf6425bc 100644 --- a/Doc/includes/mp_benchmarks.py +++ b/Doc/includes/mp_benchmarks.py @@ -5,7 +5,12 @@ # All rights reserved. # -import time, sys, multiprocessing, threading, queue, gc +import time +import sys +import multiprocessing +import threading +import queue +import gc if sys.platform == 'win32': _timer = time.clock @@ -111,7 +116,7 @@ def test_seqspeed(seq): for i in range(iterations): a = seq[5] - elapsed = _timer()-t + elapsed = _timer() - t print(iterations, 'iterations in', elapsed, 'seconds') print('average number/sec:', iterations/elapsed) @@ -132,7 +137,7 @@ def test_lockspeed(l): l.acquire() l.release() - elapsed = _timer()-t + elapsed = _timer() - t print(iterations, 'iterations in', elapsed, 'seconds') print('average number/sec:', iterations/elapsed) @@ -169,7 +174,7 @@ def test_conditionspeed(Process, c): c.notify() c.wait() - elapsed = _timer()-t + elapsed = _timer() - t c.release() p.join() diff --git a/Doc/includes/mp_pool.py b/Doc/includes/mp_pool.py index e360703bd1..1578498754 100644 --- a/Doc/includes/mp_pool.py +++ b/Doc/includes/mp_pool.py @@ -25,18 +25,18 @@ def calculatestar(args): return calculate(*args) def mul(a, b): - time.sleep(0.5*random.random()) + time.sleep(0.5 * random.random()) return a * b def plus(a, b): - time.sleep(0.5*random.random()) + time.sleep(0.5 * random.random()) return a + b def f(x): - return 1.0 / (x-5.0) + return 1.0 / (x - 5.0) def pow3(x): - return x**3 + return x ** 3 def noop(x): pass diff --git a/Doc/includes/mp_synchronize.py b/Doc/includes/mp_synchronize.py index fd393f2888..81dbc387fc 100644 --- a/Doc/includes/mp_synchronize.py +++ b/Doc/includes/mp_synchronize.py @@ -5,7 +5,9 @@ # All rights reserved. # -import time, sys, random +import time +import sys +import random from queue import Empty import multiprocessing # may get overwritten @@ -237,9 +239,9 @@ def test(namespace=multiprocessing): multiprocessing = namespace - for func in [ test_value, test_queue, test_condition, - test_semaphore, test_join_timeout, test_event, - test_sharedvalues ]: + for func in [test_value, test_queue, test_condition, + test_semaphore, test_join_timeout, test_event, + test_sharedvalues]: print('\n\t######## %s\n' % func.__name__) func() diff --git a/Doc/includes/mp_webserver.py b/Doc/includes/mp_webserver.py index 0878de110d..651024d939 100644 --- a/Doc/includes/mp_webserver.py +++ b/Doc/includes/mp_webserver.py @@ -24,7 +24,7 @@ if sys.platform == 'win32': def note(format, *args): - sys.stderr.write('[%s]\t%s\n' % (current_process().name, format%args)) + sys.stderr.write('[%s]\t%s\n' % (current_process().name, format % args)) class RequestHandler(SimpleHTTPRequestHandler): @@ -45,7 +45,7 @@ def runpool(address, number_of_processes): server = HTTPServer(address, RequestHandler) # create child processes to act as workers - for i in range(number_of_processes-1): + for i in range(number_of_processes - 1): Process(target=serve_forever, args=(server,)).start() # main process also acts as a worker diff --git a/Doc/includes/sqlite3/adapter_datetime.py b/Doc/includes/sqlite3/adapter_datetime.py index 5869e22b32..be33395100 100644 --- a/Doc/includes/sqlite3/adapter_datetime.py +++ b/Doc/includes/sqlite3/adapter_datetime.py @@ -1,5 +1,6 @@ import sqlite3 -import datetime, time +import datetime +import time def adapt_datetime(ts): return time.mktime(ts.timetuple()) diff --git a/Doc/includes/sqlite3/converter_point.py b/Doc/includes/sqlite3/converter_point.py index a8861bcf2e..5df828e336 100644 --- a/Doc/includes/sqlite3/converter_point.py +++ b/Doc/includes/sqlite3/converter_point.py @@ -8,10 +8,10 @@ class Point: return "(%f;%f)" % (self.x, self.y) def adapt_point(point): - return "%f;%f" % (point.x, point.y) + return ("%f;%f" % (point.x, point.y)).encode('ascii') def convert_point(s): - x, y = list(map(float, s.split(";"))) + x, y = list(map(float, s.split(b";"))) return Point(x, y) # Register the adapter diff --git a/Doc/includes/sqlite3/ctx_manager.py b/Doc/includes/sqlite3/ctx_manager.py index b8e43325a9..7af4ad1ecf 100644 --- a/Doc/includes/sqlite3/ctx_manager.py +++ b/Doc/includes/sqlite3/ctx_manager.py @@ -8,7 +8,7 @@ with con: con.execute("insert into person(firstname) values (?)", ("Joe",)) # con.rollback() is called after the with block finishes with an exception, the -# exception is still raised and must be catched +# exception is still raised and must be caught try: with con: con.execute("insert into person(firstname) values (?)", ("Joe",)) diff --git a/Doc/includes/sqlite3/execute_1.py b/Doc/includes/sqlite3/execute_1.py index 3d08840b86..f864a8984e 100644 --- a/Doc/includes/sqlite3/execute_1.py +++ b/Doc/includes/sqlite3/execute_1.py @@ -1,11 +1,16 @@ import sqlite3 -con = sqlite3.connect("mydb") - +con = sqlite3.connect(":memory:") cur = con.cursor() +cur.execute("create table people (name_last, age)") who = "Yeltsin" age = 72 -cur.execute("select name_last, age from people where name_last=? and age=?", (who, age)) +# This is the qmark style: +cur.execute("insert into people values (?, ?)", (who, age)) + +# And this is the named style: +cur.execute("select * from people where name_last=:who and age=:age", {"who": who, "age": age}) + print(cur.fetchone()) diff --git a/Doc/includes/sqlite3/execute_2.py b/Doc/includes/sqlite3/execute_2.py deleted file mode 100644 index 84734f967f..0000000000 --- a/Doc/includes/sqlite3/execute_2.py +++ /dev/null @@ -1,12 +0,0 @@ -import sqlite3 - -con = sqlite3.connect("mydb") - -cur = con.cursor() - -who = "Yeltsin" -age = 72 - -cur.execute("select name_last, age from people where name_last=:who and age=:age", - {"who": who, "age": age}) -print(cur.fetchone()) diff --git a/Doc/includes/sqlite3/executemany_2.py b/Doc/includes/sqlite3/executemany_2.py index 518cd9430d..527358ebc2 100644 --- a/Doc/includes/sqlite3/executemany_2.py +++ b/Doc/includes/sqlite3/executemany_2.py @@ -1,8 +1,8 @@ import sqlite3 +import string def char_generator(): - import string - for c in string.letters[:26]: + for c in string.ascii_lowercase: yield (c,) con = sqlite3.connect(":memory:") diff --git a/Doc/includes/sqlite3/load_extension.py b/Doc/includes/sqlite3/load_extension.py new file mode 100644 index 0000000000..015aa0de8c --- /dev/null +++ b/Doc/includes/sqlite3/load_extension.py @@ -0,0 +1,26 @@ +import sqlite3 + +con = sqlite3.connect(":memory:") + +# enable extension loading +con.enable_load_extension(True) + +# Load the fulltext search extension +con.execute("select load_extension('./fts3.so')") + +# alternatively you can load the extension using an API call: +# con.load_extension("./fts3.so") + +# disable extension laoding again +con.enable_load_extension(False) + +# example from SQLite wiki +con.execute("create virtual table recipe using fts3(name, ingredients)") +con.executescript(""" + insert into recipe (name, ingredients) values ('broccoli stew', 'broccoli peppers cheese tomatoes'); + insert into recipe (name, ingredients) values ('pumpkin stew', 'pumpkin onions garlic celery'); + insert into recipe (name, ingredients) values ('broccoli pie', 'broccoli cheese onions flour'); + insert into recipe (name, ingredients) values ('pumpkin pie', 'pumpkin sugar flour butter'); + """) +for row in con.execute("select rowid, name, ingredients from recipe where name match 'pie'"): + print(row) diff --git a/Doc/includes/sqlite3/md5func.py b/Doc/includes/sqlite3/md5func.py index b7bc05b379..0056b2d6ce 100644 --- a/Doc/includes/sqlite3/md5func.py +++ b/Doc/includes/sqlite3/md5func.py @@ -7,5 +7,5 @@ def md5sum(t): con = sqlite3.connect(":memory:") con.create_function("md5", 1, md5sum) cur = con.cursor() -cur.execute("select md5(?)", ("foo",)) +cur.execute("select md5(?)", (b"foo",)) print(cur.fetchone()[0]) diff --git a/Doc/includes/sqlite3/rowclass.py b/Doc/includes/sqlite3/rowclass.py index 3fa0b87389..92b5ad60cb 100644 --- a/Doc/includes/sqlite3/rowclass.py +++ b/Doc/includes/sqlite3/rowclass.py @@ -1,12 +1,12 @@ import sqlite3 -con = sqlite3.connect("mydb") +con = sqlite3.connect(":memory:") con.row_factory = sqlite3.Row cur = con.cursor() -cur.execute("select name_last, age from people") +cur.execute("select 'John' as name, 42 as age") for row in cur: - assert row[0] == row["name_last"] - assert row["name_last"] == row["nAmE_lAsT"] + assert row[0] == row["name"] + assert row["name"] == row["nAmE"] assert row[1] == row["age"] assert row[1] == row["AgE"] diff --git a/Doc/includes/sqlite3/shortcut_methods.py b/Doc/includes/sqlite3/shortcut_methods.py index 596d87c18e..71600d4f60 100644 --- a/Doc/includes/sqlite3/shortcut_methods.py +++ b/Doc/includes/sqlite3/shortcut_methods.py @@ -17,5 +17,4 @@ con.executemany("insert into person(firstname, lastname) values (?, ?)", persons for row in con.execute("select firstname, lastname from person"): print(row) -# Using a dummy WHERE clause to not let SQLite take the shortcut table deletes. -print("I just deleted", con.execute("delete from person where 1=1").rowcount, "rows") +print("I just deleted", con.execute("delete from person").rowcount, "rows") diff --git a/Doc/includes/sqlite3/text_factory.py b/Doc/includes/sqlite3/text_factory.py index 22c2970086..5f96cdb58d 100644 --- a/Doc/includes/sqlite3/text_factory.py +++ b/Doc/includes/sqlite3/text_factory.py @@ -3,9 +3,6 @@ import sqlite3 con = sqlite3.connect(":memory:") cur = con.cursor() -# Create the table -con.execute("create table person(lastname, firstname)") - AUSTRIA = "\xd6sterreich" # by default, rows are returned as Unicode @@ -14,30 +11,17 @@ row = cur.fetchone() assert row[0] == AUSTRIA # but we can make sqlite3 always return bytestrings ... -con.text_factory = str +con.text_factory = bytes cur.execute("select ?", (AUSTRIA,)) row = cur.fetchone() -assert type(row[0]) == str +assert type(row[0]) is bytes # the bytestrings will be encoded in UTF-8, unless you stored garbage in the # database ... assert row[0] == AUSTRIA.encode("utf-8") # we can also implement a custom text_factory ... -# here we implement one that will ignore Unicode characters that cannot be -# decoded from UTF-8 -con.text_factory = lambda x: str(x, "utf-8", "ignore") -cur.execute("select ?", ("this is latin1 and would normally create errors" + - "\xe4\xf6\xfc".encode("latin1"),)) -row = cur.fetchone() -assert type(row[0]) == str - -# sqlite3 offers a built-in optimized text_factory that will return bytestring -# objects, if the data is in ASCII only, and otherwise return unicode objects -con.text_factory = sqlite3.OptimizedUnicode -cur.execute("select ?", (AUSTRIA,)) -row = cur.fetchone() -assert type(row[0]) == str - -cur.execute("select ?", ("Germany",)) +# here we implement one that appends "foo" to all strings +con.text_factory = lambda x: x.decode("utf-8") + "foo" +cur.execute("select ?", ("bar",)) row = cur.fetchone() -assert type(row[0]) == str +assert row[0] == "barfoo" diff --git a/Doc/includes/turtle-star.py b/Doc/includes/turtle-star.py new file mode 100644 index 0000000000..1a5db761b3 --- /dev/null +++ b/Doc/includes/turtle-star.py @@ -0,0 +1,10 @@ +from turtle import * +color('red', 'yellow') +begin_fill() +while True: + forward(200) + left(170) + if abs(pos()) < 1: + break +end_fill() +done() diff --git a/Doc/includes/tzinfo-examples.py b/Doc/includes/tzinfo-examples.py index 5132429fb4..3a8cf47eaf 100644 --- a/Doc/includes/tzinfo-examples.py +++ b/Doc/includes/tzinfo-examples.py @@ -27,7 +27,7 @@ class FixedOffset(tzinfo): """Fixed offset in minutes east from UTC.""" def __init__(self, offset, name): - self.__offset = timedelta(minutes = offset) + self.__offset = timedelta(minutes=offset) self.__name = name def utcoffset(self, dt): diff --git a/Doc/install/index.rst b/Doc/install/index.rst index 6a11c5accc..b20f1fbd17 100644 --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -72,7 +72,7 @@ In that case, you would download the installer appropriate to your platform and do the obvious thing with it: run it if it's an executable installer, ``rpm --install`` it if it's an RPM, etc. You don't need to run Python or a setup script, you don't need to compile anything---you might not even need to read any -instructions (although it's always a good idea to do so anyways). +instructions (although it's always a good idea to do so anyway). Of course, things will not always be that easy. You might be interested in a module distribution that doesn't have an easy-to-use installer for your @@ -96,10 +96,16 @@ in the name of the downloaded archive, e.g. :file:`foo-1.0.tar.gz` or directory: :file:`foo-1.0` or :file:`widget-0.9.7`. Additionally, the distribution will contain a setup script :file:`setup.py`, and a file named :file:`README.txt` or possibly just :file:`README`, which should explain that -building and installing the module distribution is a simple matter of running :: +building and installing the module distribution is a simple matter of running +one command from a terminal:: python setup.py install +For Windows, this command should be run from a command prompt window +(:menuselection:`Start --> Accessories`):: + + setup.py install + If all these things are true, then you already know how to build and install the modules you've just downloaded: Run the command above. Unless you need to install things in a non-standard way or customize the build process, you don't @@ -113,14 +119,11 @@ Standard Build and Install ========================== As described in section :ref:`inst-new-standard`, building and installing a module -distribution using the Distutils is usually one simple command:: +distribution using the Distutils is usually one simple command to run from a +terminal:: python setup.py install -On Unix, you'd run this command from a shell prompt; on Windows, you have to -open a command prompt window ("DOS box") and do it there; on Mac OS X, you open -a :command:`Terminal` window to get a shell prompt. - .. _inst-platform-variations: @@ -141,7 +144,7 @@ archive file to :file:`C:\\Temp`, then it would unpack into :file:`C:\\Temp\\foo-1.0`; you can use either a archive manipulator with a graphical user interface (such as WinZip) or a command-line tool (such as :program:`unzip` or :program:`pkunzip`) to unpack the archive. Then, open a -command prompt window ("DOS box"), and run:: +command prompt window and run:: cd c:\Temp\foo-1.0 python setup.py install @@ -276,6 +279,14 @@ statements shown below, and get the output as shown, to find out my >>> sys.exec_prefix '/usr' +A few other placeholders are used in this document: :file:`{X.Y}` stands for the +version of Python, for example ``3.2``; :file:`{abiflags}` will be replaced by +the value of :data:`sys.abiflags` or the empty string for platforms which don't +define ABI flags; :file:`{distname}` will be replaced by the name of the module +distribution being installed. Dots and capitalization are important in the +paths; for example, a value that uses ``python3.2`` on UNIX will typically use +``Python32`` on Windows. + If you don't want to install modules to the standard location, or if you don't have permission to write there, then you need to read about alternate installations in section :ref:`inst-alt-install`. If you want to customize your @@ -304,8 +315,61 @@ scheme*) under this base directory in which to install files. The details differ across platforms, so read whichever of the following sections applies to you. +Note that the various alternate installation schemes are mutually exclusive: you +can pass ``--user``, or ``--home``, or ``--prefix`` and ``--exec-prefix``, or +``--install-base`` and ``--install-platbase``, but you can't mix from these +groups. + + +.. _inst-alt-install-user: + +Alternate installation: the user scheme +--------------------------------------- + +This scheme is designed to be the most convenient solution for users that don't +have write permission to the global site-packages directory or don't want to +install into it. It is enabled with a simple option:: + + python setup.py install --user + +Files will be installed into subdirectories of :data:`site.USER_BASE` (written +as :file:`{userbase}` hereafter). This scheme installs pure Python modules and +extension modules in the same location (also known as :data:`site.USER_SITE`). +Here are the values for UNIX, including Mac OS X: + +=============== =========================================================== +Type of file Installation directory +=============== =========================================================== +modules :file:`{userbase}/lib/python{X.Y}/site-packages` +scripts :file:`{userbase}/bin` +data :file:`{userbase}` +C headers :file:`{userbase}/include/python{X.Y}{abiflags}/{distname}` +=============== =========================================================== + +And here are the values used on Windows: + +=============== =========================================================== +Type of file Installation directory +=============== =========================================================== +modules :file:`{userbase}\\Python{XY}\\site-packages` +scripts :file:`{userbase}\\Scripts` +data :file:`{userbase}` +C headers :file:`{userbase}\\Python{XY}\\Include\\{distname}` +=============== =========================================================== + +The advantage of using this scheme compared to the other ones described below is +that the user site-packages directory is under normal conditions always included +in :data:`sys.path` (see :mod:`site` for more information), which means that +there is no additional step to perform after running the :file:`setup.py` script +to finalize the installation. + +The :command:`build_ext` command also has a ``--user`` option to add +:file:`{userbase}/include` to the compiler search path for header files and +:file:`{userbase}/lib` to the compiler search path for libraries as well as to +the runtime search path for shared C libraries (rpath). + -.. _inst-alt-install-prefix: +.. _inst-alt-install-home: Alternate installation: the home scheme --------------------------------------- @@ -327,23 +391,27 @@ will expand this to your home directory:: python setup.py install --home=~ +To make Python find the distributions installed with this scheme, you may have +to :ref:`modify Python's search path <inst-search-path>` or edit +:mod:`sitecustomize` (see :mod:`site`) to call :func:`site.addsitedir` or edit +:data:`sys.path`. + The :option:`--home` option defines the installation base directory. Files are installed to the following directories under the installation base as follows: -+------------------------------+---------------------------+-----------------------------+ -| Type of file | Installation Directory | Override option | -+==============================+===========================+=============================+ -| pure module distribution | :file:`{home}/lib/python` | :option:`--install-purelib` | -+------------------------------+---------------------------+-----------------------------+ -| non-pure module distribution | :file:`{home}/lib/python` | :option:`--install-platlib` | -+------------------------------+---------------------------+-----------------------------+ -| scripts | :file:`{home}/bin` | :option:`--install-scripts` | -+------------------------------+---------------------------+-----------------------------+ -| data | :file:`{home}/share` | :option:`--install-data` | -+------------------------------+---------------------------+-----------------------------+ +=============== =========================================================== +Type of file Installation directory +=============== =========================================================== +modules :file:`{home}/lib/python` +scripts :file:`{home}/bin` +data :file:`{home}` +C headers :file:`{home}/include/python/{distname}` +=============== =========================================================== +(Mentally replace slashes with backslashes if you're on Windows.) -.. _inst-alt-install-home: + +.. _inst-alt-install-prefix-unix: Alternate installation: Unix (the prefix scheme) ------------------------------------------------ @@ -352,7 +420,7 @@ The "prefix scheme" is useful when you wish to use one Python installation to perform the build/install (i.e., to run the setup script), but install modules into the third-party module directory of a different Python installation (or something that looks like a different Python installation). If this sounds a -trifle unusual, it is---that's why the "home scheme" comes first. However, +trifle unusual, it is---that's why the user and home schemes come before. However, there are at least two known cases where the prefix scheme will be useful. First, consider that many Linux distributions put Python in :file:`/usr`, rather @@ -380,17 +448,15 @@ non-pure module distributions, but could be expanded to C libraries, binary executables, etc.) If :option:`--exec-prefix` is not supplied, it defaults to :option:`--prefix`. Files are installed as follows: -+------------------------------+-----------------------------------------------------+-----------------------------+ -| Type of file | Installation Directory | Override option | -+==============================+=====================================================+=============================+ -| pure module distribution | :file:`{prefix}/lib/python{X.Y}/site-packages` | :option:`--install-purelib` | -+------------------------------+-----------------------------------------------------+-----------------------------+ -| non-pure module distribution | :file:`{exec-prefix}/lib/python{X.Y}/site-packages` | :option:`--install-platlib` | -+------------------------------+-----------------------------------------------------+-----------------------------+ -| scripts | :file:`{prefix}/bin` | :option:`--install-scripts` | -+------------------------------+-----------------------------------------------------+-----------------------------+ -| data | :file:`{prefix}/share` | :option:`--install-data` | -+------------------------------+-----------------------------------------------------+-----------------------------+ +================= ========================================================== +Type of file Installation directory +================= ========================================================== +Python modules :file:`{prefix}/lib/python{X.Y}/site-packages` +extension modules :file:`{exec-prefix}/lib/python{X.Y}/site-packages` +scripts :file:`{prefix}/bin` +data :file:`{prefix}` +C headers :file:`{prefix}/include/python{X.Y}{abiflags}/{distname}` +================= ========================================================== There is no requirement that :option:`--prefix` or :option:`--exec-prefix` actually point to an alternate Python installation; if the directories listed @@ -415,7 +481,7 @@ if your :option:`--prefix` and :option:`--exec-prefix` don't even point to an alternate Python installation, this is immaterial.) -.. _inst-alt-install-windows: +.. _inst-alt-install-prefix-windows: Alternate installation: Windows (the prefix scheme) --------------------------------------------------- @@ -430,20 +496,18 @@ locations on Windows. :: to install modules to the :file:`\\Temp\\Python` directory on the current drive. The installation base is defined by the :option:`--prefix` option; the -:option:`--exec-prefix` option is not supported under Windows. Files are -installed as follows: - -+------------------------------+---------------------------+-----------------------------+ -| Type of file | Installation Directory | Override option | -+==============================+===========================+=============================+ -| pure module distribution | :file:`{prefix}` | :option:`--install-purelib` | -+------------------------------+---------------------------+-----------------------------+ -| non-pure module distribution | :file:`{prefix}` | :option:`--install-platlib` | -+------------------------------+---------------------------+-----------------------------+ -| scripts | :file:`{prefix}\\Scripts` | :option:`--install-scripts` | -+------------------------------+---------------------------+-----------------------------+ -| data | :file:`{prefix}\\Data` | :option:`--install-data` | -+------------------------------+---------------------------+-----------------------------+ +:option:`--exec-prefix` option is not supported under Windows, which means that +pure Python modules and extension modules are installed into the same location. +Files are installed as follows: + +=============== ========================================================== +Type of file Installation directory +=============== ========================================================== +modules :file:`{prefix}\\Lib\\site-packages` +scripts :file:`{prefix}\\Scripts` +data :file:`{prefix}` +C headers :file:`{prefix}\\Include\\{distname}` +=============== ========================================================== .. _inst-custom-install: @@ -457,13 +521,29 @@ one or two directories while keeping everything under the same base directory, or you might want to completely redefine the installation scheme. In either case, you're creating a *custom installation scheme*. -You probably noticed the column of "override options" in the tables describing -the alternate installation schemes above. Those options are how you define a -custom installation scheme. These override options can be relative, absolute, +To create a custom installation scheme, you start with one of the alternate +schemes and override some of the installation directories used for the various +types of files, using these options: + +====================== ======================= +Type of file Override option +====================== ======================= +Python modules ``--install-purelib`` +extension modules ``--install-platlib`` +all modules ``--install-lib`` +scripts ``--install-scripts`` +data ``--install-data`` +C headers ``--install-headers`` +====================== ======================= + +These override options can be relative, absolute, or explicitly defined in terms of one of the installation base directories. (There are two installation base directories, and they are normally the same--- they only differ when you use the Unix "prefix scheme" and supply different -:option:`--prefix` and :option:`--exec-prefix` options.) +``--prefix`` and ``--exec-prefix`` options; using ``--install-lib`` will +override values computed or given for ``--install-purelib`` and +``--install-platlib``, and is recommended for schemes that don't make a +difference between Python and extension modules.) For example, say you're installing a module distribution to your home directory under Unix---but you want scripts to go in :file:`~/scripts` rather than @@ -490,15 +570,16 @@ If you maintain Python on Windows, you might want third-party modules to live in a subdirectory of :file:`{prefix}`, rather than right in :file:`{prefix}` itself. This is almost as easy as customizing the script installation directory ---you just have to remember that there are two types of modules to worry about, -pure modules and non-pure modules (i.e., modules from a non-pure distribution). -For example:: +Python and extension modules, which can conveniently be both controlled by one +option:: - python setup.py install --install-purelib=Site --install-platlib=Site + python setup.py install --install-lib=Site -The specified installation directories are relative to :file:`{prefix}`. Of -course, you also have to ensure that these directories are in Python's module -search path, such as by putting a :file:`.pth` file in :file:`{prefix}`. See -section :ref:`inst-search-path` to find out how to modify Python's search path. +The specified installation directory is relative to :file:`{prefix}`. Of +course, you also have to ensure that this directory is in Python's module +search path, such as by putting a :file:`.pth` file in a site directory (see +:mod:`site`). See section :ref:`inst-search-path` to find out how to modify +Python's search path. If you want to define an entire installation scheme, you just have to supply all of the installation directory options. The recommended way to do this is to @@ -550,8 +631,8 @@ base directory when you run the setup script. For example, :: python setup.py install --install-base=/tmp -would install pure modules to :file:`{/tmp/python/lib}` in the first case, and -to :file:`{/tmp/lib}` in the second case. (For the second case, you probably +would install pure modules to :file:`/tmp/python/lib` in the first case, and +to :file:`/tmp/lib` in the second case. (For the second case, you probably want to supply an installation base of :file:`/tmp/python`.) You probably noticed the use of ``$HOME`` and ``$PLAT`` in the sample @@ -568,7 +649,7 @@ for details. needed on those platforms? -.. XXX I'm not sure where this section should go. +.. XXX Move this to Doc/using .. _inst-search-path: @@ -691,6 +772,9 @@ And on Windows, the configuration files are: | local | :file:`setup.cfg` | \(3) | +--------------+-------------------------------------------------+-------+ +On all platforms, the "personal" file can be temporarily disabled by +passing the `--no-user-cfg` option. + Notes: (1) diff --git a/Doc/library/2to3.rst b/Doc/library/2to3.rst index de31251cec..a13fb65024 100644 --- a/Doc/library/2to3.rst +++ b/Doc/library/2to3.rst @@ -94,6 +94,38 @@ change can also be enabled manually with the :option:`-p` flag. Use :option:`-p` to run fixers on code that already has had its print statements converted. +The :option:`-o` or :option:`--output-dir` option allows specification of an +alternate directory for processed output files to be written to. The +:option:`-n` flag is required when using this as backup files do not make sense +when not overwriting the input files. + +.. versionadded:: 3.2.3 + The :option:`-o` option was added. + +The :option:`-W` or :option:`--write-unchanged-files` flag tells 2to3 to always +write output files even if no changes were required to the file. This is most +useful with :option:`-o` so that an entire Python source tree is copied with +translation from one directory to another. +This option implies the :option:`-w` flag as it would not make sense otherwise. + +.. versionadded:: 3.2.3 + The :option:`-W` flag was added. + +The :option:`--add-suffix` option specifies a string to append to all output +filenames. The :option:`-n` flag is required when specifying this as backups +are not necessary when writing to different filenames. Example:: + + $ 2to3 -n -W --add-suffix=3 example.py + +Will cause a converted file named ``example.py3`` to be written. + +.. versionadded:: 3.2.3 + The :option:`--add-suffix` option was added. + +To translate an entire project from one directory tree to another use:: + + $ 2to3 --output-dir=python3-version/mycode -W -n python2-version/mycode + .. _2to3-fixers: @@ -123,7 +155,9 @@ and off individually. They are described here in more detail. .. 2to3fixer:: callable Converts ``callable(x)`` to ``isinstance(x, collections.Callable)``, adding - an import to :mod:`collections` if needed. + an import to :mod:`collections` if needed. Note ``callable(x)`` has returned + in Python 3.2, so if you do not intend to support Python 3.1, you can disable + this fixer. .. 2to3fixer:: dict @@ -141,7 +175,7 @@ and off individually. They are described here in more detail. .. 2to3fixer:: exec - Converts the :keyword:`exec` statement to the :func:`exec` function. + Converts the ``exec`` statement to the :func:`exec` function. .. 2to3fixer:: execfile @@ -267,6 +301,25 @@ and off individually. They are described here in more detail. Converts octal literals into the new syntax. +.. 2to3fixer:: operator + + Converts calls to various functions in the :mod:`operator` module to other, + but equivalent, function calls. When needed, the appropriate ``import`` + statements are added, e.g. ``import collections``. The following mapping + are made: + + ================================== ========================================== + From To + ================================== ========================================== + ``operator.isCallable(obj)`` ``hasattr(obj, '__call__')`` + ``operator.sequenceIncludes(obj)`` ``operator.contains(obj)`` + ``operator.isSequenceType(obj)`` ``isinstance(obj, collections.Sequence)`` + ``operator.isMappingType(obj)`` ``isinstance(obj, collections.Mapping)`` + ``operator.isNumberType(obj)`` ``isinstance(obj, numbers.Number)`` + ``operator.repeat(obj, n)`` ``operator.mul(obj, n)`` + ``operator.irepeat(obj, n)`` ``operator.imul(obj, n)`` + ================================== ========================================== + .. 2to3fixer:: paren Add extra parenthesis where they are required in list comprehensions. For @@ -274,7 +327,7 @@ and off individually. They are described here in more detail. .. 2to3fixer:: print - Converts the :keyword:`print` statement to the :func:`print` function. + Converts the ``print`` statement to the :func:`print` function. .. 2to3fixer:: raise diff --git a/Doc/library/__future__.rst b/Doc/library/__future__.rst index 2348217f3d..0acccc591b 100644 --- a/Doc/library/__future__.rst +++ b/Doc/library/__future__.rst @@ -4,6 +4,9 @@ .. module:: __future__ :synopsis: Future statement definitions +**Source code:** :source:`Lib/__future__.py` + +-------------- :mod:`__future__` is a real module, and serves three purposes: diff --git a/Doc/library/_dummy_thread.rst b/Doc/library/_dummy_thread.rst index 62e57080a7..83aec12bbb 100644 --- a/Doc/library/_dummy_thread.rst +++ b/Doc/library/_dummy_thread.rst @@ -4,6 +4,9 @@ .. module:: _dummy_thread :synopsis: Drop-in replacement for the _thread module. +**Source code:** :source:`Lib/_dummy_thread.py` + +-------------- This module provides a duplicate interface to the :mod:`_thread` module. It is meant to be imported when the :mod:`_thread` module is not provided on a diff --git a/Doc/library/_thread.rst b/Doc/library/_thread.rst index cb624078ca..369e9cd01e 100644 --- a/Doc/library/_thread.rst +++ b/Doc/library/_thread.rst @@ -28,7 +28,7 @@ implementation. For systems lacking the :mod:`_thread` module, the :mod:`_dummy_thread` module is available. It duplicates this module's interface and can be used as a drop-in replacement. -It defines the following constant and functions: +It defines the following constants and functions: .. exception:: error @@ -103,18 +103,42 @@ It defines the following constant and functions: Availability: Windows, systems with POSIX threads. +.. data:: TIMEOUT_MAX + + The maximum value allowed for the *timeout* parameter of + :meth:`Lock.acquire`. Specifying a timeout greater than this value will + raise an :exc:`OverflowError`. + + .. versionadded:: 3.2 + + Lock objects have the following methods: -.. method:: lock.acquire([waitflag]) +.. method:: lock.acquire(waitflag=1, timeout=-1) - Without the optional argument, this method acquires the lock unconditionally, if + Without any optional argument, this method acquires the lock unconditionally, if necessary waiting until it is released by another thread (only one thread at a - time can acquire a lock --- that's their reason for existence). If the integer - *waitflag* argument is present, the action depends on its value: if it is zero, - the lock is only acquired if it can be acquired immediately without waiting, - while if it is nonzero, the lock is acquired unconditionally as before. The - return value is ``True`` if the lock is acquired successfully, ``False`` if not. + time can acquire a lock --- that's their reason for existence). + + If the integer *waitflag* argument is present, the action depends on its + value: if it is zero, the lock is only acquired if it can be acquired + immediately without waiting, while if it is nonzero, the lock is acquired + unconditionally as above. + + If the floating-point *timeout* argument is present and positive, it + specifies the maximum wait time in seconds before returning. A negative + *timeout* argument specifies an unbounded wait. You cannot specify + a *timeout* if *waitflag* is zero. + + The return value is ``True`` if the lock is acquired successfully, + ``False`` if not. + + .. versionchanged:: 3.2 + The *timeout* parameter is new. + + .. versionchanged:: 3.2 + Lock acquires can now be interrupted by signals on POSIX. .. method:: lock.release() @@ -156,12 +180,10 @@ In addition to these methods, lock objects can also be used via the * It is not possible to interrupt the :meth:`acquire` method on a lock --- the :exc:`KeyboardInterrupt` exception will happen after the lock has been acquired. - .. index:: pair: threads; IRIX - * When the main thread exits, it is system defined whether the other threads - survive. On SGI IRIX using the native thread implementation, they survive. On - most other systems, they are killed without executing :keyword:`try` ... - :keyword:`finally` clauses or executing object destructors. + survive. On most systems, they are killed without executing + :keyword:`try` ... :keyword:`finally` clauses or executing object + destructors. * When the main thread exits, it does not do any of its usual cleanup (except that :keyword:`try` ... :keyword:`finally` clauses are honored), and the diff --git a/Doc/library/abc.rst b/Doc/library/abc.rst index aa1cc7871e..1048b24c08 100644 --- a/Doc/library/abc.rst +++ b/Doc/library/abc.rst @@ -7,8 +7,12 @@ .. sectionauthor:: Georg Brandl .. much of the content adapted from docstrings -This module provides the infrastructure for defining an :term:`abstract base -class` (ABCs) in Python, as outlined in :pep:`3119`; see the PEP for why this +**Source code:** :source:`Lib/abc.py` + +-------------- + +This module provides the infrastructure for defining :term:`abstract base +classes <abstract base class>` (ABCs) in Python, as outlined in :pep:`3119`; see the PEP for why this was added to Python. (See also :pep:`3141` and the :mod:`numbers` module regarding a type hierarchy for numbers based on ABCs.) @@ -122,7 +126,7 @@ This module provides the following class: It also provides the following decorators: -.. function:: abstractmethod(function) +.. decorator:: abstractmethod(function) A decorator indicating abstract methods. @@ -157,6 +161,36 @@ It also provides the following decorators: multiple-inheritance. +.. decorator:: abstractclassmethod(function) + + A subclass of the built-in :func:`classmethod`, indicating an abstract + classmethod. Otherwise it is similar to :func:`abstractmethod`. + + Usage:: + + class C(metaclass=ABCMeta): + @abstractclassmethod + def my_abstract_classmethod(cls, ...): + ... + + .. versionadded:: 3.2 + + +.. decorator:: abstractstaticmethod(function) + + A subclass of the built-in :func:`staticmethod`, indicating an abstract + staticmethod. Otherwise it is similar to :func:`abstractmethod`. + + Usage:: + + class C(metaclass=ABCMeta): + @abstractstaticmethod + def my_abstract_staticmethod(...): + ... + + .. versionadded:: 3.2 + + .. function:: abstractproperty(fget=None, fset=None, fdel=None, doc=None) A subclass of the built-in :func:`property`, indicating an abstract property. diff --git a/Doc/library/aifc.rst b/Doc/library/aifc.rst index 304437d9a2..999bad83ca 100644 --- a/Doc/library/aifc.rst +++ b/Doc/library/aifc.rst @@ -10,6 +10,10 @@ single: AIFF single: AIFF-C +**Source code:** :source:`Lib/aifc.py` + +-------------- + This module provides support for reading and writing AIFF and AIFF-C files. AIFF is Audio Interchange File Format, a format for storing digital audio samples in a file. AIFF-C is a newer version of the format that includes the diff --git a/Doc/library/allos.rst b/Doc/library/allos.rst index f25c3b8751..bf91717447 100644 --- a/Doc/library/allos.rst +++ b/Doc/library/allos.rst @@ -15,9 +15,12 @@ but they are available on most other systems as well. Here's an overview: os.rst io.rst time.rst + argparse.rst optparse.rst getopt.rst logging.rst + logging.config.rst + logging.handlers.rst getpass.rst curses.rst curses.ascii.rst diff --git a/Doc/library/archiving.rst b/Doc/library/archiving.rst index f2d08ada0c..75d137c9a8 100644 --- a/Doc/library/archiving.rst +++ b/Doc/library/archiving.rst @@ -6,6 +6,7 @@ Data Compression and Archiving The modules described in this chapter support data compression with the zlib, gzip, and bzip2 algorithms, and the creation of ZIP- and tar-format archives. +See also :ref:`archiving-operations` provided by the :mod:`shutil` module. .. toctree:: diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst new file mode 100644 index 0000000000..79a98cbcfd --- /dev/null +++ b/Doc/library/argparse.rst @@ -0,0 +1,1855 @@ +:mod:`argparse` --- Parser for command-line options, arguments and sub-commands +=============================================================================== + +.. module:: argparse + :synopsis: Command-line option and argument parsing library. +.. moduleauthor:: Steven Bethard <steven.bethard@gmail.com> +.. sectionauthor:: Steven Bethard <steven.bethard@gmail.com> + +.. versionadded:: 3.2 + +**Source code:** :source:`Lib/argparse.py` + +-------------- + +The :mod:`argparse` module makes it easy to write user-friendly command-line +interfaces. The program defines what arguments it requires, and :mod:`argparse` +will figure out how to parse those out of :data:`sys.argv`. The :mod:`argparse` +module also automatically generates help and usage messages and issues errors +when users give the program invalid arguments. + + +Example +------- + +The following code is a Python program that takes a list of integers and +produces either the sum or the max:: + + import argparse + + parser = argparse.ArgumentParser(description='Process some integers.') + parser.add_argument('integers', metavar='N', type=int, nargs='+', + help='an integer for the accumulator') + parser.add_argument('--sum', dest='accumulate', action='store_const', + const=sum, default=max, + help='sum the integers (default: find the max)') + + args = parser.parse_args() + print(args.accumulate(args.integers)) + +Assuming the Python code above is saved into a file called ``prog.py``, it can +be run at the command line and provides useful help messages:: + + $ prog.py -h + usage: prog.py [-h] [--sum] N [N ...] + + Process some integers. + + positional arguments: + N an integer for the accumulator + + optional arguments: + -h, --help show this help message and exit + --sum sum the integers (default: find the max) + +When run with the appropriate arguments, it prints either the sum or the max of +the command-line integers:: + + $ prog.py 1 2 3 4 + 4 + + $ prog.py 1 2 3 4 --sum + 10 + +If invalid arguments are passed in, it will issue an error:: + + $ prog.py a b c + usage: prog.py [-h] [--sum] N [N ...] + prog.py: error: argument N: invalid int value: 'a' + +The following sections walk you through this example. + + +Creating a parser +^^^^^^^^^^^^^^^^^ + +The first step in using the :mod:`argparse` is creating an +:class:`ArgumentParser` object:: + + >>> parser = argparse.ArgumentParser(description='Process some integers.') + +The :class:`ArgumentParser` object will hold all the information necessary to +parse the command line into Python data types. + + +Adding arguments +^^^^^^^^^^^^^^^^ + +Filling an :class:`ArgumentParser` with information about program arguments is +done by making calls to the :meth:`~ArgumentParser.add_argument` method. +Generally, these calls tell the :class:`ArgumentParser` how to take the strings +on the command line and turn them into objects. This information is stored and +used when :meth:`~ArgumentParser.parse_args` is called. For example:: + + >>> parser.add_argument('integers', metavar='N', type=int, nargs='+', + ... help='an integer for the accumulator') + >>> parser.add_argument('--sum', dest='accumulate', action='store_const', + ... const=sum, default=max, + ... help='sum the integers (default: find the max)') + +Later, calling :meth:`~ArgumentParser.parse_args` will return an object with +two attributes, ``integers`` and ``accumulate``. The ``integers`` attribute +will be a list of one or more ints, and the ``accumulate`` attribute will be +either the :func:`sum` function, if ``--sum`` was specified at the command line, +or the :func:`max` function if it was not. + + +Parsing arguments +^^^^^^^^^^^^^^^^^ + +:class:`ArgumentParser` parses arguments through the +:meth:`~ArgumentParser.parse_args` method. This will inspect the command line, +convert each argument to the appropriate type and then invoke the appropriate action. +In most cases, this means a simple :class:`Namespace` object will be built up from +attributes parsed out of the command line:: + + >>> parser.parse_args(['--sum', '7', '-1', '42']) + Namespace(accumulate=<built-in function sum>, integers=[7, -1, 42]) + +In a script, :meth:`~ArgumentParser.parse_args` will typically be called with no +arguments, and the :class:`ArgumentParser` will automatically determine the +command-line arguments from :data:`sys.argv`. + + +ArgumentParser objects +---------------------- + +.. class:: ArgumentParser([description], [epilog], [prog], [usage], [add_help], \ + [argument_default], [parents], [prefix_chars], \ + [conflict_handler], [formatter_class]) + + Create a new :class:`ArgumentParser` object. Each parameter has its own more + detailed description below, but in short they are: + + * description_ - Text to display before the argument help. + + * epilog_ - Text to display after the argument help. + + * add_help_ - Add a -h/--help option to the parser. (default: ``True``) + + * argument_default_ - Set the global default value for arguments. + (default: ``None``) + + * parents_ - A list of :class:`ArgumentParser` objects whose arguments should + also be included. + + * prefix_chars_ - The set of characters that prefix optional arguments. + (default: '-') + + * fromfile_prefix_chars_ - The set of characters that prefix files from + which additional arguments should be read. (default: ``None``) + + * formatter_class_ - A class for customizing the help output. + + * conflict_handler_ - Usually unnecessary, defines strategy for resolving + conflicting optionals. + + * prog_ - The name of the program (default: + ``sys.argv[0]``) + + * usage_ - The string describing the program usage (default: generated) + +The following sections describe how each of these are used. + + +description +^^^^^^^^^^^ + +Most calls to the :class:`ArgumentParser` constructor will use the +``description=`` keyword argument. This argument gives a brief description of +what the program does and how it works. In help messages, the description is +displayed between the command-line usage string and the help messages for the +various arguments:: + + >>> parser = argparse.ArgumentParser(description='A foo that bars') + >>> parser.print_help() + usage: argparse.py [-h] + + A foo that bars + + optional arguments: + -h, --help show this help message and exit + +By default, the description will be line-wrapped so that it fits within the +given space. To change this behavior, see the formatter_class_ argument. + + +epilog +^^^^^^ + +Some programs like to display additional description of the program after the +description of the arguments. Such text can be specified using the ``epilog=`` +argument to :class:`ArgumentParser`:: + + >>> parser = argparse.ArgumentParser( + ... description='A foo that bars', + ... epilog="And that's how you'd foo a bar") + >>> parser.print_help() + usage: argparse.py [-h] + + A foo that bars + + optional arguments: + -h, --help show this help message and exit + + And that's how you'd foo a bar + +As with the description_ argument, the ``epilog=`` text is by default +line-wrapped, but this behavior can be adjusted with the formatter_class_ +argument to :class:`ArgumentParser`. + + +add_help +^^^^^^^^ + +By default, ArgumentParser objects add an option which simply displays +the parser's help message. For example, consider a file named +``myprogram.py`` containing the following code:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument('--foo', help='foo help') + args = parser.parse_args() + +If ``-h`` or ``--help`` is supplied at the command line, the ArgumentParser +help will be printed:: + + $ python myprogram.py --help + usage: myprogram.py [-h] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + --foo FOO foo help + +Occasionally, it may be useful to disable the addition of this help option. +This can be achieved by passing ``False`` as the ``add_help=`` argument to +:class:`ArgumentParser`:: + + >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False) + >>> parser.add_argument('--foo', help='foo help') + >>> parser.print_help() + usage: PROG [--foo FOO] + + optional arguments: + --foo FOO foo help + +The help option is typically ``-h/--help``. The exception to this is +if the ``prefix_chars=`` is specified and does not include ``-``, in +which case ``-h`` and ``--help`` are not valid options. In +this case, the first character in ``prefix_chars`` is used to prefix +the help options:: + + >>> parser = argparse.ArgumentParser(prog='PROG', prefix_chars='+/') + >>> parser.print_help() + usage: PROG [+h] + + optional arguments: + +h, ++help show this help message and exit + + +prefix_chars +^^^^^^^^^^^^ + +Most command-line options will use ``-`` as the prefix, e.g. ``-f/--foo``. +Parsers that need to support different or additional prefix +characters, e.g. for options +like ``+f`` or ``/foo``, may specify them using the ``prefix_chars=`` argument +to the ArgumentParser constructor:: + + >>> parser = argparse.ArgumentParser(prog='PROG', prefix_chars='-+') + >>> parser.add_argument('+f') + >>> parser.add_argument('++bar') + >>> parser.parse_args('+f X ++bar Y'.split()) + Namespace(bar='Y', f='X') + +The ``prefix_chars=`` argument defaults to ``'-'``. Supplying a set of +characters that does not include ``-`` will cause ``-f/--foo`` options to be +disallowed. + + +fromfile_prefix_chars +^^^^^^^^^^^^^^^^^^^^^ + +Sometimes, for example when dealing with a particularly long argument lists, it +may make sense to keep the list of arguments in a file rather than typing it out +at the command line. If the ``fromfile_prefix_chars=`` argument is given to the +:class:`ArgumentParser` constructor, then arguments that start with any of the +specified characters will be treated as files, and will be replaced by the +arguments they contain. For example:: + + >>> with open('args.txt', 'w') as fp: + ... fp.write('-f\nbar') + >>> parser = argparse.ArgumentParser(fromfile_prefix_chars='@') + >>> parser.add_argument('-f') + >>> parser.parse_args(['-f', 'foo', '@args.txt']) + Namespace(f='bar') + +Arguments read from a file must by default be one per line (but see also +:meth:`~ArgumentParser.convert_arg_line_to_args`) and are treated as if they +were in the same place as the original file referencing argument on the command +line. So in the example above, the expression ``['-f', 'foo', '@args.txt']`` +is considered equivalent to the expression ``['-f', 'foo', '-f', 'bar']``. + +The ``fromfile_prefix_chars=`` argument defaults to ``None``, meaning that +arguments will never be treated as file references. + + +argument_default +^^^^^^^^^^^^^^^^ + +Generally, argument defaults are specified either by passing a default to +:meth:`~ArgumentParser.add_argument` or by calling the +:meth:`~ArgumentParser.set_defaults` methods with a specific set of name-value +pairs. Sometimes however, it may be useful to specify a single parser-wide +default for arguments. This can be accomplished by passing the +``argument_default=`` keyword argument to :class:`ArgumentParser`. For example, +to globally suppress attribute creation on :meth:`~ArgumentParser.parse_args` +calls, we supply ``argument_default=SUPPRESS``:: + + >>> parser = argparse.ArgumentParser(argument_default=argparse.SUPPRESS) + >>> parser.add_argument('--foo') + >>> parser.add_argument('bar', nargs='?') + >>> parser.parse_args(['--foo', '1', 'BAR']) + Namespace(bar='BAR', foo='1') + >>> parser.parse_args([]) + Namespace() + + +parents +^^^^^^^ + +Sometimes, several parsers share a common set of arguments. Rather than +repeating the definitions of these arguments, a single parser with all the +shared arguments and passed to ``parents=`` argument to :class:`ArgumentParser` +can be used. The ``parents=`` argument takes a list of :class:`ArgumentParser` +objects, collects all the positional and optional actions from them, and adds +these actions to the :class:`ArgumentParser` object being constructed:: + + >>> parent_parser = argparse.ArgumentParser(add_help=False) + >>> parent_parser.add_argument('--parent', type=int) + + >>> foo_parser = argparse.ArgumentParser(parents=[parent_parser]) + >>> foo_parser.add_argument('foo') + >>> foo_parser.parse_args(['--parent', '2', 'XXX']) + Namespace(foo='XXX', parent=2) + + >>> bar_parser = argparse.ArgumentParser(parents=[parent_parser]) + >>> bar_parser.add_argument('--bar') + >>> bar_parser.parse_args(['--bar', 'YYY']) + Namespace(bar='YYY', parent=None) + +Note that most parent parsers will specify ``add_help=False``. Otherwise, the +:class:`ArgumentParser` will see two ``-h/--help`` options (one in the parent +and one in the child) and raise an error. + +.. note:: + You must fully initialize the parsers before passing them via ``parents=``. + If you change the parent parsers after the child parser, those changes will + not be reflected in the child. + + +formatter_class +^^^^^^^^^^^^^^^ + +:class:`ArgumentParser` objects allow the help formatting to be customized by +specifying an alternate formatting class. Currently, there are three such +classes: + +.. class:: RawDescriptionHelpFormatter + RawTextHelpFormatter + ArgumentDefaultsHelpFormatter + +The first two allow more control over how textual descriptions are displayed, +while the last automatically adds information about argument default values. + +By default, :class:`ArgumentParser` objects line-wrap the description_ and +epilog_ texts in command-line help messages:: + + >>> parser = argparse.ArgumentParser( + ... prog='PROG', + ... description='''this description + ... was indented weird + ... but that is okay''', + ... epilog=''' + ... likewise for this epilog whose whitespace will + ... be cleaned up and whose words will be wrapped + ... across a couple lines''') + >>> parser.print_help() + usage: PROG [-h] + + this description was indented weird but that is okay + + optional arguments: + -h, --help show this help message and exit + + likewise for this epilog whose whitespace will be cleaned up and whose words + will be wrapped across a couple lines + +Passing :class:`RawDescriptionHelpFormatter` as ``formatter_class=`` +indicates that description_ and epilog_ are already correctly formatted and +should not be line-wrapped:: + + >>> parser = argparse.ArgumentParser( + ... prog='PROG', + ... formatter_class=argparse.RawDescriptionHelpFormatter, + ... description=textwrap.dedent('''\ + ... Please do not mess up this text! + ... -------------------------------- + ... I have indented it + ... exactly the way + ... I want it + ... ''')) + >>> parser.print_help() + usage: PROG [-h] + + Please do not mess up this text! + -------------------------------- + I have indented it + exactly the way + I want it + + optional arguments: + -h, --help show this help message and exit + +:class:`RawTextHelpFormatter` maintains whitespace for all sorts of help text, +including argument descriptions. + +The other formatter class available, :class:`ArgumentDefaultsHelpFormatter`, +will add information about the default value of each of the arguments:: + + >>> parser = argparse.ArgumentParser( + ... prog='PROG', + ... formatter_class=argparse.ArgumentDefaultsHelpFormatter) + >>> parser.add_argument('--foo', type=int, default=42, help='FOO!') + >>> parser.add_argument('bar', nargs='*', default=[1, 2, 3], help='BAR!') + >>> parser.print_help() + usage: PROG [-h] [--foo FOO] [bar [bar ...]] + + positional arguments: + bar BAR! (default: [1, 2, 3]) + + optional arguments: + -h, --help show this help message and exit + --foo FOO FOO! (default: 42) + + +conflict_handler +^^^^^^^^^^^^^^^^ + +:class:`ArgumentParser` objects do not allow two actions with the same option +string. By default, :class:`ArgumentParser` objects raises an exception if an +attempt is made to create an argument with an option string that is already in +use:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-f', '--foo', help='old foo help') + >>> parser.add_argument('--foo', help='new foo help') + Traceback (most recent call last): + .. + ArgumentError: argument --foo: conflicting option string(s): --foo + +Sometimes (e.g. when using parents_) it may be useful to simply override any +older arguments with the same option string. To get this behavior, the value +``'resolve'`` can be supplied to the ``conflict_handler=`` argument of +:class:`ArgumentParser`:: + + >>> parser = argparse.ArgumentParser(prog='PROG', conflict_handler='resolve') + >>> parser.add_argument('-f', '--foo', help='old foo help') + >>> parser.add_argument('--foo', help='new foo help') + >>> parser.print_help() + usage: PROG [-h] [-f FOO] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + -f FOO old foo help + --foo FOO new foo help + +Note that :class:`ArgumentParser` objects only remove an action if all of its +option strings are overridden. So, in the example above, the old ``-f/--foo`` +action is retained as the ``-f`` action, because only the ``--foo`` option +string was overridden. + + +prog +^^^^ + +By default, :class:`ArgumentParser` objects uses ``sys.argv[0]`` to determine +how to display the name of the program in help messages. This default is almost +always desirable because it will make the help messages match how the program was +invoked on the command line. For example, consider a file named +``myprogram.py`` with the following code:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument('--foo', help='foo help') + args = parser.parse_args() + +The help for this program will display ``myprogram.py`` as the program name +(regardless of where the program was invoked from):: + + $ python myprogram.py --help + usage: myprogram.py [-h] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + --foo FOO foo help + $ cd .. + $ python subdir\myprogram.py --help + usage: myprogram.py [-h] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + --foo FOO foo help + +To change this default behavior, another value can be supplied using the +``prog=`` argument to :class:`ArgumentParser`:: + + >>> parser = argparse.ArgumentParser(prog='myprogram') + >>> parser.print_help() + usage: myprogram [-h] + + optional arguments: + -h, --help show this help message and exit + +Note that the program name, whether determined from ``sys.argv[0]`` or from the +``prog=`` argument, is available to help messages using the ``%(prog)s`` format +specifier. + +:: + + >>> parser = argparse.ArgumentParser(prog='myprogram') + >>> parser.add_argument('--foo', help='foo of the %(prog)s program') + >>> parser.print_help() + usage: myprogram [-h] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + --foo FOO foo of the myprogram program + + +usage +^^^^^ + +By default, :class:`ArgumentParser` calculates the usage message from the +arguments it contains:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--foo', nargs='?', help='foo help') + >>> parser.add_argument('bar', nargs='+', help='bar help') + >>> parser.print_help() + usage: PROG [-h] [--foo [FOO]] bar [bar ...] + + positional arguments: + bar bar help + + optional arguments: + -h, --help show this help message and exit + --foo [FOO] foo help + +The default message can be overridden with the ``usage=`` keyword argument:: + + >>> parser = argparse.ArgumentParser(prog='PROG', usage='%(prog)s [options]') + >>> parser.add_argument('--foo', nargs='?', help='foo help') + >>> parser.add_argument('bar', nargs='+', help='bar help') + >>> parser.print_help() + usage: PROG [options] + + positional arguments: + bar bar help + + optional arguments: + -h, --help show this help message and exit + --foo [FOO] foo help + +The ``%(prog)s`` format specifier is available to fill in the program name in +your usage messages. + + +The add_argument() method +------------------------- + +.. method:: ArgumentParser.add_argument(name or flags..., [action], [nargs], \ + [const], [default], [type], [choices], [required], \ + [help], [metavar], [dest]) + + Define how a single command-line argument should be parsed. Each parameter + has its own more detailed description below, but in short they are: + + * `name or flags`_ - Either a name or a list of option strings, e.g. ``foo`` + or ``-f, --foo``. + + * action_ - The basic type of action to be taken when this argument is + encountered at the command line. + + * nargs_ - The number of command-line arguments that should be consumed. + + * const_ - A constant value required by some action_ and nargs_ selections. + + * default_ - The value produced if the argument is absent from the + command line. + + * type_ - The type to which the command-line argument should be converted. + + * choices_ - A container of the allowable values for the argument. + + * required_ - Whether or not the command-line option may be omitted + (optionals only). + + * help_ - A brief description of what the argument does. + + * metavar_ - A name for the argument in usage messages. + + * dest_ - The name of the attribute to be added to the object returned by + :meth:`parse_args`. + +The following sections describe how each of these are used. + + +name or flags +^^^^^^^^^^^^^ + +The :meth:`~ArgumentParser.add_argument` method must know whether an optional +argument, like ``-f`` or ``--foo``, or a positional argument, like a list of +filenames, is expected. The first arguments passed to +:meth:`~ArgumentParser.add_argument` must therefore be either a series of +flags, or a simple argument name. For example, an optional argument could +be created like:: + + >>> parser.add_argument('-f', '--foo') + +while a positional argument could be created like:: + + >>> parser.add_argument('bar') + +When :meth:`~ArgumentParser.parse_args` is called, optional arguments will be +identified by the ``-`` prefix, and the remaining arguments will be assumed to +be positional:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-f', '--foo') + >>> parser.add_argument('bar') + >>> parser.parse_args(['BAR']) + Namespace(bar='BAR', foo=None) + >>> parser.parse_args(['BAR', '--foo', 'FOO']) + Namespace(bar='BAR', foo='FOO') + >>> parser.parse_args(['--foo', 'FOO']) + usage: PROG [-h] [-f FOO] bar + PROG: error: too few arguments + + +action +^^^^^^ + +:class:`ArgumentParser` objects associate command-line arguments with actions. These +actions can do just about anything with the command-line arguments associated with +them, though most actions simply add an attribute to the object returned by +:meth:`~ArgumentParser.parse_args`. The ``action`` keyword argument specifies +how the command-line arguments should be handled. The supported actions are: + +* ``'store'`` - This just stores the argument's value. This is the default + action. For example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> parser.parse_args('--foo 1'.split()) + Namespace(foo='1') + +* ``'store_const'`` - This stores the value specified by the const_ keyword + argument. (Note that the const_ keyword argument defaults to the rather + unhelpful ``None``.) The ``'store_const'`` action is most commonly used with + optional arguments that specify some sort of flag. For example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='store_const', const=42) + >>> parser.parse_args('--foo'.split()) + Namespace(foo=42) + +* ``'store_true'`` and ``'store_false'`` - These store the values ``True`` and + ``False`` respectively. These are special cases of ``'store_const'``. For + example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='store_true') + >>> parser.add_argument('--bar', action='store_false') + >>> parser.parse_args('--foo --bar'.split()) + Namespace(bar=False, foo=True) + +* ``'append'`` - This stores a list, and appends each argument value to the + list. This is useful to allow an option to be specified multiple times. + Example usage:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='append') + >>> parser.parse_args('--foo 1 --foo 2'.split()) + Namespace(foo=['1', '2']) + +* ``'append_const'`` - This stores a list, and appends the value specified by + the const_ keyword argument to the list. (Note that the const_ keyword + argument defaults to ``None``.) The ``'append_const'`` action is typically + useful when multiple arguments need to store constants to the same list. For + example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--str', dest='types', action='append_const', const=str) + >>> parser.add_argument('--int', dest='types', action='append_const', const=int) + >>> parser.parse_args('--str --int'.split()) + Namespace(types=[<class 'str'>, <class 'int'>]) + +* ``'count'`` - This counts the number of times a keyword argument occurs. For + example, this is useful for increasing verbosity levels:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--verbose', '-v', action='count') + >>> parser.parse_args('-vvv'.split()) + Namespace(verbose=3) + +* ``'help'`` - This prints a complete help message for all the options in the + current parser and then exits. By default a help action is automatically + added to the parser. See :class:`ArgumentParser` for details of how the + output is created. + +* ``'version'`` - This expects a ``version=`` keyword argument in the + :meth:`~ArgumentParser.add_argument` call, and prints version information + and exits when invoked:: + + >>> import argparse + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--version', action='version', version='%(prog)s 2.0') + >>> parser.parse_args(['--version']) + PROG 2.0 + +You can also specify an arbitrary action by passing an object that implements +the Action API. The easiest way to do this is to extend +:class:`argparse.Action`, supplying an appropriate ``__call__`` method. The +``__call__`` method should accept four parameters: + +* ``parser`` - The ArgumentParser object which contains this action. + +* ``namespace`` - The :class:`Namespace` object that will be returned by + :meth:`~ArgumentParser.parse_args`. Most actions add an attribute to this + object. + +* ``values`` - The associated command-line arguments, with any type conversions + applied. (Type conversions are specified with the type_ keyword argument to + :meth:`~ArgumentParser.add_argument`. + +* ``option_string`` - The option string that was used to invoke this action. + The ``option_string`` argument is optional, and will be absent if the action + is associated with a positional argument. + +An example of a custom action:: + + >>> class FooAction(argparse.Action): + ... def __call__(self, parser, namespace, values, option_string=None): + ... print('%r %r %r' % (namespace, values, option_string)) + ... setattr(namespace, self.dest, values) + ... + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action=FooAction) + >>> parser.add_argument('bar', action=FooAction) + >>> args = parser.parse_args('1 --foo 2'.split()) + Namespace(bar=None, foo=None) '1' None + Namespace(bar='1', foo=None) '2' '--foo' + >>> args + Namespace(bar='1', foo='2') + + +nargs +^^^^^ + +ArgumentParser objects usually associate a single command-line argument with a +single action to be taken. The ``nargs`` keyword argument associates a +different number of command-line arguments with a single action. The supported +values are: + +* ``N`` (an integer). ``N`` arguments from the command line will be gathered + together into a list. For example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', nargs=2) + >>> parser.add_argument('bar', nargs=1) + >>> parser.parse_args('c --foo a b'.split()) + Namespace(bar=['c'], foo=['a', 'b']) + + Note that ``nargs=1`` produces a list of one item. This is different from + the default, in which the item is produced by itself. + +* ``'?'``. One argument will be consumed from the command line if possible, and + produced as a single item. If no command-line argument is present, the value from + default_ will be produced. Note that for optional arguments, there is an + additional case - the option string is present but not followed by a + command-line argument. In this case the value from const_ will be produced. Some + examples to illustrate this:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', nargs='?', const='c', default='d') + >>> parser.add_argument('bar', nargs='?', default='d') + >>> parser.parse_args('XX --foo YY'.split()) + Namespace(bar='XX', foo='YY') + >>> parser.parse_args('XX --foo'.split()) + Namespace(bar='XX', foo='c') + >>> parser.parse_args(''.split()) + Namespace(bar='d', foo='d') + + One of the more common uses of ``nargs='?'`` is to allow optional input and + output files:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('infile', nargs='?', type=argparse.FileType('r'), + ... default=sys.stdin) + >>> parser.add_argument('outfile', nargs='?', type=argparse.FileType('w'), + ... default=sys.stdout) + >>> parser.parse_args(['input.txt', 'output.txt']) + Namespace(infile=<_io.TextIOWrapper name='input.txt' encoding='UTF-8'>, + outfile=<_io.TextIOWrapper name='output.txt' encoding='UTF-8'>) + >>> parser.parse_args([]) + Namespace(infile=<_io.TextIOWrapper name='<stdin>' encoding='UTF-8'>, + outfile=<_io.TextIOWrapper name='<stdout>' encoding='UTF-8'>) + +* ``'*'``. All command-line arguments present are gathered into a list. Note that + it generally doesn't make much sense to have more than one positional argument + with ``nargs='*'``, but multiple optional arguments with ``nargs='*'`` is + possible. For example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', nargs='*') + >>> parser.add_argument('--bar', nargs='*') + >>> parser.add_argument('baz', nargs='*') + >>> parser.parse_args('a b --foo x y --bar 1 2'.split()) + Namespace(bar=['1', '2'], baz=['a', 'b'], foo=['x', 'y']) + +* ``'+'``. Just like ``'*'``, all command-line args present are gathered into a + list. Additionally, an error message will be generated if there wasn't at + least one command-line argument present. For example:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', nargs='+') + >>> parser.parse_args('a b'.split()) + Namespace(foo=['a', 'b']) + >>> parser.parse_args(''.split()) + usage: PROG [-h] foo [foo ...] + PROG: error: too few arguments + +* ``argparse.REMAINDER``. All the remaining command-line arguments are gathered + into a list. This is commonly useful for command line utilities that dispatch + to other command line utilities:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--foo') + >>> parser.add_argument('command') + >>> parser.add_argument('args', nargs=argparse.REMAINDER) + >>> print(parser.parse_args('--foo B cmd --arg1 XX ZZ'.split())) + Namespace(args=['--arg1', 'XX', 'ZZ'], command='cmd', foo='B') + +If the ``nargs`` keyword argument is not provided, the number of arguments consumed +is determined by the action_. Generally this means a single command-line argument +will be consumed and a single item (not a list) will be produced. + + +const +^^^^^ + +The ``const`` argument of :meth:`~ArgumentParser.add_argument` is used to hold +constant values that are not read from the command line but are required for +the various :class:`ArgumentParser` actions. The two most common uses of it are: + +* When :meth:`~ArgumentParser.add_argument` is called with + ``action='store_const'`` or ``action='append_const'``. These actions add the + ``const`` value to one of the attributes of the object returned by + :meth:`~ArgumentParser.parse_args`. See the action_ description for examples. + +* When :meth:`~ArgumentParser.add_argument` is called with option strings + (like ``-f`` or ``--foo``) and ``nargs='?'``. This creates an optional + argument that can be followed by zero or one command-line arguments. + When parsing the command line, if the option string is encountered with no + command-line argument following it, the value of ``const`` will be assumed instead. + See the nargs_ description for examples. + +The ``const`` keyword argument defaults to ``None``. + + +default +^^^^^^^ + +All optional arguments and some positional arguments may be omitted at the +command line. The ``default`` keyword argument of +:meth:`~ArgumentParser.add_argument`, whose value defaults to ``None``, +specifies what value should be used if the command-line argument is not present. +For optional arguments, the ``default`` value is used when the option string +was not present at the command line:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default=42) + >>> parser.parse_args('--foo 2'.split()) + Namespace(foo='2') + >>> parser.parse_args(''.split()) + Namespace(foo=42) + +For positional arguments with nargs_ equal to ``?`` or ``*``, the ``default`` value +is used when no command-line argument was present:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('foo', nargs='?', default=42) + >>> parser.parse_args('a'.split()) + Namespace(foo='a') + >>> parser.parse_args(''.split()) + Namespace(foo=42) + + +Providing ``default=argparse.SUPPRESS`` causes no attribute to be added if the +command-line argument was not present.:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default=argparse.SUPPRESS) + >>> parser.parse_args([]) + Namespace() + >>> parser.parse_args(['--foo', '1']) + Namespace(foo='1') + + +type +^^^^ + +By default, :class:`ArgumentParser` objects read command-line arguments in as simple +strings. However, quite often the command-line string should instead be +interpreted as another type, like a :class:`float` or :class:`int`. The +``type`` keyword argument of :meth:`~ArgumentParser.add_argument` allows any +necessary type-checking and type conversions to be performed. Common built-in +types and functions can be used directly as the value of the ``type`` argument:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('foo', type=int) + >>> parser.add_argument('bar', type=open) + >>> parser.parse_args('2 temp.txt'.split()) + Namespace(bar=<_io.TextIOWrapper name='temp.txt' encoding='UTF-8'>, foo=2) + +To ease the use of various types of files, the argparse module provides the +factory FileType which takes the ``mode=`` and ``bufsize=`` arguments of the +:func:`open` function. For example, ``FileType('w')`` can be used to create a +writable file:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('bar', type=argparse.FileType('w')) + >>> parser.parse_args(['out.txt']) + Namespace(bar=<_io.TextIOWrapper name='out.txt' encoding='UTF-8'>) + +``type=`` can take any callable that takes a single string argument and returns +the converted value:: + + >>> def perfect_square(string): + ... value = int(string) + ... sqrt = math.sqrt(value) + ... if sqrt != int(sqrt): + ... msg = "%r is not a perfect square" % string + ... raise argparse.ArgumentTypeError(msg) + ... return value + ... + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', type=perfect_square) + >>> parser.parse_args('9'.split()) + Namespace(foo=9) + >>> parser.parse_args('7'.split()) + usage: PROG [-h] foo + PROG: error: argument foo: '7' is not a perfect square + +The choices_ keyword argument may be more convenient for type checkers that +simply check against a range of values:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', type=int, choices=range(5, 10)) + >>> parser.parse_args('7'.split()) + Namespace(foo=7) + >>> parser.parse_args('11'.split()) + usage: PROG [-h] {5,6,7,8,9} + PROG: error: argument foo: invalid choice: 11 (choose from 5, 6, 7, 8, 9) + +See the choices_ section for more details. + + +choices +^^^^^^^ + +Some command-line arguments should be selected from a restricted set of values. +These can be handled by passing a container object as the ``choices`` keyword +argument to :meth:`~ArgumentParser.add_argument`. When the command line is +parsed, argument values will be checked, and an error message will be displayed if +the argument was not one of the acceptable values:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', choices='abc') + >>> parser.parse_args('c'.split()) + Namespace(foo='c') + >>> parser.parse_args('X'.split()) + usage: PROG [-h] {a,b,c} + PROG: error: argument foo: invalid choice: 'X' (choose from 'a', 'b', 'c') + +Note that inclusion in the ``choices`` container is checked after any type_ +conversions have been performed, so the type of the objects in the ``choices`` +container should match the type_ specified:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', type=complex, choices=[1, 1j]) + >>> parser.parse_args('1j'.split()) + Namespace(foo=1j) + >>> parser.parse_args('-- -4'.split()) + usage: PROG [-h] {1,1j} + PROG: error: argument foo: invalid choice: (-4+0j) (choose from 1, 1j) + +Any object that supports the ``in`` operator can be passed as the ``choices`` +value, so :class:`dict` objects, :class:`set` objects, custom containers, +etc. are all supported. + + +required +^^^^^^^^ + +In general, the :mod:`argparse` module assumes that flags like ``-f`` and ``--bar`` +indicate *optional* arguments, which can always be omitted at the command line. +To make an option *required*, ``True`` can be specified for the ``required=`` +keyword argument to :meth:`~ArgumentParser.add_argument`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', required=True) + >>> parser.parse_args(['--foo', 'BAR']) + Namespace(foo='BAR') + >>> parser.parse_args([]) + usage: argparse.py [-h] [--foo FOO] + argparse.py: error: option --foo is required + +As the example shows, if an option is marked as ``required``, +:meth:`~ArgumentParser.parse_args` will report an error if that option is not +present at the command line. + +.. note:: + + Required options are generally considered bad form because users expect + *options* to be *optional*, and thus they should be avoided when possible. + + +help +^^^^ + +The ``help`` value is a string containing a brief description of the argument. +When a user requests help (usually by using ``-h`` or ``--help`` at the +command line), these ``help`` descriptions will be displayed with each +argument:: + + >>> parser = argparse.ArgumentParser(prog='frobble') + >>> parser.add_argument('--foo', action='store_true', + ... help='foo the bars before frobbling') + >>> parser.add_argument('bar', nargs='+', + ... help='one of the bars to be frobbled') + >>> parser.parse_args('-h'.split()) + usage: frobble [-h] [--foo] bar [bar ...] + + positional arguments: + bar one of the bars to be frobbled + + optional arguments: + -h, --help show this help message and exit + --foo foo the bars before frobbling + +The ``help`` strings can include various format specifiers to avoid repetition +of things like the program name or the argument default_. The available +specifiers include the program name, ``%(prog)s`` and most keyword arguments to +:meth:`~ArgumentParser.add_argument`, e.g. ``%(default)s``, ``%(type)s``, etc.:: + + >>> parser = argparse.ArgumentParser(prog='frobble') + >>> parser.add_argument('bar', nargs='?', type=int, default=42, + ... help='the bar to %(prog)s (default: %(default)s)') + >>> parser.print_help() + usage: frobble [-h] [bar] + + positional arguments: + bar the bar to frobble (default: 42) + + optional arguments: + -h, --help show this help message and exit + +:mod:`argparse` supports silencing the help entry for certain options, by +setting the ``help`` value to ``argparse.SUPPRESS``:: + + >>> parser = argparse.ArgumentParser(prog='frobble') + >>> parser.add_argument('--foo', help=argparse.SUPPRESS) + >>> parser.print_help() + usage: frobble [-h] + + optional arguments: + -h, --help show this help message and exit + + +metavar +^^^^^^^ + +When :class:`ArgumentParser` generates help messages, it need some way to refer +to each expected argument. By default, ArgumentParser objects use the dest_ +value as the "name" of each object. By default, for positional argument +actions, the dest_ value is used directly, and for optional argument actions, +the dest_ value is uppercased. So, a single positional argument with +``dest='bar'`` will be referred to as ``bar``. A single +optional argument ``--foo`` that should be followed by a single command-line argument +will be referred to as ``FOO``. An example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> parser.add_argument('bar') + >>> parser.parse_args('X --foo Y'.split()) + Namespace(bar='X', foo='Y') + >>> parser.print_help() + usage: [-h] [--foo FOO] bar + + positional arguments: + bar + + optional arguments: + -h, --help show this help message and exit + --foo FOO + +An alternative name can be specified with ``metavar``:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', metavar='YYY') + >>> parser.add_argument('bar', metavar='XXX') + >>> parser.parse_args('X --foo Y'.split()) + Namespace(bar='X', foo='Y') + >>> parser.print_help() + usage: [-h] [--foo YYY] XXX + + positional arguments: + XXX + + optional arguments: + -h, --help show this help message and exit + --foo YYY + +Note that ``metavar`` only changes the *displayed* name - the name of the +attribute on the :meth:`~ArgumentParser.parse_args` object is still determined +by the dest_ value. + +Different values of ``nargs`` may cause the metavar to be used multiple times. +Providing a tuple to ``metavar`` specifies a different display for each of the +arguments:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-x', nargs=2) + >>> parser.add_argument('--foo', nargs=2, metavar=('bar', 'baz')) + >>> parser.print_help() + usage: PROG [-h] [-x X X] [--foo bar baz] + + optional arguments: + -h, --help show this help message and exit + -x X X + --foo bar baz + + +dest +^^^^ + +Most :class:`ArgumentParser` actions add some value as an attribute of the +object returned by :meth:`~ArgumentParser.parse_args`. The name of this +attribute is determined by the ``dest`` keyword argument of +:meth:`~ArgumentParser.add_argument`. For positional argument actions, +``dest`` is normally supplied as the first argument to +:meth:`~ArgumentParser.add_argument`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('bar') + >>> parser.parse_args('XXX'.split()) + Namespace(bar='XXX') + +For optional argument actions, the value of ``dest`` is normally inferred from +the option strings. :class:`ArgumentParser` generates the value of ``dest`` by +taking the first long option string and stripping away the initial ``--`` +string. If no long option strings were supplied, ``dest`` will be derived from +the first short option string by stripping the initial ``-`` character. Any +internal ``-`` characters will be converted to ``_`` characters to make sure +the string is a valid attribute name. The examples below illustrate this +behavior:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('-f', '--foo-bar', '--foo') + >>> parser.add_argument('-x', '-y') + >>> parser.parse_args('-f 1 -x 2'.split()) + Namespace(foo_bar='1', x='2') + >>> parser.parse_args('--foo 1 -y 2'.split()) + Namespace(foo_bar='1', x='2') + +``dest`` allows a custom attribute name to be provided:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', dest='bar') + >>> parser.parse_args('--foo XXX'.split()) + Namespace(bar='XXX') + + +The parse_args() method +----------------------- + +.. method:: ArgumentParser.parse_args(args=None, namespace=None) + + Convert argument strings to objects and assign them as attributes of the + namespace. Return the populated namespace. + + Previous calls to :meth:`add_argument` determine exactly what objects are + created and how they are assigned. See the documentation for + :meth:`add_argument` for details. + + By default, the argument strings are taken from :data:`sys.argv`, and a new empty + :class:`Namespace` object is created for the attributes. + + +Option value syntax +^^^^^^^^^^^^^^^^^^^ + +The :meth:`~ArgumentParser.parse_args` method supports several ways of +specifying the value of an option (if it takes one). In the simplest case, the +option and its value are passed as two separate arguments:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-x') + >>> parser.add_argument('--foo') + >>> parser.parse_args('-x X'.split()) + Namespace(foo=None, x='X') + >>> parser.parse_args('--foo FOO'.split()) + Namespace(foo='FOO', x=None) + +For long options (options with names longer than a single character), the option +and value can also be passed as a single command-line argument, using ``=`` to +separate them:: + + >>> parser.parse_args('--foo=FOO'.split()) + Namespace(foo='FOO', x=None) + +For short options (options only one character long), the option and its value +can be concatenated:: + + >>> parser.parse_args('-xX'.split()) + Namespace(foo=None, x='X') + +Several short options can be joined together, using only a single ``-`` prefix, +as long as only the last option (or none of them) requires a value:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-x', action='store_true') + >>> parser.add_argument('-y', action='store_true') + >>> parser.add_argument('-z') + >>> parser.parse_args('-xyzZ'.split()) + Namespace(x=True, y=True, z='Z') + + +Invalid arguments +^^^^^^^^^^^^^^^^^ + +While parsing the command line, :meth:`~ArgumentParser.parse_args` checks for a +variety of errors, including ambiguous options, invalid types, invalid options, +wrong number of positional arguments, etc. When it encounters such an error, +it exits and prints the error along with a usage message:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--foo', type=int) + >>> parser.add_argument('bar', nargs='?') + + >>> # invalid type + >>> parser.parse_args(['--foo', 'spam']) + usage: PROG [-h] [--foo FOO] [bar] + PROG: error: argument --foo: invalid int value: 'spam' + + >>> # invalid option + >>> parser.parse_args(['--bar']) + usage: PROG [-h] [--foo FOO] [bar] + PROG: error: no such option: --bar + + >>> # wrong number of arguments + >>> parser.parse_args(['spam', 'badger']) + usage: PROG [-h] [--foo FOO] [bar] + PROG: error: extra arguments found: badger + + +Arguments containing ``-`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The :meth:`~ArgumentParser.parse_args` method attempts to give errors whenever +the user has clearly made a mistake, but some situations are inherently +ambiguous. For example, the command-line argument ``-1`` could either be an +attempt to specify an option or an attempt to provide a positional argument. +The :meth:`~ArgumentParser.parse_args` method is cautious here: positional +arguments may only begin with ``-`` if they look like negative numbers and +there are no options in the parser that look like negative numbers:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-x') + >>> parser.add_argument('foo', nargs='?') + + >>> # no negative number options, so -1 is a positional argument + >>> parser.parse_args(['-x', '-1']) + Namespace(foo=None, x='-1') + + >>> # no negative number options, so -1 and -5 are positional arguments + >>> parser.parse_args(['-x', '-1', '-5']) + Namespace(foo='-5', x='-1') + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-1', dest='one') + >>> parser.add_argument('foo', nargs='?') + + >>> # negative number options present, so -1 is an option + >>> parser.parse_args(['-1', 'X']) + Namespace(foo=None, one='X') + + >>> # negative number options present, so -2 is an option + >>> parser.parse_args(['-2']) + usage: PROG [-h] [-1 ONE] [foo] + PROG: error: no such option: -2 + + >>> # negative number options present, so both -1s are options + >>> parser.parse_args(['-1', '-1']) + usage: PROG [-h] [-1 ONE] [foo] + PROG: error: argument -1: expected one argument + +If you have positional arguments that must begin with ``-`` and don't look +like negative numbers, you can insert the pseudo-argument ``'--'`` which tells +:meth:`~ArgumentParser.parse_args` that everything after that is a positional +argument:: + + >>> parser.parse_args(['--', '-f']) + Namespace(foo='-f', one=None) + + +Argument abbreviations +^^^^^^^^^^^^^^^^^^^^^^ + +The :meth:`~ArgumentParser.parse_args` method allows long options to be +abbreviated if the abbreviation is unambiguous:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-bacon') + >>> parser.add_argument('-badger') + >>> parser.parse_args('-bac MMM'.split()) + Namespace(bacon='MMM', badger=None) + >>> parser.parse_args('-bad WOOD'.split()) + Namespace(bacon=None, badger='WOOD') + >>> parser.parse_args('-ba BA'.split()) + usage: PROG [-h] [-bacon BACON] [-badger BADGER] + PROG: error: ambiguous option: -ba could match -badger, -bacon + +An error is produced for arguments that could produce more than one options. + + +Beyond ``sys.argv`` +^^^^^^^^^^^^^^^^^^^ + +Sometimes it may be useful to have an ArgumentParser parse arguments other than those +of :data:`sys.argv`. This can be accomplished by passing a list of strings to +:meth:`~ArgumentParser.parse_args`. This is useful for testing at the +interactive prompt:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument( + ... 'integers', metavar='int', type=int, choices=range(10), + ... nargs='+', help='an integer in the range 0..9') + >>> parser.add_argument( + ... '--sum', dest='accumulate', action='store_const', const=sum, + ... default=max, help='sum the integers (default: find the max)') + >>> parser.parse_args(['1', '2', '3', '4']) + Namespace(accumulate=<built-in function max>, integers=[1, 2, 3, 4]) + >>> parser.parse_args('1 2 3 4 --sum'.split()) + Namespace(accumulate=<built-in function sum>, integers=[1, 2, 3, 4]) + + +The Namespace object +^^^^^^^^^^^^^^^^^^^^ + +.. class:: Namespace + + Simple class used by default by :meth:`~ArgumentParser.parse_args` to create + an object holding attributes and return it. + +This class is deliberately simple, just an :class:`object` subclass with a +readable string representation. If you prefer to have dict-like view of the +attributes, you can use the standard Python idiom, :func:`vars`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> args = parser.parse_args(['--foo', 'BAR']) + >>> vars(args) + {'foo': 'BAR'} + +It may also be useful to have an :class:`ArgumentParser` assign attributes to an +already existing object, rather than a new :class:`Namespace` object. This can +be achieved by specifying the ``namespace=`` keyword argument:: + + >>> class C: + ... pass + ... + >>> c = C() + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> parser.parse_args(args=['--foo', 'BAR'], namespace=c) + >>> c.foo + 'BAR' + + +Other utilities +--------------- + +Sub-commands +^^^^^^^^^^^^ + +.. method:: ArgumentParser.add_subparsers() + + Many programs split up their functionality into a number of sub-commands, + for example, the ``svn`` program can invoke sub-commands like ``svn + checkout``, ``svn update``, and ``svn commit``. Splitting up functionality + this way can be a particularly good idea when a program performs several + different functions which require different kinds of command-line arguments. + :class:`ArgumentParser` supports the creation of such sub-commands with the + :meth:`add_subparsers` method. The :meth:`add_subparsers` method is normally + called with no arguments and returns an special action object. This object + has a single method, :meth:`~ArgumentParser.add_parser`, which takes a + command name and any :class:`ArgumentParser` constructor arguments, and + returns an :class:`ArgumentParser` object that can be modified as usual. + + Some example usage:: + + >>> # create the top-level parser + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--foo', action='store_true', help='foo help') + >>> subparsers = parser.add_subparsers(help='sub-command help') + >>> + >>> # create the parser for the "a" command + >>> parser_a = subparsers.add_parser('a', help='a help') + >>> parser_a.add_argument('bar', type=int, help='bar help') + >>> + >>> # create the parser for the "b" command + >>> parser_b = subparsers.add_parser('b', help='b help') + >>> parser_b.add_argument('--baz', choices='XYZ', help='baz help') + >>> + >>> # parse some argument lists + >>> parser.parse_args(['a', '12']) + Namespace(bar=12, foo=False) + >>> parser.parse_args(['--foo', 'b', '--baz', 'Z']) + Namespace(baz='Z', foo=True) + + Note that the object returned by :meth:`parse_args` will only contain + attributes for the main parser and the subparser that was selected by the + command line (and not any other subparsers). So in the example above, when + the ``a`` command is specified, only the ``foo`` and ``bar`` attributes are + present, and when the ``b`` command is specified, only the ``foo`` and + ``baz`` attributes are present. + + Similarly, when a help message is requested from a subparser, only the help + for that particular parser will be printed. The help message will not + include parent parser or sibling parser messages. (A help message for each + subparser command, however, can be given by supplying the ``help=`` argument + to :meth:`add_parser` as above.) + + :: + + >>> parser.parse_args(['--help']) + usage: PROG [-h] [--foo] {a,b} ... + + positional arguments: + {a,b} sub-command help + a a help + b b help + + optional arguments: + -h, --help show this help message and exit + --foo foo help + + >>> parser.parse_args(['a', '--help']) + usage: PROG a [-h] bar + + positional arguments: + bar bar help + + optional arguments: + -h, --help show this help message and exit + + >>> parser.parse_args(['b', '--help']) + usage: PROG b [-h] [--baz {X,Y,Z}] + + optional arguments: + -h, --help show this help message and exit + --baz {X,Y,Z} baz help + + The :meth:`add_subparsers` method also supports ``title`` and ``description`` + keyword arguments. When either is present, the subparser's commands will + appear in their own group in the help output. For example:: + + >>> parser = argparse.ArgumentParser() + >>> subparsers = parser.add_subparsers(title='subcommands', + ... description='valid subcommands', + ... help='additional help') + >>> subparsers.add_parser('foo') + >>> subparsers.add_parser('bar') + >>> parser.parse_args(['-h']) + usage: [-h] {foo,bar} ... + + optional arguments: + -h, --help show this help message and exit + + subcommands: + valid subcommands + + {foo,bar} additional help + + Furthermore, ``add_parser`` supports an additional ``aliases`` argument, + which allows multiple strings to refer to the same subparser. This example, + like ``svn``, aliases ``co`` as a shorthand for ``checkout``:: + + >>> parser = argparse.ArgumentParser() + >>> subparsers = parser.add_subparsers() + >>> checkout = subparsers.add_parser('checkout', aliases=['co']) + >>> checkout.add_argument('foo') + >>> parser.parse_args(['co', 'bar']) + Namespace(foo='bar') + + One particularly effective way of handling sub-commands is to combine the use + of the :meth:`add_subparsers` method with calls to :meth:`set_defaults` so + that each subparser knows which Python function it should execute. For + example:: + + >>> # sub-command functions + >>> def foo(args): + ... print(args.x * args.y) + ... + >>> def bar(args): + ... print('((%s))' % args.z) + ... + >>> # create the top-level parser + >>> parser = argparse.ArgumentParser() + >>> subparsers = parser.add_subparsers() + >>> + >>> # create the parser for the "foo" command + >>> parser_foo = subparsers.add_parser('foo') + >>> parser_foo.add_argument('-x', type=int, default=1) + >>> parser_foo.add_argument('y', type=float) + >>> parser_foo.set_defaults(func=foo) + >>> + >>> # create the parser for the "bar" command + >>> parser_bar = subparsers.add_parser('bar') + >>> parser_bar.add_argument('z') + >>> parser_bar.set_defaults(func=bar) + >>> + >>> # parse the args and call whatever function was selected + >>> args = parser.parse_args('foo 1 -x 2'.split()) + >>> args.func(args) + 2.0 + >>> + >>> # parse the args and call whatever function was selected + >>> args = parser.parse_args('bar XYZYX'.split()) + >>> args.func(args) + ((XYZYX)) + + This way, you can let :meth:`parse_args` do the job of calling the + appropriate function after argument parsing is complete. Associating + functions with actions like this is typically the easiest way to handle the + different actions for each of your subparsers. However, if it is necessary + to check the name of the subparser that was invoked, the ``dest`` keyword + argument to the :meth:`add_subparsers` call will work:: + + >>> parser = argparse.ArgumentParser() + >>> subparsers = parser.add_subparsers(dest='subparser_name') + >>> subparser1 = subparsers.add_parser('1') + >>> subparser1.add_argument('-x') + >>> subparser2 = subparsers.add_parser('2') + >>> subparser2.add_argument('y') + >>> parser.parse_args(['2', 'frobble']) + Namespace(subparser_name='2', y='frobble') + + +FileType objects +^^^^^^^^^^^^^^^^ + +.. class:: FileType(mode='r', bufsize=None) + + The :class:`FileType` factory creates objects that can be passed to the type + argument of :meth:`ArgumentParser.add_argument`. Arguments that have + :class:`FileType` objects as their type will open command-line arguments as files + with the requested modes and buffer sizes:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--output', type=argparse.FileType('wb', 0)) + >>> parser.parse_args(['--output', 'out']) + Namespace(output=<_io.BufferedWriter name='out'>) + + FileType objects understand the pseudo-argument ``'-'`` and automatically + convert this into ``sys.stdin`` for readable :class:`FileType` objects and + ``sys.stdout`` for writable :class:`FileType` objects:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('infile', type=argparse.FileType('r')) + >>> parser.parse_args(['-']) + Namespace(infile=<_io.TextIOWrapper name='<stdin>' encoding='UTF-8'>) + + +Argument groups +^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.add_argument_group(title=None, description=None) + + By default, :class:`ArgumentParser` groups command-line arguments into + "positional arguments" and "optional arguments" when displaying help + messages. When there is a better conceptual grouping of arguments than this + default one, appropriate groups can be created using the + :meth:`add_argument_group` method:: + + >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False) + >>> group = parser.add_argument_group('group') + >>> group.add_argument('--foo', help='foo help') + >>> group.add_argument('bar', help='bar help') + >>> parser.print_help() + usage: PROG [--foo FOO] bar + + group: + bar bar help + --foo FOO foo help + + The :meth:`add_argument_group` method returns an argument group object which + has an :meth:`~ArgumentParser.add_argument` method just like a regular + :class:`ArgumentParser`. When an argument is added to the group, the parser + treats it just like a normal argument, but displays the argument in a + separate group for help messages. The :meth:`add_argument_group` method + accepts *title* and *description* arguments which can be used to + customize this display:: + + >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False) + >>> group1 = parser.add_argument_group('group1', 'group1 description') + >>> group1.add_argument('foo', help='foo help') + >>> group2 = parser.add_argument_group('group2', 'group2 description') + >>> group2.add_argument('--bar', help='bar help') + >>> parser.print_help() + usage: PROG [--bar BAR] foo + + group1: + group1 description + + foo foo help + + group2: + group2 description + + --bar BAR bar help + + Note that any arguments not your user defined groups will end up back in the + usual "positional arguments" and "optional arguments" sections. + + +Mutual exclusion +^^^^^^^^^^^^^^^^ + +.. method:: add_mutually_exclusive_group(required=False) + + Create a mutually exclusive group. :mod:`argparse` will make sure that only + one of the arguments in the mutually exclusive group was present on the + command line:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> group = parser.add_mutually_exclusive_group() + >>> group.add_argument('--foo', action='store_true') + >>> group.add_argument('--bar', action='store_false') + >>> parser.parse_args(['--foo']) + Namespace(bar=True, foo=True) + >>> parser.parse_args(['--bar']) + Namespace(bar=False, foo=False) + >>> parser.parse_args(['--foo', '--bar']) + usage: PROG [-h] [--foo | --bar] + PROG: error: argument --bar: not allowed with argument --foo + + The :meth:`add_mutually_exclusive_group` method also accepts a *required* + argument, to indicate that at least one of the mutually exclusive arguments + is required:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> group = parser.add_mutually_exclusive_group(required=True) + >>> group.add_argument('--foo', action='store_true') + >>> group.add_argument('--bar', action='store_false') + >>> parser.parse_args([]) + usage: PROG [-h] (--foo | --bar) + PROG: error: one of the arguments --foo --bar is required + + Note that currently mutually exclusive argument groups do not support the + *title* and *description* arguments of + :meth:`~ArgumentParser.add_argument_group`. + + +Parser defaults +^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.set_defaults(**kwargs) + + Most of the time, the attributes of the object returned by :meth:`parse_args` + will be fully determined by inspecting the command-line arguments and the argument + actions. :meth:`set_defaults` allows some additional + attributes that are determined without any inspection of the command line to + be added:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('foo', type=int) + >>> parser.set_defaults(bar=42, baz='badger') + >>> parser.parse_args(['736']) + Namespace(bar=42, baz='badger', foo=736) + + Note that parser-level defaults always override argument-level defaults:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default='bar') + >>> parser.set_defaults(foo='spam') + >>> parser.parse_args([]) + Namespace(foo='spam') + + Parser-level defaults can be particularly useful when working with multiple + parsers. See the :meth:`~ArgumentParser.add_subparsers` method for an + example of this type. + +.. method:: ArgumentParser.get_default(dest) + + Get the default value for a namespace attribute, as set by either + :meth:`~ArgumentParser.add_argument` or by + :meth:`~ArgumentParser.set_defaults`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default='badger') + >>> parser.get_default('foo') + 'badger' + + +Printing help +^^^^^^^^^^^^^ + +In most typical applications, :meth:`~ArgumentParser.parse_args` will take +care of formatting and printing any usage or error messages. However, several +formatting methods are available: + +.. method:: ArgumentParser.print_usage(file=None) + + Print a brief description of how the :class:`ArgumentParser` should be + invoked on the command line. If *file* is ``None``, :data:`sys.stdout` is + assumed. + +.. method:: ArgumentParser.print_help(file=None) + + Print a help message, including the program usage and information about the + arguments registered with the :class:`ArgumentParser`. If *file* is + ``None``, :data:`sys.stdout` is assumed. + +There are also variants of these methods that simply return a string instead of +printing it: + +.. method:: ArgumentParser.format_usage() + + Return a string containing a brief description of how the + :class:`ArgumentParser` should be invoked on the command line. + +.. method:: ArgumentParser.format_help() + + Return a string containing a help message, including the program usage and + information about the arguments registered with the :class:`ArgumentParser`. + + +Partial parsing +^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.parse_known_args(args=None, namespace=None) + +Sometimes a script may only parse a few of the command-line arguments, passing +the remaining arguments on to another script or program. In these cases, the +:meth:`~ArgumentParser.parse_known_args` method can be useful. It works much like +:meth:`~ArgumentParser.parse_args` except that it does not produce an error when +extra arguments are present. Instead, it returns a two item tuple containing +the populated namespace and the list of remaining argument strings. + +:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='store_true') + >>> parser.add_argument('bar') + >>> parser.parse_known_args(['--foo', '--badger', 'BAR', 'spam']) + (Namespace(bar='BAR', foo=True), ['--badger', 'spam']) + + +Customizing file parsing +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.convert_arg_line_to_args(arg_line) + + Arguments that are read from a file (see the *fromfile_prefix_chars* + keyword argument to the :class:`ArgumentParser` constructor) are read one + argument per line. :meth:`convert_arg_line_to_args` can be overriden for + fancier reading. + + This method takes a single argument *arg_line* which is a string read from + the argument file. It returns a list of arguments parsed from this string. + The method is called once per line read from the argument file, in order. + + A useful override of this method is one that treats each space-separated word + as an argument:: + + def convert_arg_line_to_args(self, arg_line): + for arg in arg_line.split(): + if not arg.strip(): + continue + yield arg + + +Exiting methods +^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.exit(status=0, message=None) + + This method terminates the program, exiting with the specified *status* + and, if given, it prints a *message* before that. + +.. method:: ArgumentParser.error(message) + + This method prints a usage message including the *message* to the + standard error and terminates the program with a status code of 2. + +.. _upgrading-optparse-code: + +Upgrading optparse code +----------------------- + +Originally, the :mod:`argparse` module had attempted to maintain compatibility +with :mod:`optparse`. However, :mod:`optparse` was difficult to extend +transparently, particularly with the changes required to support the new +``nargs=`` specifiers and better usage messages. When most everything in +:mod:`optparse` had either been copy-pasted over or monkey-patched, it no +longer seemed practical to try to maintain the backwards compatibility. + +A partial upgrade path from :mod:`optparse` to :mod:`argparse`: + +* Replace all :meth:`optparse.OptionParser.add_option` calls with + :meth:`ArgumentParser.add_argument` calls. + +* Replace ``options, args = parser.parse_args()`` with ``args = + parser.parse_args()`` and add additional :meth:`ArgumentParser.add_argument` + calls for the positional arguments. + +* Replace callback actions and the ``callback_*`` keyword arguments with + ``type`` or ``action`` arguments. + +* Replace string names for ``type`` keyword arguments with the corresponding + type objects (e.g. int, float, complex, etc). + +* Replace :class:`optparse.Values` with :class:`Namespace` and + :exc:`optparse.OptionError` and :exc:`optparse.OptionValueError` with + :exc:`ArgumentError`. + +* Replace strings with implicit arguments such as ``%default`` or ``%prog`` with + the standard Python syntax to use dictionaries to format strings, that is, + ``%(default)s`` and ``%(prog)s``. + +* Replace the OptionParser constructor ``version`` argument with a call to + ``parser.add_argument('--version', action='version', version='<the version>')`` diff --git a/Doc/library/array.rst b/Doc/library/array.rst index 3ffc64d037..d563cceb77 100644 --- a/Doc/library/array.rst +++ b/Doc/library/array.rst @@ -60,7 +60,7 @@ The module defines the following type: appropriate type. If given a list or string, the initializer is passed to the new array's - :meth:`fromlist`, :meth:`fromstring`, or :meth:`fromunicode` method (see below) + :meth:`fromlist`, :meth:`frombytes`, or :meth:`fromunicode` method (see below) to add initial items to the array. Otherwise, the iterable initializer is passed to the :meth:`extend` method. @@ -99,7 +99,7 @@ The following data items and methods are also supported: memory buffer in bytes can be computed as ``array.buffer_info()[1] * array.itemsize``. This is occasionally useful when working with low-level (and inherently unsafe) I/O interfaces that require memory addresses, such as certain - :cfunc:`ioctl` operations. The returned numbers are valid as long as the array + :c:func:`ioctl` operations. The returned numbers are valid as long as the array exists and no length-changing operations are applied to it. .. note:: @@ -132,6 +132,15 @@ The following data items and methods are also supported: must be the right type to be appended to the array. +.. method:: array.frombytes(s) + + Appends items from the string, interpreting the string as an array of machine + values (as if it had been read from a file using the :meth:`fromfile` method). + + .. versionadded:: 3.2 + :meth:`fromstring` is renamed to :meth:`frombytes` for clarity. + + .. method:: array.fromfile(f, n) Read *n* items (as machine values) from the :term:`file object` *f* and append @@ -147,17 +156,16 @@ The following data items and methods are also supported: a.append(x)`` except that if there is a type error, the array is unchanged. -.. method:: array.fromstring(s) +.. method:: array.fromstring() - Appends items from the string, interpreting the string as an array of machine - values (as if it had been read from a file using the :meth:`fromfile` method). + Deprecated alias for :meth:`frombytes`. .. method:: array.fromunicode(s) Extends this array with data from the given unicode string. The array must be a type ``'u'`` array; otherwise a :exc:`ValueError` is raised. Use - ``array.fromstring(unicodestring.encode(enc))`` to append Unicode data to an + ``array.frombytes(unicodestring.encode(enc))`` to append Unicode data to an array of some other type. @@ -190,6 +198,16 @@ The following data items and methods are also supported: Reverse the order of the items in the array. +.. method:: array.tobytes() + + Convert the array to an array of machine values and return the bytes + representation (the same sequence of bytes that would be written to a file by + the :meth:`tofile` method.) + + .. versionadded:: 3.2 + :meth:`tostring` is renamed to :meth:`tobytes` for clarity. + + .. method:: array.tofile(f) Write all items (as machine values) to the :term:`file object` *f*. @@ -202,15 +220,13 @@ The following data items and methods are also supported: .. method:: array.tostring() - Convert the array to an array of machine values and return the string - representation (the same sequence of bytes that would be written to a file by - the :meth:`tofile` method.) + Deprecated alias for :meth:`tobytes`. .. method:: array.tounicode() Convert the array to a unicode string. The array must be a type ``'u'`` array; - otherwise a :exc:`ValueError` is raised. Use ``array.tostring().decode(enc)`` to + otherwise a :exc:`ValueError` is raised. Use ``array.tobytes().decode(enc)`` to obtain a unicode string from an array of some other type. diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index 08ee714d53..e2c0b6d17d 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -7,6 +7,9 @@ .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> .. sectionauthor:: Georg Brandl <georg@python.org> +**Source code:** :source:`Lib/ast.py` + +-------------- The :mod:`ast` module helps Python applications to process trees of the Python abstract syntax grammar. The abstract syntax itself might change with each @@ -117,12 +120,15 @@ and classes for traversing abstract syntax trees: Safely evaluate an expression node or a string containing a Python expression. The string or node provided may only consist of the following - Python literal structures: strings, numbers, tuples, lists, dicts, booleans, - and ``None``. + Python literal structures: strings, bytes, numbers, tuples, lists, dicts, + sets, booleans, and ``None``. This can be used for safely evaluating strings containing Python expressions from untrusted sources without the need to parse the values oneself. + .. versionchanged:: 3.2 + Now allows bytes and set literals. + .. function:: get_docstring(node, clean=True) diff --git a/Doc/library/asynchat.rst b/Doc/library/asynchat.rst index 3b8eb1240f..75b3cdaad3 100644 --- a/Doc/library/asynchat.rst +++ b/Doc/library/asynchat.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Sam Rushing <rushing@nightmare.com> .. sectionauthor:: Steve Holden <sholden@holdenweb.com> +**Source code:** :source:`Lib/asynchat.py` + +-------------- This module builds on the :mod:`asyncore` infrastructure, simplifying asynchronous clients and servers and making it easier to handle protocols @@ -31,7 +34,7 @@ connection requests. Like :class:`asyncore.dispatcher`, :class:`async_chat` defines a set of events that are generated by an analysis of socket conditions after a - :cfunc:`select` call. Once the polling loop has been started the + :c:func:`select` call. Once the polling loop has been started the :class:`async_chat` object's methods are called by the event-processing framework with no action on the part of the programmer. diff --git a/Doc/library/asyncore.rst b/Doc/library/asyncore.rst index b6fe2bbea2..619b7bb639 100644 --- a/Doc/library/asyncore.rst +++ b/Doc/library/asyncore.rst @@ -9,6 +9,9 @@ .. sectionauthor:: Steve Holden <sholden@holdenweb.com> .. heavily adapted from original documentation by Sam Rushing +**Source code:** :source:`Lib/asyncore.py` + +-------------- This module provides the basic infrastructure for writing asynchronous socket service clients and servers. @@ -22,7 +25,7 @@ bound. If your program is processor bound, then pre-emptive scheduled threads are probably what you really need. Network servers are rarely processor bound, however. -If your operating system supports the :cfunc:`select` system call in its I/O +If your operating system supports the :c:func:`select` system call in its I/O library (and nearly all do), then you can use it to juggle multiple communication channels at once; doing other work while your I/O is taking place in the "background." Although this strategy can seem strange and @@ -86,14 +89,14 @@ any that have been added to the map during asynchronous service) is closed. | ``handle_close()`` | Implied by a read event with no data | | | available | +----------------------+----------------------------------------+ - | ``handle_accept()`` | Implied by a read event on a listening | + | ``handle_accepted()``| Implied by a read event on a listening | | | socket | +----------------------+----------------------------------------+ During asynchronous processing, each mapped channel's :meth:`readable` and :meth:`writable` methods are used to determine whether the channel's socket - should be added to the list of channels :cfunc:`select`\ ed or - :cfunc:`poll`\ ed for read and write events. + should be added to the list of channels :c:func:`select`\ ed or + :c:func:`poll`\ ed for read and write events. Thus, the set of channel events is larger than the basic socket events. The full set of methods that can be overridden in your subclass follows: @@ -144,7 +147,21 @@ any that have been added to the map during asynchronous service) is closed. Called on listening channels (passive openers) when a connection can be established with a new remote endpoint that has issued a :meth:`connect` - call for the local endpoint. + call for the local endpoint. Deprecated in version 3.2; use + :meth:`handle_accepted` instead. + + .. deprecated:: 3.2 + + + .. method:: handle_accepted(sock, addr) + + Called on listening channels (passive openers) when a connection has been + established with a new remote endpoint that has issued a :meth:`connect` + call for the local endpoint. *sock* is a *new* socket object usable to + send and receive data on the connection, and *addr* is the address + bound to the socket on the other end of the connection. + + .. versionadded:: 3.2 .. method:: readable() @@ -201,7 +218,8 @@ any that have been added to the map during asynchronous service) is closed. .. method:: bind(address) Bind the socket to *address*. The socket must not already be bound. (The - format of *address* depends on the address family --- see above.) To mark + format of *address* depends on the address family --- refer to the + :mod:`socket` documentation for more information.) To mark the socket as re-usable (setting the :const:`SO_REUSEADDR` option), call the :class:`dispatcher` object's :meth:`set_reuse_addr` method. @@ -225,6 +243,7 @@ any that have been added to the map during asynchronous service) is closed. flushed). Sockets are automatically closed when they are garbage-collected. + .. class:: dispatcher_with_send() A :class:`dispatcher` subclass which adds simple buffered output capability, @@ -234,9 +253,9 @@ any that have been added to the map during asynchronous service) is closed. .. class:: file_dispatcher() A file_dispatcher takes a file descriptor or :term:`file object` along - with an optional map argument and wraps it for use with the :cfunc:`poll` - or :cfunc:`loop` functions. If provided a file object or anything with a - :cfunc:`fileno` method, that method will be called and passed to the + with an optional map argument and wraps it for use with the :c:func:`poll` + or :c:func:`loop` functions. If provided a file object or anything with a + :c:func:`fileno` method, that method will be called and passed to the :class:`file_wrapper` constructor. Availability: UNIX. .. class:: file_wrapper() @@ -283,15 +302,15 @@ implement its socket handling:: self.buffer = self.buffer[sent:] - client = HTTPClient('www.python.org', '/') - asyncore.loop() + client = HTTPClient('www.python.org', '/') + asyncore.loop() .. _asyncore-example-2: asyncore Example basic echo server ---------------------------------- -Here is abasic echo server that uses the :class:`dispatcher` class to accept +Here is a basic echo server that uses the :class:`dispatcher` class to accept connections and dispatches the incoming connections to a handler:: import asyncore @@ -301,7 +320,8 @@ connections and dispatches the incoming connections to a handler:: def handle_read(self): data = self.recv(8192) - self.send(data) + if data: + self.send(data) class EchoServer(asyncore.dispatcher): @@ -312,14 +332,9 @@ connections and dispatches the incoming connections to a handler:: self.bind((host, port)) self.listen(5) - def handle_accept(self): - pair = self.accept() - if pair is None: - return - else: - sock, addr = pair - print('Incoming connection from %s' % repr(addr)) - handler = EchoHandler(sock) + def handle_accepted(self, sock, addr): + print('Incoming connection from %s' % repr(addr)) + handler = EchoHandler(sock) server = EchoServer('localhost', 8080) asyncore.loop() diff --git a/Doc/library/atexit.rst b/Doc/library/atexit.rst index 104c73027d..7c76bab50d 100644 --- a/Doc/library/atexit.rst +++ b/Doc/library/atexit.rst @@ -9,7 +9,9 @@ The :mod:`atexit` module defines functions to register and unregister cleanup functions. Functions thus registered are automatically executed upon normal -interpreter termination. +interpreter termination. The order in which the functions are called is not +defined; if you have cleanup operations that depend on each other, you should +wrap them in a function and register that one. This keeps :mod:`atexit` simple. Note: the functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error @@ -20,7 +22,8 @@ is detected, or when :func:`os._exit` is called. Register *func* as a function to be executed at termination. Any optional arguments that are to be passed to *func* must be passed as arguments to - :func:`register`. + :func:`register`. It is possible to register the same function and arguments + more than once. At normal program termination (for instance, if :func:`sys.exit` is called or the main module's execution completes), all functions registered are called in @@ -33,15 +36,17 @@ is detected, or when :func:`os._exit` is called. saved. After all exit handlers have had a chance to run the last exception to be raised is re-raised. - This function returns *func* which makes it possible to use it as a decorator - without binding the original name to ``None``. + This function returns *func*, which makes it possible to use it as a + decorator. .. function:: unregister(func) - Remove a function *func* from the list of functions to be run at interpreter- + Remove *func* from the list of functions to be run at interpreter shutdown. After calling :func:`unregister`, *func* is guaranteed not to be - called when the interpreter shuts down. + called when the interpreter shuts down, even if it was registered more than + once. :func:`unregister` silently does nothing if *func* was not previously + registered. .. seealso:: @@ -96,5 +101,4 @@ Usage as a :term:`decorator`:: def goodbye(): print("You are now leaving the Python sector.") -This obviously only works with functions that don't take arguments. - +This only works with functions that can be called without arguments. diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst index c10a74ac8a..06f3ab1282 100644 --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -37,7 +37,7 @@ The modern interface provides: The encoded byte string is returned. -.. function:: b64decode(s, altchars=None) +.. function:: b64decode(s, altchars=None, validate=False) Decode a Base64 encoded byte string. @@ -45,9 +45,13 @@ The modern interface provides: at least length 2 (additional characters are ignored) which specifies the alternative alphabet used instead of the ``+`` and ``/`` characters. - The decoded byte string is returned. A :exc:`TypeError` is raised if *s* were - incorrectly padded or if there are non-alphabet characters present in the - string. + The decoded string is returned. A :exc:`binascii.Error` exception is raised + if *s* is incorrectly padded. + + If *validate* is ``False`` (the default), non-base64-alphabet characters are + discarded prior to the padding check. If *validate* is ``True``, + non-base64-alphabet characters in the input result in a + :exc:`binascii.Error`. .. function:: standard_b64encode(s) diff --git a/Doc/library/bdb.rst b/Doc/library/bdb.rst index 4795ec943b..07376024b8 100644 --- a/Doc/library/bdb.rst +++ b/Doc/library/bdb.rst @@ -4,6 +4,10 @@ .. module:: bdb :synopsis: Debugger framework. +**Source code:** :source:`Lib/bdb.py` + +-------------- + The :mod:`bdb` module handles basic debugger functions, like setting breakpoints or managing execution via the debugger. @@ -50,9 +54,10 @@ The :mod:`bdb` module also defines two classes: Mark the breakpoint as disabled. - .. method:: bpprint(out=None) + .. method:: bpformat() - Print all the information about the breakpoint: + Return a string with all the information about the breakpoint, nicely + formatted: * The breakpoint number. * If it is temporary or not. @@ -61,6 +66,13 @@ The :mod:`bdb` module also defines two classes: * If it must be ignored the next N times. * The breakpoint hit count. + .. versionadded:: 3.2 + + .. method:: bpprint(out=None) + + Print the output of :meth:`bpformat` to the file *out*, or if it is + ``None``, to standard output. + .. class:: Bdb(skip=None) @@ -267,6 +279,15 @@ The :mod:`bdb` module also defines two classes: Delete all existing breakpoints. + .. method:: get_bpbynumber(arg) + + Return a breakpoint specified by the given number. If *arg* is a string, + it will be converted to a number. If *arg* is a non-numeric string, if + the given breakpoint never existed or has been deleted, a + :exc:`ValueError` is raised. + + .. versionadded:: 3.2 + .. method:: get_break(filename, lineno) Check if there is a breakpoint for *lineno* of *filename*. diff --git a/Doc/library/binascii.rst b/Doc/library/binascii.rst index 2f7851a0f9..2aa3702cfe 100644 --- a/Doc/library/binascii.rst +++ b/Doc/library/binascii.rst @@ -18,6 +18,11 @@ use these functions directly but use wrapper modules like :mod:`uu`, low-level functions written in C for greater speed that are used by the higher-level modules. +.. note:: + + Encoding and decoding functions do not accept Unicode strings. Only bytestring + and bytearray objects can be processed. + The :mod:`binascii` module defines the following functions: @@ -54,6 +59,9 @@ The :mod:`binascii` module defines the following functions: data. More than one line may be passed at a time. If the optional argument *header* is present and true, underscores will be decoded as spaces. + .. versionchanged:: 3.2 + Accept only bytestring or bytearray objects as input. + .. function:: b2a_qp(data, quotetabs=False, istext=True, header=False) @@ -83,6 +91,9 @@ The :mod:`binascii` module defines the following functions: decompressed data, unless data input data ends in an orphaned repeat indicator, in which case the :exc:`Incomplete` exception is raised. + .. versionchanged:: 3.2 + Accept only bytestring or bytearray objects as input. + .. function:: rlecode_hqx(data) @@ -139,6 +150,9 @@ The :mod:`binascii` module defines the following functions: of hexadecimal digits (which can be upper or lower case), otherwise a :exc:`TypeError` is raised. + .. versionchanged:: 3.2 + Accept only bytestring or bytearray objects as input. + .. exception:: Error @@ -164,4 +178,3 @@ The :mod:`binascii` module defines the following functions: Module :mod:`quopri` Support for quoted-printable encoding used in MIME email messages. - diff --git a/Doc/library/bisect.rst b/Doc/library/bisect.rst index eb231598ed..13b0147190 100644 --- a/Doc/library/bisect.rst +++ b/Doc/library/bisect.rst @@ -7,6 +7,10 @@ .. sectionauthor:: Raymond Hettinger <python at rcn.com> .. example based on the PyModules FAQ entry by Aaron Watters <arw@pythonpros.com> +**Source code:** :source:`Lib/bisect.py` + +-------------- + This module provides support for maintaining a list in sorted order without having to sort the list after each insertion. For long lists of items with expensive comparison operations, this can be an improvement over the more common diff --git a/Doc/library/builtins.rst b/Doc/library/builtins.rst index f0d2a60a3c..2cca1d05df 100644 --- a/Doc/library/builtins.rst +++ b/Doc/library/builtins.rst @@ -7,7 +7,9 @@ This module provides direct access to all 'built-in' identifiers of Python; for example, ``builtins.open`` is the full name for the built-in function -:func:`open`. See chapter :ref:`builtin`. +:func:`open`. See :ref:`built-in-funcs` and :ref:`built-in-consts` for +documentation. + This module is not normally accessed explicitly by most applications, but can be useful in modules that provide objects with the same name as a built-in value, @@ -34,6 +36,6 @@ that wants to implement an :func:`open` function that wraps the built-in As an implementation detail, most modules have the name ``__builtins__`` made available as part of their globals. The value of ``__builtins__`` is normally -either this module or the value of this modules's :attr:`__dict__` attribute. +either this module or the value of this module's :attr:`__dict__` attribute. Since this is an implementation detail, it may not be used by alternate implementations of Python. diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index d9a2bad756..d13f6e0910 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -12,9 +12,6 @@ This module provides a comprehensive interface for the bz2 compression library. It implements a complete file interface, one-shot (de)compression functions, and types for sequential (de)compression. -For other archive formats, see the :mod:`gzip`, :mod:`zipfile`, and -:mod:`tarfile` modules. - Here is a summary of the features offered by the bz2 module: * :class:`BZ2File` class implements a complete file interface, including @@ -65,6 +62,18 @@ Handling of compressed files is offered by the :class:`BZ2File` class. Support for the :keyword:`with` statement was added. + .. note:: + + This class does not support input files containing multiple streams (such + as those produced by the :program:`pbzip2` tool). When reading such an + input file, only the first stream will be accessible. If you require + support for multi-stream files, consider using the third-party + :mod:`bz2file` module (available from + `PyPI <http://pypi.python.org/pypi/bz2file>`_). This module provides a + backport of Python 3.3's :class:`BZ2File` class, which does support + multi-stream files. + + .. method:: close() Close the file. Sets data attribute :attr:`closed` to true. A closed file diff --git a/Doc/library/calendar.rst b/Doc/library/calendar.rst index c8dac49e3c..f4952711cb 100644 --- a/Doc/library/calendar.rst +++ b/Doc/library/calendar.rst @@ -6,6 +6,9 @@ of the Unix cal program. .. sectionauthor:: Drew Csillag <drew_csillag@geocities.com> +**Source code:** :source:`Lib/calendar.py` + +-------------- This module allows you to output calendars like the Unix :program:`cal` program, and provides additional useful functions related to the calendar. By default, @@ -16,7 +19,7 @@ are given as integers. For related functionality, see also the :mod:`datetime` and :mod:`time` modules. Most of these functions and classes rely on the :mod:`datetime` module which -uses an idealized calendar, the current Gregorian calendar indefinitely extended +uses an idealized calendar, the current Gregorian calendar extended in both directions. This matches the definition of the "proleptic Gregorian" calendar in Dershowitz and Reingold's book "Calendrical Calculations", where it's the base calendar for all computations. @@ -309,4 +312,3 @@ The :mod:`calendar` module exports the following data attributes: Module :mod:`time` Low-level time related functions. - diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst index 734031dd09..1e2498d460 100644 --- a/Doc/library/cgi.rst +++ b/Doc/library/cgi.rst @@ -13,6 +13,10 @@ single: URL single: Common Gateway Interface +**Source code:** :source:`Lib/cgi.py` + +-------------- + Support module for Common Gateway Interface (CGI) scripts. This module defines a number of utilities for use by CGI scripts written in @@ -324,15 +328,13 @@ algorithms implemented in this module in other circumstances. Convert the characters ``'&'``, ``'<'`` and ``'>'`` in string *s* to HTML-safe sequences. Use this if you need to display text that might contain such characters in HTML. If the optional flag *quote* is true, the quotation mark - character (``'"'``) is also translated; this helps for inclusion in an HTML - attribute value, as in ``<A HREF="...">``. If the value to be quoted might - include single- or double-quote characters, or both, consider using the - :func:`~xml.sax.saxutils.quoteattr` function in the :mod:`xml.sax.saxutils` - module instead. + character (``"``) is also translated; this helps for inclusion in an HTML + attribute value delimited by double quotes, as in ``<a href="...">``. Note + that single quotes are never translated. - If the value to be quoted might include single- or double-quote characters, - or both, consider using the :func:`quoteattr` function in the - :mod:`xml.sax.saxutils` module instead. + .. deprecated:: 3.2 + This function is unsafe because *quote* is false by default, and therefore + deprecated. Use :func:`html.escape` instead. .. _cgi-security: @@ -510,8 +512,8 @@ Common problems and solutions .. rubric:: Footnotes -.. [#] Note that some recent versions of the HTML specification do state what order the - field values should be supplied in, but knowing whether a request was - received from a conforming browser, or even from a browser at all, is tedious - and error-prone. +.. [#] Note that some recent versions of the HTML specification do state what + order the field values should be supplied in, but knowing whether a request + was received from a conforming browser, or even from a browser at all, is + tedious and error-prone. diff --git a/Doc/library/cmath.rst b/Doc/library/cmath.rst index 14b909bd28..d7778df317 100644 --- a/Doc/library/cmath.rst +++ b/Doc/library/cmath.rst @@ -187,15 +187,24 @@ Hyperbolic functions Classification functions ------------------------ +.. function:: isfinite(x) + + Return ``True`` if both the real and imaginary parts of *x* are finite, and + ``False`` otherwise. + + .. versionadded:: 3.2 + + .. function:: isinf(x) - Return *True* if the real or the imaginary part of x is positive - or negative infinity. + Return ``True`` if either the real or the imaginary part of *x* is an + infinity, and ``False`` otherwise. .. function:: isnan(x) - Return *True* if the real or imaginary part of x is not a number (NaN). + Return ``True`` if either the real or the imaginary part of *x* is a NaN, + and ``False`` otherwise. Constants diff --git a/Doc/library/cmd.rst b/Doc/library/cmd.rst index d789270eb9..fd7f4537c6 100644 --- a/Doc/library/cmd.rst +++ b/Doc/library/cmd.rst @@ -5,13 +5,15 @@ :synopsis: Build line-oriented command interpreters. .. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com> +**Source code:** :source:`Lib/cmd.py` + +-------------- The :class:`Cmd` class provides a simple framework for writing line-oriented command interpreters. These are often useful for test harnesses, administrative tools, and prototypes that will later be wrapped in a more sophisticated interface. - .. class:: Cmd(completekey='tab', stdin=None, stdout=None) A :class:`Cmd` instance or subclass instance is a line-oriented interpreter @@ -48,7 +50,7 @@ A :class:`Cmd` instance has the following methods: the line as argument. The optional argument is a banner or intro string to be issued before the first - prompt (this overrides the :attr:`intro` class member). + prompt (this overrides the :attr:`intro` class attribute). If the :mod:`readline` module is loaded, input will automatically inherit :program:`bash`\ -like history-list editing (e.g. :kbd:`Control-P` scrolls back @@ -203,3 +205,164 @@ Instances of :class:`Cmd` subclasses have some public instance variables: :mod:`readline`, on systems that support it, the interpreter will automatically support :program:`Emacs`\ -like line editing and command-history keystrokes.) + +.. _cmd-example: + +Cmd Example +----------- + +.. sectionauthor:: Raymond Hettinger <python at rcn dot com> + +The :mod:`cmd` module is mainly useful for building custom shells that let a +user work with a program interactively. + +This section presents a simple example of how to build a shell around a few of +the commands in the :mod:`turtle` module. + +Basic turtle commands such as :meth:`~turtle.forward` are added to a +:class:`Cmd` subclass with method named :meth:`do_forward`. The argument is +converted to a number and dispatched to the turtle module. The docstring is +used in the help utility provided by the shell. + +The example also includes a basic record and playback facility implemented with +the :meth:`~Cmd.precmd` method which is responsible for converting the input to +lowercase and writing the commands to a file. The :meth:`do_playback` method +reads the file and adds the recorded commands to the :attr:`cmdqueue` for +immediate playback:: + + import cmd, sys + from turtle import * + + class TurtleShell(cmd.Cmd): + intro = 'Welcome to the turtle shell. Type help or ? to list commands.\n' + prompt = '(turtle) ' + file = None + + # ----- basic turtle commands ----- + def do_forward(self, arg): + 'Move the turtle forward by the specified distance: FORWARD 10' + forward(*parse(arg)) + def do_right(self, arg): + 'Turn turtle right by given number of degrees: RIGHT 20' + right(*parse(arg)) + def do_left(self, arg): + 'Turn turtle left by given number of degrees: LEFT 90' + left(*parse(arg)) + def do_goto(self, arg): + 'Move turtle to an absolute position with changing orientation. GOTO 100 200' + goto(*parse(arg)) + def do_home(self, arg): + 'Return turtle to the home postion: HOME' + home() + def do_circle(self, arg): + 'Draw circle with given radius an options extent and steps: CIRCLE 50' + circle(*parse(arg)) + def do_position(self, arg): + 'Print the current turle position: POSITION' + print('Current position is %d %d\n' % position()) + def do_heading(self, arg): + 'Print the current turle heading in degrees: HEADING' + print('Current heading is %d\n' % (heading(),)) + def do_color(self, arg): + 'Set the color: COLOR BLUE' + color(arg.lower()) + def do_undo(self, arg): + 'Undo (repeatedly) the last turtle action(s): UNDO' + def do_reset(self, arg): + 'Clear the screen and return turtle to center: RESET' + reset() + def do_bye(self, arg): + 'Stop recording, close the turtle window, and exit: BYE' + print('Thank you for using Turtle') + self.close() + bye() + sys.exit(0) + + # ----- record and playback ----- + def do_record(self, arg): + 'Save future commands to filename: RECORD rose.cmd' + self.file = open(arg, 'w') + def do_playback(self, arg): + 'Playback commands from a file: PLAYBACK rose.cmd' + self.close() + cmds = open(arg).read().splitlines() + self.cmdqueue.extend(cmds) + def precmd(self, line): + line = line.lower() + if self.file and 'playback' not in line: + print(line, file=self.file) + return line + def close(self): + if self.file: + self.file.close() + self.file = None + + def parse(arg): + 'Convert a series of zero or more numbers to an argument tuple' + return tuple(map(int, arg.split())) + + if __name__ == '__main__': + TurtleShell().cmdloop() + + +Here is a sample session with the turtle shell showing the help functions, using +blank lines to repeat commands, and the simple record and playback facility:: + + Welcome to the turtle shell. Type help or ? to list commands. + + (turtle) ? + + Documented commands (type help <topic>): + ======================================== + bye color goto home playback record right + circle forward heading left position reset undo + + (turtle) help forward + Move the turtle forward by the specified distance: FORWARD 10 + (turtle) record spiral.cmd + (turtle) position + Current position is 0 0 + + (turtle) heading + Current heading is 0 + + (turtle) reset + (turtle) circle 20 + (turtle) right 30 + (turtle) circle 40 + (turtle) right 30 + (turtle) circle 60 + (turtle) right 30 + (turtle) circle 80 + (turtle) right 30 + (turtle) circle 100 + (turtle) right 30 + (turtle) circle 120 + (turtle) right 30 + (turtle) circle 120 + (turtle) heading + Current heading is 180 + + (turtle) forward 100 + (turtle) + (turtle) right 90 + (turtle) forward 100 + (turtle) + (turtle) right 90 + (turtle) forward 400 + (turtle) right 90 + (turtle) forward 500 + (turtle) right 90 + (turtle) forward 400 + (turtle) right 90 + (turtle) forward 300 + (turtle) playback spiral.cmd + Current position is 0 0 + + Current heading is 0 + + Current heading is 180 + + (turtle) bye + Thank you for using Turtle + diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index d07dafaa98..7747794091 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -787,9 +787,9 @@ Encodings and Unicode --------------------- Strings are stored internally as sequences of codepoints (to be precise -as :ctype:`Py_UNICODE` arrays). Depending on the way Python is compiled (either +as :c:type:`Py_UNICODE` arrays). Depending on the way Python is compiled (either via ``--without-wide-unicode`` or ``--with-wide-unicode``, with the -former being the default) :ctype:`Py_UNICODE` is either a 16-bit or 32-bit data +former being the default) :c:type:`Py_UNICODE` is either a 16-bit or 32-bit data type. Once a string object is used outside of CPU and memory, CPU endianness and how these arrays are stored as bytes become an issue. Transforming a string object into a sequence of bytes is called encoding and recreating the @@ -810,27 +810,28 @@ e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on Windows). There's a string constant with 256 characters that shows you which character is mapped to which byte value. -All of these encodings can only encode 256 of the 65536 (or 1114111) codepoints +All of these encodings can only encode 256 of the 1114112 codepoints defined in Unicode. A simple and straightforward way that can store each Unicode -code point, is to store each codepoint as two consecutive bytes. There are two -possibilities: Store the bytes in big endian or in little endian order. These -two encodings are called UTF-16-BE and UTF-16-LE respectively. Their -disadvantage is that if e.g. you use UTF-16-BE on a little endian machine you -will always have to swap bytes on encoding and decoding. UTF-16 avoids this -problem: Bytes will always be in natural endianness. When these bytes are read +code point, is to store each codepoint as four consecutive bytes. There are two +possibilities: store the bytes in big endian or in little endian order. These +two encodings are called ``UTF-32-BE`` and ``UTF-32-LE`` respectively. Their +disadvantage is that if e.g. you use ``UTF-32-BE`` on a little endian machine you +will always have to swap bytes on encoding and decoding. ``UTF-32`` avoids this +problem: bytes will always be in natural endianness. When these bytes are read by a CPU with a different endianness, then bytes have to be swapped though. To -be able to detect the endianness of a UTF-16 byte sequence, there's the so -called BOM (the "Byte Order Mark"). This is the Unicode character ``U+FEFF``. -This character will be prepended to every UTF-16 byte sequence. The byte swapped -version of this character (``0xFFFE``) is an illegal character that may not -appear in a Unicode text. So when the first character in an UTF-16 byte sequence +be able to detect the endianness of a ``UTF-16`` or ``UTF-32`` byte sequence, +there's the so called BOM ("Byte Order Mark"). This is the Unicode character +``U+FEFF``. This character can be prepended to every ``UTF-16`` or ``UTF-32`` +byte sequence. The byte swapped version of this character (``0xFFFE``) is an +illegal character that may not appear in a Unicode text. So when the +first character in an ``UTF-16`` or ``UTF-32`` byte sequence appears to be a ``U+FFFE`` the bytes have to be swapped on decoding. -Unfortunately upto Unicode 4.0 the character ``U+FEFF`` had a second purpose as -a ``ZERO WIDTH NO-BREAK SPACE``: A character that has no width and doesn't allow +Unfortunately the character ``U+FEFF`` had a second purpose as +a ``ZERO WIDTH NO-BREAK SPACE``: a character that has no width and doesn't allow a word to be split. It can e.g. be used to give hints to a ligature algorithm. With Unicode 4.0 using ``U+FEFF`` as a ``ZERO WIDTH NO-BREAK SPACE`` has been deprecated (with ``U+2060`` (``WORD JOINER``) assuming this role). Nevertheless -Unicode software still must be able to handle ``U+FEFF`` in both roles: As a BOM +Unicode software still must be able to handle ``U+FEFF`` in both roles: as a BOM it's a device to determine the storage layout of the encoded bytes, and vanishes once the byte sequence has been decoded into a string; as a ``ZERO WIDTH NO-BREAK SPACE`` it's a normal character that will be decoded like any other. @@ -838,8 +839,8 @@ NO-BREAK SPACE`` it's a normal character that will be decoded like any other. There's another encoding that is able to encoding the full range of Unicode characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two -parts: Marker bits (the most significant bits) and payload bits. The marker bits -are a sequence of zero to six 1 bits followed by a 0 bit. Unicode characters are +parts: marker bits (the most significant bits) and payload bits. The marker bits +are a sequence of zero to four ``1`` bits followed by a ``0`` bit. Unicode characters are encoded like this (with x being payload bits, which when concatenated give the Unicode character): @@ -852,12 +853,7 @@ Unicode character): +-----------------------------------+----------------------------------------------+ | ``U-00000800`` ... ``U-0000FFFF`` | 1110xxxx 10xxxxxx 10xxxxxx | +-----------------------------------+----------------------------------------------+ -| ``U-00010000`` ... ``U-001FFFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx | -+-----------------------------------+----------------------------------------------+ -| ``U-00200000`` ... ``U-03FFFFFF`` | 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx | -+-----------------------------------+----------------------------------------------+ -| ``U-04000000`` ... ``U-7FFFFFFF`` | 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx | -| | 10xxxxxx | +| ``U-00010000`` ... ``U-0010FFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx | +-----------------------------------+----------------------------------------------+ The least significant bit of the Unicode character is the rightmost x bit. @@ -882,13 +878,14 @@ map to | RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK | INVERTED QUESTION MARK -in iso-8859-1), this increases the probability that a utf-8-sig encoding can be +in iso-8859-1), this increases the probability that a ``utf-8-sig`` encoding can be correctly guessed from the byte sequence. So here the BOM is not used to be able to determine the byte order used for generating the byte sequence, but as a signature that helps in guessing the encoding. On encoding the utf-8-sig codec will write ``0xef``, ``0xbb``, ``0xbf`` as the first three bytes to the file. On -decoding utf-8-sig will skip those three bytes if they appear as the first three -bytes in the file. +decoding ``utf-8-sig`` will skip those three bytes if they appear as the first +three bytes in the file. In UTF-8, the use of the BOM is discouraged and +should generally be avoided. .. _standard-encodings: @@ -936,6 +933,8 @@ particular, the following variants typically exist: | cp500 | EBCDIC-CP-BE, EBCDIC-CP-CH, | Western Europe | | | IBM500 | | +-----------------+--------------------------------+--------------------------------+ +| cp720 | | Arabic | ++-----------------+--------------------------------+--------------------------------+ | cp737 | | Greek | +-----------------+--------------------------------+--------------------------------+ | cp775 | IBM775 | Baltic languages | @@ -951,6 +950,8 @@ particular, the following variants typically exist: +-----------------+--------------------------------+--------------------------------+ | cp857 | 857, IBM857 | Turkish | +-----------------+--------------------------------+--------------------------------+ +| cp858 | 858, IBM858 | Western Europe | ++-----------------+--------------------------------+--------------------------------+ | cp860 | 860, IBM860 | Portuguese | +-----------------+--------------------------------+--------------------------------+ | cp861 | 861, CP-IS, IBM861 | Icelandic | @@ -1086,7 +1087,7 @@ particular, the following variants typically exist: +-----------------+--------------------------------+--------------------------------+ | mac_latin2 | maclatin2, maccentraleurope | Central and Eastern Europe | +-----------------+--------------------------------+--------------------------------+ -| mac_roman | macroman | Western Europe | +| mac_roman | macroman, macintosh | Western Europe | +-----------------+--------------------------------+--------------------------------+ | mac_turkish | macturkish | Turkish | +-----------------+--------------------------------+--------------------------------+ @@ -1110,9 +1111,9 @@ particular, the following variants typically exist: +-----------------+--------------------------------+--------------------------------+ | utf_16 | U16, utf16 | all languages | +-----------------+--------------------------------+--------------------------------+ -| utf_16_be | UTF-16BE | all languages (BMP only) | +| utf_16_be | UTF-16BE | all languages | +-----------------+--------------------------------+--------------------------------+ -| utf_16_le | UTF-16LE | all languages (BMP only) | +| utf_16_le | UTF-16LE | all languages | +-----------------+--------------------------------+--------------------------------+ | utf_7 | U7, unicode-1-1-utf-7 | all languages | +-----------------+--------------------------------+--------------------------------+ @@ -1161,6 +1162,44 @@ particular, the following variants typically exist: | | | operand | +--------------------+---------+---------------------------+ +The following codecs provide bytes-to-bytes mappings. + ++--------------------+---------------------------+---------------------------+ +| Codec | Aliases | Purpose | ++====================+===========================+===========================+ +| base64_codec | base64, base-64 | Convert operand to MIME | +| | | base64 | ++--------------------+---------------------------+---------------------------+ +| bz2_codec | bz2 | Compress the operand | +| | | using bz2 | ++--------------------+---------------------------+---------------------------+ +| hex_codec | hex | Convert operand to | +| | | hexadecimal | +| | | representation, with two | +| | | digits per byte | ++--------------------+---------------------------+---------------------------+ +| quopri_codec | quopri, quoted-printable, | Convert operand to MIME | +| | quotedprintable | quoted printable | ++--------------------+---------------------------+---------------------------+ +| uu_codec | uu | Convert the operand using | +| | | uuencode | ++--------------------+---------------------------+---------------------------+ +| zlib_codec | zip, zlib | Compress the operand | +| | | using gzip | ++--------------------+---------------------------+---------------------------+ + +The following codecs provide string-to-string mappings. + ++--------------------+---------------------------+---------------------------+ +| Codec | Aliases | Purpose | ++====================+===========================+===========================+ +| rot_13 | rot13 | Returns the Caesar-cypher | +| | | encryption of the operand | ++--------------------+---------------------------+---------------------------+ + +.. versionadded:: 3.2 + bytes-to-bytes and string-to-string codecs. + :mod:`encodings.idna` --- Internationalized Domain Names in Applications ------------------------------------------------------------------------ @@ -1227,6 +1266,23 @@ functions can be used directly if desired. Convert a label to Unicode, as specified in :rfc:`3490`. +:mod:`encodings.mbcs` --- Windows ANSI codepage +----------------------------------------------- + +.. module:: encodings.mbcs + :synopsis: Windows ANSI codepage + +Encode operand according to the ANSI codepage (CP_ACP). This codec only +supports ``'strict'`` and ``'replace'`` error handlers to encode, and +``'strict'`` and ``'ignore'`` error handlers to decode. + +Availability: Windows only. + +.. versionchanged:: 3.2 + Before 3.2, the *errors* argument was ignored; ``'replace'`` was always used + to encode, and ``'ignore'`` to decode. + + :mod:`encodings.utf_8_sig` --- UTF-8 codec with BOM signature ------------------------------------------------------------- diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 68a79f12d0..e90b25e870 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -1,4 +1,3 @@ - :mod:`collections` --- Container datatypes ========================================== @@ -13,6 +12,10 @@ import itertools __name__ = '<doctest>' +**Source code:** :source:`Lib/collections.py` and :source:`Lib/_abcoll.py` + +-------------- + This module implements specialized container datatypes providing alternatives to Python's general purpose built-in containers, :class:`dict`, :class:`list`, :class:`set`, and :class:`tuple`. @@ -29,8 +32,9 @@ Python's general purpose built-in containers, :class:`dict`, :class:`list`, ===================== ==================================================================== In addition to the concrete container classes, the collections module provides -ABCs (abstract base classes) that can be used to test whether a class provides a -particular interface, for example, whether it is hashable or a mapping. +:ref:`abstract base classes <collections-abstract-base-classes>` that can be +used to test whether a class provides a particular interface, for example, +whether it is hashable or a mapping. :class:`Counter` objects @@ -85,7 +89,7 @@ For example:: .. versionadded:: 3.1 - Counter objects support two methods beyond those available for all + Counter objects support three methods beyond those available for all dictionaries: .. method:: elements() @@ -108,6 +112,19 @@ For example:: >>> Counter('abracadabra').most_common(3) [('a', 5), ('r', 2), ('b', 2)] + .. method:: subtract([iterable-or-mapping]) + + Elements are subtracted from an *iterable* or from another *mapping* + (or counter). Like :meth:`dict.update` but subtracts counts instead + of replacing them. Both inputs and outputs may be zero or negative. + + >>> c = Counter(a=4, b=2, c=0, d=-2) + >>> d = Counter(a=1, b=2, c=3, d=4) + >>> c.subtract(d) + Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6}) + + .. versionadded:: 3.2 + The usual dictionary methods are available for :class:`Counter` objects except for two which work differently for counters. @@ -174,7 +191,7 @@ counts, but the output will exclude results with counts of zero or less. * The multiset methods are designed only for use cases with positive values. The inputs may be negative or zero, but only outputs with positive values are created. There are no type restrictions, but the value type needs to - support support addition, subtraction, and comparison. + support addition, subtraction, and comparison. * The :meth:`elements` method requires integer counts. It ignores zero and negative counts. @@ -188,14 +205,14 @@ counts, but the output will exclude results with counts of zero or less. * `Bag class <http://www.gnu.org/software/smalltalk/manual-base/html_node/Bag.html>`_ in Smalltalk. - * Wikipedia entry for `Multisets <http://en.wikipedia.org/wiki/Multiset>`_\. + * Wikipedia entry for `Multisets <http://en.wikipedia.org/wiki/Multiset>`_. * `C++ multisets <http://www.demo2s.com/Tutorial/Cpp/0380__set-multiset/Catalog0380__set-multiset.htm>`_ tutorial with examples. * For mathematical operations on multisets and their use cases, see *Knuth, Donald. The Art of Computer Programming Volume II, - Section 4.6.3, Exercise 19*\. + Section 4.6.3, Exercise 19*. * To enumerate all distinct multisets of a given size over a given set of elements, see :func:`itertools.combinations_with_replacement`. @@ -248,6 +265,13 @@ counts, but the output will exclude results with counts of zero or less. Remove all elements from the deque leaving it with length 0. + .. method:: count(x) + + Count the number of deque elements equal to *x*. + + .. versionadded:: 3.2 + + .. method:: extend(iterable) Extend the right side of the deque by appending elements from the iterable @@ -279,6 +303,13 @@ counts, but the output will exclude results with counts of zero or less. :exc:`ValueError`. + .. method:: reverse() + + Reverse the elements of the deque in-place and then return ``None``. + + .. versionadded:: 3.2 + + .. method:: rotate(n) Rotate the deque *n* steps to the right. If *n* is negative, rotate to @@ -437,6 +468,11 @@ stack manipulations such as ``dup``, ``drop``, ``swap``, ``over``, ``pick``, :class:`dict` class when the requested key is not found; whatever it returns or raises is then returned or raised by :meth:`__getitem__`. + Note that :meth:`__missing__` is *not* called for any operations besides + :meth:`__getitem__`. This means that :meth:`get` will, like normal + dictionaries, return ``None`` as a default rather than using + :attr:`default_factory`. + :class:`defaultdict` objects support the following instance variable: @@ -555,11 +591,15 @@ they add the ability to access fields by name instead of position index. .. versionchanged:: 3.1 Added support for *rename*. -Example: .. doctest:: :options: +NORMALIZE_WHITESPACE + >>> # Basic example + >>> Point = namedtuple('Point', ['x', 'y']) + >>> p = Point(x=10, y=11) + + >>> # Example using the verbose option to print the class definition >>> Point = namedtuple('Point', 'x y', verbose=True) class Point(tuple): 'Point(x, y)' @@ -569,6 +609,7 @@ Example: _fields = ('x', 'y') <BLANKLINE> def __new__(_cls, x, y): + 'Create a new instance of Point(x, y)' return _tuple.__new__(_cls, (x, y)) <BLANKLINE> @classmethod @@ -580,12 +621,15 @@ Example: return result <BLANKLINE> def __repr__(self): - return 'Point(x=%r, y=%r)' % self + 'Return a nicely formatted representation string' + return self.__class__.__name__ + '(x=%r, y=%r)' % self <BLANKLINE> def _asdict(self): 'Return a new OrderedDict which maps field names to their values' return OrderedDict(zip(self._fields, self)) <BLANKLINE> + __dict__ = property(_asdict) + <BLANKLINE> def _replace(_self, **kwds): 'Return a new Point object replacing specified fields with new values' result = _self._make(map(kwds.pop, ('x', 'y'), _self)) @@ -594,10 +638,11 @@ Example: return result <BLANKLINE> def __getnewargs__(self): + 'Return self as a plain tuple. Used by copy and pickle.' return tuple(self) <BLANKLINE> - x = _property(_itemgetter(0)) - y = _property(_itemgetter(1)) + x = _property(_itemgetter(0), doc='Alias for field number 0') + y = _property(_itemgetter(1), doc='Alias for field number 1') >>> p = Point(11, y=22) # instantiate with positional or keyword arguments >>> p[0] + p[1] # indexable like the plain tuple (11, 22) @@ -698,15 +743,15 @@ functionality with a subclass. Here is how to add a calculated field and a fixed-width print format: >>> class Point(namedtuple('Point', 'x y')): - ... __slots__ = () - ... @property - ... def hypot(self): - ... return (self.x ** 2 + self.y ** 2) ** 0.5 - ... def __str__(self): - ... return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot) + __slots__ = () + @property + def hypot(self): + return (self.x ** 2 + self.y ** 2) ** 0.5 + def __str__(self): + return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot) >>> for p in Point(3, 4), Point(14, 5/7): - ... print(p) + print(p) Point: x= 3.000 y= 4.000 hypot= 5.000 Point: x=14.000 y= 0.714 hypot=14.018 @@ -733,12 +778,19 @@ and more efficient to use a simple class declaration: >>> Status.open, Status.pending, Status.closed (0, 1, 2) >>> class Status: - ... open, pending, closed = range(3) + open, pending, closed = range(3) .. seealso:: - `Named tuple recipe <http://code.activestate.com/recipes/500261/>`_ - adapted for Python 2.4. + * `Named tuple recipe <http://code.activestate.com/recipes/500261/>`_ + adapted for Python 2.4. + + * `Recipe for named tuple abstract base class with a metaclass mix-in + <http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/>`_ + by Jan Kaliszewski. Besides providing an :term:`abstract base class` for + named tuples, it also supports an alternate :term:`metaclass`-based + constructor that is convenient for use cases where named tuples are being + subclassed. :class:`OrderedDict` objects @@ -764,6 +816,23 @@ the items are returned in the order their keys were first added. (key, value) pair. The pairs are returned in LIFO order if *last* is true or FIFO order if false. + .. method:: move_to_end(key, last=True) + + Move an existing *key* to either end of an ordered dictionary. The item + is moved to the right end if *last* is true (the default) or to the + beginning if *last* is false. Raises :exc:`KeyError` if the *key* does + not exist:: + + >>> d = OrderedDict.fromkeys('abcde') + >>> d.move_to_end('b') + >>> ''.join(d.keys()) + 'acdeb' + >>> d.move_to_end('b', last=False) + >>> ''.join(d.keys()) + 'bacde' + + .. versionadded:: 3.2 + In addition to the usual mapping methods, ordered dictionaries also support reverse iteration using :func:`reversed`. @@ -821,16 +890,12 @@ original insertion position is changed and moved to the end:: del self[key] OrderedDict.__setitem__(self, key, value) -An ordered dictionary can combined with the :class:`Counter` class +An ordered dictionary can be combined with the :class:`Counter` class so that the counter remembers the order elements are first encountered:: class OrderedCounter(Counter, OrderedDict): 'Counter that remembers the order elements are first encountered' - def __init__(self, iterable=None, **kwds): - OrderedDict.__init__(self) - Counter.__init__(self, iterable, **kwds) - def __repr__(self): return '%s(%r)' % (self.__class__.__name__, OrderedDict(self)) @@ -864,6 +929,7 @@ attribute. class. + :class:`UserList` objects ------------------------- @@ -924,6 +990,8 @@ attribute. the built-in :func:`str` function. +.. _collections-abstract-base-classes: + ABCs - abstract base classes ---------------------------- @@ -939,7 +1007,7 @@ ABC Inherits from Abstract Methods Mixin :class:`Sized` ``__len__`` :class:`Callable` ``__call__`` -:class:`Sequence` :class:`Sized`, ``__getitem__`` ``__contains__``. ``__iter__``, ``__reversed__``, +:class:`Sequence` :class:`Sized`, ``__getitem__`` ``__contains__``, ``__iter__``, ``__reversed__``, :class:`Iterable`, ``index``, and ``count`` :class:`Container` diff --git a/Doc/library/colorsys.rst b/Doc/library/colorsys.rst index 2cbc704db1..dbab7063e1 100644 --- a/Doc/library/colorsys.rst +++ b/Doc/library/colorsys.rst @@ -5,6 +5,9 @@ :synopsis: Conversion functions between RGB and other color systems. .. sectionauthor:: David Ascher <da@python.net> +**Source code:** :source:`Lib/colorsys.py` + +-------------- The :mod:`colorsys` module defines bidirectional conversions of color values between colors expressed in the RGB (Red Green Blue) color space used in diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 8a93f9b719..cb7a09c0fb 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -52,11 +52,30 @@ compile Python sources. regex is used to search the full path to each file considered for compilation, and if the regex produces a match, the file is skipped. +.. cmdoption:: -i list + + Read the file ``list`` and add each line that it contains to the list of + files and directories to compile. If ``list`` is ``-``, read lines from + ``stdin``. + +.. cmdoption:: -b + + Write the byte-code files to their legacy locations and names, which may + overwrite byte-code files created by another version of Python. The default + is to write files to their :pep:`3147` locations and names, which allows + byte-code files from multiple versions of Python to coexist. + +.. versionchanged:: 3.2 + Added the ``-i``, ``-b`` and ``-h`` options. + +There is no command-line option to control the optimization level used by the +:func:`compile` function, because the Python interpreter itself already +provides the option: :program:`python -O -m compileall`. Public functions ---------------- -.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False) +.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) Recursively descend the directory tree named by *dir*, compiling all :file:`.py` files along the way. @@ -80,7 +99,49 @@ Public functions If *quiet* is true, nothing is printed to the standard output unless errors occur. -.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False) + If *legacy* is true, byte-code files are written to their legacy locations + and names, which may overwrite byte-code files created by another version of + Python. The default is to write files to their :pep:`3147` locations and + names, which allows byte-code files from multiple versions of Python to + coexist. + + *optimize* specifies the optimization level for the compiler. It is passed to + the built-in :func:`compile` function. + + .. versionchanged:: 3.2 + Added the *legacy* and *optimize* parameter. + + +.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) + + Compile the file with path *fullname*. + + If *ddir* is given, it is prepended to the path to the file being compiled + for use in compilation time tracebacks, and is also compiled in to the + byte-code file, where it will be used in tracebacks and other messages in + cases where the source file does not exist at the time the byte-code file is + executed. + + If *rx* is given, its search method is passed the full path name to the + file being compiled, and if it returns a true value, the file is not + compiled and ``True`` is returned. + + If *quiet* is true, nothing is printed to the standard output unless errors + occur. + + If *legacy* is true, byte-code files are written to their legacy locations + and names, which may overwrite byte-code files created by another version of + Python. The default is to write files to their :pep:`3147` locations and + names, which allows byte-code files from multiple versions of Python to + coexist. + + *optimize* specifies the optimization level for the compiler. It is passed to + the built-in :func:`compile` function. + + .. versionadded:: 3.2 + + +.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, legacy=False, optimize=-1) Byte-compile all the :file:`.py` files found along ``sys.path``. If *skip_curdir* is true (the default), the current directory is not included @@ -88,6 +149,10 @@ Public functions function. Note that unlike the other compile functions, ``maxlevels`` defaults to ``0``. + .. versionchanged:: 3.2 + Added the *legacy* and *optimize* parameter. + + To force a recompile of all the :file:`.py` files in the :file:`Lib/` subdirectory and all its subdirectories:: diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst new file mode 100644 index 0000000000..29ffc0d057 --- /dev/null +++ b/Doc/library/concurrent.futures.rst @@ -0,0 +1,371 @@ +:mod:`concurrent.futures` --- Launching parallel tasks +====================================================== + +.. module:: concurrent.futures + :synopsis: Execute computations concurrently using threads or processes. + +.. versionadded:: 3.2 + +**Source code:** :source:`Lib/concurrent/futures/thread.py` +and :source:`Lib/concurrent/futures/process.py` + +-------------- + +The :mod:`concurrent.futures` module provides a high-level interface for +asynchronously executing callables. + +The asynchronous execution can be performed with threads, using +:class:`ThreadPoolExecutor`, or separate processes, using +:class:`ProcessPoolExecutor`. Both implement the same interface, which is +defined by the abstract :class:`Executor` class. + + +Executor Objects +---------------- + +.. class:: Executor + + An abstract class that provides methods to execute calls asynchronously. It + should not be used directly, but through its concrete subclasses. + + .. method:: submit(fn, *args, **kwargs) + + Schedules the callable, *fn*, to be executed as ``fn(*args **kwargs)`` + and returns a :class:`Future` object representing the execution of the + callable. :: + + with ThreadPoolExecutor(max_workers=1) as executor: + future = executor.submit(pow, 323, 1235) + print(future.result()) + + .. method:: map(func, *iterables, timeout=None) + + Equivalent to ``map(func, *iterables)`` except *func* is executed + asynchronously and several calls to *func* may be made concurrently. The + returned iterator raises a :exc:`TimeoutError` if :meth:`__next__()` is + called and the result isn't available after *timeout* seconds from the + original call to :meth:`Executor.map`. *timeout* can be an int or a + float. If *timeout* is not specified or ``None``, there is no limit to + the wait time. If a call raises an exception, then that exception will + be raised when its value is retrieved from the iterator. + + .. method:: shutdown(wait=True) + + Signal the executor that it should free any resources that it is using + when the currently pending futures are done executing. Calls to + :meth:`Executor.submit` and :meth:`Executor.map` made after shutdown will + raise :exc:`RuntimeError`. + + If *wait* is ``True`` then this method will not return until all the + pending futures are done executing and the resources associated with the + executor have been freed. If *wait* is ``False`` then this method will + return immediately and the resources associated with the executor will be + freed when all pending futures are done executing. Regardless of the + value of *wait*, the entire Python program will not exit until all + pending futures are done executing. + + You can avoid having to call this method explicitly if you use the + :keyword:`with` statement, which will shutdown the :class:`Executor` + (waiting as if :meth:`Executor.shutdown` were called with *wait* set to + ``True``):: + + import shutil + with ThreadPoolExecutor(max_workers=4) as e: + e.submit(shutil.copy, 'src1.txt', 'dest1.txt') + e.submit(shutil.copy, 'src2.txt', 'dest2.txt') + e.submit(shutil.copy, 'src3.txt', 'dest3.txt') + e.submit(shutil.copy, 'src3.txt', 'dest4.txt') + + +ThreadPoolExecutor +------------------ + +:class:`ThreadPoolExecutor` is a :class:`Executor` subclass that uses a pool of +threads to execute calls asynchronously. + +Deadlocks can occur when the callable associated with a :class:`Future` waits on +the results of another :class:`Future`. For example:: + + import time + def wait_on_b(): + time.sleep(5) + print(b.result()) # b will never complete because it is waiting on a. + return 5 + + def wait_on_a(): + time.sleep(5) + print(a.result()) # a will never complete because it is waiting on b. + return 6 + + + executor = ThreadPoolExecutor(max_workers=2) + a = executor.submit(wait_on_b) + b = executor.submit(wait_on_a) + +And:: + + def wait_on_future(): + f = executor.submit(pow, 5, 2) + # This will never complete because there is only one worker thread and + # it is executing this function. + print(f.result()) + + executor = ThreadPoolExecutor(max_workers=1) + executor.submit(wait_on_future) + + +.. class:: ThreadPoolExecutor(max_workers) + + An :class:`Executor` subclass that uses a pool of at most *max_workers* + threads to execute calls asynchronously. + + +.. _threadpoolexecutor-example: + +ThreadPoolExecutor Example +~~~~~~~~~~~~~~~~~~~~~~~~~~ +:: + + import concurrent.futures + import urllib.request + + URLS = ['http://www.foxnews.com/', + 'http://www.cnn.com/', + 'http://europe.wsj.com/', + 'http://www.bbc.co.uk/', + 'http://some-made-up-domain.com/'] + + def load_url(url, timeout): + return urllib.request.urlopen(url, timeout=timeout).read() + + with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: + future_to_url = dict((executor.submit(load_url, url, 60), url) + for url in URLS) + + for future in concurrent.futures.as_completed(future_to_url): + url = future_to_url[future] + if future.exception() is not None: + print('%r generated an exception: %s' % (url, + future.exception())) + else: + print('%r page is %d bytes' % (url, len(future.result()))) + + +ProcessPoolExecutor +------------------- + +The :class:`ProcessPoolExecutor` class is an :class:`Executor` subclass that +uses a pool of processes to execute calls asynchronously. +:class:`ProcessPoolExecutor` uses the :mod:`multiprocessing` module, which +allows it to side-step the :term:`Global Interpreter Lock` but also means that +only picklable objects can be executed and returned. + +Calling :class:`Executor` or :class:`Future` methods from a callable submitted +to a :class:`ProcessPoolExecutor` will result in deadlock. + +.. class:: ProcessPoolExecutor(max_workers=None) + + An :class:`Executor` subclass that executes calls asynchronously using a pool + of at most *max_workers* processes. If *max_workers* is ``None`` or not + given, it will default to the number of processors on the machine. + + +.. _processpoolexecutor-example: + +ProcessPoolExecutor Example +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +:: + + import concurrent.futures + import math + + PRIMES = [ + 112272535095293, + 112582705942171, + 112272535095293, + 115280095190773, + 115797848077099, + 1099726899285419] + + def is_prime(n): + if n % 2 == 0: + return False + + sqrt_n = int(math.floor(math.sqrt(n))) + for i in range(3, sqrt_n + 1, 2): + if n % i == 0: + return False + return True + + def main(): + with concurrent.futures.ProcessPoolExecutor() as executor: + for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)): + print('%d is prime: %s' % (number, prime)) + + if __name__ == '__main__': + main() + + +Future Objects +-------------- + +The :class:`Future` class encapsulates the asynchronous execution of a callable. +:class:`Future` instances are created by :meth:`Executor.submit`. + +.. class:: Future + + Encapsulates the asynchronous execution of a callable. :class:`Future` + instances are created by :meth:`Executor.submit` and should not be created + directly except for testing. + + .. method:: cancel() + + Attempt to cancel the call. If the call is currently being executed and + cannot be cancelled then the method will return ``False``, otherwise the + call will be cancelled and the method will return ``True``. + + .. method:: cancelled() + + Return ``True`` if the call was successfully cancelled. + + .. method:: running() + + Return ``True`` if the call is currently being executed and cannot be + cancelled. + + .. method:: done() + + Return ``True`` if the call was successfully cancelled or finished + running. + + .. method:: result(timeout=None) + + Return the value returned by the call. If the call hasn't yet completed + then this method will wait up to *timeout* seconds. If the call hasn't + completed in *timeout* seconds, then a :exc:`TimeoutError` will be + raised. *timeout* can be an int or float. If *timeout* is not specified + or ``None``, there is no limit to the wait time. + + If the future is cancelled before completing then :exc:`CancelledError` + will be raised. + + If the call raised, this method will raise the same exception. + + .. method:: exception(timeout=None) + + Return the exception raised by the call. If the call hasn't yet + completed then this method will wait up to *timeout* seconds. If the + call hasn't completed in *timeout* seconds, then a :exc:`TimeoutError` + will be raised. *timeout* can be an int or float. If *timeout* is not + specified or ``None``, there is no limit to the wait time. + + If the future is cancelled before completing then :exc:`CancelledError` + will be raised. + + If the call completed without raising, ``None`` is returned. + + .. method:: add_done_callback(fn) + + Attaches the callable *fn* to the future. *fn* will be called, with the + future as its only argument, when the future is cancelled or finishes + running. + + Added callables are called in the order that they were added and are + always called in a thread belonging to the process that added them. If + the callable raises a :exc:`Exception` subclass, it will be logged and + ignored. If the callable raises a :exc:`BaseException` subclass, the + behavior is undefined. + + If the future has already completed or been cancelled, *fn* will be + called immediately. + + The following :class:`Future` methods are meant for use in unit tests and + :class:`Executor` implementations. + + .. method:: set_running_or_notify_cancel() + + This method should only be called by :class:`Executor` implementations + before executing the work associated with the :class:`Future` and by unit + tests. + + If the method returns ``False`` then the :class:`Future` was cancelled, + i.e. :meth:`Future.cancel` was called and returned `True`. Any threads + waiting on the :class:`Future` completing (i.e. through + :func:`as_completed` or :func:`wait`) will be woken up. + + If the method returns ``True`` then the :class:`Future` was not cancelled + and has been put in the running state, i.e. calls to + :meth:`Future.running` will return `True`. + + This method can only be called once and cannot be called after + :meth:`Future.set_result` or :meth:`Future.set_exception` have been + called. + + .. method:: set_result(result) + + Sets the result of the work associated with the :class:`Future` to + *result*. + + This method should only be used by :class:`Executor` implementations and + unit tests. + + .. method:: set_exception(exception) + + Sets the result of the work associated with the :class:`Future` to the + :class:`Exception` *exception*. + + This method should only be used by :class:`Executor` implementations and + unit tests. + + +Module Functions +---------------- + +.. function:: wait(fs, timeout=None, return_when=ALL_COMPLETED) + + Wait for the :class:`Future` instances (possibly created by different + :class:`Executor` instances) given by *fs* to complete. Returns a named + 2-tuple of sets. The first set, named ``done``, contains the futures that + completed (finished or were cancelled) before the wait completed. The second + set, named ``not_done``, contains uncompleted futures. + + *timeout* can be used to control the maximum number of seconds to wait before + returning. *timeout* can be an int or float. If *timeout* is not specified + or ``None``, there is no limit to the wait time. + + *return_when* indicates when this function should return. It must be one of + the following constants: + + +-----------------------------+----------------------------------------+ + | Constant | Description | + +=============================+========================================+ + | :const:`FIRST_COMPLETED` | The function will return when any | + | | future finishes or is cancelled. | + +-----------------------------+----------------------------------------+ + | :const:`FIRST_EXCEPTION` | The function will return when any | + | | future finishes by raising an | + | | exception. If no future raises an | + | | exception then it is equivalent to | + | | :const:`ALL_COMPLETED`. | + +-----------------------------+----------------------------------------+ + | :const:`ALL_COMPLETED` | The function will return when all | + | | futures finish or are cancelled. | + +-----------------------------+----------------------------------------+ + +.. function:: as_completed(fs, timeout=None) + + Returns an iterator over the :class:`Future` instances (possibly created by + different :class:`Executor` instances) given by *fs* that yields futures as + they complete (finished or were cancelled). Any futures that completed + before :func:`as_completed` is called will be yielded first. The returned + iterator raises a :exc:`TimeoutError` if :meth:`__next__` is called and the + result isn't available after *timeout* seconds from the original call to + :func:`as_completed`. *timeout* can be an int or float. If *timeout* is not + specified or ``None``, there is no limit to the wait time. + + +.. seealso:: + + :pep:`3148` -- futures - execute computations asynchronously + The proposal which described this feature for inclusion in the Python + standard library. diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst index 2cef232bb5..afceb8d30a 100644 --- a/Doc/library/configparser.rst +++ b/Doc/library/configparser.rst @@ -7,7 +7,9 @@ .. moduleauthor:: Ken Manheimer <klm@zope.com> .. moduleauthor:: Barry Warsaw <bwarsaw@python.org> .. moduleauthor:: Eric S. Raymond <esr@thyrsus.com> +.. moduleauthor:: Łukasz Langa <lukasz@langa.pl> .. sectionauthor:: Christopher G. Petrilli <petrilli@amber.org> +.. sectionauthor:: Łukasz Langa <lukasz@langa.pl> .. index:: pair: .ini; file @@ -15,11 +17,10 @@ single: ini file single: Windows ini file -This module defines the class :class:`ConfigParser`. The :class:`ConfigParser` -class implements a basic configuration file parser language which provides a -structure similar to what you would find on Microsoft Windows INI files. You -can use this to write Python programs which can be customized by end users -easily. +This module provides the :class:`ConfigParser` class which implements a basic +configuration language which provides a structure similar to what's found in +Microsoft Windows INI files. You can use this to write Python programs which +can be customized by end users easily. .. note:: @@ -36,444 +37,1234 @@ easily. The json module implements a subset of JavaScript syntax which can also be used for this purpose. -The configuration file consists of sections, led by a ``[section]`` header and -followed by ``name: value`` entries, with continuations in the style of -:rfc:`822` (see section 3.1.1, "LONG HEADER FIELDS"); ``name=value`` is also -accepted. Note that leading whitespace is removed from values. The optional -values can contain format strings which refer to other values in the same -section, or values in a special ``DEFAULT`` section. Additional defaults can be -provided on initialization and retrieval. Lines beginning with ``'#'`` or -``';'`` are ignored and may be used to provide comments. -Configuration files may include comments, prefixed by specific characters (``#`` -and ``;``). Comments may appear on their own in an otherwise empty line, or may -be entered in lines holding values or section names. In the latter case, they -need to be preceded by a whitespace character to be recognized as a comment. -(For backwards compatibility, only ``;`` starts an inline comment, while ``#`` -does not.) +Quick Start +----------- + +Let's take a very basic configuration file that looks like this: + +.. code-block:: ini + + [DEFAULT] + ServerAliveInterval = 45 + Compression = yes + CompressionLevel = 9 + ForwardX11 = yes + + [bitbucket.org] + User = hg + + [topsecret.server.com] + Port = 50022 + ForwardX11 = no + +The structure of INI files is described `in the following section +<#supported-ini-file-structure>`_. Essentially, the file +consists of sections, each of which contains keys with values. +:mod:`configparser` classes can read and write such files. Let's start by +creating the above configuration file programatically. + +.. doctest:: + + >>> import configparser + >>> config = configparser.ConfigParser() + >>> config['DEFAULT'] = {'ServerAliveInterval': '45', + ... 'Compression': 'yes', + ... 'CompressionLevel': '9'} + >>> config['bitbucket.org'] = {} + >>> config['bitbucket.org']['User'] = 'hg' + >>> config['topsecret.server.com'] = {} + >>> topsecret = config['topsecret.server.com'] + >>> topsecret['Port'] = '50022' # mutates the parser + >>> topsecret['ForwardX11'] = 'no' # same here + >>> config['DEFAULT']['ForwardX11'] = 'yes' + >>> with open('example.ini', 'w') as configfile: + ... config.write(configfile) + ... + +As you can see, we can treat a config parser much like a dictionary. +There are differences, `outlined later <#mapping-protocol-access>`_, but +the behavior is very close to what you would expect from a dictionary. + +Now that we have created and saved a configuration file, let's read it +back and explore the data it holds. + +.. doctest:: + + >>> import configparser + >>> config = configparser.ConfigParser() + >>> config.sections() + [] + >>> config.read('example.ini') + ['example.ini'] + >>> config.sections() + ['bitbucket.org', 'topsecret.server.com'] + >>> 'bitbucket.org' in config + True + >>> 'bytebong.com' in config + False + >>> config['bitbucket.org']['User'] + 'hg' + >>> config['DEFAULT']['Compression'] + 'yes' + >>> topsecret = config['topsecret.server.com'] + >>> topsecret['ForwardX11'] + 'no' + >>> topsecret['Port'] + '50022' + >>> for key in config['bitbucket.org']: print(key) + ... + user + compressionlevel + serveraliveinterval + compression + forwardx11 + >>> config['bitbucket.org']['ForwardX11'] + 'yes' + +As we can see above, the API is pretty straightforward. The only bit of magic +involves the ``DEFAULT`` section which provides default values for all other +sections [1]_. Note also that keys in sections are +case-insensitive and stored in lowercase [1]_. + + +Supported Datatypes +------------------- + +Config parsers do not guess datatypes of values in configuration files, always +storing them internally as strings. This means that if you need other +datatypes, you should convert on your own: + +.. doctest:: + + >>> int(topsecret['Port']) + 50022 + >>> float(topsecret['CompressionLevel']) + 9.0 + +Extracting Boolean values is not that simple, though. Passing the value +to ``bool()`` would do no good since ``bool('False')`` is still +``True``. This is why config parsers also provide :meth:`getboolean`. +This method is case-insensitive and recognizes Boolean values from +``'yes'``/``'no'``, ``'on'``/``'off'`` and ``'1'``/``'0'`` [1]_. +For example: + +.. doctest:: + + >>> topsecret.getboolean('ForwardX11') + False + >>> config['bitbucket.org'].getboolean('ForwardX11') + True + >>> config.getboolean('bitbucket.org', 'Compression') + True + +Apart from :meth:`getboolean`, config parsers also provide equivalent +:meth:`getint` and :meth:`getfloat` methods, but these are far less +useful since conversion using :func:`int` and :func:`float` is +sufficient for these types. + + +Fallback Values +--------------- + +As with a dictionary, you can use a section's :meth:`get` method to +provide fallback values: + +.. doctest:: + + >>> topsecret.get('Port') + '50022' + >>> topsecret.get('CompressionLevel') + '9' + >>> topsecret.get('Cipher') + >>> topsecret.get('Cipher', '3des-cbc') + '3des-cbc' + +Please note that default values have precedence over fallback values. +For instance, in our example the ``'CompressionLevel'`` key was +specified only in the ``'DEFAULT'`` section. If we try to get it from +the section ``'topsecret.server.com'``, we will always get the default, +even if we specify a fallback: + +.. doctest:: + + >>> topsecret.get('CompressionLevel', '3') + '9' + +One more thing to be aware of is that the parser-level :meth:`get` method +provides a custom, more complex interface, maintained for backwards +compatibility. When using this method, a fallback value can be provided via +the ``fallback`` keyword-only argument: + +.. doctest:: + + >>> config.get('bitbucket.org', 'monster', + ... fallback='No such things as monsters') + 'No such things as monsters' + +The same ``fallback`` argument can be used with the :meth:`getint`, +:meth:`getfloat` and :meth:`getboolean` methods, for example: + +.. doctest:: + + >>> 'BatchMode' in topsecret + False + >>> topsecret.getboolean('BatchMode', fallback=True) + True + >>> config['DEFAULT']['BatchMode'] = 'no' + >>> topsecret.getboolean('BatchMode', fallback=True) + False + + +Supported INI File Structure +---------------------------- + +A configuration file consists of sections, each led by a ``[section]`` header, +followed by key/value entries separated by a specific string (``=`` or ``:`` by +default [1]_). By default, section names are case sensitive but keys are not +[1]_. Leading and trailing whitespace is removed from keys and values. +Values can be omitted, in which case the key/value delimiter may also be left +out. Values can also span multiple lines, as long as they are indented deeper +than the first line of the value. Depending on the parser's mode, blank lines +may be treated as parts of multiline values or ignored. + +Configuration files may include comments, prefixed by specific +characters (``#`` and ``;`` by default [1]_). Comments may appear on +their own on an otherwise empty line, possibly indented. [1]_ + +For example: + +.. code-block:: ini + + [Simple Values] + key=value + spaces in keys=allowed + spaces in values=allowed as well + spaces around the delimiter = obviously + you can also use : to delimit keys from values + + [All Values Are Strings] + values like this: 1000000 + or this: 3.14159265359 + are they treated as numbers? : no + integers, floats and booleans are held as: strings + can use the API to get converted values directly: true + + [Multiline Values] + chorus: I'm a lumberjack, and I'm okay + I sleep all night and I work all day + + [No Values] + key_without_value + empty string value here = -On top of the core functionality, :class:`SafeConfigParser` supports -interpolation. This means values can contain format strings which refer to -other values in the same section, or values in a special ``DEFAULT`` section. -Additional defaults can be provided on initialization. + [You can use comments] + # like this + ; or this -For example:: + # By default only in an empty line. + # Inline comments can be harmful because they prevent users + # from using the delimiting characters as parts of values. + # That being said, this can be customized. + + [Sections Can Be Indented] + can_values_be_as_well = True + does_that_mean_anything_special = False + purpose = formatting for readability + multiline_values = are + handled just fine as + long as they are indented + deeper than the first line + of a value + # Did I mention we can indent comments, too? + + +Interpolation of values +----------------------- - [My Section] - foodir: %(dir)s/whatever - dir=frob - long: this value continues - in the next line +On top of the core functionality, :class:`ConfigParser` supports +interpolation. This means values can be preprocessed before returning them +from ``get()`` calls. -would resolve the ``%(dir)s`` to the value of ``dir`` (``frob`` in this case). -All reference expansions are done on demand. +.. class:: BasicInterpolation() -Default values can be specified by passing them into the :class:`ConfigParser` -constructor as a dictionary. Additional defaults may be passed into the -:meth:`get` method which will override all others. + The default implementation used by :class:`ConfigParser`. It enables + values to contain format strings which refer to other values in the same + section, or values in the special default section [1]_. Additional default + values can be provided on initialization. -Sections are normally stored in a built-in dictionary. An alternative dictionary -type can be passed to the :class:`ConfigParser` constructor. For example, if a -dictionary type is passed that sorts its keys, the sections will be sorted on -write-back, as will be the keys within each section. + For example: + .. code-block:: ini -.. class:: RawConfigParser(defaults=None, dict_type=collections.OrderedDict) + [Paths] + home_dir: /Users + my_dir: %(home_dir)s/lumberjack + my_pictures: %(my_dir)s/Pictures - The basic configuration object. When *defaults* is given, it is initialized - into the dictionary of intrinsic defaults. When *dict_type* is given, it will - be used to create the dictionary objects for the list of sections, for the - options within a section, and for the default values. This class does not - support the magical interpolation behavior. - .. versionchanged:: 3.1 - The default *dict_type* is :class:`collections.OrderedDict`. + In the example above, :class:`ConfigParser` with *interpolation* set to + ``BasicInterpolation()`` would resolve ``%(home_dir)s`` to the value of + ``home_dir`` (``/Users`` in this case). ``%(my_dir)s`` in effect would + resolve to ``/Users/lumberjack``. All interpolations are done on demand so + keys used in the chain of references do not have to be specified in any + specific order in the configuration file. + With ``interpolation`` set to ``None``, the parser would simply return + ``%(my_dir)s/Pictures`` as the value of ``my_pictures`` and + ``%(home_dir)s/lumberjack`` as the value of ``my_dir``. -.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict) +.. class:: ExtendedInterpolation() - Derived class of :class:`RawConfigParser` that implements the magical - interpolation feature and adds optional arguments to the :meth:`get` and - :meth:`items` methods. The values in *defaults* must be appropriate for the - ``%()s`` string interpolation. Note that *__name__* is an intrinsic default; - its value is the section name, and will override any value provided in - *defaults*. + An alternative handler for interpolation which implements a more advanced + syntax, used for instance in ``zc.buildout``. Extended interpolation is + using ``${section:option}`` to denote a value from a foreign section. + Interpolation can span multiple levels. For convenience, if the ``section:`` + part is omitted, interpolation defaults to the current section (and possibly + the default values from the special section). - All option names used in interpolation will be passed through the - :meth:`optionxform` method just like any other option name reference. For - example, using the default implementation of :meth:`optionxform` (which converts - option names to lower case), the values ``foo %(bar)s`` and ``foo %(BAR)s`` are - equivalent. + For example, the configuration specified above with basic interpolation, + would look like this with extended interpolation: - .. versionchanged:: 3.1 - The default *dict_type* is :class:`collections.OrderedDict`. + .. code-block:: ini + + [Paths] + home_dir: /Users + my_dir: ${home_dir}/lumberjack + my_pictures: ${my_dir}/Pictures + + Values from other sections can be fetched as well: + + .. code-block:: ini + + [Common] + home_dir: /Users + library_dir: /Library + system_dir: /System + macports_dir: /opt/local + [Frameworks] + Python: 3.2 + path: ${Common:system_dir}/Library/Frameworks/ -.. class:: SafeConfigParser(defaults=None, dict_type=collections.OrderedDict) + [Arthur] + nickname: Two Sheds + last_name: Jackson + my_dir: ${Common:home_dir}/twosheds + my_pictures: ${my_dir}/Pictures + python_dir: ${Frameworks:path}/Python/Versions/${Frameworks:Python} - Derived class of :class:`ConfigParser` that implements a more-sane variant of - the magical interpolation feature. This implementation is more predictable as - well. New applications should prefer this version if they don't need to be - compatible with older versions of Python. +Mapping Protocol Access +----------------------- + +.. versionadded:: 3.2 + +Mapping protocol access is a generic name for functionality that enables using +custom objects as if they were dictionaries. In case of :mod:`configparser`, +the mapping interface implementation is using the +``parser['section']['option']`` notation. + +``parser['section']`` in particular returns a proxy for the section's data in +the parser. This means that the values are not copied but they are taken from +the original parser on demand. What's even more important is that when values +are changed on a section proxy, they are actually mutated in the original +parser. + +:mod:`configparser` objects behave as close to actual dictionaries as possible. +The mapping interface is complete and adheres to the ``MutableMapping`` ABC. +However, there are a few differences that should be taken into account: + +* By default, all keys in sections are accessible in a case-insensitive manner + [1]_. E.g. ``for option in parser["section"]`` yields only ``optionxform``'ed + option key names. This means lowercased keys by default. At the same time, + for a section that holds the key ``'a'``, both expressions return ``True``:: + + "a" in parser["section"] + "A" in parser["section"] + +* All sections include ``DEFAULTSECT`` values as well which means that + ``.clear()`` on a section may not leave the section visibly empty. This is + because default values cannot be deleted from the section (because technically + they are not there). If they are overriden in the section, deleting causes + the default value to be visible again. Trying to delete a default value + causes a ``KeyError``. + +* Trying to delete the ``DEFAULTSECT`` raises ``ValueError``. + +* ``parser.get(section, option, **kwargs)`` - the second argument is **not** + a fallback value. Note however that the section-level ``get()`` methods are + compatible both with the mapping protocol and the classic configparser API. + +* ``parser.items()`` is compatible with the mapping protocol (returns a list of + *section_name*, *section_proxy* pairs including the DEFAULTSECT). However, + this method can also be invoked with arguments: ``parser.items(section, raw, + vars)``. The latter call returns a list of *option*, *value* pairs for + a specified ``section``, with all interpolations expanded (unless + ``raw=True`` is provided). + +The mapping protocol is implemented on top of the existing legacy API so that +subclasses overriding the original interface still should have mappings working +as expected. + + +Customizing Parser Behaviour +---------------------------- + +There are nearly as many INI format variants as there are applications using it. +:mod:`configparser` goes a long way to provide support for the largest sensible +set of INI styles available. The default functionality is mainly dictated by +historical background and it's very likely that you will want to customize some +of the features. + +The most common way to change the way a specific config parser works is to use +the :meth:`__init__` options: + +* *defaults*, default value: ``None`` + + This option accepts a dictionary of key-value pairs which will be initially + put in the ``DEFAULT`` section. This makes for an elegant way to support + concise configuration files that don't specify values which are the same as + the documented default. + + Hint: if you want to specify default values for a specific section, use + :meth:`read_dict` before you read the actual file. + +* *dict_type*, default value: :class:`collections.OrderedDict` + + This option has a major impact on how the mapping protocol will behave and how + the written configuration files look. With the default ordered + dictionary, every section is stored in the order they were added to the + parser. Same goes for options within sections. + + An alternative dictionary type can be used for example to sort sections and + options on write-back. You can also use a regular dictionary for performance + reasons. + + Please note: there are ways to add a set of key-value pairs in a single + operation. When you use a regular dictionary in those operations, the order + of the keys may be random. For example: + + .. doctest:: + + >>> parser = configparser.ConfigParser() + >>> parser.read_dict({'section1': {'key1': 'value1', + ... 'key2': 'value2', + ... 'key3': 'value3'}, + ... 'section2': {'keyA': 'valueA', + ... 'keyB': 'valueB', + ... 'keyC': 'valueC'}, + ... 'section3': {'foo': 'x', + ... 'bar': 'y', + ... 'baz': 'z'} + ... }) + >>> parser.sections() + ['section3', 'section2', 'section1'] + >>> [option for option in parser['section3']] + ['baz', 'foo', 'bar'] + + In these operations you need to use an ordered dictionary as well: + + .. doctest:: + + >>> from collections import OrderedDict + >>> parser = configparser.ConfigParser() + >>> parser.read_dict( + ... OrderedDict(( + ... ('s1', + ... OrderedDict(( + ... ('1', '2'), + ... ('3', '4'), + ... ('5', '6'), + ... )) + ... ), + ... ('s2', + ... OrderedDict(( + ... ('a', 'b'), + ... ('c', 'd'), + ... ('e', 'f'), + ... )) + ... ), + ... )) + ... ) + >>> parser.sections() + ['s1', 's2'] + >>> [option for option in parser['s1']] + ['1', '3', '5'] + >>> [option for option in parser['s2'].values()] + ['b', 'd', 'f'] + +* *allow_no_value*, default value: ``False`` + + Some configuration files are known to include settings without values, but + which otherwise conform to the syntax supported by :mod:`configparser`. The + *allow_no_value* parameter to the constructor can be used to + indicate that such values should be accepted: + + .. doctest:: + + >>> import configparser + + >>> sample_config = """ + ... [mysqld] + ... user = mysql + ... pid-file = /var/run/mysqld/mysqld.pid + ... skip-external-locking + ... old_passwords = 1 + ... skip-bdb + ... # we don't need ACID today + ... skip-innodb + ... """ + >>> config = configparser.ConfigParser(allow_no_value=True) + >>> config.read_string(sample_config) + + >>> # Settings with values are treated as before: + >>> config["mysqld"]["user"] + 'mysql' + + >>> # Settings without values provide None: + >>> config["mysqld"]["skip-bdb"] + + >>> # Settings which aren't specified still raise an error: + >>> config["mysqld"]["does-not-exist"] + Traceback (most recent call last): + ... + KeyError: 'does-not-exist' + +* *delimiters*, default value: ``('=', ':')`` + + Delimiters are substrings that delimit keys from values within a section. The + first occurence of a delimiting substring on a line is considered a delimiter. + This means values (but not keys) can contain the delimiters. + + See also the *space_around_delimiters* argument to + :meth:`ConfigParser.write`. + +* *comment_prefixes*, default value: ``('#', ';')`` + +* *inline_comment_prefixes*, default value: ``None`` + + Comment prefixes are strings that indicate the start of a valid comment within + a config file. *comment_prefixes* are used only on otherwise empty lines + (optionally indented) whereas *inline_comment_prefixes* can be used after + every valid value (e.g. section names, options and empty lines as well). By + default inline comments are disabled and ``'#'`` and ``';'`` are used as + prefixes for whole line comments. + + .. versionchanged:: 3.2 + In previous versions of :mod:`configparser` behaviour matched + ``comment_prefixes=('#',';')`` and ``inline_comment_prefixes=(';',)``. + + Please note that config parsers don't support escaping of comment prefixes so + using *inline_comment_prefixes* may prevent users from specifying option + values with characters used as comment prefixes. When in doubt, avoid setting + *inline_comment_prefixes*. In any circumstances, the only way of storing + comment prefix characters at the beginning of a line in multiline values is to + interpolate the prefix, for example:: + + >>> from configparser import ConfigParser, ExtendedInterpolation + >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) + >>> # the default BasicInterpolation could be used as well + >>> parser.read_string(""" + ... [DEFAULT] + ... hash = # + ... + ... [hashes] + ... shebang = + ... ${hash}!/usr/bin/env python + ... ${hash} -*- coding: utf-8 -*- + ... + ... extensions = + ... enabled_extension + ... another_extension + ... #disabled_by_comment + ... yet_another_extension + ... + ... interpolation not necessary = if # is not at line start + ... even in multiline values = line #1 + ... line #2 + ... line #3 + ... """) + >>> print(parser['hashes']['shebang']) + + #!/usr/bin/env python + # -*- coding: utf-8 -*- + >>> print(parser['hashes']['extensions']) + + enabled_extension + another_extension + yet_another_extension + >>> print(parser['hashes']['interpolation not necessary']) + if # is not at line start + >>> print(parser['hashes']['even in multiline values']) + line #1 + line #2 + line #3 + +* *strict*, default value: ``True`` + + When set to ``True``, the parser will not allow for any section or option + duplicates while reading from a single source (using :meth:`read_file`, + :meth:`read_string` or :meth:`read_dict`). It is recommended to use strict + parsers in new applications. + + .. versionchanged:: 3.2 + In previous versions of :mod:`configparser` behaviour matched + ``strict=False``. + +* *empty_lines_in_values*, default value: ``True`` + + In config parsers, values can span multiple lines as long as they are + indented more than the key that holds them. By default parsers also let + empty lines to be parts of values. At the same time, keys can be arbitrarily + indented themselves to improve readability. In consequence, when + configuration files get big and complex, it is easy for the user to lose + track of the file structure. Take for instance: + + .. code-block:: ini + + [Section] + key = multiline + value with a gotcha + + this = is still a part of the multiline value of 'key' + + This can be especially problematic for the user to see if she's using a + proportional font to edit the file. That is why when your application does + not need values with empty lines, you should consider disallowing them. This + will make empty lines split keys every time. In the example above, it would + produce two keys, ``key`` and ``this``. + +* *default_section*, default value: ``configparser.DEFAULTSECT`` (that is: + ``"DEFAULT"``) + + The convention of allowing a special section of default values for other + sections or interpolation purposes is a powerful concept of this library, + letting users create complex declarative configurations. This section is + normally called ``"DEFAULT"`` but this can be customized to point to any + other valid section name. Some typical values include: ``"general"`` or + ``"common"``. The name provided is used for recognizing default sections when + reading from any source and is used when writing configuration back to + a file. Its current value can be retrieved using the + ``parser_instance.default_section`` attribute and may be modified at runtime + (i.e. to convert files from one format to another). + +* *interpolation*, default value: ``configparser.BasicInterpolation`` + + Interpolation behaviour may be customized by providing a custom handler + through the *interpolation* argument. ``None`` can be used to turn off + interpolation completely, ``ExtendedInterpolation()`` provides a more + advanced variant inspired by ``zc.buildout``. More on the subject in the + `dedicated documentation section <#interpolation-of-values>`_. + :class:`RawConfigParser` has a default value of ``None``. + + +More advanced customization may be achieved by overriding default values of +these parser attributes. The defaults are defined on the classes, so they +may be overriden by subclasses or by attribute assignment. + +.. attribute:: BOOLEAN_STATES + + By default when using :meth:`getboolean`, config parsers consider the + following values ``True``: ``'1'``, ``'yes'``, ``'true'``, ``'on'`` and the + following values ``False``: ``'0'``, ``'no'``, ``'false'``, ``'off'``. You + can override this by specifying a custom dictionary of strings and their + Boolean outcomes. For example: + + .. doctest:: + + >>> custom = configparser.ConfigParser() + >>> custom['section1'] = {'funky': 'nope'} + >>> custom['section1'].getboolean('funky') + Traceback (most recent call last): + ... + ValueError: Not a boolean: nope + >>> custom.BOOLEAN_STATES = {'sure': True, 'nope': False} + >>> custom['section1'].getboolean('funky') + False + + Other typical Boolean pairs include ``accept``/``reject`` or + ``enabled``/``disabled``. + +.. method:: optionxform(option) + + This method transforms option names on every read, get, or set + operation. The default converts the name to lowercase. This also + means that when a configuration file gets written, all keys will be + lowercase. Override this method if that's unsuitable. + For example: + + .. doctest:: + + >>> config = """ + ... [Section1] + ... Key = Value + ... + ... [Section2] + ... AnotherKey = Value + ... """ + >>> typical = configparser.ConfigParser() + >>> typical.read_string(config) + >>> list(typical['Section1'].keys()) + ['key'] + >>> list(typical['Section2'].keys()) + ['anotherkey'] + >>> custom = configparser.RawConfigParser() + >>> custom.optionxform = lambda option: option + >>> custom.read_string(config) + >>> list(custom['Section1'].keys()) + ['Key'] + >>> list(custom['Section2'].keys()) + ['AnotherKey'] + +.. attribute:: SECTCRE + + A compiled regular expression used to parse section headers. The default + matches ``[section]`` to the name ``"section"``. Whitespace is considered part + of the section name, thus ``[ larch ]`` will be read as a section of name + ``" larch "``. Override this attribute if that's unsuitable. For example: + + .. doctest:: + + >>> config = """ + ... [Section 1] + ... option = value + ... + ... [ Section 2 ] + ... another = val + ... """ + >>> typical = ConfigParser() + >>> typical.read_string(config) + >>> typical.sections() + ['Section 1', ' Section 2 '] + >>> custom = ConfigParser() + >>> custom.SECTCRE = re.compile(r"\[ *(?P<header>[^]]+?) *\]") + >>> custom.read_string(config) + >>> custom.sections() + ['Section 1', 'Section 2'] + + .. note:: + + While ConfigParser objects also use an ``OPTCRE`` attribute for recognizing + option lines, it's not recommended to override it because that would + interfere with constructor options *allow_no_value* and *delimiters*. + + +Legacy API Examples +------------------- + +Mainly because of backwards compatibility concerns, :mod:`configparser` +provides also a legacy API with explicit ``get``/``set`` methods. While there +are valid use cases for the methods outlined below, mapping protocol access is +preferred for new projects. The legacy API is at times more advanced, +low-level and downright counterintuitive. + +An example of writing to a configuration file:: + + import configparser - .. XXX Need to explain what's safer/more predictable about it. + config = configparser.RawConfigParser() + + # Please note that using RawConfigParser's set functions, you can assign + # non-string values to keys internally, but will receive an error when + # attempting to write to a file or when you get it in non-raw mode. Setting + # values using the mapping protocol or ConfigParser's set() does not allow + # such assignments to take place. + config.add_section('Section1') + config.set('Section1', 'int', '15') + config.set('Section1', 'bool', 'true') + config.set('Section1', 'float', '3.1415') + config.set('Section1', 'baz', 'fun') + config.set('Section1', 'bar', 'Python') + config.set('Section1', 'foo', '%(bar)s is %(baz)s!') + + # Writing our configuration file to 'example.cfg' + with open('example.cfg', 'w') as configfile: + config.write(configfile) + +An example of reading the configuration file again:: + + import configparser + + config = configparser.RawConfigParser() + config.read('example.cfg') + + # getfloat() raises an exception if the value is not a float + # getint() and getboolean() also do this for their respective types + float = config.getfloat('Section1', 'float') + int = config.getint('Section1', 'int') + print(float + int) + + # Notice that the next output does not interpolate '%(bar)s' or '%(baz)s'. + # This is because we are using a RawConfigParser(). + if config.getboolean('Section1', 'bool'): + print(config.get('Section1', 'foo')) + +To get interpolation, use :class:`ConfigParser`:: + + import configparser + + cfg = configparser.ConfigParser() + cfg.read('example.cfg') + + # Set the optional *raw* argument of get() to True if you wish to disable + # interpolation in a single get operation. + print(cfg.get('Section1', 'foo', raw=False)) # -> "Python is fun!" + print(cfg.get('Section1', 'foo', raw=True)) # -> "%(bar)s is %(baz)s!" + + # The optional *vars* argument is a dict with members that will take + # precedence in interpolation. + print(cfg.get('Section1', 'foo', vars={'bar': 'Documentation', + 'baz': 'evil'})) + + # The optional *fallback* argument can be used to provide a fallback value + print(cfg.get('Section1', 'foo')) + # -> "Python is fun!" + + print(cfg.get('Section1', 'foo', fallback='Monty is not.')) + # -> "Python is fun!" + + print(cfg.get('Section1', 'monster', fallback='No such things as monsters.')) + # -> "No such things as monsters." + + # A bare print(cfg.get('Section1', 'monster')) would raise NoOptionError + # but we can also use: + + print(cfg.get('Section1', 'monster', fallback=None)) + # -> None + +Default values are available in both types of ConfigParsers. They are used in +interpolation if an option used is not defined elsewhere. :: + + import configparser + + # New instance with 'bar' and 'baz' defaulting to 'Life' and 'hard' each + config = configparser.ConfigParser({'bar': 'Life', 'baz': 'hard'}) + config.read('example.cfg') + + print(config.get('Section1', 'foo')) # -> "Python is fun!" + config.remove_option('Section1', 'bar') + config.remove_option('Section1', 'baz') + print(config.get('Section1', 'foo')) # -> "Life is hard!" + + +.. _configparser-objects: + +ConfigParser Objects +-------------------- + +.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation()) + + The main configuration parser. When *defaults* is given, it is initialized + into the dictionary of intrinsic defaults. When *dict_type* is given, it + will be used to create the dictionary objects for the list of sections, for + the options within a section, and for the default values. + + When *delimiters* is given, it is used as the set of substrings that + divide keys from values. When *comment_prefixes* is given, it will be used + as the set of substrings that prefix comments in otherwise empty lines. + Comments can be indented. When *inline_comment_prefixes* is given, it will be + used as the set of substrings that prefix comments in non-empty lines. + + When *strict* is ``True`` (the default), the parser won't allow for + any section or option duplicates while reading from a single source (file, + string or dictionary), raising :exc:`DuplicateSectionError` or + :exc:`DuplicateOptionError`. When *empty_lines_in_values* is ``False`` + (default: ``True``), each empty line marks the end of an option. Otherwise, + internal empty lines of a multiline option are kept as part of the value. + When *allow_no_value* is ``True`` (default: ``False``), options without + values are accepted; the value held for these is ``None`` and they are + serialized without the trailing delimiter. + + When *default_section* is given, it specifies the name for the special + section holding default values for other sections and interpolation purposes + (normally named ``"DEFAULT"``). This value can be retrieved and changed on + runtime using the ``default_section`` instance attribute. + + Interpolation behaviour may be customized by providing a custom handler + through the *interpolation* argument. ``None`` can be used to turn off + interpolation completely, ``ExtendedInterpolation()`` provides a more + advanced variant inspired by ``zc.buildout``. More on the subject in the + `dedicated documentation section <#interpolation-of-values>`_. + + All option names used in interpolation will be passed through the + :meth:`optionxform` method just like any other option name reference. For + example, using the default implementation of :meth:`optionxform` (which + converts option names to lower case), the values ``foo %(bar)s`` and ``foo + %(BAR)s`` are equivalent. .. versionchanged:: 3.1 The default *dict_type* is :class:`collections.OrderedDict`. + .. versionchanged:: 3.2 + *allow_no_value*, *delimiters*, *comment_prefixes*, *strict*, + *empty_lines_in_values*, *default_section* and *interpolation* were + added. -.. exception:: Error - Base class for all other configparser exceptions. + .. method:: defaults() + Return a dictionary containing the instance-wide defaults. -.. exception:: NoSectionError - Exception raised when a specified section is not found. + .. method:: sections() + Return a list of the sections available; the *default section* is not + included in the list. -.. exception:: DuplicateSectionError - Exception raised if :meth:`add_section` is called with the name of a section - that is already present. + .. method:: add_section(section) + Add a section named *section* to the instance. If a section by the given + name already exists, :exc:`DuplicateSectionError` is raised. If the + *default section* name is passed, :exc:`ValueError` is raised. The name + of the section must be a string; if not, :exc:`TypeError` is raised. -.. exception:: NoOptionError + .. versionchanged:: 3.2 + Non-string section names raise :exc:`TypeError`. - Exception raised when a specified option is not found in the specified section. + .. method:: has_section(section) -.. exception:: InterpolationError + Indicates whether the named *section* is present in the configuration. + The *default section* is not acknowledged. - Base class for exceptions raised when problems occur performing string - interpolation. + .. method:: options(section) -.. exception:: InterpolationDepthError + Return a list of options available in the specified *section*. - Exception raised when string interpolation cannot be completed because the - number of iterations exceeds :const:`MAX_INTERPOLATION_DEPTH`. Subclass of - :exc:`InterpolationError`. + .. method:: has_option(section, option) -.. exception:: InterpolationMissingOptionError + If the given *section* exists, and contains the given *option*, return + :const:`True`; otherwise return :const:`False`. If the specified + *section* is :const:`None` or an empty string, DEFAULT is assumed. - Exception raised when an option referenced from a value does not exist. Subclass - of :exc:`InterpolationError`. + .. method:: read(filenames, encoding=None) -.. exception:: InterpolationSyntaxError + Attempt to read and parse a list of filenames, returning a list of + filenames which were successfully parsed. If *filenames* is a string, it + is treated as a single filename. If a file named in *filenames* cannot + be opened, that file will be ignored. This is designed so that you can + specify a list of potential configuration file locations (for example, + the current directory, the user's home directory, and some system-wide + directory), and all existing configuration files in the list will be + read. If none of the named files exist, the :class:`ConfigParser` + instance will contain an empty dataset. An application which requires + initial values to be loaded from a file should load the required file or + files using :meth:`read_file` before calling :meth:`read` for any + optional files:: - Exception raised when the source text into which substitutions are made does not - conform to the required syntax. Subclass of :exc:`InterpolationError`. + import configparser, os + config = configparser.ConfigParser() + config.read_file(open('defaults.cfg')) + config.read(['site.cfg', os.path.expanduser('~/.myapp.cfg')], + encoding='cp1250') -.. exception:: MissingSectionHeaderError + .. versionadded:: 3.2 + The *encoding* parameter. Previously, all files were read using the + default encoding for :func:`open`. - Exception raised when attempting to parse a file which has no section headers. + .. method:: read_file(f, source=None) -.. exception:: ParsingError + Read and parse configuration data from *f* which must be an iterable + yielding Unicode strings (for example files opened in text mode). - Exception raised when errors occur attempting to parse a file. + Optional argument *source* specifies the name of the file being read. If + not given and *f* has a :attr:`name` attribute, that is used for + *source*; the default is ``'<???>'``. + .. versionadded:: 3.2 + Replaces :meth:`readfp`. -.. data:: MAX_INTERPOLATION_DEPTH + .. method:: read_string(string, source='<string>') - The maximum depth for recursive interpolation for :meth:`get` when the *raw* - parameter is false. This is relevant only for the :class:`ConfigParser` class. + Parse configuration data from a string. + Optional argument *source* specifies a context-specific name of the + string passed. If not given, ``'<string>'`` is used. This should + commonly be a filesystem path or a URL. -.. seealso:: + .. versionadded:: 3.2 - Module :mod:`shlex` - Support for a creating Unix shell-like mini-languages which can be used as an - alternate format for application configuration files. + .. method:: read_dict(dictionary, source='<dict>') -.. _rawconfigparser-objects: + Load configuration from any object that provides a dict-like ``items()`` + method. Keys are section names, values are dictionaries with keys and + values that should be present in the section. If the used dictionary + type preserves order, sections and their keys will be added in order. + Values are automatically converted to strings. -RawConfigParser Objects ------------------------ + Optional argument *source* specifies a context-specific name of the + dictionary passed. If not given, ``<dict>`` is used. -:class:`RawConfigParser` instances have the following methods: + This method can be used to copy state between parsers. + .. versionadded:: 3.2 -.. method:: RawConfigParser.defaults() - Return a dictionary containing the instance-wide defaults. + .. method:: get(section, option, raw=False, [vars, fallback]) + Get an *option* value for the named *section*. If *vars* is provided, it + must be a dictionary. The *option* is looked up in *vars* (if provided), + *section*, and in *DEFAULTSECT* in that order. If the key is not found + and *fallback* is provided, it is used as a fallback value. ``None`` can + be provided as a *fallback* value. -.. method:: RawConfigParser.sections() + All the ``'%'`` interpolations are expanded in the return values, unless + the *raw* argument is true. Values for interpolation keys are looked up + in the same manner as the option. - Return a list of the sections available; ``DEFAULT`` is not included in the - list. + .. versionchanged:: 3.2 + Arguments *raw*, *vars* and *fallback* are keyword only to protect + users from trying to use the third argument as the *fallback* fallback + (especially when using the mapping protocol). -.. method:: RawConfigParser.add_section(section) + .. method:: getint(section, option, raw=False, [vars, fallback]) - Add a section named *section* to the instance. If a section by the given name - already exists, :exc:`DuplicateSectionError` is raised. If the name - ``DEFAULT`` (or any of it's case-insensitive variants) is passed, - :exc:`ValueError` is raised. + A convenience method which coerces the *option* in the specified *section* + to an integer. See :meth:`get` for explanation of *raw*, *vars* and + *fallback*. -.. method:: RawConfigParser.has_section(section) - Indicates whether the named section is present in the configuration. The - ``DEFAULT`` section is not acknowledged. + .. method:: getfloat(section, option, raw=False, [vars, fallback]) + A convenience method which coerces the *option* in the specified *section* + to a floating point number. See :meth:`get` for explanation of *raw*, + *vars* and *fallback*. -.. method:: RawConfigParser.options(section) - Returns a list of options available in the specified *section*. + .. method:: getboolean(section, option, raw=False, [vars, fallback]) + A convenience method which coerces the *option* in the specified *section* + to a Boolean value. Note that the accepted values for the option are + ``'1'``, ``'yes'``, ``'true'``, and ``'on'``, which cause this method to + return ``True``, and ``'0'``, ``'no'``, ``'false'``, and ``'off'``, which + cause it to return ``False``. These string values are checked in a + case-insensitive manner. Any other value will cause it to raise + :exc:`ValueError`. See :meth:`get` for explanation of *raw*, *vars* and + *fallback*. -.. method:: RawConfigParser.has_option(section, option) - If the given section exists, and contains the given option, return - :const:`True`; otherwise return :const:`False`. + .. method:: items([section], raw=False, vars=None) + When *section* is not given, return a list of *section_name*, + *section_proxy* pairs, including DEFAULTSECT. -.. method:: RawConfigParser.read(filenames) + Otherwise, return a list of *name*, *value* pairs for the options in the + given *section*. Optional arguments have the same meaning as for the + :meth:`get` method. - Attempt to read and parse a list of filenames, returning a list of filenames - which were successfully parsed. If *filenames* is a string, - it is treated as a single filename. If a file named in *filenames* cannot be - opened, that file will be ignored. This is designed so that you can specify a - list of potential configuration file locations (for example, the current - directory, the user's home directory, and some system-wide directory), and all - existing configuration files in the list will be read. If none of the named - files exist, the :class:`ConfigParser` instance will contain an empty dataset. - An application which requires initial values to be loaded from a file should - load the required file or files using :meth:`readfp` before calling :meth:`read` - for any optional files:: + .. versionchanged:: 3.2 + Items present in *vars* no longer appear in the result. The previous + behaviour mixed actual parser options with variables provided for + interpolation. - import configparser, os + .. method:: set(section, option, value) - config = configparser.ConfigParser() - config.readfp(open('defaults.cfg')) - config.read(['site.cfg', os.path.expanduser('~/.myapp.cfg')]) + If the given section exists, set the given option to the specified value; + otherwise raise :exc:`NoSectionError`. *option* and *value* must be + strings; if not, :exc:`TypeError` is raised. -.. method:: RawConfigParser.readfp(fp, filename=None) + .. method:: write(fileobject, space_around_delimiters=True) - Read and parse configuration data from the file or file-like object in *fp* - (only the :meth:`readline` method is used). The file-like object must - operate in text mode, i.e. return strings from :meth:`readline`. + Write a representation of the configuration to the specified :term:`file + object`, which must be opened in text mode (accepting strings). This + representation can be parsed by a future :meth:`read` call. If + *space_around_delimiters* is true, delimiters between + keys and values are surrounded by spaces. - If *filename* is omitted and *fp* has a :attr:`name` attribute, that is used - for *filename*; the default is ``<???>``. + .. method:: remove_option(section, option) -.. method:: RawConfigParser.get(section, option) + Remove the specified *option* from the specified *section*. If the + section does not exist, raise :exc:`NoSectionError`. If the option + existed to be removed, return :const:`True`; otherwise return + :const:`False`. - Get an *option* value for the named *section*. + .. method:: remove_section(section) -.. method:: RawConfigParser.getint(section, option) + Remove the specified *section* from the configuration. If the section in + fact existed, return ``True``. Otherwise return ``False``. - A convenience method which coerces the *option* in the specified *section* to an - integer. + .. method:: optionxform(option) -.. method:: RawConfigParser.getfloat(section, option) + Transforms the option name *option* as found in an input file or as passed + in by client code to the form that should be used in the internal + structures. The default implementation returns a lower-case version of + *option*; subclasses may override this or client code can set an attribute + of this name on instances to affect this behavior. - A convenience method which coerces the *option* in the specified *section* to a - floating point number. + You don't need to subclass the parser to use this method, you can also + set it on an instance, to a function that takes a string argument and + returns a string. Setting it to ``str``, for example, would make option + names case sensitive:: + cfgparser = ConfigParser() + cfgparser.optionxform = str -.. method:: RawConfigParser.getboolean(section, option) + Note that when reading configuration files, whitespace around the option + names is stripped before :meth:`optionxform` is called. - A convenience method which coerces the *option* in the specified *section* to a - Boolean value. Note that the accepted values for the option are ``"1"``, - ``"yes"``, ``"true"``, and ``"on"``, which cause this method to return ``True``, - and ``"0"``, ``"no"``, ``"false"``, and ``"off"``, which cause it to return - ``False``. These string values are checked in a case-insensitive manner. Any - other value will cause it to raise :exc:`ValueError`. + .. method:: readfp(fp, filename=None) -.. method:: RawConfigParser.items(section) + .. deprecated:: 3.2 + Use :meth:`read_file` instead. - Return a list of ``(name, value)`` pairs for each option in the given *section*. + .. versionchanged:: 3.2 + :meth:`readfp` now iterates on *f* instead of calling ``f.readline()``. + For existing code calling :meth:`readfp` with arguments which don't + support iteration, the following generator may be used as a wrapper + around the file-like object:: -.. method:: RawConfigParser.set(section, option, value) + def readline_generator(f): + line = f.readline() + while line: + yield line + line = f.readline() - If the given section exists, set the given option to the specified value; - otherwise raise :exc:`NoSectionError`. While it is possible to use - :class:`RawConfigParser` (or :class:`ConfigParser` with *raw* parameters set to - true) for *internal* storage of non-string values, full functionality (including - interpolation and output to files) can only be achieved using string values. + Instead of ``parser.readfp(f)`` use + ``parser.read_file(readline_generator(f))``. -.. method:: RawConfigParser.write(fileobject) +.. data:: MAX_INTERPOLATION_DEPTH - Write a representation of the configuration to the specified :term:`file object`, - which must be opened in text mode (accepting strings). This representation - can be parsed by a future :meth:`read` call. + The maximum depth for recursive interpolation for :meth:`get` when the *raw* + parameter is false. This is relevant only when the default *interpolation* + is used. -.. method:: RawConfigParser.remove_option(section, option) +.. _rawconfigparser-objects: - Remove the specified *option* from the specified *section*. If the section does - not exist, raise :exc:`NoSectionError`. If the option existed to be removed, - return :const:`True`; otherwise return :const:`False`. +RawConfigParser Objects +----------------------- +.. class:: RawConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configaparser.DEFAULTSECT, interpolation=None) -.. method:: RawConfigParser.remove_section(section) + Legacy variant of the :class:`ConfigParser` with interpolation disabled + by default and unsafe ``add_section`` and ``set`` methods. - Remove the specified *section* from the configuration. If the section in fact - existed, return ``True``. Otherwise return ``False``. + .. note:: + Consider using :class:`ConfigParser` instead which checks types of + the values to be stored internally. If you don't want interpolation, you + can use ``ConfigParser(interpolation=None)``. -.. method:: RawConfigParser.optionxform(option) + .. method:: add_section(section) - Transforms the option name *option* as found in an input file or as passed in - by client code to the form that should be used in the internal structures. - The default implementation returns a lower-case version of *option*; - subclasses may override this or client code can set an attribute of this name - on instances to affect this behavior. + Add a section named *section* to the instance. If a section by the given + name already exists, :exc:`DuplicateSectionError` is raised. If the + *default section* name is passed, :exc:`ValueError` is raised. - You don't necessarily need to subclass a ConfigParser to use this method, you - can also re-set it on an instance, to a function that takes a string - argument. Setting it to ``str``, for example, would make option names case - sensitive:: + Type of *section* is not checked which lets users create non-string named + sections. This behaviour is unsupported and may cause internal errors. - cfgparser = ConfigParser() - ... - cfgparser.optionxform = str - Note that when reading configuration files, whitespace around the - option names are stripped before :meth:`optionxform` is called. + .. method:: set(section, option, value) + If the given section exists, set the given option to the specified value; + otherwise raise :exc:`NoSectionError`. While it is possible to use + :class:`RawConfigParser` (or :class:`ConfigParser` with *raw* parameters + set to true) for *internal* storage of non-string values, full + functionality (including interpolation and output to files) can only be + achieved using string values. -.. _configparser-objects: + This method lets users assign non-string values to keys internally. This + behaviour is unsupported and will cause errors when attempting to write + to a file or get it in non-raw mode. **Use the mapping protocol API** + which does not allow such assignments to take place. -ConfigParser Objects --------------------- -The :class:`ConfigParser` class extends some methods of the -:class:`RawConfigParser` interface, adding some optional arguments. +Exceptions +---------- +.. exception:: Error -.. method:: ConfigParser.get(section, option, raw=False, vars=None) + Base class for all other :mod:`configparser` exceptions. - Get an *option* value for the named *section*. If *vars* is provided, it - must be a dictionary. The *option* is looked up in *vars* (if provided), - *section*, and in *defaults* in that order. - All the ``'%'`` interpolations are expanded in the return values, unless the - *raw* argument is true. Values for interpolation keys are looked up in the - same manner as the option. +.. exception:: NoSectionError + Exception raised when a specified section is not found. -.. method:: ConfigParser.items(section, raw=False, vars=None) - Return a list of ``(name, value)`` pairs for each option in the given *section*. - Optional arguments have the same meaning as for the :meth:`get` method. +.. exception:: DuplicateSectionError + Exception raised if :meth:`add_section` is called with the name of a section + that is already present or in strict parsers when a section if found more + than once in a single input file, string or dictionary. -.. _safeconfigparser-objects: + .. versionadded:: 3.2 + Optional ``source`` and ``lineno`` attributes and arguments to + :meth:`__init__` were added. -SafeConfigParser Objects ------------------------- -The :class:`SafeConfigParser` class implements the same extended interface as -:class:`ConfigParser`, with the following addition: +.. exception:: DuplicateOptionError + Exception raised by strict parsers if a single option appears twice during + reading from a single file, string or dictionary. This catches misspellings + and case sensitivity-related errors, e.g. a dictionary may have two keys + representing the same case-insensitive configuration key. -.. method:: SafeConfigParser.set(section, option, value) - If the given section exists, set the given option to the specified value; - otherwise raise :exc:`NoSectionError`. *value* must be a string; if it is - not, :exc:`TypeError` is raised. +.. exception:: NoOptionError + Exception raised when a specified option is not found in the specified + section. -Examples --------- -An example of writing to a configuration file:: +.. exception:: InterpolationError - import configparser + Base class for exceptions raised when problems occur performing string + interpolation. - config = configparser.RawConfigParser() - # When adding sections or items, add them in the reverse order of - # how you want them to be displayed in the actual file. - # In addition, please note that using RawConfigParser's and the raw - # mode of ConfigParser's respective set functions, you can assign - # non-string values to keys internally, but will receive an error - # when attempting to write to a file or when you get it in non-raw - # mode. SafeConfigParser does not allow such assignments to take place. - config.add_section('Section1') - config.set('Section1', 'int', '15') - config.set('Section1', 'bool', 'true') - config.set('Section1', 'float', '3.1415') - config.set('Section1', 'baz', 'fun') - config.set('Section1', 'bar', 'Python') - config.set('Section1', 'foo', '%(bar)s is %(baz)s!') +.. exception:: InterpolationDepthError - # Writing our configuration file to 'example.cfg' - with open('example.cfg', 'w') as configfile: - config.write(configfile) + Exception raised when string interpolation cannot be completed because the + number of iterations exceeds :const:`MAX_INTERPOLATION_DEPTH`. Subclass of + :exc:`InterpolationError`. -An example of reading the configuration file again:: - import configparser +.. exception:: InterpolationMissingOptionError - config = configparser.RawConfigParser() - config.read('example.cfg') + Exception raised when an option referenced from a value does not exist. + Subclass of :exc:`InterpolationError`. - # getfloat() raises an exception if the value is not a float - # getint() and getboolean() also do this for their respective types - float = config.getfloat('Section1', 'float') - int = config.getint('Section1', 'int') - print(float + int) - # Notice that the next output does not interpolate '%(bar)s' or '%(baz)s'. - # This is because we are using a RawConfigParser(). - if config.getboolean('Section1', 'bool'): - print(config.get('Section1', 'foo')) +.. exception:: InterpolationSyntaxError -To get interpolation, you will need to use a :class:`ConfigParser` or -:class:`SafeConfigParser`:: + Exception raised when the source text into which substitutions are made does + not conform to the required syntax. Subclass of :exc:`InterpolationError`. - import configparser - config = configparser.ConfigParser() - config.read('example.cfg') +.. exception:: MissingSectionHeaderError - # Set the third, optional argument of get to 1 if you wish to use raw mode. - print(config.get('Section1', 'foo', 0)) # -> "Python is fun!" - print(config.get('Section1', 'foo', 1)) # -> "%(bar)s is %(baz)s!" + Exception raised when attempting to parse a file which has no section + headers. - # The optional fourth argument is a dict with members that will take - # precedence in interpolation. - print(config.get('Section1', 'foo', 0, {'bar': 'Documentation', - 'baz': 'evil'})) -Defaults are available in all three types of ConfigParsers. They are used in -interpolation if an option used is not defined elsewhere. :: +.. exception:: ParsingError - import configparser + Exception raised when errors occur attempting to parse a file. - # New instance with 'bar' and 'baz' defaulting to 'Life' and 'hard' each - config = configparser.SafeConfigParser({'bar': 'Life', 'baz': 'hard'}) - config.read('example.cfg') + .. versionchanged:: 3.2 + The ``filename`` attribute and :meth:`__init__` argument were renamed to + ``source`` for consistency. - print(config.get('Section1', 'foo')) # -> "Python is fun!" - config.remove_option('Section1', 'bar') - config.remove_option('Section1', 'baz') - print(config.get('Section1', 'foo')) # -> "Life is hard!" -The function ``opt_move`` below can be used to move options between sections:: - - def opt_move(config, section1, section2, option): - try: - config.set(section2, option, config.get(section1, option, 1)) - except configparser.NoSectionError: - # Create non-existent section - config.add_section(section2) - opt_move(config, section1, section2, option) - else: - config.remove_option(section1, option) +.. rubric:: Footnotes + +.. [1] Config parsers allow for heavy customization. If you are interested in + changing the behaviour outlined by the footnote reference, consult the + `Customizing Parser Behaviour`_ section. diff --git a/Doc/library/constants.rst b/Doc/library/constants.rst index 51a1c262d6..fa61f68852 100644 --- a/Doc/library/constants.rst +++ b/Doc/library/constants.rst @@ -1,3 +1,5 @@ +.. _built-in-consts: + Built-in Constants ================== diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index ca37f0f16a..e8dc17fed3 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -4,6 +4,9 @@ .. module:: contextlib :synopsis: Utilities for with-statement contexts. +**Source code:** :source:`Lib/contextlib.py` + +-------------- This module provides utilities for common tasks involving the :keyword:`with` statement. For more information see also :ref:`typecontextmanager` and @@ -12,7 +15,7 @@ statement. For more information see also :ref:`typecontextmanager` and Functions provided: -.. function:: contextmanager(func) +.. decorator:: contextmanager This function is a :term:`decorator` that can be used to define a factory function for :keyword:`with` statement context managers, without needing to @@ -51,54 +54,15 @@ Functions provided: the exception has been handled, and execution will resume with the statement immediately following the :keyword:`with` statement. + :func:`contextmanager` uses :class:`ContextDecorator` so the context managers + it creates can be used as decorators as well as in :keyword:`with` statements. + When used as a decorator, a new generator instance is implicitly created on + each function call (this allows the otherwise "one-shot" context managers + created by :func:`contextmanager` to meet the requirement that context + managers support multiple invocations in order to be used as decorators). -.. function:: nested(mgr1[, mgr2[, ...]]) - - Combine multiple context managers into a single nested context manager. - - This function has been deprecated in favour of the multiple manager form - of the :keyword:`with` statement. - - The one advantage of this function over the multiple manager form of the - :keyword:`with` statement is that argument unpacking allows it to be - used with a variable number of context managers as follows:: - - from contextlib import nested - - with nested(*managers): - do_something() - - Note that if the :meth:`__exit__` method of one of the nested context managers - indicates an exception should be suppressed, no exception information will be - passed to any remaining outer context managers. Similarly, if the - :meth:`__exit__` method of one of the nested managers raises an exception, any - previous exception state will be lost; the new exception will be passed to the - :meth:`__exit__` methods of any remaining outer context managers. In general, - :meth:`__exit__` methods should avoid raising exceptions, and in particular they - should not re-raise a passed-in exception. - - This function has two major quirks that have led to it being deprecated. Firstly, - as the context managers are all constructed before the function is invoked, the - :meth:`__new__` and :meth:`__init__` methods of the inner context managers are - not actually covered by the scope of the outer context managers. That means, for - example, that using :func:`nested` to open two files is a programming error as the - first file will not be closed promptly if an exception is thrown when opening - the second file. - - Secondly, if the :meth:`__enter__` method of one of the inner context managers - raises an exception that is caught and suppressed by the :meth:`__exit__` method - of one of the outer context managers, this construct will raise - :exc:`RuntimeError` rather than skipping the body of the :keyword:`with` - statement. - - Developers that need to support nesting of a variable number of context managers - can either use the :mod:`warnings` module to suppress the DeprecationWarning - raised by this function or else use this function as a model for an application - specific implementation. - - .. deprecated:: 3.1 - The with-statement now supports this functionality directly (without the - confusing error prone quirks). + .. versionchanged:: 3.2 + Use of :class:`ContextDecorator`. .. function:: closing(thing) @@ -128,6 +92,82 @@ Functions provided: ``page.close()`` will be called when the :keyword:`with` block is exited. +.. class:: ContextDecorator() + + A base class that enables a context manager to also be used as a decorator. + + Context managers inheriting from ``ContextDecorator`` have to implement + ``__enter__`` and ``__exit__`` as normal. ``__exit__`` retains its optional + exception handling even when used as a decorator. + + ``ContextDecorator`` is used by :func:`contextmanager`, so you get this + functionality automatically. + + Example of ``ContextDecorator``:: + + from contextlib import ContextDecorator + + class mycontext(ContextDecorator): + def __enter__(self): + print('Starting') + return self + + def __exit__(self, *exc): + print('Finishing') + return False + + >>> @mycontext() + ... def function(): + ... print('The bit in the middle') + ... + >>> function() + Starting + The bit in the middle + Finishing + + >>> with mycontext(): + ... print('The bit in the middle') + ... + Starting + The bit in the middle + Finishing + + This change is just syntactic sugar for any construct of the following form:: + + def f(): + with cm(): + # Do stuff + + ``ContextDecorator`` lets you instead write:: + + @cm() + def f(): + # Do stuff + + It makes it clear that the ``cm`` applies to the whole function, rather than + just a piece of it (and saving an indentation level is nice, too). + + Existing context managers that already have a base class can be extended by + using ``ContextDecorator`` as a mixin class:: + + from contextlib import ContextDecorator + + class mycontext(ContextBaseClass, ContextDecorator): + def __enter__(self): + return self + + def __exit__(self, *exc): + return False + + .. note:: + As the decorated function must be able to be called multiple times, the + underlying context manager must support use in multiple :keyword:`with` + statements. If this is not the case, then the original construct with the + explicit :keyword:`with` statement inside the function should be used. + + .. versionadded:: 3.2 + + .. seealso:: :pep:`0343` - The "with" statement diff --git a/Doc/library/copy.rst b/Doc/library/copy.rst index 5ae79aa91d..0c68bd9d0c 100644 --- a/Doc/library/copy.rst +++ b/Doc/library/copy.rst @@ -4,7 +4,11 @@ .. module:: copy :synopsis: Shallow and deep copy operations. -This module provides generic (shallow and deep) copying operations. +Assignment statements in Python do not copy objects, they create bindings +between a target and an object. For collections that are mutable or contain +mutable items, a copy is sometimes needed so one can change one copy without +changing the other. This module provides generic shallow and deep copy +operations (explained below). Interface summary: diff --git a/Doc/library/crypto.rst b/Doc/library/crypto.rst index 8650094c57..a233561d1e 100644 --- a/Doc/library/crypto.rst +++ b/Doc/library/crypto.rst @@ -25,5 +25,5 @@ Hardcore cypherpunks will probably find the cryptographic modules written by A.M. Kuchling of further interest; the package contains modules for various encryption algorithms, most notably AES. These modules are not distributed with Python but available separately. See the URL -http://www.amk.ca/python/code/crypto.html for more information. +http://www.pycrypto.org for more information. diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index 92ba6da7d1..edbe7263df 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -188,6 +188,15 @@ The :mod:`csv` module defines the following classes: TAB-delimited file. It is registered with the dialect name ``'excel-tab'``. +.. class:: unix_dialect() + + The :class:`unix_dialect` class defines the usual properties of a CSV file + generated on UNIX systems, i.e. using ``'\n'`` as line terminator and quoting + all fields. It is registered with the dialect name ``'unix'``. + + .. versionadded:: 3.2 + + .. class:: Sniffer() The :class:`Sniffer` class is used to deduce the format of a CSV file. @@ -393,6 +402,16 @@ Writer objects have the following public attribute: A read-only description of the dialect in use by the writer. +DictWriter objects have the following public method: + + +.. method:: DictWriter.writeheader() + + Write a row with the field names (as specified in the constructor). + + .. versionadded:: 3.2 + + .. _csv-examples: Examples diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index d1d025bf52..df39c28cc9 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -38,7 +38,7 @@ You load libraries by accessing them as attributes of these objects. *cdll* loads libraries which export functions using the standard ``cdecl`` calling convention, while *windll* libraries call functions using the ``stdcall`` calling convention. *oledll* also uses the ``stdcall`` calling convention, and -assumes the functions return a Windows :ctype:`HRESULT` error code. The error +assumes the functions return a Windows :c:type:`HRESULT` error code. The error code is used to automatically raise a :class:`WindowsError` exception when the function call fails. @@ -198,9 +198,9 @@ should be careful anyway. ``None``, integers, bytes objects and (unicode) strings are the only native Python objects that can directly be used as parameters in these function calls. ``None`` is passed as a C ``NULL`` pointer, bytes objects and strings are passed -as pointer to the memory block that contains their data (:ctype:`char *` or -:ctype:`wchar_t *`). Python integers are passed as the platforms default C -:ctype:`int` type, their value is masked to fit into the C type. +as pointer to the memory block that contains their data (:c:type:`char *` or +:c:type:`wchar_t *`). Python integers are passed as the platforms default C +:c:type:`int` type, their value is masked to fit into the C type. Before we move on calling functions with other parameter types, we have to learn more about :mod:`ctypes` data types. @@ -213,48 +213,48 @@ Fundamental data types :mod:`ctypes` defines a number of primitive C compatible data types : -+----------------------+----------------------------------------+----------------------------+ -| ctypes type | C type | Python type | -+======================+========================================+============================+ -| :class:`c_bool` | :ctype:`_Bool` | bool (1) | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_char` | :ctype:`char` | 1-character bytes object | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_wchar` | :ctype:`wchar_t` | 1-character string | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_byte` | :ctype:`char` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_ubyte` | :ctype:`unsigned char` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_short` | :ctype:`short` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_ushort` | :ctype:`unsigned short` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_int` | :ctype:`int` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_uint` | :ctype:`unsigned int` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_long` | :ctype:`long` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_ulong` | :ctype:`unsigned long` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_longlong` | :ctype:`__int64` or :ctype:`long long` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_ulonglong` | :ctype:`unsigned __int64` or | int | -| | :ctype:`unsigned long long` | | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_float` | :ctype:`float` | float | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_double` | :ctype:`double` | float | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_longdouble`| :ctype:`long double` | float | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_char_p` | :ctype:`char *` (NUL terminated) | bytes object or ``None`` | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_wchar_p` | :ctype:`wchar_t *` (NUL terminated) | string or ``None`` | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_void_p` | :ctype:`void *` | int or ``None`` | -+----------------------+----------------------------------------+----------------------------+ ++----------------------+------------------------------------------+----------------------------+ +| ctypes type | C type | Python type | ++======================+==========================================+============================+ +| :class:`c_bool` | :c:type:`_Bool` | bool (1) | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_char` | :c:type:`char` | 1-character bytes object | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_wchar` | :c:type:`wchar_t` | 1-character string | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_byte` | :c:type:`char` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_ubyte` | :c:type:`unsigned char` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_short` | :c:type:`short` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_ushort` | :c:type:`unsigned short` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_int` | :c:type:`int` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_uint` | :c:type:`unsigned int` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_long` | :c:type:`long` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_ulong` | :c:type:`unsigned long` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_longlong` | :c:type:`__int64` or :c:type:`long long` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_ulonglong` | :c:type:`unsigned __int64` or | int | +| | :c:type:`unsigned long long` | | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_float` | :c:type:`float` | float | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_double` | :c:type:`double` | float | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_longdouble`| :c:type:`long double` | float | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_char_p` | :c:type:`char *` (NUL terminated) | bytes object or ``None`` | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_wchar_p` | :c:type:`wchar_t *` (NUL terminated) | string or ``None`` | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_void_p` | :c:type:`void *` | int or ``None`` | ++----------------------+------------------------------------------+----------------------------+ (1) The constructor accepts any object with a truth value. @@ -325,7 +325,7 @@ property:: The :func:`create_string_buffer` function replaces the :func:`c_buffer` function (which is still available as an alias), as well as the :func:`c_string` function from earlier ctypes releases. To create a mutable memory block containing -unicode characters of the C type :ctype:`wchar_t` use the +unicode characters of the C type :c:type:`wchar_t` use the :func:`create_unicode_buffer` function. @@ -436,7 +436,7 @@ integer, string, bytes, a :mod:`ctypes` instance, or an object with an Return types ^^^^^^^^^^^^ -By default functions are assumed to return the C :ctype:`int` type. Other +By default functions are assumed to return the C :c:type:`int` type. Other return types can be specified by setting the :attr:`restype` attribute of the function object. @@ -865,10 +865,10 @@ later:: struct cell; /* forward declaration */ - struct { + struct cell { char *name; struct cell *next; - } cell; + }; The straightforward translation into ctypes code would be this, but it does not work:: @@ -935,8 +935,8 @@ argument, and the callback functions expected argument types as the remaining arguments. I will present an example here which uses the standard C library's -:cfunc:`qsort` function, this is used to sort items with the help of a callback -function. :cfunc:`qsort` will be used to sort an array of integers:: +:c:func:`qsort` function, this is used to sort items with the help of a callback +function. :c:func:`qsort` will be used to sort an array of integers:: >>> IntArray5 = c_int * 5 >>> ia = IntArray5(5, 1, 7, 33, 99) @@ -1077,7 +1077,7 @@ Accessing values exported from dlls ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Some shared libraries not only export functions, they also export variables. An -example in the Python library itself is the :cdata:`Py_OptimizeFlag`, an integer +example in the Python library itself is the :c:data:`Py_OptimizeFlag`, an integer set to 0, 1, or 2, depending on the :option:`-O` or :option:`-OO` flag given on startup. @@ -1095,11 +1095,11 @@ have printed ``c_long(1)``, or ``c_long(2)`` if :option:`-OO` would have been specified. An extended example which also demonstrates the use of pointers accesses the -:cdata:`PyImport_FrozenModules` pointer exported by Python. +:c:data:`PyImport_FrozenModules` pointer exported by Python. Quoting the docs for that value: - This pointer is initialized to point to an array of :ctype:`struct _frozen` + This pointer is initialized to point to an array of :c:type:`struct _frozen` records, terminated by one whose members are all *NULL* or zero. When a frozen module is imported, it is searched in this table. Third-party code could play tricks with this to provide a dynamically created collection of frozen modules. @@ -1116,7 +1116,7 @@ size, we show only how this table can be read with :mod:`ctypes`:: ... >>> -We have defined the :ctype:`struct _frozen` data type, so we can get the pointer +We have defined the :c:type:`struct _frozen` data type, so we can get the pointer to the table:: >>> FrozenTable = POINTER(struct_frozen) @@ -1335,7 +1335,7 @@ way is to instantiate one of the following classes: Instances of this class represent loaded shared libraries. Functions in these libraries use the standard C calling convention, and are assumed to return - :ctype:`int`. + :c:type:`int`. .. class:: OleDLL(name, mode=DEFAULT_MODE, handle=None, use_errno=False, use_last_error=False) @@ -1352,7 +1352,7 @@ way is to instantiate one of the following classes: Windows only: Instances of this class represent loaded shared libraries, functions in these libraries use the ``stdcall`` calling convention, and are - assumed to return :ctype:`int` by default. + assumed to return :c:type:`int` by default. On Windows CE only the standard calling convention is used, for convenience the :class:`WinDLL` and :class:`OleDLL` use the standard calling convention on this @@ -1493,7 +1493,7 @@ object is available: An instance of :class:`PyDLL` that exposes Python C API functions as attributes. Note that all these functions are assumed to return C - :ctype:`int`, which is of course not always the truth, so you have to assign + :c:type:`int`, which is of course not always the truth, so you have to assign the correct :attr:`restype` attribute to use these functions. @@ -1522,10 +1522,10 @@ They are instances of a private class: .. attribute:: restype Assign a ctypes type to specify the result type of the foreign function. - Use ``None`` for :ctype:`void`, a function not returning anything. + Use ``None`` for :c:type:`void`, a function not returning anything. It is possible to assign a callable Python object that is not a ctypes - type, in this case the function is assumed to return a C :ctype:`int`, and + type, in this case the function is assumed to return a C :c:type:`int`, and the callable will be called with this integer, allowing to do further processing or error checking. Using this is deprecated, for more flexible post processing or error checking use a ctypes data type as @@ -1815,8 +1815,6 @@ Utility functions termination character. An integer can be passed as second argument which allows to specify the size of the array if the length of the bytes should not be used. - If the first parameter is a string, it is converted into a bytes object - according to ctypes conversion rules. .. function:: create_unicode_buffer(init_or_size, size=None) @@ -1833,8 +1831,6 @@ Utility functions allows to specify the size of the array if the length of the string should not be used. - If the first parameter is a bytes object, it is converted into an unicode string - according to ctypes conversion rules. .. function:: DllCanUnloadNow() @@ -1935,22 +1931,6 @@ Utility functions but it is possible to enlarge the buffer. -.. function:: set_conversion_mode(encoding, errors) - - This function sets the rules that ctypes objects use when converting between - bytes objects and (unicode) strings. *encoding* must be a string specifying an - encoding, like ``'utf-8'`` or ``'mbcs'``, *errors* must be a string specifying - the error handling on encoding/decoding errors. Examples of possible values are - ``'strict'``, ``'replace'``, or ``'ignore'``. - - :func:`set_conversion_mode` returns a 2-tuple containing the previous - conversion rules. On windows, the initial conversion rules are ``('mbcs', - 'ignore')``, on other systems ``('ascii', 'strict')``. - - You can set the *encoding* to ``'undefined'`` to completely disable automatic - conversions. - - .. function:: set_errno(value) Set the current value of the ctypes-private copy of the system :data:`errno` @@ -1974,7 +1954,7 @@ Utility functions .. function:: string_at(address, size=-1) - This function returns the C string starting at memory address address as a bytes + This function returns the C string starting at memory address *address* as a bytes object. If size is specified, it is used as size, otherwise the string is assumed to be zero-terminated. @@ -2120,21 +2100,21 @@ These are the fundamental ctypes data types: .. class:: c_byte - Represents the C :ctype:`signed char` datatype, and interprets the value as + Represents the C :c:type:`signed char` datatype, and interprets the value as small integer. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_char - Represents the C :ctype:`char` datatype, and interprets the value as a single + Represents the C :c:type:`char` datatype, and interprets the value as a single character. The constructor accepts an optional string initializer, the length of the string must be exactly one character. .. class:: c_char_p - Represents the C :ctype:`char *` datatype when it points to a zero-terminated + Represents the C :c:type:`char *` datatype when it points to a zero-terminated string. For a general character pointer that may also point to binary data, ``POINTER(c_char)`` must be used. The constructor accepts an integer address, or a bytes object. @@ -2142,173 +2122,180 @@ These are the fundamental ctypes data types: .. class:: c_double - Represents the C :ctype:`double` datatype. The constructor accepts an + Represents the C :c:type:`double` datatype. The constructor accepts an optional float initializer. .. class:: c_longdouble - Represents the C :ctype:`long double` datatype. The constructor accepts an + Represents the C :c:type:`long double` datatype. The constructor accepts an optional float initializer. On platforms where ``sizeof(long double) == sizeof(double)`` it is an alias to :class:`c_double`. .. class:: c_float - Represents the C :ctype:`float` datatype. The constructor accepts an + Represents the C :c:type:`float` datatype. The constructor accepts an optional float initializer. .. class:: c_int - Represents the C :ctype:`signed int` datatype. The constructor accepts an + Represents the C :c:type:`signed int` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. On platforms where ``sizeof(int) == sizeof(long)`` it is an alias to :class:`c_long`. .. class:: c_int8 - Represents the C 8-bit :ctype:`signed int` datatype. Usually an alias for + Represents the C 8-bit :c:type:`signed int` datatype. Usually an alias for :class:`c_byte`. .. class:: c_int16 - Represents the C 16-bit :ctype:`signed int` datatype. Usually an alias for + Represents the C 16-bit :c:type:`signed int` datatype. Usually an alias for :class:`c_short`. .. class:: c_int32 - Represents the C 32-bit :ctype:`signed int` datatype. Usually an alias for + Represents the C 32-bit :c:type:`signed int` datatype. Usually an alias for :class:`c_int`. .. class:: c_int64 - Represents the C 64-bit :ctype:`signed int` datatype. Usually an alias for + Represents the C 64-bit :c:type:`signed int` datatype. Usually an alias for :class:`c_longlong`. .. class:: c_long - Represents the C :ctype:`signed long` datatype. The constructor accepts an + Represents the C :c:type:`signed long` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_longlong - Represents the C :ctype:`signed long long` datatype. The constructor accepts + Represents the C :c:type:`signed long long` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_short - Represents the C :ctype:`signed short` datatype. The constructor accepts an + Represents the C :c:type:`signed short` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_size_t - Represents the C :ctype:`size_t` datatype. + Represents the C :c:type:`size_t` datatype. + + +.. class:: c_ssize_t + + Represents the C :c:type:`ssize_t` datatype. + + .. versionadded:: 3.2 .. class:: c_ubyte - Represents the C :ctype:`unsigned char` datatype, it interprets the value as + Represents the C :c:type:`unsigned char` datatype, it interprets the value as small integer. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_uint - Represents the C :ctype:`unsigned int` datatype. The constructor accepts an + Represents the C :c:type:`unsigned int` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. On platforms where ``sizeof(int) == sizeof(long)`` it is an alias for :class:`c_ulong`. .. class:: c_uint8 - Represents the C 8-bit :ctype:`unsigned int` datatype. Usually an alias for + Represents the C 8-bit :c:type:`unsigned int` datatype. Usually an alias for :class:`c_ubyte`. .. class:: c_uint16 - Represents the C 16-bit :ctype:`unsigned int` datatype. Usually an alias for + Represents the C 16-bit :c:type:`unsigned int` datatype. Usually an alias for :class:`c_ushort`. .. class:: c_uint32 - Represents the C 32-bit :ctype:`unsigned int` datatype. Usually an alias for + Represents the C 32-bit :c:type:`unsigned int` datatype. Usually an alias for :class:`c_uint`. .. class:: c_uint64 - Represents the C 64-bit :ctype:`unsigned int` datatype. Usually an alias for + Represents the C 64-bit :c:type:`unsigned int` datatype. Usually an alias for :class:`c_ulonglong`. .. class:: c_ulong - Represents the C :ctype:`unsigned long` datatype. The constructor accepts an + Represents the C :c:type:`unsigned long` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_ulonglong - Represents the C :ctype:`unsigned long long` datatype. The constructor + Represents the C :c:type:`unsigned long long` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_ushort - Represents the C :ctype:`unsigned short` datatype. The constructor accepts + Represents the C :c:type:`unsigned short` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_void_p - Represents the C :ctype:`void *` type. The value is represented as integer. + Represents the C :c:type:`void *` type. The value is represented as integer. The constructor accepts an optional integer initializer. .. class:: c_wchar - Represents the C :ctype:`wchar_t` datatype, and interprets the value as a + Represents the C :c:type:`wchar_t` datatype, and interprets the value as a single character unicode string. The constructor accepts an optional string initializer, the length of the string must be exactly one character. .. class:: c_wchar_p - Represents the C :ctype:`wchar_t *` datatype, which must be a pointer to a + Represents the C :c:type:`wchar_t *` datatype, which must be a pointer to a zero-terminated wide character string. The constructor accepts an integer address, or a string. .. class:: c_bool - Represent the C :ctype:`bool` datatype (more accurately, :ctype:`_Bool` from + Represent the C :c:type:`bool` datatype (more accurately, :c:type:`_Bool` from C99). Its value can be True or False, and the constructor accepts any object that has a truth value. .. class:: HRESULT - Windows only: Represents a :ctype:`HRESULT` value, which contains success or + Windows only: Represents a :c:type:`HRESULT` value, which contains success or error information for a function or method call. .. class:: py_object - Represents the C :ctype:`PyObject *` datatype. Calling this without an - argument creates a ``NULL`` :ctype:`PyObject *` pointer. + Represents the C :c:type:`PyObject *` datatype. Calling this without an + argument creates a ``NULL`` :c:type:`PyObject *` pointer. The :mod:`ctypes.wintypes` module provides quite some other Windows specific -data types, for example :ctype:`HWND`, :ctype:`WPARAM`, or :ctype:`DWORD`. Some -useful structures like :ctype:`MSG` or :ctype:`RECT` are also defined. +data types, for example :c:type:`HWND`, :c:type:`WPARAM`, or :c:type:`DWORD`. Some +useful structures like :c:type:`MSG` or :c:type:`RECT` are also defined. .. _ctypes-structured-data-types: diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index 6d2baa0f99..f31b9c536b 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -41,15 +41,11 @@ Linux and the BSD variants of Unix. Module :mod:`curses.textpad` Editable text widget for curses supporting :program:`Emacs`\ -like bindings. - Module :mod:`curses.wrapper` - Convenience function to ensure proper terminal setup and resetting on - application entry and exit. - :ref:`curses-howto` Tutorial material on using curses with Python, by Andrew Kuchling and Eric Raymond. - The :file:`Demo/curses/` directory in the Python source distribution contains + The :file:`Tools/demo/` directory in the Python source distribution contains some example programs using the curses bindings provided by this module. @@ -76,7 +72,7 @@ The module :mod:`curses` defines the following functions: .. function:: baudrate() - Returns the output speed of the terminal in bits per second. On software + Return the output speed of the terminal in bits per second. On software terminal emulators it will have a fixed high value. Included for historical reasons; in former times, it was used to write output loops for time delays and occasionally to change interfaces depending on the line speed. @@ -89,7 +85,7 @@ The module :mod:`curses` defines the following functions: .. function:: can_change_color() - Returns true or false, depending on whether the programmer can change the colors + Return ``True`` or ``False``, depending on whether the programmer can change the colors displayed by the terminal. @@ -104,7 +100,7 @@ The module :mod:`curses` defines the following functions: .. function:: color_content(color_number) - Returns the intensity of the red, green, and blue (RGB) components in the color + Return the intensity of the red, green, and blue (RGB) components in the color *color_number*, which must be between ``0`` and :const:`COLORS`. A 3-tuple is returned, containing the R,G,B values for the given color, which will be between ``0`` (no component) and ``1000`` (maximum amount of component). @@ -112,7 +108,7 @@ The module :mod:`curses` defines the following functions: .. function:: color_pair(color_number) - Returns the attribute value for displaying text in the specified color. This + Return the attribute value for displaying text in the specified color. This attribute value can be combined with :const:`A_STANDOUT`, :const:`A_REVERSE`, and the other :const:`A_\*` attributes. :func:`pair_number` is the counterpart to this function. @@ -120,7 +116,7 @@ The module :mod:`curses` defines the following functions: .. function:: curs_set(visibility) - Sets the cursor state. *visibility* can be set to 0, 1, or 2, for invisible, + Set the cursor state. *visibility* can be set to 0, 1, or 2, for invisible, normal, or very visible. If the terminal supports the visibility requested, the previous cursor state is returned; otherwise, an exception is raised. On many terminals, the "visible" mode is an underline cursor and the "very visible" mode @@ -129,7 +125,7 @@ The module :mod:`curses` defines the following functions: .. function:: def_prog_mode() - Saves the current terminal mode as the "program" mode, the mode when the running + Save the current terminal mode as the "program" mode, the mode when the running program is using curses. (Its counterpart is the "shell" mode, for when the program is not in curses.) Subsequent calls to :func:`reset_prog_mode` will restore this mode. @@ -137,7 +133,7 @@ The module :mod:`curses` defines the following functions: .. function:: def_shell_mode() - Saves the current terminal mode as the "shell" mode, the mode when the running + Save the current terminal mode as the "shell" mode, the mode when the running program is not using curses. (Its counterpart is the "program" mode, when the program is using curses capabilities.) Subsequent calls to :func:`reset_shell_mode` will restore this mode. @@ -145,7 +141,7 @@ The module :mod:`curses` defines the following functions: .. function:: delay_output(ms) - Inserts an *ms* millisecond pause in output. + Insert an *ms* millisecond pause in output. .. function:: doupdate() @@ -176,7 +172,7 @@ The module :mod:`curses` defines the following functions: .. function:: erasechar() - Returns the user's current erase character. Under Unix operating systems this + Return the user's current erase character. Under Unix operating systems this is a property of the controlling tty of the curses program, and is not set by the curses library itself. @@ -184,7 +180,7 @@ The module :mod:`curses` defines the following functions: .. function:: filter() The :func:`.filter` routine, if used, must be called before :func:`initscr` is - called. The effect is that, during those calls, LINES is set to 1; the + called. The effect is that, during those calls, :envvar:`LINES` is set to 1; the capabilities clear, cup, cud, cud1, cuu1, cuu, vpa are disabled; and the home string is set to the value of cr. The effect is that the cursor is confined to the current line, and so are screen updates. This may be used for enabling @@ -210,7 +206,7 @@ The module :mod:`curses` defines the following functions: method should be call to retrieve the queued mouse event, represented as a 5-tuple ``(id, x, y, z, bstate)``. *id* is an ID value used to distinguish multiple devices, and *x*, *y*, *z* are the event's coordinates. (*z* is - currently unused.). *bstate* is an integer value whose bits will be set to + currently unused.) *bstate* is an integer value whose bits will be set to indicate the type of event, and will be the bitwise OR of one or more of the following constants, where *n* is the button number from 1 to 4: :const:`BUTTONn_PRESSED`, :const:`BUTTONn_RELEASED`, :const:`BUTTONn_CLICKED`, @@ -220,32 +216,32 @@ The module :mod:`curses` defines the following functions: .. function:: getsyx() - Returns the current coordinates of the virtual screen cursor in y and x. If + Return the current coordinates of the virtual screen cursor in y and x. If leaveok is currently true, then -1,-1 is returned. .. function:: getwin(file) - Reads window related data stored in the file by an earlier :func:`putwin` call. + Read window related data stored in the file by an earlier :func:`putwin` call. The routine then creates and initializes a new window using that data, returning the new window object. .. function:: has_colors() - Returns true if the terminal can display colors; otherwise, it returns false. + Return ``True`` if the terminal can display colors; otherwise, return ``False``. .. function:: has_ic() - Returns true if the terminal has insert- and delete- character capabilities. + Return ``True`` if the terminal has insert- and delete-character capabilities. This function is included for historical reasons only, as all modern software terminal emulators have such capabilities. .. function:: has_il() - Returns true if the terminal has insert- and delete-line capabilities, or can + Return ``True`` if the terminal has insert- and delete-line capabilities, or can simulate them using scrolling regions. This function is included for historical reasons only, as all modern software terminal emulators have such capabilities. @@ -253,7 +249,7 @@ The module :mod:`curses` defines the following functions: .. function:: has_key(ch) - Takes a key value *ch*, and returns true if the current terminal type recognizes + Take a key value *ch*, and return ``True`` if the current terminal type recognizes a key with that value. @@ -262,13 +258,13 @@ The module :mod:`curses` defines the following functions: Used for half-delay mode, which is similar to cbreak mode in that characters typed by the user are immediately available to the program. However, after blocking for *tenths* tenths of seconds, an exception is raised if nothing has - been typed. The value of *tenths* must be a number between 1 and 255. Use + been typed. The value of *tenths* must be a number between ``1`` and ``255``. Use :func:`nocbreak` to leave half-delay mode. .. function:: init_color(color_number, r, g, b) - Changes the definition of a color, taking the number of the color to be changed + Change the definition of a color, taking the number of the color to be changed followed by three RGB values (for the amounts of red, green, and blue components). The value of *color_number* must be between ``0`` and :const:`COLORS`. Each of *r*, *g*, *b*, must be a value between ``0`` and @@ -279,7 +275,7 @@ The module :mod:`curses` defines the following functions: .. function:: init_pair(pair_number, fg, bg) - Changes the definition of a color-pair. It takes three arguments: the number of + Change the definition of a color-pair. It takes three arguments: the number of the color-pair to be changed, the foreground color number, and the background color number. The value of *pair_number* must be between ``1`` and ``COLOR_PAIRS - 1`` (the ``0`` color pair is wired to white on black and cannot @@ -291,7 +287,7 @@ The module :mod:`curses` defines the following functions: .. function:: initscr() - Initialize the library. Returns a :class:`WindowObject` which represents the + Initialize the library. Return a :class:`WindowObject` which represents the whole screen. .. note:: @@ -300,9 +296,15 @@ The module :mod:`curses` defines the following functions: cause the interpreter to exit. +.. function:: is_term_resized(nlines, ncols) + + Return ``True`` if :func:`resize_term` would modify the window structure, + ``False`` otherwise. + + .. function:: isendwin() - Returns true if :func:`endwin` has been called (that is, the curses library has + Return ``True`` if :func:`endwin` has been called (that is, the curses library has been deinitialized). @@ -318,14 +320,14 @@ The module :mod:`curses` defines the following functions: .. function:: killchar() - Returns the user's current line kill character. Under Unix operating systems + Return the user's current line kill character. Under Unix operating systems this is a property of the controlling tty of the curses program, and is not set by the curses library itself. .. function:: longname() - Returns a string containing the terminfo long name field describing the current + Return a string containing the terminfo long name field describing the current terminal. The maximum length of a verbose description is 128 characters. It is defined only after the call to :func:`initscr`. @@ -338,14 +340,14 @@ The module :mod:`curses` defines the following functions: .. function:: mouseinterval(interval) - Sets the maximum time in milliseconds that can elapse between press and release - events in order for them to be recognized as a click, and returns the previous + Set the maximum time in milliseconds that can elapse between press and release + events in order for them to be recognized as a click, and return the previous interval value. The default value is 200 msec, or one fifth of a second. .. function:: mousemask(mousemask) - Sets the mouse events to be reported, and returns a tuple ``(availmask, + Set the mouse events to be reported, and return a tuple ``(availmask, oldmask)``. *availmask* indicates which of the specified mouse events can be reported; on complete failure it returns 0. *oldmask* is the previous value of the given window's mouse event mask. If this function is never called, no mouse @@ -359,7 +361,7 @@ The module :mod:`curses` defines the following functions: .. function:: newpad(nlines, ncols) - Creates and returns a pointer to a new pad data structure with the given number + Create and return a pointer to a new pad data structure with the given number of lines and columns. A pad is returned as a window object. A pad is like a window, except that it is not restricted by the screen size, and @@ -369,9 +371,9 @@ The module :mod:`curses` defines the following functions: echoing of input) do not occur. The :meth:`refresh` and :meth:`noutrefresh` methods of a pad require 6 arguments to specify the part of the pad to be displayed and the location on the screen to be used for the display. The - arguments are pminrow, pmincol, sminrow, smincol, smaxrow, smaxcol; the p + arguments are *pminrow*, *pmincol*, *sminrow*, *smincol*, *smaxrow*, *smaxcol*; the *p* arguments refer to the upper left corner of the pad region to be displayed and - the s arguments define a clipping box on the screen within which the pad region + the *s* arguments define a clipping box on the screen within which the pad region is to be displayed. @@ -413,7 +415,7 @@ The module :mod:`curses` defines the following functions: .. function:: noqiflush() - When the noqiflush routine is used, normal flush of input and output queues + When the :func:`noqiflush` routine is used, normal flush of input and output queues associated with the INTR, QUIT and SUSP characters will not be done. You may want to call :func:`noqiflush` in a signal handler if you want output to continue as though the interrupt had not occurred, after the handler exits. @@ -426,27 +428,27 @@ The module :mod:`curses` defines the following functions: .. function:: pair_content(pair_number) - Returns a tuple ``(fg, bg)`` containing the colors for the requested color pair. + Return a tuple ``(fg, bg)`` containing the colors for the requested color pair. The value of *pair_number* must be between ``1`` and ``COLOR_PAIRS - 1``. .. function:: pair_number(attr) - Returns the number of the color-pair set by the attribute value *attr*. + Return the number of the color-pair set by the attribute value *attr*. :func:`color_pair` is the counterpart to this function. .. function:: putp(string) - Equivalent to ``tputs(str, 1, putchar)``; emits the value of a specified - terminfo capability for the current terminal. Note that the output of putp + Equivalent to ``tputs(str, 1, putchar)``; emit the value of a specified + terminfo capability for the current terminal. Note that the output of :func:`putp` always goes to standard output. .. function:: qiflush( [flag] ) - If *flag* is false, the effect is the same as calling :func:`noqiflush`. If - *flag* is true, or no argument is provided, the queues will be flushed when + If *flag* is ``False``, the effect is the same as calling :func:`noqiflush`. If + *flag* is ``True``, or no argument is provided, the queues will be flushed when these control characters are read. @@ -459,26 +461,55 @@ The module :mod:`curses` defines the following functions: .. function:: reset_prog_mode() - Restores the terminal to "program" mode, as previously saved by + Restore the terminal to "program" mode, as previously saved by :func:`def_prog_mode`. .. function:: reset_shell_mode() - Restores the terminal to "shell" mode, as previously saved by + Restore the terminal to "shell" mode, as previously saved by :func:`def_shell_mode`. +.. function:: resetty() + + Restore the state of the terminal modes to what it was at the last call to + :func:`savetty`. + + +.. function:: resize_term(nlines, ncols) + + Backend function used by :func:`resizeterm`, performing most of the work; + when resizing the windows, :func:`resize_term` blank-fills the areas that are + extended. The calling application should fill in these areas with + appropriate data. The :func:`resize_term` function attempts to resize all + windows. However, due to the calling convention of pads, it is not possible + to resize these without additional interaction with the application. + + +.. function:: resizeterm(nlines, ncols) + + Resize the standard and current windows to the specified dimensions, and + adjusts other bookkeeping data used by the curses library that record the + window dimensions (in particular the SIGWINCH handler). + + +.. function:: savetty() + + Save the current state of the terminal modes in a buffer, usable by + :func:`resetty`. + + .. function:: setsyx(y, x) - Sets the virtual screen cursor to *y*, *x*. If *y* and *x* are both -1, then + Set the virtual screen cursor to *y*, *x*. If *y* and *x* are both -1, then leaveok is set. .. function:: setupterm([termstr, fd]) - Initializes the terminal. *termstr* is a string giving the terminal name; if - omitted, the value of the TERM environment variable will be used. *fd* is the + Initialize the terminal. *termstr* is a string giving the terminal name; if + omitted, the value of the :envvar:`TERM` environment variable will be used. *fd* is the file descriptor to which any initialization sequences will be sent; if not supplied, the file descriptor for ``sys.stdout`` will be used. @@ -498,19 +529,19 @@ The module :mod:`curses` defines the following functions: .. function:: termattrs() - Returns a logical OR of all video attributes supported by the terminal. This + Return a logical OR of all video attributes supported by the terminal. This information is useful when a curses program needs complete control over the appearance of the screen. .. function:: termname() - Returns the value of the environment variable TERM, truncated to 14 characters. + Return the value of the environment variable :envvar:`TERM`, truncated to 14 characters. .. function:: tigetflag(capname) - Returns the value of the Boolean capability corresponding to the terminfo + Return the value of the Boolean capability corresponding to the terminfo capability name *capname*. The value ``-1`` is returned if *capname* is not a Boolean capability, or ``0`` if it is canceled or absent from the terminal description. @@ -518,7 +549,7 @@ The module :mod:`curses` defines the following functions: .. function:: tigetnum(capname) - Returns the value of the numeric capability corresponding to the terminfo + Return the value of the numeric capability corresponding to the terminfo capability name *capname*. The value ``-2`` is returned if *capname* is not a numeric capability, or ``-1`` if it is canceled or absent from the terminal description. @@ -526,22 +557,22 @@ The module :mod:`curses` defines the following functions: .. function:: tigetstr(capname) - Returns the value of the string capability corresponding to the terminfo + Return the value of the string capability corresponding to the terminfo capability name *capname*. ``None`` is returned if *capname* is not a string capability, or is canceled or absent from the terminal description. .. function:: tparm(str[, ...]) - Instantiates the string *str* with the supplied parameters, where *str* should - be a parameterized string obtained from the terminfo database. E.g. - ``tparm(tigetstr("cup"), 5, 3)`` could result in ``'\033[6;4H'``, the exact + Instantiate the string *str* with the supplied parameters, where *str* should + be a parameterized string obtained from the terminfo database. E.g. + ``tparm(tigetstr("cup"), 5, 3)`` could result in ``b'\033[6;4H'``, the exact result depending on terminal type. .. function:: typeahead(fd) - Specifies that the file descriptor *fd* be used for typeahead checking. If *fd* + Specify that the file descriptor *fd* be used for typeahead checking. If *fd* is ``-1``, then no typeahead checking is done. The curses library does "line-breakout optimization" by looking for typeahead @@ -553,7 +584,7 @@ The module :mod:`curses` defines the following functions: .. function:: unctrl(ch) - Returns a string which is a printable representation of the character *ch*. + Return a string which is a printable representation of the character *ch*. Control characters are displayed as a caret followed by the character, for example as ``^C``. Printing characters are left as they are. @@ -576,7 +607,7 @@ The module :mod:`curses` defines the following functions: .. function:: use_env(flag) If used, this function should be called before :func:`initscr` or newterm are - called. When *flag* is false, the values of lines and columns specified in the + called. When *flag* is ``False``, the values of lines and columns specified in the terminfo database will be used, even if environment variables :envvar:`LINES` and :envvar:`COLUMNS` (used by default) are set, or if curses is running in a window (in which case default behavior would be to use the window size if @@ -592,6 +623,19 @@ The module :mod:`curses` defines the following functions: foreground color on the default background. +.. function:: wrapper(func, ...) + + Initialize curses and call another callable object, *func*, which should be the + rest of your curses-using application. If the application raises an exception, + this function will restore the terminal to a sane state before re-raising the + exception and generating a traceback. The callable object *func* is then passed + the main window 'stdscr' as its first argument, followed by any other arguments + passed to :func:`wrapper`. Before calling *func*, :func:`wrapper` turns on + cbreak mode, turns off echo, enables the terminal keypad, and initializes colors + if the terminal has color support. On exit (whether normally or by exception) + it restores cooked mode, turns on echo, and disables the terminal keypad. + + .. _curses-window-objects: Window Objects @@ -605,7 +649,7 @@ the following methods: .. note:: - A *character* means a C character (an ASCII code), rather then a Python + A *character* means a C character (an ASCII code), rather than a Python character (a string of length 1). (This note is true whenever the documentation mentions a character.) The built-in :func:`ord` is handy for conveying strings to codes. @@ -647,7 +691,7 @@ the following methods: .. method:: window.bkgd(ch[, attr]) - Sets the background property of the window to the character *ch*, with + Set the background property of the window to the character *ch*, with attributes *attr*. The change is then applied to every character position in that window: @@ -660,7 +704,7 @@ the following methods: .. method:: window.bkgdset(ch[, attr]) - Sets the window's background. A window's background consists of a character and + Set the window's background. A window's background consists of a character and any combination of attributes. The attribute part of the background is combined (OR'ed) with all non-blank characters that are written into the window. Both the character and attribute parts of the background are combined with the blank @@ -705,12 +749,12 @@ the following methods: .. method:: window.box([vertch, horch]) Similar to :meth:`border`, but both *ls* and *rs* are *vertch* and both *ts* and - bs are *horch*. The default corner characters are always used by this function. + *bs* are *horch*. The default corner characters are always used by this function. .. method:: window.chgat([y, x, ] [num,] attr) - Sets the attributes of *num* characters at the current cursor position, or at + Set the attributes of *num* characters at the current cursor position, or at position ``(y, x)`` if supplied. If no value of *num* is given or *num* = -1, the attribute will be set on all the characters to the end of the line. This function does not move the cursor. The changed line will be touched using the @@ -720,7 +764,7 @@ the following methods: .. method:: window.clear() - Like :meth:`erase`, but also causes the whole window to be repainted upon next + Like :meth:`erase`, but also cause the whole window to be repainted upon next call to :meth:`refresh`. @@ -743,7 +787,7 @@ the following methods: .. method:: window.cursyncup() - Updates the current cursor position of all the ancestors of the window to + Update the current cursor position of all the ancestors of the window to reflect the current cursor position of the window. @@ -754,14 +798,14 @@ the following methods: .. method:: window.deleteln() - Delete the line under the cursor. All following lines are moved up by 1 line. + Delete the line under the cursor. All following lines are moved up by one line. .. method:: window.derwin([nlines, ncols,] begin_y, begin_x) An abbreviation for "derive window", :meth:`derwin` is the same as calling :meth:`subwin`, except that *begin_y* and *begin_x* are relative to the origin - of the window, rather than relative to the entire screen. Returns a window + of the window, rather than relative to the entire screen. Return a window object for the derived window. @@ -773,8 +817,8 @@ the following methods: .. method:: window.enclose(y, x) - Tests whether the given pair of screen-relative character-cell coordinates are - enclosed by the given window, returning true or false. It is useful for + Test whether the given pair of screen-relative character-cell coordinates are + enclosed by the given window, returning ``True`` or ``False``. It is useful for determining what subset of the screen windows enclose the location of a mouse event. @@ -789,6 +833,11 @@ the following methods: Return a tuple ``(y, x)`` of co-ordinates of upper-left corner. +.. method:: window.getbkgd() + + Return the given window's current background character/attribute pair. + + .. method:: window.getch([y, x]) Get a character. Note that the integer returned does *not* have to be in ASCII @@ -811,8 +860,8 @@ the following methods: .. method:: window.getparyx() - Returns the beginning coordinates of this window relative to its parent window - into two integer variables y and x. Returns ``-1,-1`` if this window has no + Return the beginning coordinates of this window relative to its parent window + into two integer variables y and x. Return ``-1, -1`` if this window has no parent. @@ -835,8 +884,8 @@ the following methods: .. method:: window.idcok(flag) - If *flag* is false, curses no longer considers using the hardware insert/delete - character feature of the terminal; if *flag* is true, use of character insertion + If *flag* is ``False``, curses no longer considers using the hardware insert/delete + character feature of the terminal; if *flag* is ``True``, use of character insertion and deletion is enabled. When curses is first initialized, use of character insert/delete is enabled by default. @@ -849,7 +898,7 @@ the following methods: .. method:: window.immedok(flag) - If *flag* is true, any change in the window image automatically causes the + If *flag* is ``True``, any change in the window image automatically causes the window to be refreshed; you no longer have to call :meth:`refresh` yourself. However, it may degrade performance considerably, due to repeated calls to wrefresh. This option is disabled by default. @@ -869,7 +918,7 @@ the following methods: .. method:: window.insdelln(nlines) - Inserts *nlines* lines into the specified window above the current line. The + Insert *nlines* lines into the specified window above the current line. The *nlines* bottom lines are lost. For negative *nlines*, delete *nlines* lines starting with the one under the cursor, and move the remaining lines up. The bottom *nlines* lines are cleared. The current cursor position remains the @@ -878,7 +927,7 @@ the following methods: .. method:: window.insertln() - Insert a blank line under the cursor. All following lines are moved down by 1 + Insert a blank line under the cursor. All following lines are moved down by one line. @@ -901,23 +950,23 @@ the following methods: .. method:: window.instr([y, x] [, n]) - Returns a string of characters, extracted from the window starting at the + Return a string of characters, extracted from the window starting at the current cursor position, or at *y*, *x* if specified. Attributes are stripped - from the characters. If *n* is specified, :meth:`instr` returns return a string + from the characters. If *n* is specified, :meth:`instr` returns a string at most *n* characters long (exclusive of the trailing NUL). .. method:: window.is_linetouched(line) - Returns true if the specified line was modified since the last call to - :meth:`refresh`; otherwise returns false. Raises a :exc:`curses.error` + Return ``True`` if the specified line was modified since the last call to + :meth:`refresh`; otherwise return ``False``. Raise a :exc:`curses.error` exception if *line* is not valid for the given window. .. method:: window.is_wintouched() - Returns true if the specified window was modified since the last call to - :meth:`refresh`; otherwise returns false. + Return ``True`` if the specified window was modified since the last call to + :meth:`refresh`; otherwise return ``False``. .. method:: window.keypad(yes) @@ -943,7 +992,7 @@ the following methods: .. method:: window.mvderwin(y, x) - Moves the window inside its parent window. The screen-relative parameters of + Move the window inside its parent window. The screen-relative parameters of the window are not changed. This routine is used to display different parts of the parent window at the same physical position on the screen. @@ -1001,19 +1050,19 @@ the following methods: .. method:: window.putwin(file) - Writes all data associated with the window into the provided file object. This + Write all data associated with the window into the provided file object. This information can be later retrieved using the :func:`getwin` function. .. method:: window.redrawln(beg, num) - Indicates that the *num* screen lines, starting at line *beg*, are corrupted and + Indicate that the *num* screen lines, starting at line *beg*, are corrupted and should be completely redrawn on the next :meth:`refresh` call. .. method:: window.redrawwin() - Touches the entire window, causing it to be completely redrawn on the next + Touch the entire window, causing it to be completely redrawn on the next :meth:`refresh` call. @@ -1034,6 +1083,14 @@ the following methods: *sminrow*, or *smincol* are treated as if they were zero. +.. method:: window.resize(nlines, ncols) + + Reallocate storage for a curses window to adjust its dimensions to the + specified values. If either dimension is larger than the current values, the + window's data is filled with blanks that have the current background + rendition (as set by :meth:`bkgdset`) merged into them. + + .. method:: window.scroll([lines=1]) Scroll the screen or scrolling region upward by *lines* lines. @@ -1041,7 +1098,7 @@ the following methods: .. method:: window.scrollok(flag) - Controls what happens when the cursor of a window is moved off the edge of the + Control what happens when the cursor of a window is moved off the edge of the window or scrolling region, either as a result of a newline action on the bottom line, or typing the last character of the last line. If *flag* is false, the cursor is left on the bottom line. If *flag* is true, the window is scrolled up @@ -1083,26 +1140,26 @@ the following methods: .. method:: window.syncdown() - Touches each location in the window that has been touched in any of its ancestor + Touch each location in the window that has been touched in any of its ancestor windows. This routine is called by :meth:`refresh`, so it should almost never be necessary to call it manually. .. method:: window.syncok(flag) - If called with *flag* set to true, then :meth:`syncup` is called automatically + If called with *flag* set to ``True``, then :meth:`syncup` is called automatically whenever there is a change in the window. .. method:: window.syncup() - Touches all locations in ancestors of the window that have been changed in the + Touch all locations in ancestors of the window that have been changed in the window. .. method:: window.timeout(delay) - Sets blocking or non-blocking read behavior for the window. If *delay* is + Set blocking or non-blocking read behavior for the window. If *delay* is negative, blocking read is used (which will wait indefinitely for input). If *delay* is zero, then non-blocking read is used, and -1 will be returned by :meth:`getch` if no input is waiting. If *delay* is positive, then @@ -1125,7 +1182,7 @@ the following methods: .. method:: window.untouchwin() - Marks all lines in the window as unchanged since the last call to + Mark all lines in the window as unchanged since the last call to :meth:`refresh`. @@ -1582,7 +1639,7 @@ You can instantiate a :class:`Textbox` object as follows: each keystroke entered with the keystroke as a parameter; command dispatch is done on the result. This method returns the window contents as a string; whether blanks in the window are included is affected by the - :attr:`stripspaces` member. + :attr:`stripspaces` attribute. .. method:: do_command(ch) @@ -1648,43 +1705,14 @@ You can instantiate a :class:`Textbox` object as follows: .. method:: gather() - This method returns the window contents as a string; whether blanks in the + Return the window contents as a string; whether blanks in the window are included is affected by the :attr:`stripspaces` member. .. attribute:: stripspaces - This data member is a flag which controls the interpretation of blanks in + This attribute is a flag which controls the interpretation of blanks in the window. When it is on, trailing blanks on each line are ignored; any cursor motion that would land the cursor on a trailing blank goes to the end of that line instead, and trailing blanks are stripped when the window contents are gathered. - - -:mod:`curses.wrapper` --- Terminal handler for curses programs -============================================================== - -.. module:: curses.wrapper - :synopsis: Terminal configuration wrapper for curses programs. -.. moduleauthor:: Eric Raymond <esr@thyrsus.com> -.. sectionauthor:: Eric Raymond <esr@thyrsus.com> - - -This module supplies one function, :func:`wrapper`, which runs another function -which should be the rest of your curses-using application. If the application -raises an exception, :func:`wrapper` will restore the terminal to a sane state -before re-raising the exception and generating a traceback. - - -.. function:: wrapper(func, ...) - - Wrapper function that initializes curses and calls another function, *func*, - restoring normal keyboard/screen behavior on error. The callable object *func* - is then passed the main window 'stdscr' as its first argument, followed by any - other arguments passed to :func:`wrapper`. - -Before calling the hook function, :func:`wrapper` turns on cbreak mode, turns -off echo, enables the terminal keypad, and initializes colors if the terminal -has color support. On exit (whether normally or by exception) it restores -cooked mode, turns on echo, and disables the terminal keypad. - diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 26d9946630..1f4cfba5c4 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -11,40 +11,43 @@ The :mod:`datetime` module supplies classes for manipulating dates and times in both simple and complex ways. While date and time arithmetic is supported, the -focus of the implementation is on efficient member extraction for output +focus of the implementation is on efficient attribute extraction for output formatting and manipulation. For related functionality, see also the :mod:`time` and :mod:`calendar` modules. There are two kinds of date and time objects: "naive" and "aware". This distinction refers to whether the object has any notion of time zone, daylight saving time, or other kind of algorithmic or political time adjustment. Whether -a naive :class:`datetime` object represents Coordinated Universal Time (UTC), +a naive :class:`.datetime` object represents Coordinated Universal Time (UTC), local time, or time in some other timezone is purely up to the program, just like it's up to the program whether a particular number represents metres, -miles, or mass. Naive :class:`datetime` objects are easy to understand and to +miles, or mass. Naive :class:`.datetime` objects are easy to understand and to work with, at the cost of ignoring some aspects of reality. -For applications requiring more, :class:`datetime` and :class:`time` objects -have an optional time zone information member, :attr:`tzinfo`, that can contain -an instance of a subclass of the abstract :class:`tzinfo` class. These +For applications requiring more, :class:`.datetime` and :class:`.time` objects +have an optional time zone information attribute, :attr:`tzinfo`, that can be +set to an instance of a subclass of the abstract :class:`tzinfo` class. These :class:`tzinfo` objects capture information about the offset from UTC time, the -time zone name, and whether Daylight Saving Time is in effect. Note that no -concrete :class:`tzinfo` classes are supplied by the :mod:`datetime` module. -Supporting timezones at whatever level of detail is required is up to the -application. The rules for time adjustment across the world are more political -than rational, and there is no standard suitable for every application. +time zone name, and whether Daylight Saving Time is in effect. Note that only +one concrete :class:`tzinfo` class, the :class:`timezone` class, is supplied by the +:mod:`datetime` module. The :class:`timezone` class can represent simple +timezones with fixed offset from UTC such as UTC itself or North American EST and +EDT timezones. Supporting timezones at whatever level of detail is +required is up to the application. The rules for time adjustment across the +world are more political than rational, change frequently, and there is no +standard suitable for every application aside from UTC. The :mod:`datetime` module exports the following constants: .. data:: MINYEAR - The smallest year number allowed in a :class:`date` or :class:`datetime` object. + The smallest year number allowed in a :class:`date` or :class:`.datetime` object. :const:`MINYEAR` is ``1``. .. data:: MAXYEAR - The largest year number allowed in a :class:`date` or :class:`datetime` object. + The largest year number allowed in a :class:`date` or :class:`.datetime` object. :const:`MAXYEAR` is ``9999``. @@ -88,22 +91,30 @@ Available Types .. class:: timedelta :noindex: - A duration expressing the difference between two :class:`date`, :class:`time`, - or :class:`datetime` instances to microsecond resolution. + A duration expressing the difference between two :class:`date`, :class:`.time`, + or :class:`.datetime` instances to microsecond resolution. .. class:: tzinfo An abstract base class for time zone information objects. These are used by the - :class:`datetime` and :class:`time` classes to provide a customizable notion of + :class:`.datetime` and :class:`.time` classes to provide a customizable notion of time adjustment (for example, to account for time zone and/or daylight saving time). +.. class:: timezone + + A class that implements the :class:`tzinfo` abstract base class as a + fixed offset from the UTC. + + .. versionadded:: 3.2 + + Objects of these types are immutable. Objects of the :class:`date` type are always naive. -An object *d* of type :class:`time` or :class:`datetime` may be naive or aware. +An object *d* of type :class:`.time` or :class:`.datetime` may be naive or aware. *d* is aware if ``d.tzinfo`` is not ``None`` and ``d.tzinfo.utcoffset(d)`` does not return ``None``. If ``d.tzinfo`` is ``None``, or if ``d.tzinfo`` is not ``None`` but ``d.tzinfo.utcoffset(d)`` returns ``None``, *d* is naive. @@ -116,6 +127,7 @@ Subclass relationships:: object timedelta tzinfo + timezone time date datetime @@ -220,8 +232,28 @@ Supported operations: | | In general, *t1* \* i == *t1* \* (i-1) + *t1* | | | is true. (1) | +--------------------------------+-----------------------------------------------+ -| ``t1 = t2 // i`` | The floor is computed and the remainder (if | -| | any) is thrown away. (3) | +| ``t1 = t2 * f or t1 = f * t2`` | Delta multiplied by a float. The result is | +| | rounded to the nearest multiple of | +| | timedelta.resolution using round-half-to-even.| ++--------------------------------+-----------------------------------------------+ +| ``f = t2 / t3`` | Division (3) of *t2* by *t3*. Returns a | +| | :class:`float` object. | ++--------------------------------+-----------------------------------------------+ +| ``t1 = t2 / f or t1 = t2 / i`` | Delta divided by a float or an int. The result| +| | is rounded to the nearest multiple of | +| | timedelta.resolution using round-half-to-even.| ++--------------------------------+-----------------------------------------------+ +| ``t1 = t2 // i`` or | The floor is computed and the remainder (if | +| ``t1 = t2 // t3`` | any) is thrown away. In the second case, an | +| | integer is returned. (3) | ++--------------------------------+-----------------------------------------------+ +| ``t1 = t2 % t3`` | The remainder is computed as a | +| | :class:`timedelta` object. (3) | ++--------------------------------+-----------------------------------------------+ +| ``q, r = divmod(t1, t2)`` | Computes the quotient and the remainder: | +| | ``q = t1 // t2`` (3) and ``r = t1 % t2``. | +| | q is an integer and r is a :class:`timedelta` | +| | object. | +--------------------------------+-----------------------------------------------+ | ``+t1`` | Returns a :class:`timedelta` object with the | | | same value. (2) | @@ -267,9 +299,16 @@ Notes: -1 day, 19:00:00 In addition to the operations listed above :class:`timedelta` objects support -certain additions and subtractions with :class:`date` and :class:`datetime` +certain additions and subtractions with :class:`date` and :class:`.datetime` objects (see below). +.. versionchanged:: 3.2 + Floor division and true division of a :class:`timedelta` object by another + :class:`timedelta` object are now supported, as are remainder operations and + the :func:`divmod` function. True division and multiplication of a + :class:`timedelta` object by a :class:`float` object are now supported. + + Comparisons of :class:`timedelta` objects are supported with the :class:`timedelta` object representing the smaller duration considered to be the smaller timedelta. In order to stop mixed-type comparisons from falling back to @@ -282,12 +321,27 @@ comparison is ``==`` or ``!=``. The latter cases return :const:`False` or efficient pickling, and in Boolean contexts, a :class:`timedelta` object is considered to be true if and only if it isn't equal to ``timedelta(0)``. +Instance methods: + +.. method:: timedelta.total_seconds() + + Return the total number of seconds contained in the duration. Equivalent to + ``td / timedelta(seconds=1)``. + + Note that for very large time intervals (greater than 270 years on + most platforms) this method will lose microsecond accuracy. + + .. versionadded:: 3.2 + + Example usage: >>> from datetime import timedelta >>> year = timedelta(days=365) >>> another_year = timedelta(weeks=40, days=84, hours=23, ... minutes=50, seconds=600) # adds up to 365 days + >>> year.total_seconds() + 31536000.0 >>> year == another_year True >>> ten_years = 10 * year @@ -342,7 +396,7 @@ Other constructors, all class methods: Return the local date corresponding to the POSIX timestamp, such as is returned by :func:`time.time`. This may raise :exc:`ValueError`, if the timestamp is out - of the range of values supported by the platform C :cfunc:`localtime` function. + of the range of values supported by the platform C :c:func:`localtime` function. It's common for this to be restricted to years from 1970 through 2038. Note that on non-POSIX systems that include leap seconds in their notion of a timestamp, leap seconds are ignored by :meth:`fromtimestamp`. @@ -445,9 +499,9 @@ Instance methods: .. method:: date.replace(year, month, day) - Return a date with the same value, except for those members given new values by - whichever keyword arguments are specified. For example, if ``d == date(2002, - 12, 31)``, then ``d.replace(day=26) == date(2002, 12, 26)``. + Return a date with the same value, except for those parameters given new + values by whichever keyword arguments are specified. For example, if ``d == + date(2002, 12, 31)``, then ``d.replace(day=26) == date(2002, 12, 26)``. .. method:: date.timetuple() @@ -516,7 +570,7 @@ Instance methods: Return a string representing the date, for example ``date(2002, 12, 4).ctime() == 'Wed Dec 4 00:00:00 2002'``. ``d.ctime()`` is equivalent to ``time.ctime(time.mktime(d.timetuple()))`` on platforms where the native C - :cfunc:`ctime` function (which :func:`time.ctime` invokes, but which + :c:func:`ctime` function (which :func:`time.ctime` invokes, but which :meth:`date.ctime` does not invoke) conforms to the C standard. @@ -584,10 +638,10 @@ Example of working with :class:`date`: :class:`datetime` Objects ------------------------- -A :class:`datetime` object is a single object containing all the information -from a :class:`date` object and a :class:`time` object. Like a :class:`date` -object, :class:`datetime` assumes the current Gregorian calendar extended in -both directions; like a time object, :class:`datetime` assumes there are exactly +A :class:`.datetime` object is a single object containing all the information +from a :class:`date` object and a :class:`.time` object. Like a :class:`date` +object, :class:`.datetime` assumes the current Gregorian calendar extended in +both directions; like a time object, :class:`.datetime` assumes there are exactly 3600\*24 seconds in every day. Constructor: @@ -623,7 +677,7 @@ Other constructors, all class methods: or not specified, this is like :meth:`today`, but, if possible, supplies more precision than can be gotten from going through a :func:`time.time` timestamp (for example, this may be possible on platforms supplying the C - :cfunc:`gettimeofday` function). + :c:func:`gettimeofday` function). Else *tz* must be an instance of a class :class:`tzinfo` subclass, and the current date and time are converted to *tz*'s time zone. In this case the @@ -635,15 +689,15 @@ Other constructors, all class methods: Return the current UTC date and time, with :attr:`tzinfo` ``None``. This is like :meth:`now`, but returns the current UTC date and time, as a naive - :class:`datetime` object. See also :meth:`now`. - + :class:`.datetime` object. An aware current UTC datetime can be obtained by + calling ``datetime.now(timezone.utc)``. See also :meth:`now`. .. classmethod:: datetime.fromtimestamp(timestamp, tz=None) Return the local date and time corresponding to the POSIX timestamp, such as is returned by :func:`time.time`. If optional argument *tz* is ``None`` or not specified, the timestamp is converted to the platform's local date and time, and - the returned :class:`datetime` object is naive. + the returned :class:`.datetime` object is naive. Else *tz* must be an instance of a class :class:`tzinfo` subclass, and the timestamp is converted to *tz*'s time zone. In this case the result is @@ -651,26 +705,26 @@ Other constructors, all class methods: ``tz.fromutc(datetime.utcfromtimestamp(timestamp).replace(tzinfo=tz))``. :meth:`fromtimestamp` may raise :exc:`ValueError`, if the timestamp is out of - the range of values supported by the platform C :cfunc:`localtime` or - :cfunc:`gmtime` functions. It's common for this to be restricted to years in + the range of values supported by the platform C :c:func:`localtime` or + :c:func:`gmtime` functions. It's common for this to be restricted to years in 1970 through 2038. Note that on non-POSIX systems that include leap seconds in their notion of a timestamp, leap seconds are ignored by :meth:`fromtimestamp`, and then it's possible to have two timestamps differing by a second that yield - identical :class:`datetime` objects. See also :meth:`utcfromtimestamp`. + identical :class:`.datetime` objects. See also :meth:`utcfromtimestamp`. .. classmethod:: datetime.utcfromtimestamp(timestamp) - Return the UTC :class:`datetime` corresponding to the POSIX timestamp, with + Return the UTC :class:`.datetime` corresponding to the POSIX timestamp, with :attr:`tzinfo` ``None``. This may raise :exc:`ValueError`, if the timestamp is - out of the range of values supported by the platform C :cfunc:`gmtime` function. + out of the range of values supported by the platform C :c:func:`gmtime` function. It's common for this to be restricted to years in 1970 through 2038. See also :meth:`fromtimestamp`. .. classmethod:: datetime.fromordinal(ordinal) - Return the :class:`datetime` corresponding to the proleptic Gregorian ordinal, + Return the :class:`.datetime` corresponding to the proleptic Gregorian ordinal, where January 1 of year 1 has ordinal 1. :exc:`ValueError` is raised unless ``1 <= ordinal <= datetime.max.toordinal()``. The hour, minute, second and microsecond of the result are all 0, and :attr:`tzinfo` is ``None``. @@ -678,16 +732,18 @@ Other constructors, all class methods: .. classmethod:: datetime.combine(date, time) - Return a new :class:`datetime` object whose date members are equal to the given - :class:`date` object's, and whose time and :attr:`tzinfo` members are equal to - the given :class:`time` object's. For any :class:`datetime` object *d*, ``d == - datetime.combine(d.date(), d.timetz())``. If date is a :class:`datetime` - object, its time and :attr:`tzinfo` members are ignored. + Return a new :class:`.datetime` object whose date components are equal to the + given :class:`date` object's, and whose time components and :attr:`tzinfo` + attributes are equal to the given :class:`.time` object's. For any + :class:`.datetime` object *d*, + ``d == datetime.combine(d.date(), d.timetz())``. If date is a + :class:`.datetime` object, its time components and :attr:`tzinfo` attributes + are ignored. .. classmethod:: datetime.strptime(date_string, format) - Return a :class:`datetime` corresponding to *date_string*, parsed according to + Return a :class:`.datetime` corresponding to *date_string*, parsed according to *format*. This is equivalent to ``datetime(*(time.strptime(date_string, format)[0:6]))``. :exc:`ValueError` is raised if the date_string and format can't be parsed by :func:`time.strptime` or if it returns a value which isn't a @@ -699,19 +755,19 @@ Class attributes: .. attribute:: datetime.min - The earliest representable :class:`datetime`, ``datetime(MINYEAR, 1, 1, + The earliest representable :class:`.datetime`, ``datetime(MINYEAR, 1, 1, tzinfo=None)``. .. attribute:: datetime.max - The latest representable :class:`datetime`, ``datetime(MAXYEAR, 12, 31, 23, 59, + The latest representable :class:`.datetime`, ``datetime(MAXYEAR, 12, 31, 23, 59, 59, 999999, tzinfo=None)``. .. attribute:: datetime.resolution - The smallest possible difference between non-equal :class:`datetime` objects, + The smallest possible difference between non-equal :class:`.datetime` objects, ``timedelta(microseconds=1)``. @@ -754,80 +810,81 @@ Instance attributes (read-only): .. attribute:: datetime.tzinfo - The object passed as the *tzinfo* argument to the :class:`datetime` constructor, + The object passed as the *tzinfo* argument to the :class:`.datetime` constructor, or ``None`` if none was passed. Supported operations: -+---------------------------------------+-------------------------------+ -| Operation | Result | -+=======================================+===============================+ -| ``datetime2 = datetime1 + timedelta`` | \(1) | -+---------------------------------------+-------------------------------+ -| ``datetime2 = datetime1 - timedelta`` | \(2) | -+---------------------------------------+-------------------------------+ -| ``timedelta = datetime1 - datetime2`` | \(3) | -+---------------------------------------+-------------------------------+ -| ``datetime1 < datetime2`` | Compares :class:`datetime` to | -| | :class:`datetime`. (4) | -+---------------------------------------+-------------------------------+ ++---------------------------------------+--------------------------------+ +| Operation | Result | ++=======================================+================================+ +| ``datetime2 = datetime1 + timedelta`` | \(1) | ++---------------------------------------+--------------------------------+ +| ``datetime2 = datetime1 - timedelta`` | \(2) | ++---------------------------------------+--------------------------------+ +| ``timedelta = datetime1 - datetime2`` | \(3) | ++---------------------------------------+--------------------------------+ +| ``datetime1 < datetime2`` | Compares :class:`.datetime` to | +| | :class:`.datetime`. (4) | ++---------------------------------------+--------------------------------+ (1) datetime2 is a duration of timedelta removed from datetime1, moving forward in time if ``timedelta.days`` > 0, or backward if ``timedelta.days`` < 0. The - result has the same :attr:`tzinfo` member as the input datetime, and datetime2 - - datetime1 == timedelta after. :exc:`OverflowError` is raised if datetime2.year - would be smaller than :const:`MINYEAR` or larger than :const:`MAXYEAR`. Note - that no time zone adjustments are done even if the input is an aware object. + result has the same :attr:`tzinfo` attribute as the input datetime, and + datetime2 - datetime1 == timedelta after. :exc:`OverflowError` is raised if + datetime2.year would be smaller than :const:`MINYEAR` or larger than + :const:`MAXYEAR`. Note that no time zone adjustments are done even if the + input is an aware object. (2) Computes the datetime2 such that datetime2 + timedelta == datetime1. As for - addition, the result has the same :attr:`tzinfo` member as the input datetime, - and no time zone adjustments are done even if the input is aware. This isn't - quite equivalent to datetime1 + (-timedelta), because -timedelta in isolation - can overflow in cases where datetime1 - timedelta does not. + addition, the result has the same :attr:`tzinfo` attribute as the input + datetime, and no time zone adjustments are done even if the input is aware. + This isn't quite equivalent to datetime1 + (-timedelta), because -timedelta + in isolation can overflow in cases where datetime1 - timedelta does not. (3) - Subtraction of a :class:`datetime` from a :class:`datetime` is defined only if + Subtraction of a :class:`.datetime` from a :class:`.datetime` is defined only if both operands are naive, or if both are aware. If one is aware and the other is naive, :exc:`TypeError` is raised. - If both are naive, or both are aware and have the same :attr:`tzinfo` member, - the :attr:`tzinfo` members are ignored, and the result is a :class:`timedelta` + If both are naive, or both are aware and have the same :attr:`tzinfo` attribute, + the :attr:`tzinfo` attributes are ignored, and the result is a :class:`timedelta` object *t* such that ``datetime2 + t == datetime1``. No time zone adjustments are done in this case. - If both are aware and have different :attr:`tzinfo` members, ``a-b`` acts as if - *a* and *b* were first converted to naive UTC datetimes first. The result is - ``(a.replace(tzinfo=None) - a.utcoffset()) - (b.replace(tzinfo=None) - - b.utcoffset())`` except that the implementation never overflows. + If both are aware and have different :attr:`tzinfo` attributes, ``a-b`` acts + as if *a* and *b* were first converted to naive UTC datetimes first. The + result is ``(a.replace(tzinfo=None) - a.utcoffset()) - (b.replace(tzinfo=None) + - b.utcoffset())`` except that the implementation never overflows. (4) *datetime1* is considered less than *datetime2* when *datetime1* precedes *datetime2* in time. If one comparand is naive and the other is aware, :exc:`TypeError` is raised. - If both comparands are aware, and have the same :attr:`tzinfo` member, the - common :attr:`tzinfo` member is ignored and the base datetimes are compared. If - both comparands are aware and have different :attr:`tzinfo` members, the - comparands are first adjusted by subtracting their UTC offsets (obtained from - ``self.utcoffset()``). + If both comparands are aware, and have the same :attr:`tzinfo` attribute, the + common :attr:`tzinfo` attribute is ignored and the base datetimes are + compared. If both comparands are aware and have different :attr:`tzinfo` + attributes, the comparands are first adjusted by subtracting their UTC + offsets (obtained from ``self.utcoffset()``). .. note:: In order to stop comparison from falling back to the default scheme of comparing object addresses, datetime comparison normally raises :exc:`TypeError` if the - other comparand isn't also a :class:`datetime` object. However, + other comparand isn't also a :class:`.datetime` object. However, ``NotImplemented`` is returned instead if the other comparand has a :meth:`timetuple` attribute. This hook gives other kinds of date objects a - chance at implementing mixed-type comparison. If not, when a :class:`datetime` + chance at implementing mixed-type comparison. If not, when a :class:`.datetime` object is compared to an object of a different type, :exc:`TypeError` is raised unless the comparison is ``==`` or ``!=``. The latter cases return :const:`False` or :const:`True`, respectively. -:class:`datetime` objects can be used as dictionary keys. In Boolean contexts, -all :class:`datetime` objects are considered to be true. +:class:`.datetime` objects can be used as dictionary keys. In Boolean contexts, +all :class:`.datetime` objects are considered to be true. Instance methods: @@ -838,29 +895,29 @@ Instance methods: .. method:: datetime.time() - Return :class:`time` object with same hour, minute, second and microsecond. + Return :class:`.time` object with same hour, minute, second and microsecond. :attr:`tzinfo` is ``None``. See also method :meth:`timetz`. .. method:: datetime.timetz() - Return :class:`time` object with same hour, minute, second, microsecond, and - tzinfo members. See also method :meth:`time`. + Return :class:`.time` object with same hour, minute, second, microsecond, and + tzinfo attributes. See also method :meth:`time`. .. method:: datetime.replace([year[, month[, day[, hour[, minute[, second[, microsecond[, tzinfo]]]]]]]]) - Return a datetime with the same members, except for those members given new - values by whichever keyword arguments are specified. Note that ``tzinfo=None`` - can be specified to create a naive datetime from an aware datetime with no - conversion of date and time members. + Return a datetime with the same attributes, except for those attributes given + new values by whichever keyword arguments are specified. Note that + ``tzinfo=None`` can be specified to create a naive datetime from an aware + datetime with no conversion of date and time data. .. method:: datetime.astimezone(tz) - Return a :class:`datetime` object with new :attr:`tzinfo` member *tz*, adjusting - the date and time members so the result is the same UTC time as *self*, but in - *tz*'s local time. + Return a :class:`.datetime` object with new :attr:`tzinfo` attribute *tz*, + adjusting the date and time data so the result is the same UTC time as + *self*, but in *tz*'s local time. *tz* must be an instance of a :class:`tzinfo` subclass, and its :meth:`utcoffset` and :meth:`dst` methods must not return ``None``. *self* must @@ -868,18 +925,18 @@ Instance methods: not return ``None``). If ``self.tzinfo`` is *tz*, ``self.astimezone(tz)`` is equal to *self*: no - adjustment of date or time members is performed. Else the result is local time - in time zone *tz*, representing the same UTC time as *self*: after ``astz = - dt.astimezone(tz)``, ``astz - astz.utcoffset()`` will usually have the same date - and time members as ``dt - dt.utcoffset()``. The discussion of class - :class:`tzinfo` explains the cases at Daylight Saving Time transition boundaries - where this cannot be achieved (an issue only if *tz* models both standard and - daylight time). + adjustment of date or time data is performed. Else the result is local + time in time zone *tz*, representing the same UTC time as *self*: after + ``astz = dt.astimezone(tz)``, ``astz - astz.utcoffset()`` will usually have + the same date and time data as ``dt - dt.utcoffset()``. The discussion + of class :class:`tzinfo` explains the cases at Daylight Saving Time transition + boundaries where this cannot be achieved (an issue only if *tz* models both + standard and daylight time). If you merely want to attach a time zone object *tz* to a datetime *dt* without - adjustment of date and time members, use ``dt.replace(tzinfo=tz)``. If you + adjustment of date and time data, use ``dt.replace(tzinfo=tz)``. If you merely want to remove the time zone object from an aware datetime *dt* without - conversion of date and time members, use ``dt.replace(tzinfo=None)``. + conversion of date and time data, use ``dt.replace(tzinfo=None)``. Note that the default :meth:`tzinfo.fromutc` method can be overridden in a :class:`tzinfo` subclass to affect the result returned by :meth:`astimezone`. @@ -927,20 +984,20 @@ Instance methods: of the result is set according to the :meth:`dst` method: :attr:`tzinfo` is ``None`` or :meth:`dst` returns ``None``, :attr:`tm_isdst` is set to ``-1``; else if :meth:`dst` returns a non-zero value, :attr:`tm_isdst` is set to ``1``; - else ``tm_isdst`` is set to ``0``. + else :attr:`tm_isdst` is set to ``0``. .. method:: datetime.utctimetuple() - If :class:`datetime` instance *d* is naive, this is the same as + If :class:`.datetime` instance *d* is naive, this is the same as ``d.timetuple()`` except that :attr:`tm_isdst` is forced to 0 regardless of what ``d.dst()`` returns. DST is never in effect for a UTC time. If *d* is aware, *d* is normalized to UTC time, by subtracting - ``d.utcoffset()``, and a :class:`time.struct_time` for the normalized time is - returned. :attr:`tm_isdst` is forced to 0. Note that the result's - :attr:`tm_year` member may be :const:`MINYEAR`\ -1 or :const:`MAXYEAR`\ +1, if - *d*.year was ``MINYEAR`` or ``MAXYEAR`` and UTC adjustment spills over a year + ``d.utcoffset()``, and a :class:`time.struct_time` for the + normalized time is returned. :attr:`tm_isdst` is forced to 0. Note + that an :exc:`OverflowError` may be raised if *d*.year was + ``MINYEAR`` or ``MAXYEAR`` and UTC adjustment spills over a year boundary. @@ -993,7 +1050,7 @@ Instance methods: .. method:: datetime.__str__() - For a :class:`datetime` instance *d*, ``str(d)`` is equivalent to + For a :class:`.datetime` instance *d*, ``str(d)`` is equivalent to ``d.isoformat(' ')``. @@ -1002,7 +1059,7 @@ Instance methods: Return a string representing the date and time, for example ``datetime(2002, 12, 4, 20, 30, 40).ctime() == 'Wed Dec 4 20:30:40 2002'``. ``d.ctime()`` is equivalent to ``time.ctime(time.mktime(d.timetuple()))`` on platforms where the - native C :cfunc:`ctime` function (which :func:`time.ctime` invokes, but which + native C :c:func:`ctime` function (which :func:`time.ctime` invokes, but which :meth:`datetime.ctime` does not invoke) conforms to the C standard. @@ -1142,19 +1199,19 @@ Class attributes: .. attribute:: time.min - The earliest representable :class:`time`, ``time(0, 0, 0, 0)``. + The earliest representable :class:`.time`, ``time(0, 0, 0, 0)``. .. attribute:: time.max - The latest representable :class:`time`, ``time(23, 59, 59, 999999)``. + The latest representable :class:`.time`, ``time(23, 59, 59, 999999)``. .. attribute:: time.resolution - The smallest possible difference between non-equal :class:`time` objects, - ``timedelta(microseconds=1)``, although note that arithmetic on :class:`time` - objects is not supported. + The smallest possible difference between non-equal :class:`.time` objects, + ``timedelta(microseconds=1)``, although note that arithmetic on + :class:`.time` objects is not supported. Instance attributes (read-only): @@ -1181,29 +1238,29 @@ Instance attributes (read-only): .. attribute:: time.tzinfo - The object passed as the tzinfo argument to the :class:`time` constructor, or + The object passed as the tzinfo argument to the :class:`.time` constructor, or ``None`` if none was passed. Supported operations: -* comparison of :class:`time` to :class:`time`, where *a* is considered less +* comparison of :class:`.time` to :class:`.time`, where *a* is considered less than *b* when *a* precedes *b* in time. If one comparand is naive and the other is aware, :exc:`TypeError` is raised. If both comparands are aware, and have - the same :attr:`tzinfo` member, the common :attr:`tzinfo` member is ignored and - the base times are compared. If both comparands are aware and have different - :attr:`tzinfo` members, the comparands are first adjusted by subtracting their - UTC offsets (obtained from ``self.utcoffset()``). In order to stop mixed-type - comparisons from falling back to the default comparison by object address, when - a :class:`time` object is compared to an object of a different type, - :exc:`TypeError` is raised unless the comparison is ``==`` or ``!=``. The - latter cases return :const:`False` or :const:`True`, respectively. + the same :attr:`tzinfo` attribute, the common :attr:`tzinfo` attribute is + ignored and the base times are compared. If both comparands are aware and + have different :attr:`tzinfo` attributes, the comparands are first adjusted by + subtracting their UTC offsets (obtained from ``self.utcoffset()``). In order + to stop mixed-type comparisons from falling back to the default comparison by + object address, when a :class:`.time` object is compared to an object of a + different type, :exc:`TypeError` is raised unless the comparison is ``==`` or + ``!=``. The latter cases return :const:`False` or :const:`True`, respectively. * hash, use as dict key * efficient pickling -* in Boolean contexts, a :class:`time` object is considered to be true if and +* in Boolean contexts, a :class:`.time` object is considered to be true if and only if, after converting it to minutes and subtracting :meth:`utcoffset` (or ``0`` if that's ``None``), the result is non-zero. @@ -1212,10 +1269,10 @@ Instance methods: .. method:: time.replace([hour[, minute[, second[, microsecond[, tzinfo]]]]]) - Return a :class:`time` with the same value, except for those members given new - values by whichever keyword arguments are specified. Note that ``tzinfo=None`` - can be specified to create a naive :class:`time` from an aware :class:`time`, - without conversion of the time members. + Return a :class:`.time` with the same value, except for those attributes given + new values by whichever keyword arguments are specified. Note that + ``tzinfo=None`` can be specified to create a naive :class:`.time` from an + aware :class:`.time`, without conversion of the time data. .. method:: time.isoformat() @@ -1293,12 +1350,14 @@ Example: :class:`tzinfo` is an abstract base class, meaning that this class should not be instantiated directly. You need to derive a concrete subclass, and (at least) supply implementations of the standard :class:`tzinfo` methods needed by the -:class:`datetime` methods you use. The :mod:`datetime` module does not supply -any concrete subclasses of :class:`tzinfo`. +:class:`.datetime` methods you use. The :mod:`datetime` module supplies +a simple concrete subclass of :class:`tzinfo` :class:`timezone` which can reprsent +timezones with fixed offset from UTC such as UTC itself or North American EST and +EDT. An instance of (a concrete subclass of) :class:`tzinfo` can be passed to the -constructors for :class:`datetime` and :class:`time` objects. The latter objects -view their members as being in local time, and the :class:`tzinfo` object +constructors for :class:`.datetime` and :class:`.time` objects. The latter objects +view their attributes as being in local time, and the :class:`tzinfo` object supports methods revealing offset of local time from UTC, the name of the time zone, and DST offset, all relative to a date or time object passed to them. @@ -1312,7 +1371,7 @@ methods. Exactly which methods are needed depends on the uses made of aware :mod:`datetime` objects. If in doubt, simply implement all of them. -.. method:: tzinfo.utcoffset(self, dt) +.. method:: tzinfo.utcoffset(dt) Return offset of local time from UTC, in minutes east of UTC. If local time is west of UTC, this should be negative. Note that this is intended to be the @@ -1334,7 +1393,7 @@ methods. Exactly which methods are needed depends on the uses made of aware :exc:`NotImplementedError`. -.. method:: tzinfo.dst(self, dt) +.. method:: tzinfo.dst(dt) Return the daylight saving time (DST) adjustment, in minutes east of UTC, or ``None`` if DST information isn't known. Return ``timedelta(0)`` if DST is not @@ -1343,16 +1402,16 @@ methods. Exactly which methods are needed depends on the uses made of aware already been added to the UTC offset returned by :meth:`utcoffset`, so there's no need to consult :meth:`dst` unless you're interested in obtaining DST info separately. For example, :meth:`datetime.timetuple` calls its :attr:`tzinfo` - member's :meth:`dst` method to determine how the :attr:`tm_isdst` flag should be - set, and :meth:`tzinfo.fromutc` calls :meth:`dst` to account for DST changes - when crossing time zones. + attribute's :meth:`dst` method to determine how the :attr:`tm_isdst` flag + should be set, and :meth:`tzinfo.fromutc` calls :meth:`dst` to account for + DST changes when crossing time zones. An instance *tz* of a :class:`tzinfo` subclass that models both standard and daylight times must be consistent in this sense: ``tz.utcoffset(dt) - tz.dst(dt)`` - must return the same result for every :class:`datetime` *dt* with ``dt.tzinfo == + must return the same result for every :class:`.datetime` *dt* with ``dt.tzinfo == tz`` For sane :class:`tzinfo` subclasses, this expression yields the time zone's "standard offset", which should not depend on the date or the time, but only on geographic location. The implementation of :meth:`datetime.astimezone` @@ -1363,13 +1422,13 @@ methods. Exactly which methods are needed depends on the uses made of aware Most implementations of :meth:`dst` will probably look like one of these two:: - def dst(self): + def dst(self, dt): # a fixed-offset class: doesn't account for DST return timedelta(0) or :: - def dst(self): + def dst(self, dt): # Code to set dston and dstoff to the time zone's DST # transition times based on the input dt.year, and expressed # in standard local time. Then @@ -1382,9 +1441,9 @@ methods. Exactly which methods are needed depends on the uses made of aware The default implementation of :meth:`dst` raises :exc:`NotImplementedError`. -.. method:: tzinfo.tzname(self, dt) +.. method:: tzinfo.tzname(dt) - Return the time zone name corresponding to the :class:`datetime` object *dt*, as + Return the time zone name corresponding to the :class:`.datetime` object *dt*, as a string. Nothing about string names is defined by the :mod:`datetime` module, and there's no requirement that it mean anything in particular. For example, "GMT", "UTC", "-500", "-5:00", "EDT", "US/Eastern", "America/New York" are all @@ -1397,11 +1456,11 @@ methods. Exactly which methods are needed depends on the uses made of aware The default implementation of :meth:`tzname` raises :exc:`NotImplementedError`. -These methods are called by a :class:`datetime` or :class:`time` object, in -response to their methods of the same names. A :class:`datetime` object passes -itself as the argument, and a :class:`time` object passes ``None`` as the +These methods are called by a :class:`.datetime` or :class:`.time` object, in +response to their methods of the same names. A :class:`.datetime` object passes +itself as the argument, and a :class:`.time` object passes ``None`` as the argument. A :class:`tzinfo` subclass's methods should therefore be prepared to -accept a *dt* argument of ``None``, or of class :class:`datetime`. +accept a *dt* argument of ``None``, or of class :class:`.datetime`. When ``None`` is passed, it's up to the class designer to decide the best response. For example, returning ``None`` is appropriate if the class wishes to @@ -1409,7 +1468,7 @@ say that time objects don't participate in the :class:`tzinfo` protocols. It may be more useful for ``utcoffset(None)`` to return the standard UTC offset, as there is no other convention for discovering the standard offset. -When a :class:`datetime` object is passed in response to a :class:`datetime` +When a :class:`.datetime` object is passed in response to a :class:`.datetime` method, ``dt.tzinfo`` is the same object as *self*. :class:`tzinfo` methods can rely on this, unless user code calls :class:`tzinfo` methods directly. The intent is that the :class:`tzinfo` methods interpret *dt* as being in local @@ -1418,13 +1477,13 @@ time, and not need worry about objects in other timezones. There is one more :class:`tzinfo` method that a subclass may wish to override: -.. method:: tzinfo.fromutc(self, dt) +.. method:: tzinfo.fromutc(dt) - This is called from the default :class:`datetime.astimezone()` implementation. - When called from that, ``dt.tzinfo`` is *self*, and *dt*'s date and time members - are to be viewed as expressing a UTC time. The purpose of :meth:`fromutc` is to - adjust the date and time members, returning an equivalent datetime in *self*'s - local time. + This is called from the default :class:`datetime.astimezone()` + implementation. When called from that, ``dt.tzinfo`` is *self*, and *dt*'s + date and time data are to be viewed as expressing a UTC time. The purpose + of :meth:`fromutc` is to adjust the date and time data, returning an + equivalent datetime in *self*'s local time. Most :class:`tzinfo` subclasses should be able to inherit the default :meth:`fromutc` implementation without problems. It's strong enough to handle @@ -1495,9 +1554,65 @@ arranged, as in the example, by expressing DST switch times in the time zone's standard local time. Applications that can't bear such ambiguities should avoid using hybrid -:class:`tzinfo` subclasses; there are no ambiguities when using UTC, or any -other fixed-offset :class:`tzinfo` subclass (such as a class representing only -EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). +:class:`tzinfo` subclasses; there are no ambiguities when using :class:`timezone`, +or any other fixed-offset :class:`tzinfo` subclass (such as a class representing +only EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). + + +.. _datetime-timezone: + +:class:`timezone` Objects +-------------------------- + +A :class:`timezone` object represents a timezone that is defined by a +fixed offset from UTC. Note that objects of this class cannot be used +to represent timezone information in the locations where different +offsets are used in different days of the year or where historical +changes have been made to civil time. + + +.. class:: timezone(offset[, name]) + + The *offset* argument must be specified as a :class:`timedelta` + object representing the difference between the local time and UTC. It must + be strictly between ``-timedelta(hours=24)`` and + ``timedelta(hours=24)`` and represent a whole number of minutes, + otherwise :exc:`ValueError` is raised. + + The *name* argument is optional. If specified it must be a string that + is used as the value returned by the ``tzname(dt)`` method. Otherwise, + ``tzname(dt)`` returns a string 'UTCsHH:MM', where s is the sign of + *offset*, HH and MM are two digits of ``offset.hours`` and + ``offset.minutes`` respectively. + +.. method:: timezone.utcoffset(dt) + + Return the fixed value specified when the :class:`timezone` instance is + constructed. The *dt* argument is ignored. The return value is a + :class:`timedelta` instance equal to the difference between the + local time and UTC. + +.. method:: timezone.tzname(dt) + + Return the fixed value specified when the :class:`timezone` instance is + constructed or a string 'UTCsHH:MM', where s is the sign of + *offset*, HH and MM are two digits of ``offset.hours`` and + ``offset.minutes`` respectively. + +.. method:: timezone.dst(dt) + + Always returns ``None``. + +.. method:: timezone.fromutc(dt) + + Return ``dt + offset``. The *dt* argument must be an aware + :class:`.datetime` instance, with ``tzinfo`` set to ``self``. + +Class attributes: + +.. attribute:: timezone.utc + + The UTC timezone, ``timezone(timedelta(0))``. .. _strftime-strptime-behavior: @@ -1505,18 +1620,18 @@ EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). :meth:`strftime` and :meth:`strptime` Behavior ---------------------------------------------- -:class:`date`, :class:`datetime`, and :class:`time` objects all support a +:class:`date`, :class:`.datetime`, and :class:`.time` objects all support a ``strftime(format)`` method, to create a string representing the time under the control of an explicit format string. Broadly speaking, ``d.strftime(fmt)`` acts like the :mod:`time` module's ``time.strftime(fmt, d.timetuple())`` although not all objects support a :meth:`timetuple` method. Conversely, the :meth:`datetime.strptime` class method creates a -:class:`datetime` object from a string representing a date and time and a +:class:`.datetime` object from a string representing a date and time and a corresponding format string. ``datetime.strptime(date_string, format)`` is equivalent to ``datetime(*(time.strptime(date_string, format)[0:6]))``. -For :class:`time` objects, the format codes for year, month, and day should not +For :class:`.time` objects, the format codes for year, month, and day should not be used, as time objects have no such values. If they're used anyway, ``1900`` is substituted for the year, and ``1`` for the month and day. @@ -1549,9 +1664,6 @@ version) requires, and these work on all platforms with a standard C implementation. Note that the 1999 version of the C standard added additional format codes. -The exact range of years for which :meth:`strftime` works also varies across -platforms. Regardless of platform, years before 1900 cannot be used. - +-----------+--------------------------------+-------+ | Directive | Meaning | Notes | +===========+================================+=======+ @@ -1594,7 +1706,7 @@ platforms. Regardless of platform, years before 1900 cannot be used. | | AM or PM. | | +-----------+--------------------------------+-------+ | ``%S`` | Second as a decimal number | \(3) | -| | [00,61]. | | +| | [00,59]. | | +-----------+--------------------------------+-------+ | ``%U`` | Week number of the year | \(4) | | | (Sunday as the first day of | | @@ -1624,10 +1736,11 @@ platforms. Regardless of platform, years before 1900 cannot be used. | ``%y`` | Year without century as a | | | | decimal number [00,99]. | | +-----------+--------------------------------+-------+ -| ``%Y`` | Year with century as a decimal | | -| | number. | | +| ``%Y`` | Year with century as a decimal | \(5) | +| | number [0001,9999] (strptime), | | +| | [1000,9999] (strftime). | | +-----------+--------------------------------+-------+ -| ``%z`` | UTC offset in the form +HHMM | \(5) | +| ``%z`` | UTC offset in the form +HHMM | \(6) | | | or -HHMM (empty string if the | | | | the object is naive). | | +-----------+--------------------------------+-------+ @@ -1651,17 +1764,30 @@ Notes: the output hour field if the ``%I`` directive is used to parse the hour. (3) - The range really is ``0`` to ``61``; according to the Posix standard this - accounts for leap seconds and the (very rare) double leap seconds. - The :mod:`time` module may produce and does accept leap seconds since - it is based on the Posix standard, but the :mod:`datetime` module - does not accept leap seconds in :meth:`strptime` input nor will it - produce them in :func:`strftime` output. + Unlike :mod:`time` module, :mod:`datetime` module does not support + leap seconds. (4) When used with the :meth:`strptime` method, ``%U`` and ``%W`` are only used in calculations when the day of the week and the year are specified. (5) + For technical reasons, :meth:`strftime` method does not support + dates before year 1000: ``t.strftime(format)`` will raise a + :exc:`ValueError` when ``t.year < 1000`` even if ``format`` does + not contain ``%Y`` directive. The :meth:`strptime` method can + parse years in the full [1, 9999] range, but years < 1000 must be + zero-filled to 4-digit width. + + .. versionchanged:: 3.2 + In previous versions, :meth:`strftime` method was restricted to + years >= 1900. + +(6) For example, if :meth:`utcoffset` returns ``timedelta(hours=-3, minutes=-30)``, ``%z`` is replaced with the string ``'-0330'``. + +.. versionchanged:: 3.2 + When the ``%z`` directive is provided to the :meth:`strptime` method, an + aware :class:`.datetime` object will be produced. The ``tzinfo`` of the + result will be set to a :class:`timezone` instance. diff --git a/Doc/library/dbm.rst b/Doc/library/dbm.rst index cbefc1aa11..e3d50b9d94 100644 --- a/Doc/library/dbm.rst +++ b/Doc/library/dbm.rst @@ -61,10 +61,15 @@ the Oracle Berkeley DB. modified by the prevailing umask). -The object returned by :func:`.open` supports most of the same functionality as +The object returned by :func:`.open` supports the same basic functionality as dictionaries; keys and their corresponding values can be stored, retrieved, and deleted, and the :keyword:`in` operator and the :meth:`keys` method are -available. Key and values are always stored as bytes. This means that when +available, as well as :meth:`get` and :meth:`setdefault`. + +.. versionchanged:: 3.2 + :meth:`get` and :meth:`setdefault` are now available in all database modules. + +Key and values are always stored as bytes. This means that when strings are used they are implicitly converted to the default encoding before being stored. diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index 758dcce9e3..ef8b43f7f6 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -29,7 +29,7 @@ arithmetic. It offers several advantages over the :class:`float` datatype: people learn at school." -- excerpt from the decimal arithmetic specification. * Decimal numbers can be represented exactly. In contrast, numbers like - :const:`1.1` and :const:`2.2` do not have an exact representations in binary + :const:`1.1` and :const:`2.2` do not have exact representations in binary floating point. End users typically would not expect ``1.1 + 2.2`` to display as :const:`3.3000000000000003` as it does with binary floating point. @@ -123,15 +123,14 @@ precision, rounding, or enabled traps:: >>> from decimal import * >>> getcontext() Context(prec=28, rounding=ROUND_HALF_EVEN, Emin=-999999999, Emax=999999999, - capitals=1, flags=[], traps=[Overflow, DivisionByZero, + capitals=1, clamp=0, flags=[], traps=[Overflow, DivisionByZero, InvalidOperation]) >>> getcontext().prec = 7 # Set a new precision -Decimal instances can be constructed from integers, strings, or tuples. To -create a Decimal from a :class:`float`, first convert it to a string. This -serves as an explicit reminder of the details of the conversion (including -representation error). Decimal numbers include special values such as +Decimal instances can be constructed from integers, strings, floats, or tuples. +Construction from an integer or a float performs an exact conversion of the +value of that integer or float. Decimal numbers include special values such as :const:`NaN` which stands for "Not a number", positive and negative :const:`Infinity`, and :const:`-0`. @@ -140,10 +139,12 @@ representation error). Decimal numbers include special values such as Decimal('10') >>> Decimal('3.14') Decimal('3.14') + >>> Decimal(3.14) + Decimal('3.140000000000000124344978758017532527446746826171875') >>> Decimal((0, (3, 1, 4), -2)) Decimal('3.14') >>> Decimal(str(2.0 ** 0.5)) - Decimal('1.41421356237') + Decimal('1.4142135623730951') >>> Decimal(2) ** Decimal('0.5') Decimal('1.414213562373095048801688724') >>> Decimal('NaN') @@ -244,7 +245,7 @@ enabled: >>> ExtendedContext Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999999, Emax=999999999, - capitals=1, flags=[], traps=[]) + capitals=1, clamp=0, flags=[], traps=[]) >>> setcontext(ExtendedContext) >>> Decimal(1) / Decimal(7) Decimal('0.142857143') @@ -269,7 +270,7 @@ using the :meth:`clear_flags` method. :: Decimal('3.14159292') >>> getcontext() Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999999, Emax=999999999, - capitals=1, flags=[Inexact, Rounded], traps=[]) + capitals=1, clamp=0, flags=[Inexact, Rounded], traps=[]) The *flags* entry shows that the rational approximation to :const:`Pi` was rounded (digits beyond the context precision were thrown away) and that the @@ -309,7 +310,7 @@ Decimal objects Construct a new :class:`Decimal` object based from *value*. - *value* can be an integer, string, tuple, or another :class:`Decimal` + *value* can be an integer, string, tuple, :class:`float`, or another :class:`Decimal` object. If no *value* is given, returns ``Decimal('0')``. If *value* is a string, it should conform to the decimal numeric string syntax after leading and trailing whitespace characters are removed:: @@ -335,6 +336,12 @@ Decimal objects digits, and an integer exponent. For example, ``Decimal((0, (1, 4, 1, 4), -3))`` returns ``Decimal('1.414')``. + If *value* is a :class:`float`, the binary floating point value is losslessly + converted to its exact decimal equivalent. This conversion can often require + 53 or more digits of precision. For example, ``Decimal(float('1.1'))`` + converts to + ``Decimal('1.100000000000000088817841970012523233890533447265625')``. + The *context* precision does not affect how many digits are stored. That is determined exclusively by the number of digits in *value*. For example, ``Decimal('3.00000')`` records all five zeros even if the context precision is @@ -347,6 +354,10 @@ Decimal objects Once constructed, :class:`Decimal` objects are immutable. + .. versionchanged:: 3.2 + The argument to the constructor is now permitted to be a :class:`float` + instance. + Decimal floating point objects share many properties with the other built-in numeric types such as :class:`float` and :class:`int`. All of the usual math operations and special methods apply. Likewise, decimal objects can be @@ -354,6 +365,18 @@ Decimal objects compared, sorted, and coerced to another type (such as :class:`float` or :class:`int`). + Decimal objects cannot generally be combined with floats or + instances of :class:`fractions.Fraction` in arithmetic operations: + an attempt to add a :class:`Decimal` to a :class:`float`, for + example, will raise a :exc:`TypeError`. However, it is possible to + use Python's comparison operators to compare a :class:`Decimal` + instance ``x`` with another number ``y``. This avoids confusing results + when doing equality comparisons between numbers of different types. + + .. versionchanged:: 3.2 + Mixed-type comparisons between :class:`Decimal` instances and other + numeric types are now fully supported. + In addition to the standard numeric properties, decimal floating point objects also have a number of specialized methods: @@ -468,6 +491,9 @@ Decimal objects `0x1.999999999999ap-4`. That equivalent value in decimal is `0.1000000000000000055511151231257827021181583404541015625`. + .. note:: From Python 3.2 onwards, a :class:`Decimal` instance + can also be constructed directly from a :class:`float`. + .. doctest:: >>> Decimal.from_float(0.1) @@ -628,7 +654,7 @@ Decimal objects Normalize the number by stripping the rightmost trailing zeros and converting any result equal to :const:`Decimal('0')` to - :const:`Decimal('0e0')`. Used for producing canonical values for members + :const:`Decimal('0e0')`. Used for producing canonical values for attributes of an equivalence class. For example, ``Decimal('32.100')`` and ``Decimal('0.321000e+2')`` both normalize to the equivalent value ``Decimal('32.1')``. @@ -861,7 +887,7 @@ In addition to the three supplied contexts, new contexts can be created with the :class:`Context` constructor. -.. class:: Context(prec=None, rounding=None, traps=None, flags=None, Emin=None, Emax=None, capitals=1) +.. class:: Context(prec=None, rounding=None, traps=None, flags=None, Emin=None, Emax=None, capitals=None, clamp=None) Creates a new context. If a field is not specified or is :const:`None`, the default values are copied from the :const:`DefaultContext`. If the *flags* @@ -892,13 +918,33 @@ In addition to the three supplied contexts, new contexts can be created with the :const:`1`, exponents are printed with a capital :const:`E`; otherwise, a lowercase :const:`e` is used: :const:`Decimal('6.02e+23')`. + The *clamp* field is either :const:`0` (the default) or :const:`1`. + If set to :const:`1`, the exponent ``e`` of a :class:`Decimal` + instance representable in this context is strictly limited to the + range ``Emin - prec + 1 <= e <= Emax - prec + 1``. If *clamp* is + :const:`0` then a weaker condition holds: the adjusted exponent of + the :class:`Decimal` instance is at most ``Emax``. When *clamp* is + :const:`1`, a large normal number will, where possible, have its + exponent reduced and a corresponding number of zeros added to its + coefficient, in order to fit the exponent constraints; this + preserves the value of the number but loses information about + significant trailing zeros. For example:: + + >>> Context(prec=6, Emax=999, clamp=1).create_decimal('1.23e999') + Decimal('1.23000E+999') + + A *clamp* value of :const:`1` allows compatibility with the + fixed-width decimal interchange formats specified in IEEE 754. The :class:`Context` class defines several general purpose methods as well as a large number of methods for doing arithmetic directly in a given context. In addition, for each of the :class:`Decimal` methods described above (with the exception of the :meth:`adjusted` and :meth:`as_tuple` methods) there is - a corresponding :class:`Context` method. For example, ``C.exp(x)`` is - equivalent to ``x.exp(context=C)``. + a corresponding :class:`Context` method. For example, for a :class:`Context` + instance ``C`` and :class:`Decimal` instance ``x``, ``C.exp(x)`` is + equivalent to ``x.exp(context=C)``. Each :class:`Context` method accepts a + Python integer (an instance of :class:`int`) anywhere that a + Decimal instance is accepted. .. method:: clear_flags() @@ -963,7 +1009,6 @@ In addition to the three supplied contexts, new contexts can be created with the value for subnormal results. When underflow occurs, the exponent is set to :const:`Etiny`. - .. method:: Etop() Returns a value equal to ``Emax - prec + 1``. @@ -1612,7 +1657,8 @@ to work with the :class:`Decimal` class:: build(trailneg) for i in range(places): build(next() if digits else '0') - build(dp) + if places: + build(dp) if not digits: build('0') i = 0 @@ -1672,6 +1718,9 @@ to work with the :class:`Decimal` class:: def cos(x): """Return the cosine of x as measured in radians. + The Taylor series approximation works best for a small value of x. + For larger values, first compute x = x % (2 * pi). + >>> print(cos(Decimal('0.5'))) 0.8775825618903727161162815826 >>> print(cos(0.5)) @@ -1695,6 +1744,9 @@ to work with the :class:`Decimal` class:: def sin(x): """Return the sine of x as measured in radians. + The Taylor series approximation works best for a small value of x. + For larger values, first compute x = x % (2 * pi). + >>> print(sin(Decimal('0.5'))) 0.4794255386042030002732879352 >>> print(sin(0.5)) @@ -1821,37 +1873,15 @@ value unchanged: Q. Is there a way to convert a regular float to a :class:`Decimal`? -A. Yes, all binary floating point numbers can be exactly expressed as a -Decimal. An exact conversion may take more precision than intuition would -suggest, so we trap :const:`Inexact` to signal a need for more precision: - -.. testcode:: - - def float_to_decimal(f): - "Convert a floating point number to a Decimal with no loss of information" - n, d = f.as_integer_ratio() - with localcontext() as ctx: - ctx.traps[Inexact] = True - while True: - try: - return Decimal(n) / Decimal(d) - except Inexact: - ctx.prec += 1 +A. Yes, any binary floating point number can be exactly expressed as a +Decimal though an exact conversion may take more precision than intuition would +suggest: .. doctest:: - >>> float_to_decimal(math.pi) + >>> Decimal(math.pi) Decimal('3.141592653589793115997963468544185161590576171875') -Q. Why isn't the :func:`float_to_decimal` routine included in the module? - -A. There is some question about whether it is advisable to mix binary and -decimal floating point. Also, its use requires some care to avoid the -representation issues associated with binary floating point: - - >>> float_to_decimal(1.1) - Decimal('1.100000000000000088817841970012523233890533447265625') - Q. Within a complex calculation, how can I make sure that I haven't gotten a spurious result because of insufficient precision or rounding anomalies. diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index 6dea8c1bb8..bdc37b3e0a 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -17,6 +17,7 @@ can be used for example, for comparing files, and can produce difference information in various formats, including HTML and context and unified diffs. For comparing directories and files, see also, the :mod:`filecmp` module. + .. class:: SequenceMatcher This is a flexible class for comparing pairs of sequences of any type, so long @@ -35,11 +36,17 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. complicated way on how many elements the sequences have in common; best case time is linear. - **Heuristic:** To speed-up matching, items whose duplicates appear more than 1% of - the time in sequences of at least 200 items are treated as junk. This has the - unfortunate side-effect of giving bad results for sequences constructed from - a small set of items. An option to turn off the heuristic will be added to - Python 3.2. + **Automatic junk heuristic:** :class:`SequenceMatcher` supports a heuristic that + automatically treats certain sequence items as junk. The heuristic counts how many + times each individual item appears in the sequence. If an item's duplicates (after + the first one) account for more than 1% of the sequence and the sequence is at least + 200 items long, this item is marked as "popular" and is treated as junk for + the purpose of sequence matching. This heuristic can be turned off by setting + the ``autojunk`` argument to ``False`` when creating the :class:`SequenceMatcher`. + + .. versionadded:: 3.2 + The *autojunk* parameter. + .. class:: Differ @@ -145,8 +152,8 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. The context diff format normally has a header for filenames and modification times. Any or all of these may be specified using strings for *fromfile*, - *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally - expressed in the format returned by :func:`time.ctime`. If not specified, the + *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally + expressed in the ISO 8601 format. If not specified, the strings default to blanks. >>> s1 = ['bacon\n', 'eggs\n', 'ham\n', 'guido\n'] @@ -277,8 +284,8 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. The context diff format normally has a header for filenames and modification times. Any or all of these may be specified using strings for *fromfile*, - *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally - expressed in the format returned by :func:`time.ctime`. If not specified, the + *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally + expressed in the ISO 8601 format. If not specified, the strings default to blanks. @@ -329,7 +336,7 @@ SequenceMatcher Objects The :class:`SequenceMatcher` class has this constructor: -.. class:: SequenceMatcher(isjunk=None, a='', b='') +.. class:: SequenceMatcher(isjunk=None, a='', b='', autojunk=True) Optional argument *isjunk* must be ``None`` (the default) or a one-argument function that takes a sequence element and returns true if and only if the @@ -345,6 +352,22 @@ The :class:`SequenceMatcher` class has this constructor: The optional arguments *a* and *b* are sequences to be compared; both default to empty strings. The elements of both sequences must be :term:`hashable`. + The optional argument *autojunk* can be used to disable the automatic junk + heuristic. + + .. versionadded:: 3.2 + The *autojunk* parameter. + + SequenceMatcher objects get three data attributes: *bjunk* is the + set of elements of *b* for which *isjunk* is True; *bpopular* is the set of + non-junk elements considered popular by the heuristic (if it is not + disabled); *b2j* is a dict mapping the remaining elements of *b* to a list + of positions where they occur. All three are reset whenever *b* is reset + with :meth:`set_seqs` or :meth:`set_seq2`. + + .. versionadded:: 3.2 + The *bjunk* and *bpopular* attributes. + :class:`SequenceMatcher` objects have the following methods: .. method:: set_seqs(a, b) @@ -460,13 +483,15 @@ The :class:`SequenceMatcher` class has this constructor: >>> b = "abycdf" >>> s = SequenceMatcher(None, a, b) >>> for tag, i1, i2, j1, j2 in s.get_opcodes(): - ... print(("%7s a[%d:%d] (%s) b[%d:%d] (%s)" % - ... (tag, i1, i2, a[i1:i2], j1, j2, b[j1:j2]))) - delete a[0:1] (q) b[0:0] () - equal a[1:3] (ab) b[0:2] (ab) - replace a[3:4] (x) b[2:3] (y) - equal a[4:6] (cd) b[3:5] (cd) - insert a[6:6] () b[5:6] (f) + print('{:7} a[{}:{}] --> b[{}:{}] {!r:>8} --> {!r}'.format( + tag, i1, i2, j1, j2, a[i1:i2], b[j1:j2])) + + + delete a[0:1] --> b[0:0] 'q' --> '' + equal a[1:3] --> b[0:2] 'ab' --> 'ab' + replace a[3:4] --> b[2:3] 'x' --> 'y' + equal a[4:6] --> b[3:5] 'cd' --> 'cd' + insert a[6:6] --> b[5:6] '' --> 'f' .. method:: get_grouped_opcodes(n=3) @@ -524,7 +549,7 @@ different results due to differing levels of approximation, although SequenceMatcher Examples ------------------------ -This example compares two strings, considering blanks to be "junk:" +This example compares two strings, considering blanks to be "junk": >>> s = SequenceMatcher(lambda x: x == " ", ... "private Thread currentThread;", diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 4aae06b60b..79cc583b75 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -4,6 +4,9 @@ .. module:: dis :synopsis: Disassembler for Python bytecode. +**Source code:** :source:`Lib/dis.py` + +-------------- The :mod:`dis` module supports the analysis of CPython :term:`bytecode` by disassembling it. The CPython bytecode which this module takes as an @@ -12,7 +15,7 @@ and the interpreter. .. impl-detail:: - Bytecode is an implementation detail of the CPython interpreter! No + Bytecode is an implementation detail of the CPython interpreter. No guarantees are made that bytecode will not be added, removed, or changed between versions of Python. Use of this module should not be considered to work across Python VMs or Python releases. @@ -36,6 +39,28 @@ the following command can be used to get the disassembly of :func:`myfunc`:: The :mod:`dis` module defines the following functions and constants: +.. function:: code_info(x) + + Return a formatted multi-line string with detailed code object information + for the supplied function, method, source code string or code object. + + Note that the exact contents of code info strings are highly implementation + dependent and they may change arbitrarily across Python VMs or Python + releases. + + .. versionadded:: 3.2 + + +.. function:: show_code(x) + + Print detailed code object information for the supplied function, method, + source code string or code object to stdout. + + This is a convenient shorthand for ``print(code_info(x))``, intended for + interactive exploration at the interpreter prompt. + + .. versionadded:: 3.2 + .. function:: dis(x=None) Disassemble the *x* object. *x* can denote either a module, a class, a @@ -172,15 +197,15 @@ The Python compiler currently generates the following bytecode instructions. three. -.. opcode:: ROT_FOUR +.. opcode:: DUP_TOP - Lifts second, third and forth stack item one position up, moves top down to - position four. + Duplicates the reference on top of the stack. -.. opcode:: DUP_TOP +.. opcode:: DUP_TOP_TWO - Duplicates the reference on top of the stack. + Duplicates the two references on top of the stack, leaving them in the + same order. **Unary operations** @@ -445,6 +470,18 @@ the stack so that it is available for further iterations of the loop. by ``CALL_FUNCTION`` to construct a class. +.. opcode:: SETUP_WITH (delta) + + This opcode performs several operations before a with block starts. First, + it loads :meth:`~object.__exit__` from the context manager and pushes it onto + the stack for later use by :opcode:`WITH_CLEANUP`. Then, + :meth:`~object.__enter__` is called, and a finally block pointing to *delta* + is pushed. Finally, the result of calling the enter method is pushed onto + the stack. The next opcode will either ignore it (:opcode:`POP_TOP`), or + store it in (a) variable(s) (:opcode:`STORE_FAST`, :opcode:`STORE_NAME`, or + :opcode:`UNPACK_SEQUENCE`). + + .. opcode:: WITH_CLEANUP Cleans up the stack when a :keyword:`with` statement block exits. TOS is @@ -507,12 +544,6 @@ the more significant byte last. are put onto the stack right-to-left. -.. opcode:: DUP_TOPX (count) - - Duplicate *count* items, keeping them in the same order. Due to implementation - limits, *count* should be between 1 and 5 inclusive. - - .. opcode:: STORE_ATTR (namei) Implements ``TOS.name = TOS1``, where *namei* is the index of name in @@ -695,6 +726,12 @@ the more significant byte last. storage. +.. opcode:: DELETE_DEREF (i) + + Empties the cell contained in slot *i* of the cell and free variable storage. + Used by the :keyword:`del` statement. + + .. opcode:: RAISE_VARARGS (argc) Raises an exception. *argc* indicates the number of parameters to the raise diff --git a/Doc/library/doctest.rst b/Doc/library/doctest.rst index 5f40432e45..cdd6c26245 100644 --- a/Doc/library/doctest.rst +++ b/Doc/library/doctest.rst @@ -452,8 +452,9 @@ Some details you should read once, but won't need to remember: with an alphanumeric is taken to be the start of the exception detail. Of course this does the right thing for genuine tracebacks. -* When the :const:`IGNORE_EXCEPTION_DETAIL` doctest option is is specified, - everything following the leftmost colon is ignored. +* When the :const:`IGNORE_EXCEPTION_DETAIL` doctest option is specified, + everything following the leftmost colon and any module information in the + exception name is ignored. * The interactive shell omits the traceback header line for some :exc:`SyntaxError`\ s. But doctest uses the traceback header line to @@ -543,20 +544,38 @@ doctest decides whether actual output matches an example's expected output: exception raised is ``ValueError: 3*14``, but will fail, e.g., if :exc:`TypeError` is raised. - Note that a similar effect can be obtained using :const:`ELLIPSIS`, and - :const:`IGNORE_EXCEPTION_DETAIL` may go away when Python releases prior to 2.4 - become uninteresting. Until then, :const:`IGNORE_EXCEPTION_DETAIL` is the only - clear way to write a doctest that doesn't care about the exception detail yet - continues to pass under Python releases prior to 2.4 (doctest directives appear - to be comments to them). For example, :: + It will also ignore the module name used in Python 3 doctest reports. Hence + both these variations will work regardless of whether the test is run under + Python 2.7 or Python 3.2 (or later versions): + + >>> raise CustomError('message') #doctest: +IGNORE_EXCEPTION_DETAIL + Traceback (most recent call last): + CustomError: message + + >>> raise CustomError('message') #doctest: +IGNORE_EXCEPTION_DETAIL + Traceback (most recent call last): + my_module.CustomError: message + + Note that :const:`ELLIPSIS` can also be used to ignore the + details of the exception message, but such a test may still fail based + on whether or not the module details are printed as part of the + exception name. Using :const:`IGNORE_EXCEPTION_DETAIL` and the details + from Python 2.3 is also the only clear way to write a doctest that doesn't + care about the exception detail yet continues to pass under Python 2.3 or + earlier (those releases do not support doctest directives and ignore them + as irrelevant comments). For example, :: >>> (1, 2)[3] = 'moo' #doctest: +IGNORE_EXCEPTION_DETAIL Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: object doesn't support item assignment - passes under Python 2.4 and Python 2.3. The detail changed in 2.4, to say "does - not" instead of "doesn't". + passes under Python 2.3 and later Python versions, even though the detail + changed in Python 2.4 to say "does not" instead of "doesn't". + + .. versionchanged:: 3.2 + :const:`IGNORE_EXCEPTION_DETAIL` now also ignores any information relating + to the module containing the exception under test. .. data:: SKIP @@ -671,7 +690,6 @@ usually the only meaningful choice. However, option flags can also be passed to functions that run doctests, establishing different defaults. In such cases, disabling an option via ``-`` in a directive can be useful. - There's also a way to register new option flag names, although this isn't useful unless you intend to extend :mod:`doctest` internals via subclassing: @@ -895,18 +913,16 @@ Unittest API As your collection of doctest'ed modules grows, you'll want a way to run all their doctests systematically. :mod:`doctest` provides two functions that can be used to create :mod:`unittest` test suites from modules and text files -containing doctests. These test suites can then be run using :mod:`unittest` -test runners:: +containing doctests. To integrate with :mod:`unittest` test discovery, include +a :func:`load_tests` function in your test module:: import unittest import doctest - import my_module_with_doctests, and_another + import my_module_with_doctests - suite = unittest.TestSuite() - for mod in my_module_with_doctests, and_another: - suite.addTest(doctest.DocTestSuite(mod)) - runner = unittest.TextTestRunner() - runner.run(suite) + def load_tests(loader, tests, ignore): + tests.addTests(doctest.DocTestSuite(my_module_with_doctests)) + return tests There are two main functions for creating :class:`unittest.TestSuite` instances from text files and modules with doctests: @@ -1111,11 +1127,10 @@ DocTest Objects .. class:: DocTest(examples, globs, name, filename, lineno, docstring) A collection of doctest examples that should be run in a single namespace. The - constructor arguments are used to initialize the member variables of the same - names. + constructor arguments are used to initialize the attributes of the same names. - :class:`DocTest` defines the following member variables. They are initialized by + :class:`DocTest` defines the following attributes. They are initialized by the constructor, and should not be modified directly. @@ -1168,11 +1183,11 @@ Example Objects .. class:: Example(source, want, exc_msg=None, lineno=0, indent=0, options=None) A single interactive example, consisting of a Python statement and its expected - output. The constructor arguments are used to initialize the member variables - of the same names. + output. The constructor arguments are used to initialize the attributes of + the same names. - :class:`Example` defines the following member variables. They are initialized by + :class:`Example` defines the following attributes. They are initialized by the constructor, and should not be modified directly. @@ -1659,9 +1674,9 @@ There are two exceptions that may be raised by :class:`DebugRunner` instances: An exception raised by :class:`DocTestRunner` to signal that a doctest example's actual output did not match its expected output. The constructor arguments are - used to initialize the member variables of the same names. + used to initialize the attributes of the same names. -:exc:`DocTestFailure` defines the following member variables: +:exc:`DocTestFailure` defines the following attributes: .. attribute:: DocTestFailure.test @@ -1683,9 +1698,9 @@ There are two exceptions that may be raised by :class:`DebugRunner` instances: An exception raised by :class:`DocTestRunner` to signal that a doctest example raised an unexpected exception. The constructor arguments are used - to initialize the member variables of the same names. + to initialize the attributes of the same names. -:exc:`UnexpectedException` defines the following member variables: +:exc:`UnexpectedException` defines the following attributes: .. attribute:: UnexpectedException.test diff --git a/Doc/library/dummy_threading.rst b/Doc/library/dummy_threading.rst index 0658df2d27..b578324873 100644 --- a/Doc/library/dummy_threading.rst +++ b/Doc/library/dummy_threading.rst @@ -4,6 +4,9 @@ .. module:: dummy_threading :synopsis: Drop-in replacement for the threading module. +**Source code:** :source:`Lib/dummy_threading.py` + +-------------- This module provides a duplicate interface to the :mod:`threading` module. It is meant to be imported when the :mod:`_thread` module is not provided on a diff --git a/Doc/library/email-examples.rst b/Doc/library/email-examples.rst index c1b16da394..32cecf3486 100644 --- a/Doc/library/email-examples.rst +++ b/Doc/library/email-examples.rst @@ -11,6 +11,12 @@ First, let's see how to create and send a simple text message: .. literalinclude:: ../includes/email-simple.py +And parsing RFC822 headers can easily be done by the parse(filename) or +parsestr(message_as_string) methods of the Parser() class: + +.. literalinclude:: ../includes/email-headers.py + + Here's an example of how to send a MIME message containing a bunch of family pictures that may be residing in a directory: diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst index 930905aee0..85b32fe7fe 100644 --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -22,6 +22,12 @@ the Generator on a :class:`~email.message.Message` constructed by program may result in changes to the :class:`~email.message.Message` object as defaults are filled in. +:class:`bytes` output can be generated using the :class:`BytesGenerator` class. +If the message object structure contains non-ASCII bytes, this generator's +:meth:`~BytesGenerator.flatten` method will emit the original bytes. Parsing a +binary message and then flattening it with :class:`BytesGenerator` should be +idempotent for standards compliant messages. + Here are the public methods of the :class:`Generator` class, imported from the :mod:`email.generator` module: @@ -50,7 +56,7 @@ Here are the public methods of the :class:`Generator` class, imported from the The other public :class:`Generator` methods are: - .. method:: flatten(msg, unixfrom=False) + .. method:: flatten(msg, unixfrom=False, linesep='\\n') Print the textual representation of the message object structure rooted at *msg* to the output file specified when the :class:`Generator` instance @@ -65,6 +71,21 @@ Here are the public methods of the :class:`Generator` class, imported from the Note that for subparts, no envelope header is ever printed. + Optional *linesep* specifies the line separator character used to + terminate lines in the output. It defaults to ``\n`` because that is + the most useful value for Python application code (other library packages + expect ``\n`` separated lines). ``linesep=\r\n`` can be used to + generate output with RFC-compliant line separators. + + Messages parsed with a Bytes parser that have a + :mailheader:`Content-Transfer-Encoding` of 8bit will be converted to a + use a 7bit Content-Transfer-Encoding. Non-ASCII bytes in the headers + will be :rfc:`2047` encoded with a charset of `unknown-8bit`. + + .. versionchanged:: 3.2 + Added support for re-encoding 8bit message bodies, and the *linesep* + argument. + .. method:: clone(fp) Return an independent clone of this :class:`Generator` instance with the @@ -76,11 +97,81 @@ Here are the public methods of the :class:`Generator` class, imported from the :class:`Generator`'s constructor. This provides just enough file-like API for :class:`Generator` instances to be used in the :func:`print` function. -As a convenience, see the methods :meth:`Message.as_string` and -``str(aMessage)``, a.k.a. :meth:`Message.__str__`, which simplify the generation -of a formatted string representation of a message object. For more detail, see +As a convenience, see the :class:`~email.message.Message` methods +:meth:`~email.message.Message.as_string` and ``str(aMessage)``, a.k.a. +:meth:`~email.message.Message.__str__`, which simplify the generation of a +formatted string representation of a message object. For more detail, see :mod:`email.message`. +.. class:: BytesGenerator(outfp, mangle_from_=True, maxheaderlen=78) + + The constructor for the :class:`BytesGenerator` class takes a binary + :term:`file-like object` called *outfp* for an argument. *outfp* must + support a :meth:`write` method that accepts binary data. + + Optional *mangle_from_* is a flag that, when ``True``, puts a ``>`` + character in front of any line in the body that starts exactly as ``From``, + i.e. ``From`` followed by a space at the beginning of the line. This is the + only guaranteed portable way to avoid having such lines be mistaken for a + Unix mailbox format envelope header separator (see `WHY THE CONTENT-LENGTH + FORMAT IS BAD <http://www.jwz.org/doc/content-length.html>`_ for details). + *mangle_from_* defaults to ``True``, but you might want to set this to + ``False`` if you are not writing Unix mailbox format files. + + Optional *maxheaderlen* specifies the longest length for a non-continued + header. When a header line is longer than *maxheaderlen* (in characters, + with tabs expanded to 8 spaces), the header will be split as defined in the + :class:`~email.header.Header` class. Set to zero to disable header + wrapping. The default is 78, as recommended (but not required) by + :rfc:`2822`. + + The other public :class:`BytesGenerator` methods are: + + + .. method:: flatten(msg, unixfrom=False, linesep='\n') + + Print the textual representation of the message object structure rooted + at *msg* to the output file specified when the :class:`BytesGenerator` + instance was created. Subparts are visited depth-first and the resulting + text will be properly MIME encoded. If the input that created the *msg* + contained bytes with the high bit set and those bytes have not been + modified, they will be copied faithfully to the output, even if doing so + is not strictly RFC compliant. (To produce strictly RFC compliant + output, use the :class:`Generator` class.) + + Messages parsed with a Bytes parser that have a + :mailheader:`Content-Transfer-Encoding` of 8bit will be reconstructed + as 8bit if they have not been modified. + + Optional *unixfrom* is a flag that forces the printing of the envelope + header delimiter before the first :rfc:`2822` header of the root message + object. If the root object has no envelope header, a standard one is + crafted. By default, this is set to ``False`` to inhibit the printing of + the envelope delimiter. + + Note that for subparts, no envelope header is ever printed. + + Optional *linesep* specifies the line separator character used to + terminate lines in the output. It defaults to ``\n`` because that is + the most useful value for Python application code (other library packages + expect ``\n`` separated lines). ``linesep=\r\n`` can be used to + generate output with RFC-compliant line separators. + + .. method:: clone(fp) + + Return an independent clone of this :class:`BytesGenerator` instance with + the exact same options. + + .. method:: write(s) + + Write the string *s* to the underlying file object. *s* is encoded using + the ``ASCII`` codec and written to the *write* method of the *outfp* + *outfp* passed to the :class:`BytesGenerator`'s constructor. This + provides just enough file-like API for :class:`BytesGenerator` instances + to be used in the :func:`print` function. + + .. versionadded:: 3.2 + The :mod:`email.generator` module also provides a derived class, called :class:`DecodedGenerator` which is like the :class:`Generator` base class, except that non-\ :mimetype:`text` parts are substituted with a format string diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst index 2202637027..47a0749d52 100644 --- a/Doc/library/email.header.rst +++ b/Doc/library/email.header.rst @@ -104,25 +104,48 @@ Here is the :class:`Header` class description: Optional *errors* is passed as the errors argument to the decode call if *s* is a byte string. - .. method:: encode(splitchars=';, \\t', maxlinelen=None) + + .. method:: encode(splitchars=';, \\t', maxlinelen=None, linesep='\\n') Encode a message header into an RFC-compliant format, possibly wrapping long lines and encapsulating non-ASCII parts in base64 or quoted-printable - encodings. Optional *splitchars* is a string containing characters to - split long ASCII lines on, in rough support of :rfc:`2822`'s *highest - level syntactic breaks*. This doesn't affect :rfc:`2047` encoded lines. + encodings. + + Optional *splitchars* is a string containing characters which should be + given extra weight by the splitting algorithm during normal header + wrapping. This is in very rough support of :RFC:`2822`\'s 'higher level + syntactic breaks': split points preceded by a splitchar are preferred + during line splitting, with the characters preferred in the order in + which they appear in the string. Space and tab may be included in the + string to indicate whether preference should be given to one over the + other as a split point when other split chars do not appear in the line + being split. Splitchars does not affect :RFC:`2047` encoded lines. *maxlinelen*, if given, overrides the instance's value for the maximum line length. + *linesep* specifies the characters used to separate the lines of the + folded header. It defaults to the most useful value for Python + application code (``\n``), but ``\r\n`` can be specified in order + to produce headers with RFC-compliant line separators. + + .. versionchanged:: 3.2 + Added the *linesep* argument. + The :class:`Header` class also provides a number of methods to support standard operators and built-in functions. .. method:: __str__() - A helper for :class:`str`'s :func:`encode` method. Returns the header as - a Unicode string. + Returns an approximation of the :class:`Header` as a string, using an + unlimited line length. All pieces are converted to unicode using the + specified encoding and joined together appropriately. Any pieces with a + charset of ``'unknown-8bit'`` are decoded as ASCII using the ``'replace'`` + error handler. + + .. versionchanged:: 3.2 + Added handling for the ``'unknown-8bit'`` charset. .. method:: __eq__(other) diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst index 4b23f6acd9..3e632581b7 100644 --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -112,9 +112,18 @@ Here are the methods of the :class:`Message` class: be decoded if this header's value is ``quoted-printable`` or ``base64``. If some other encoding is used, or :mailheader:`Content-Transfer-Encoding` header is missing, or if the payload has bogus base64 data, the payload is - returned as-is (undecoded). If the message is a multipart and the - *decode* flag is ``True``, then ``None`` is returned. The default for - *decode* is ``False``. + returned as-is (undecoded). In all cases the returned value is binary + data. If the message is a multipart and the *decode* flag is ``True``, + then ``None`` is returned. + + When *decode* is ``False`` (the default) the body is returned as a string + without decoding the :mailheader:`Content-Transfer-Encoding`. However, + for a :mailheader:`Content-Transfer-Encoding` of 8bit, an attempt is made + to decode the original bytes using the ``charset`` specified by the + :mailheader:`Content-Type` header, using the ``replace`` error handler. + If no ``charset`` is specified, or if the ``charset`` given is not + recognized by the email package, the body is decoded using the default + ASCII charset. .. method:: set_payload(payload, charset=None) @@ -168,6 +177,11 @@ Here are the methods of the :class:`Message` class: Note that in all cases, any envelope header present in the message is not included in the mapping interface. + In a model generated from bytes, any header values that (in contravention of + the RFCs) contain non-ASCII bytes will, when retrieved through this + interface, be represented as :class:`~email.header.Header` objects with + a charset of `unknown-8bit`. + .. method:: __len__() @@ -263,10 +277,10 @@ Here are the methods of the :class:`Message` class: it can be specified as a three tuple in the format ``(CHARSET, LANGUAGE, VALUE)``, where ``CHARSET`` is a string naming the charset to be used to encode the value, ``LANGUAGE`` can usually be set - to ``None`` or the empty string (see :RFC:`2231` for other possibilities), + to ``None`` or the empty string (see :rfc:`2231` for other possibilities), and ``VALUE`` is the string value containing non-ASCII code points. If a three tuple is not passed and the value contains non-ASCII characters, - it is automatically encoded in :RFC`2231` format using a ``CHARSET`` + it is automatically encoded in :rfc:`2231` format using a ``CHARSET`` of ``utf-8`` and a ``LANGUAGE`` of ``None``. Here's an example:: @@ -277,7 +291,7 @@ Here are the methods of the :class:`Message` class: Content-Disposition: attachment; filename="bud.gif" - An example with with non-ASCII characters:: + An example with non-ASCII characters:: msg.add_header('Content-Disposition', 'attachment', filename=('iso-8859-1', '', 'Fußballer.ppt')) diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst index 32f4ff164b..384c5c9ee0 100644 --- a/Doc/library/email.parser.rst +++ b/Doc/library/email.parser.rst @@ -80,6 +80,14 @@ Here is the API for the :class:`FeedParser`: if you feed more data to a closed :class:`FeedParser`. +.. class:: BytesFeedParser(_factory=email.message.Message) + + Works exactly like :class:`FeedParser` except that the input to the + :meth:`~FeedParser.feed` method must be bytes and not string. + + .. versionadded:: 3.2 + + Parser class API ^^^^^^^^^^^^^^^^ @@ -125,39 +133,92 @@ class. data or by a blank line. Following the header block is the body of the message (which may contain MIME-encoded subparts). - Optional *headersonly* is as with the :meth:`parse` method. + Optional *headersonly* is a flag specifying whether to stop parsing after + reading the headers or not. The default is ``False``, meaning it parses + the entire contents of the file. .. method:: parsestr(text, headersonly=False) Similar to the :meth:`parse` method, except it takes a string object instead of a file-like object. Calling this method on a string is exactly - equivalent to wrapping *text* in a :class:`StringIO` instance first and + equivalent to wrapping *text* in a :class:`~io.StringIO` instance first and calling :meth:`parse`. + Optional *headersonly* is as with the :meth:`parse` method. + + +.. class:: BytesParser(_class=email.message.Message, strict=None) + + This class is exactly parallel to :class:`Parser`, but handles bytes input. + The *_class* and *strict* arguments are interpreted in the same way as for + the :class:`Parser` constructor. *strict* is supported only to make porting + code easier; it is deprecated. + + .. method:: parse(fp, headeronly=False) + + Read all the data from the binary file-like object *fp*, parse the + resulting bytes, and return the message object. *fp* must support + both the :meth:`readline` and the :meth:`read` methods on file-like + objects. + + The bytes contained in *fp* must be formatted as a block of :rfc:`2822` + style headers and header continuation lines, optionally preceded by a + envelope header. The header block is terminated either by the end of the + data or by a blank line. Following the header block is the body of the + message (which may contain MIME-encoded subparts, including subparts + with a :mailheader:`Content-Transfer-Encoding` of ``8bit``. + Optional *headersonly* is a flag specifying whether to stop parsing after reading the headers or not. The default is ``False``, meaning it parses the entire contents of the file. + .. method:: parsebytes(bytes, headersonly=False) + + Similar to the :meth:`parse` method, except it takes a byte string object + instead of a file-like object. Calling this method on a byte string is + exactly equivalent to wrapping *text* in a :class:`~io.BytesIO` instance + first and calling :meth:`parse`. + + Optional *headersonly* is as with the :meth:`parse` method. + + .. versionadded:: 3.2 + Since creating a message object structure from a string or a file object is such -a common task, two functions are provided as a convenience. They are available +a common task, four functions are provided as a convenience. They are available in the top-level :mod:`email` package namespace. .. currentmodule:: email -.. function:: message_from_string(s[, _class][, strict]) +.. function:: message_from_string(s, _class=email.message.Message, strict=None) Return a message object structure from a string. This is exactly equivalent to ``Parser().parsestr(s)``. Optional *_class* and *strict* are interpreted as with the :class:`Parser` class constructor. +.. function:: message_from_bytes(s, _class=email.message.Message, strict=None) + + Return a message object structure from a byte string. This is exactly + equivalent to ``BytesParser().parsebytes(s)``. Optional *_class* and + *strict* are interpreted as with the :class:`Parser` class constructor. + + .. versionadded:: 3.2 -.. function:: message_from_file(fp[, _class][, strict]) +.. function:: message_from_file(fp, _class=email.message.Message, strict=None) Return a message object structure tree from an open :term:`file object`. This is exactly equivalent to ``Parser().parse(fp)``. Optional *_class* and *strict* are interpreted as with the :class:`Parser` class constructor. +.. function:: message_from_binary_file(fp, _class=email.message.Message, strict=None) + + Return a message object structure tree from an open binary :term:`file + object`. This is exactly equivalent to ``BytesParser().parse(fp)``. + Optional *_class* and *strict* are interpreted as with the :class:`Parser` + class constructor. + + .. versionadded:: 3.2 + Here's an example of how you might use this at an interactive Python prompt:: >>> import email diff --git a/Doc/library/email.rst b/Doc/library/email.rst index d3f1908150..4530b9506e 100644 --- a/Doc/library/email.rst +++ b/Doc/library/email.rst @@ -6,7 +6,7 @@ email messages, including MIME documents. .. moduleauthor:: Barry A. Warsaw <barry@python.org> .. sectionauthor:: Barry A. Warsaw <barry@python.org> -.. Copyright (C) 2001-2007 Python Software Foundation +.. Copyright (C) 2001-2010 Python Software Foundation The :mod:`email` package is a library for managing email messages, including @@ -92,6 +92,44 @@ table also describes the Python compatibility of each version of the package. +---------------+------------------------------+-----------------------+ | :const:`4.0` | Python 2.5 | Python 2.3 to 2.5 | +---------------+------------------------------+-----------------------+ +| :const:`5.0` | Python 3.0 and Python 3.1 | Python 3.0 to 3.2 | ++---------------+------------------------------+-----------------------+ +| :const:`5.1` | Python 3.2 | Python 3.0 to 3.2 | ++---------------+------------------------------+-----------------------+ + +Here are the major differences between :mod:`email` version 5.1 and +version 5.0: + +* It is once again possible to parse messages containing non-ASCII bytes, + and to reproduce such messages if the data containing the non-ASCII + bytes is not modified. + +* New functions :func:`message_from_bytes` and :func:`message_from_binary_file`, + and new classes :class:`~email.parser.BytesFeedParser` and + :class:`~email.parser.BytesParser` allow binary message data to be parsed + into model objects. + +* Given bytes input to the model, :meth:`~email.message.Message.get_payload` + will by default decode a message body that has a + :mailheader:`Content-Transfer-Encoding` of ``8bit`` using the charset + specified in the MIME headers and return the resulting string. + +* Given bytes input to the model, :class:`~email.generator.Generator` will + convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of + 8bit to instead have a 7bit Content-Transfer-Encoding. + +* New class :class:`~email.generator.BytesGenerator` produces bytes + as output, preserving any unchanged non-ASCII data that was + present in the input used to build the model, including message bodies + with a :mailheader:`Content-Transfer-Encoding` of 8bit. + +Here are the major differences between :mod:`email` version 5.0 and version 4: + +* All operations are on unicode strings. Text inputs must be strings, + text outputs are strings. Outputs are limited to the ASCII character + set and so can be encoded to ASCII for transmission. Inputs are also + limited to ASCII; this is an acknowledged limitation of email 5.0 and + means it can only be used to parse email that is 7bit clean. Here are the major differences between :mod:`email` version 4 and version 3: diff --git a/Doc/library/email.util.rst b/Doc/library/email.util.rst index a1ce3017f4..f7b777a01d 100644 --- a/Doc/library/email.util.rst +++ b/Doc/library/email.util.rst @@ -105,11 +105,17 @@ There are several useful utilities provided in the :mod:`email.utils` module: ``False``. The default is ``False``. -.. function:: make_msgid(idstring=None) +.. function:: make_msgid(idstring=None, domain=None) Returns a string suitable for an :rfc:`2822`\ -compliant :mailheader:`Message-ID` header. Optional *idstring* if given, is a string - used to strengthen the uniqueness of the message id. + used to strengthen the uniqueness of the message id. Optional *domain* if + given provides the portion of the msgid after the '@'. The default is the + local hostname. It is not normally necessary to override this default, but + may be useful certain cases, such as a constructing distributed system that + uses a consistent domain name across multiple hosts. + + .. versionchanged:: 3.2 domain keyword added .. function:: decode_rfc2231(s) diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 4159287eb4..ca3ad3ea4c 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -40,9 +40,9 @@ The following exceptions are used mostly as base classes for other exceptions. The base class for all built-in exceptions. It is not meant to be directly inherited by user-defined classes (for that, use :exc:`Exception`). If - :func:`bytes` or :func:`str` is called on an instance of this class, the - representation of the argument(s) to the instance are returned, or the empty - string when there were no arguments. + :func:`str` is called on an instance of this class, the representation of + the argument(s) to the instance are returned, or the empty string when + there were no arguments. .. attribute:: args @@ -197,7 +197,7 @@ The following exceptions are the exceptions that are usually raised. Raised when an operation runs out of memory but the situation may still be rescued (by deleting some objects). The associated value is a string indicating what kind of (internal) operation ran out of memory. Note that because of the - underlying memory management architecture (C's :cfunc:`malloc` function), the + underlying memory management architecture (C's :c:func:`malloc` function), the interpreter may not always be able to completely recover from this situation; it nevertheless raises an exception so that a stack traceback can be printed, in case a run-away program was the cause. @@ -224,8 +224,8 @@ The following exceptions are the exceptions that are usually raised. This exception is derived from :exc:`EnvironmentError`. It is raised when a function returns a system-related error (not for illegal argument types or other incidental errors). The :attr:`errno` attribute is a numeric error - code from :cdata:`errno`, and the :attr:`strerror` attribute is the - corresponding string, as would be printed by the C function :cfunc:`perror`. + code from :c:data:`errno`, and the :attr:`strerror` attribute is the + corresponding string, as would be printed by the C function :c:func:`perror`. See the module :mod:`errno`, which contains names for the error codes defined by the underlying operating system. @@ -261,8 +261,8 @@ The following exceptions are the exceptions that are usually raised. .. exception:: StopIteration - Raised by builtin :func:`next` and an :term:`iterator`\'s :meth:`__next__` - method to signal that there are no further values. + Raised by built-in function :func:`next` and an :term:`iterator`\'s + :meth:`__next__` method to signal that there are no further values. .. exception:: SyntaxError @@ -307,7 +307,7 @@ The following exceptions are the exceptions that are usually raised. This exception is raised by the :func:`sys.exit` function. When it is not handled, the Python interpreter exits; no stack traceback is printed. If the associated value is an integer, it specifies the system exit status (passed - to C's :cfunc:`exit` function); if it is ``None``, the exit status is zero; + to C's :c:func:`exit` function); if it is ``None``, the exit status is zero; if it has another type (such as a string), the object's value is printed and the exit status is one. @@ -380,9 +380,9 @@ The following exceptions are the exceptions that are usually raised. .. exception:: WindowsError Raised when a Windows-specific error occurs or when the error number does not - correspond to an :cdata:`errno` value. The :attr:`winerror` and + correspond to an :c:data:`errno` value. The :attr:`winerror` and :attr:`strerror` values are created from the return values of the - :cfunc:`GetLastError` and :cfunc:`FormatMessage` functions from the Windows + :c:func:`GetLastError` and :c:func:`FormatMessage` functions from the Windows Platform API. The :attr:`errno` value maps the :attr:`winerror` value to corresponding ``errno.h`` values. This is a subclass of :exc:`OSError`. @@ -442,10 +442,20 @@ module for more information. Base class for warnings related to Unicode. + .. exception:: BytesWarning Base class for warnings related to :class:`bytes` and :class:`buffer`. + +.. exception:: ResourceWarning + + Base class for warnings related to resource usage. + + .. versionadded:: 3.2 + + + Exception hierarchy ------------------- diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst index dd76d656ba..6192400c9d 100644 --- a/Doc/library/fcntl.rst +++ b/Doc/library/fcntl.rst @@ -12,7 +12,7 @@ pair: UNIX; I/O control This module performs file control and I/O control on file descriptors. It is an -interface to the :cfunc:`fcntl` and :cfunc:`ioctl` Unix routines. +interface to the :c:func:`fcntl` and :c:func:`ioctl` Unix routines. All functions in this module take a file descriptor *fd* as their first argument. This can be an integer file descriptor, such as returned by @@ -30,17 +30,17 @@ The module defines the following functions: :mod:`fcntl` module. The argument *arg* is optional, and defaults to the integer value ``0``. When present, it can either be an integer value, or a string. With the argument missing or an integer value, the return value of this function - is the integer return value of the C :cfunc:`fcntl` call. When the argument is + is the integer return value of the C :c:func:`fcntl` call. When the argument is a string it represents a binary structure, e.g. created by :func:`struct.pack`. The binary data is copied to a buffer whose address is passed to the C - :cfunc:`fcntl` call. The return value after a successful call is the contents + :c:func:`fcntl` call. The return value after a successful call is the contents of the buffer, converted to a string object. The length of the returned string will be the same as the length of the *arg* argument. This is limited to 1024 bytes. If the information returned in the buffer by the operating system is larger than 1024 bytes, this is most likely to result in a segmentation violation or a more subtle data corruption. - If the :cfunc:`fcntl` fails, an :exc:`IOError` is raised. + If the :c:func:`fcntl` fails, an :exc:`IOError` is raised. .. function:: ioctl(fd, op[, arg[, mutate_flag]]) @@ -91,7 +91,7 @@ The module defines the following functions: Perform the lock operation *op* on file descriptor *fd* (file objects providing a :meth:`fileno` method are accepted as well). See the Unix manual :manpage:`flock(2)` for details. (On some systems, this function is emulated - using :cfunc:`fcntl`.) + using :c:func:`fcntl`.) .. function:: lockf(fd, operation, [length, [start, [whence]]]) diff --git a/Doc/library/filecmp.rst b/Doc/library/filecmp.rst index f57dcceff0..e0ffff704e 100644 --- a/Doc/library/filecmp.rst +++ b/Doc/library/filecmp.rst @@ -5,6 +5,9 @@ :synopsis: Compare files efficiently. .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/filecmp.py` + +-------------- The :mod:`filecmp` module defines functions to compare files and directories, with various optional time/correctness trade-offs. For comparing files, diff --git a/Doc/library/fileformats.rst b/Doc/library/fileformats.rst index 980d4f5ffb..e9c2e1fbbd 100644 --- a/Doc/library/fileformats.rst +++ b/Doc/library/fileformats.rst @@ -5,7 +5,7 @@ File Formats ************ The modules described in this chapter parse various miscellaneous file formats -that aren't markup languages or are related to e-mail. +that aren't markup languages and are not related to e-mail. .. toctree:: diff --git a/Doc/library/fileinput.rst b/Doc/library/fileinput.rst index d98a198f5c..ac4431145d 100644 --- a/Doc/library/fileinput.rst +++ b/Doc/library/fileinput.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Guido van Rossum <guido@python.org> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/fileinput.py` + +-------------- This module implements a helper class and functions to quickly write a loop over standard input or a list of files. If you just want to read or @@ -24,7 +27,7 @@ as the first argument to :func:`.input`. A single file name is also allowed. All files are opened in text mode by default, but you can override this by specifying the *mode* parameter in the call to :func:`.input` or -:class:`FileInput()`. If an I/O error occurs during opening or reading a file, +:class:`FileInput`. If an I/O error occurs during opening or reading a file, :exc:`IOError` is raised. If ``sys.stdin`` is used more than once, the second and further use will return @@ -54,6 +57,17 @@ The following function is the primary interface of this module: during iteration. The parameters to this function will be passed along to the constructor of the :class:`FileInput` class. + The :class:`FileInput` instance can be used as a context manager in the + :keyword:`with` statement. In this example, *input* is closed after the + :keyword:`with` statement is exited, even if an exception occurs:: + + with fileinput.input(files=('spam.txt', 'eggs.txt')) as f: + for line in f: + process(line) + + .. versionchanged:: 3.2 + Can be used as a context manager. + The following functions use the global state created by :func:`fileinput.input`; if there is no active state, :exc:`RuntimeError` is raised. @@ -132,13 +146,23 @@ available for subclassing as well: *filename* and *mode*, and returns an accordingly opened file-like object. You cannot use *inplace* and *openhook* together. + A :class:`FileInput` instance can be used as a context manager in the + :keyword:`with` statement. In this example, *input* is closed after the + :keyword:`with` statement is exited, even if an exception occurs:: + + with FileInput(files=('spam.txt', 'eggs.txt')) as input: + process(input) + + .. versionchanged:: 3.2 + Can be used as a context manager. + -**Optional in-place filtering:** if the keyword argument ``inplace=1`` is passed -to :func:`fileinput.input` or to the :class:`FileInput` constructor, the file is -moved to a backup file and standard output is directed to the input file (if a -file of the same name as the backup file already exists, it will be replaced -silently). This makes it possible to write a filter that rewrites its input -file in place. If the *backup* parameter is given (typically as +**Optional in-place filtering:** if the keyword argument ``inplace=True`` is +passed to :func:`fileinput.input` or to the :class:`FileInput` constructor, the +file is moved to a backup file and standard output is directed to the input file +(if a file of the same name as the backup file already exists, it will be +replaced silently). This makes it possible to write a filter that rewrites its +input file in place. If the *backup* parameter is given (typically as ``backup='.<some extension>'``), it specifies the extension for the backup file, and the backup file remains around; by default, the extension is ``'.bak'`` and it is deleted when the output file is closed. In-place filtering is disabled diff --git a/Doc/library/fnmatch.rst b/Doc/library/fnmatch.rst index 7fa6148180..4ba6b77f8e 100644 --- a/Doc/library/fnmatch.rst +++ b/Doc/library/fnmatch.rst @@ -9,6 +9,10 @@ .. index:: module: re +**Source code:** :source:`Lib/fnmatch.py` + +-------------- + This module provides support for Unix shell-style wildcards, which are *not* the same as regular expressions (which are documented in the :mod:`re` module). The special characters used in shell-style wildcards are: @@ -70,6 +74,8 @@ patterns. Return the shell-style *pattern* converted to a regular expression. + Be aware there is no way to quote meta-characters. + Example: >>> import fnmatch, re @@ -86,4 +92,3 @@ patterns. Module :mod:`glob` Unix shell-style path expansion. - diff --git a/Doc/library/fractions.rst b/Doc/library/fractions.rst index 79600269c9..a3ad44a655 100644 --- a/Doc/library/fractions.rst +++ b/Doc/library/fractions.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Jeffrey Yasskin <jyasskin at gmail.com> .. sectionauthor:: Jeffrey Yasskin <jyasskin at gmail.com> +**Source code:** :source:`Lib/fractions.py` + +-------------- The :mod:`fractions` module provides support for rational number arithmetic. @@ -15,17 +18,24 @@ another rational number, or from a string. .. class:: Fraction(numerator=0, denominator=1) Fraction(other_fraction) + Fraction(float) + Fraction(decimal) Fraction(string) - The first version requires that *numerator* and *denominator* are - instances of :class:`numbers.Rational` and returns a new - :class:`Fraction` instance with value ``numerator/denominator``. If - *denominator* is :const:`0`, it raises a - :exc:`ZeroDivisionError`. The second version requires that - *other_fraction* is an instance of :class:`numbers.Rational` and - returns an :class:`Fraction` instance with the same value. The - last version of the constructor expects a string instance. The - usual form for this string is:: + The first version requires that *numerator* and *denominator* are instances + of :class:`numbers.Rational` and returns a new :class:`Fraction` instance + with value ``numerator/denominator``. If *denominator* is :const:`0`, it + raises a :exc:`ZeroDivisionError`. The second version requires that + *other_fraction* is an instance of :class:`numbers.Rational` and returns a + :class:`Fraction` instance with the same value. The next two versions accept + either a :class:`float` or a :class:`decimal.Decimal` instance, and return a + :class:`Fraction` instance with exactly the same value. Note that due to the + usual issues with binary floating-point (see :ref:`tut-fp-issues`), the + argument to ``Fraction(1.1)`` is not exactly equal to 11/10, and so + ``Fraction(1.1)`` does *not* return ``Fraction(11, 10)`` as one might expect. + (But see the documentation for the :meth:`limit_denominator` method below.) + The last version of the constructor expects a string or unicode instance. + The usual form for this instance is:: [sign] numerator ['/' denominator] @@ -55,6 +65,13 @@ another rational number, or from a string. Fraction(-1, 8) >>> Fraction('7e-6') Fraction(7, 1000000) + >>> Fraction(2.25) + Fraction(9, 4) + >>> Fraction(1.1) + Fraction(2476979795053773, 2251799813685248) + >>> from decimal import Decimal + >>> Fraction(Decimal('1.1')) + Fraction(11, 10) The :class:`Fraction` class inherits from the abstract base class @@ -63,6 +80,10 @@ another rational number, or from a string. and should be treated as immutable. In addition, :class:`Fraction` has the following methods: + .. versionchanged:: 3.2 + The :class:`Fraction` constructor now accepts :class:`float` and + :class:`decimal.Decimal` instances. + .. method:: from_float(flt) @@ -70,12 +91,19 @@ another rational number, or from a string. value of *flt*, which must be a :class:`float`. Beware that ``Fraction.from_float(0.3)`` is not the same value as ``Fraction(3, 10)`` + .. note:: From Python 3.2 onwards, you can also construct a + :class:`Fraction` instance directly from a :class:`float`. + .. method:: from_decimal(dec) This class method constructs a :class:`Fraction` representing the exact value of *dec*, which must be a :class:`decimal.Decimal` instance. + .. note:: From Python 3.2 onwards, you can also construct a + :class:`Fraction` instance directly from a :class:`decimal.Decimal` + instance. + .. method:: limit_denominator(max_denominator=1000000) @@ -90,10 +118,12 @@ another rational number, or from a string. or for recovering a rational number that's represented as a float: >>> from math import pi, cos - >>> Fraction.from_float(cos(pi/3)) + >>> Fraction(cos(pi/3)) Fraction(4503599627370497, 9007199254740992) - >>> Fraction.from_float(cos(pi/3)).limit_denominator() + >>> Fraction(cos(pi/3)).limit_denominator() Fraction(1, 2) + >>> Fraction(1.1).limit_denominator() + Fraction(11, 10) .. method:: __floor__() diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst index 5545505273..5bbef4f233 100644 --- a/Doc/library/ftplib.rst +++ b/Doc/library/ftplib.rst @@ -9,6 +9,10 @@ pair: FTP; protocol single: FTP; ftplib (standard module) +**Source code:** :source:`Lib/ftplib.py` + +-------------- + This module defines the class :class:`FTP` and a few related items. The :class:`FTP` class implements the client side of the FTP protocol. You can use this to write Python programs that perform a variety of automated FTP jobs, such @@ -33,8 +37,8 @@ Here's a sample session using the :mod:`ftplib` module:: '226 Transfer complete.' >>> ftp.quit() -The module defines the following items: +The module defines the following items: .. class:: FTP(host='', user='', passwd='', acct=''[, timeout]) @@ -46,6 +50,61 @@ The module defines the following items: connection attempt (if is not specified, the global default timeout setting will be used). + :class:`FTP` class supports the :keyword:`with` statement. Here is a sample + on how using it: + + >>> from ftplib import FTP + >>> with FTP("ftp1.at.proftpd.org") as ftp: + ... ftp.login() + ... ftp.dir() + ... + '230 Anonymous login ok, restrictions apply.' + dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 . + dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .. + dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS + dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora + >>> + + .. versionchanged:: 3.2 + Support for the :keyword:`with` statement was added. + + +.. class:: FTP_TLS(host='', user='', passwd='', acct='', [keyfile[, certfile[, context[, timeout]]]]) + + A :class:`FTP` subclass which adds TLS support to FTP as described in + :rfc:`4217`. + Connect as usual to port 21 implicitly securing the FTP control connection + before authenticating. Securing the data connection requires the user to + explicitly ask for it by calling the :meth:`prot_p` method. + *keyfile* and *certfile* are optional -- they can contain a PEM formatted + private key and certificate chain file name for the SSL connection. + *context* parameter is a :class:`ssl.SSLContext` object which allows + bundling SSL configuration options, certificates and private keys into a + single (potentially long-lived) structure. + + .. versionadded:: 3.2 + + Here's a sample session using the :class:`FTP_TLS` class: + + >>> from ftplib import FTP_TLS + >>> ftps = FTP_TLS('ftp.python.org') + >>> ftps.login() # login anonymously before securing control channel + >>> ftps.prot_p() # switch to secure data connection + >>> ftps.retrlines('LIST') # list directory content securely + total 9 + drwxr-xr-x 8 root wheel 1024 Jan 3 1994 . + drwxr-xr-x 8 root wheel 1024 Jan 3 1994 .. + drwxr-xr-x 2 root wheel 1024 Jan 3 1994 bin + drwxr-xr-x 2 root wheel 1024 Jan 3 1994 etc + d-wxrwxr-x 2 ftp wheel 1024 Sep 5 13:43 incoming + drwxr-xr-x 2 root wheel 1024 Nov 17 1993 lib + drwxr-xr-x 6 1094 wheel 1024 Sep 13 19:07 pub + drwxr-xr-x 3 root wheel 1024 Jan 3 1994 usr + -rw-r--r-- 1 root root 312 Aug 1 1994 welcome.msg + '226 Transfer complete.' + >>> ftps.quit() + >>> + .. exception:: error_reply @@ -197,14 +256,18 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. Passive mode is on by default. -.. method:: FTP.storbinary(cmd, file, blocksize=8192, callback=None) +.. method:: FTP.storbinary(cmd, file, blocksize=8192, callback=None, rest=None) Store a file in binary transfer mode. *cmd* should be an appropriate ``STOR`` command: ``"STOR filename"``. *file* is an open :term:`file object` which is read until EOF using its :meth:`read` method in blocks of size *blocksize* to provide the data to be stored. The *blocksize* argument defaults to 8192. *callback* is an optional single parameter callable that - is called on each block of data after it is sent. + is called on each block of data after it is sent. *rest* means the same thing + as in the :meth:`transfercmd` method. + + .. versionchanged:: 3.2 + *rest* parameter added. .. method:: FTP.storlines(cmd, file, callback=None) @@ -319,3 +382,26 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. :meth:`close` or :meth:`quit` you cannot reopen the connection by issuing another :meth:`login` method). + +FTP_TLS Objects +--------------- + +:class:`FTP_TLS` class inherits from :class:`FTP`, defining these additional objects: + +.. attribute:: FTP_TLS.ssl_version + + The SSL version to use (defaults to *TLSv1*). + +.. method:: FTP_TLS.auth() + + Set up secure control connection by using TLS or SSL, depending on what specified in :meth:`ssl_version` attribute. + +.. method:: FTP_TLS.prot_p() + + Set up secure data connection. + +.. method:: FTP_TLS.prot_c() + + Set up clear text data connection. + + diff --git a/Doc/library/functional.rst b/Doc/library/functional.rst new file mode 100644 index 0000000000..5b6185a65c --- /dev/null +++ b/Doc/library/functional.rst @@ -0,0 +1,15 @@ +****************************** +Functional Programming Modules +****************************** + +The modules described in this chapter provide functions and classes that support +a functional programming style, and general operations on callables. + +The following modules are documented in this chapter: + + +.. toctree:: + + itertools.rst + functools.rst + operator.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 42f2bc93d7..f835dcfc71 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -10,22 +10,31 @@ are always available. They are listed here in alphabetical order. =================== ================= ================== ================ ==================== .. .. Built-in Functions .. .. =================== ================= ================== ================ ==================== -:func:`abs` :func:`dir` :func:`hex` :func:`next` :func:`slice` -:func:`all` :func:`divmod` :func:`id` :func:`object` :func:`sorted` -:func:`any` :func:`enumerate` :func:`input` :func:`oct` :func:`staticmethod` -:func:`ascii` :func:`eval` :func:`int` :func:`open` :func:`str` -:func:`bin` :func:`exec` :func:`isinstance` :func:`ord` :func:`sum` -:func:`bool` :func:`filter` :func:`issubclass` :func:`pow` :func:`super` -:func:`bytearray` :func:`float` :func:`iter` :func:`print` :func:`tuple` -:func:`bytes` :func:`format` :func:`len` :func:`property` :func:`type` -:func:`chr` :func:`frozenset` :func:`list` :func:`range` :func:`vars` +:func:`abs` |func-dict|_ :func:`help` :func:`min` :func:`setattr` +:func:`all` :func:`dir` :func:`hex` :func:`next` :func:`slice` +:func:`any` :func:`divmod` :func:`id` :func:`object` :func:`sorted` +:func:`ascii` :func:`enumerate` :func:`input` :func:`oct` :func:`staticmethod` +:func:`bin` :func:`eval` :func:`int` :func:`open` :func:`str` +:func:`bool` :func:`exec` :func:`isinstance` :func:`ord` :func:`sum` +:func:`bytearray` :func:`filter` :func:`issubclass` :func:`pow` :func:`super` +:func:`bytes` :func:`float` :func:`iter` :func:`print` :func:`tuple` +:func:`callable` :func:`format` :func:`len` :func:`property` :func:`type` +:func:`chr` |func-frozenset|_ :func:`list` :func:`range` :func:`vars` :func:`classmethod` :func:`getattr` :func:`locals` :func:`repr` :func:`zip` :func:`compile` :func:`globals` :func:`map` :func:`reversed` :func:`__import__` :func:`complex` :func:`hasattr` :func:`max` :func:`round` -:func:`delattr` :func:`hash` :func:`memoryview` :func:`set` -:func:`dict` :func:`help` :func:`min` :func:`setattr` +:func:`delattr` :func:`hash` |func-memoryview|_ |func-set|_ =================== ================= ================== ================ ==================== +.. using :func:`dict` would create a link to another page, so local targets are + used, with replacement texts to make the output in the table consistent + +.. |func-dict| replace:: ``dict()`` +.. |func-frozenset| replace:: ``frozenset()`` +.. |func-memoryview| replace:: ``memoryview()`` +.. |func-set| replace:: ``set()`` + + .. function:: abs(x) Return the absolute value of a number. The argument may be an @@ -74,11 +83,12 @@ are always available. They are listed here in alphabetical order. .. function:: bool([x]) - Convert a value to a Boolean, using the standard truth testing procedure. If - *x* is false or omitted, this returns :const:`False`; otherwise it returns - :const:`True`. :class:`bool` is also a class, which is a subclass of - :class:`int`. Class :class:`bool` cannot be subclassed further. Its only - instances are :const:`False` and :const:`True`. + Convert a value to a Boolean, using the standard :ref:`truth testing + procedure <truth>`. If *x* is false or omitted, this returns ``False``; + otherwise it returns ``True``. :class:`bool` is also a class, which is a + subclass of :class:`int` (see :ref:`typesnumeric`). Class :class:`bool` + cannot be subclassed further. Its only instances are ``False`` and + ``True`` (see :ref:`bltin-boolean-values`). .. index:: pair: Boolean; type @@ -121,6 +131,19 @@ are always available. They are listed here in alphabetical order. Bytes objects can also be created with literals, see :ref:`strings`. +.. function:: callable(object) + + Return :const:`True` if the *object* argument appears callable, + :const:`False` if not. If this returns true, it is still possible that a + call fails, but if it is false, calling *object* will never succeed. + Note that classes are callable (calling a class returns a new instance); + instances are callable if their class has a :meth:`__call__` method. + + .. versionadded:: 3.2 + This function was first removed in Python 3.0 and then brought back + in Python 3.2. + + .. function:: chr(i) Return the string representing a character whose Unicode codepoint is the integer @@ -161,7 +184,7 @@ are always available. They are listed here in alphabetical order. type hierarchy in :ref:`types`. -.. function:: compile(source, filename, mode, flags=0, dont_inherit=False) +.. function:: compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1) Compile the *source* into a code or AST object. Code objects can be executed by :func:`exec` or :func:`eval`. *source* can either be a string or an AST @@ -193,16 +216,25 @@ are always available. They are listed here in alphabetical order. can be found as the :attr:`compiler_flag` attribute on the :class:`_Feature` instance in the :mod:`__future__` module. + The argument *optimize* specifies the optimization level of the compiler; the + default value of ``-1`` selects the optimization level of the interpreter as + given by :option:`-O` options. Explicit levels are ``0`` (no optimization; + ``__debug__`` is true), ``1`` (asserts are removed, ``__debug__`` is false) + or ``2`` (docstrings are removed too). + This function raises :exc:`SyntaxError` if the compiled source is invalid, and :exc:`TypeError` if the source contains null bytes. .. note:: - When compiling a string with multi-line statements, line endings must be - represented by a single newline character (``'\n'``), and the input must - be terminated by at least one newline character. If line endings are - represented by ``'\r\n'``, use :meth:`str.replace` to change them into - ``'\n'``. + When compiling a string with multi-line code in ``'single'`` or + ``'eval'`` mode, input must be terminated by at least one newline + character. This is to facilitate detection of incomplete and complete + statements in the :mod:`code` module. + + .. versionchanged:: 3.2 + Allowed use of Windows and Mac newlines. Also input in ``'exec'`` mode + does not have to end in a newline anymore. Added the *optimize* parameter. .. function:: complex([real[, imag]]) @@ -226,6 +258,7 @@ are always available. They are listed here in alphabetical order. example, ``delattr(x, 'foobar')`` is equivalent to ``del x.foobar``. +.. _func-dict: .. function:: dict([arg]) :noindex: @@ -268,19 +301,18 @@ are always available. They are listed here in alphabetical order. The resulting list is sorted alphabetically. For example: >>> import struct - >>> dir() # doctest: +SKIP + >>> dir() # show the names in the module namespace ['__builtins__', '__doc__', '__name__', 'struct'] - >>> dir(struct) # doctest: +NORMALIZE_WHITESPACE + >>> dir(struct) # show the names in the struct module ['Struct', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_clearcache', 'calcsize', 'error', 'pack', 'pack_into', 'unpack', 'unpack_from'] - >>> class Foo: - ... def __dir__(self): - ... return ["kan", "ga", "roo"] - ... - >>> f = Foo() - >>> dir(f) - ['ga', 'kan', 'roo'] + >>> class Shape(object): + def __dir__(self): + return ['area', 'perimeter', 'location'] + >>> s = Shape() + >>> dir(s) + ['area', 'perimeter', 'location'] .. note:: @@ -310,16 +342,21 @@ are always available. They are listed here in alphabetical order. :term:`iterator`, or some other object which supports iteration. The :meth:`__next__` method of the iterator returned by :func:`enumerate` returns a tuple containing a count (from *start* which defaults to 0) and the - corresponding value obtained from iterating over *iterable*. - :func:`enumerate` is useful for obtaining an indexed series: ``(0, seq[0])``, - ``(1, seq[1])``, ``(2, seq[2])``, .... For example: + values obtained from iterating over *iterable*. + + >>> seasons = ['Spring', 'Summer', 'Fall', 'Winter'] + >>> list(enumerate(seasons)) + [(0, 'Spring'), (1, 'Summer'), (2, 'Fall'), (3, 'Winter')] + >>> list(enumerate(seasons, start=1)) + [(1, 'Spring'), (2, 'Summer'), (3, 'Fall'), (4, 'Winter')] - >>> for i, season in enumerate(['Spring', 'Summer', 'Fall', 'Winter']): - ... print(i, season) - 0 Spring - 1 Summer - 2 Fall - 3 Winter + Equivalent to:: + + def enumerate(sequence, start=0): + n = start + for elem in sequence: + yield n, elem + n += 1 .. function:: eval(expression, globals=None, locals=None) @@ -346,7 +383,7 @@ are always available. They are listed here in alphabetical order. This function can also be used to execute arbitrary code objects (such as those created by :func:`compile`). In this case pass a code object instead of a string. If the code object has been compiled with ``'exec'`` as the - *kind* argument, :func:`eval`\'s return value will be ``None``. + *mode* argument, :func:`eval`\'s return value will be ``None``. Hints: dynamic execution of statements is supported by the :func:`exec` function. The :func:`globals` and :func:`locals` functions @@ -414,29 +451,58 @@ are always available. They are listed here in alphabetical order. .. function:: float([x]) - Convert a string or a number to floating point. If the argument is a string, - it must contain a possibly signed decimal or floating point number, possibly - embedded in whitespace. The argument may also be ``'[+|-]nan'`` or - ``'[+|-]inf'``. Otherwise, the argument may be an integer or a floating - point number, and a floating point number with the same value (within - Python's floating point precision) is returned. If no argument is given, - ``0.0`` is returned. - - .. note:: - - .. index:: - single: NaN - single: Infinity - - When passing in a string, values for NaN and Infinity may be returned, - depending on the underlying C library. Float accepts the strings - ``'nan'``, ``'inf'`` and ``'-inf'`` for NaN and positive or negative - infinity. The case and a leading + are ignored as well as a leading - is - ignored for NaN. Float always represents NaN and infinity as ``nan``, - ``inf`` or ``-inf``. + .. index:: + single: NaN + single: Infinity + + Convert a string or a number to floating point. + + If the argument is a string, it should contain a decimal number, optionally + preceded by a sign, and optionally embedded in whitespace. The optional + sign may be ``'+'`` or ``'-'``; a ``'+'`` sign has no effect on the value + produced. The argument may also be a string representing a NaN + (not-a-number), or a positive or negative infinity. More precisely, the + input must conform to the following grammar after leading and trailing + whitespace characters are removed: + + .. productionlist:: + sign: "+" | "-" + infinity: "Infinity" | "inf" + nan: "nan" + numeric_value: `floatnumber` | `infinity` | `nan` + numeric_string: [`sign`] `numeric_value` + + Here ``floatnumber`` is the form of a Python floating-point literal, + described in :ref:`floating`. Case is not significant, so, for example, + "inf", "Inf", "INFINITY" and "iNfINity" are all acceptable spellings for + positive infinity. + + Otherwise, if the argument is an integer or a floating point number, a + floating point number with the same value (within Python's floating point + precision) is returned. If the argument is outside the range of a Python + float, an :exc:`OverflowError` will be raised. + + For a general Python object ``x``, ``float(x)`` delegates to + ``x.__float__()``. + + If no argument is given, ``0.0`` is returned. + + Examples:: + + >>> float('+1.23') + 1.23 + >>> float(' -12345\n') + -12345.0 + >>> float('1e-003') + 0.001 + >>> float('+1E6') + 1000000.0 + >>> float('-Infinity') + -inf The float type is described in :ref:`typesnumeric`. + .. function:: format(value[, format_spec]) .. index:: @@ -448,12 +514,17 @@ are always available. They are listed here in alphabetical order. of the *value* argument, however there is a standard formatting syntax that is used by most built-in types: :ref:`formatspec`. - .. note:: + The default *format_spec* is an empty string which usually gives the same + effect as calling ``str(value)``. - ``format(value, format_spec)`` merely calls - ``value.__format__(format_spec)``. + A call to ``format(value, format_spec)`` is translated to + ``type(value).__format__(format_spec)`` which bypasses the instance + dictionary when searching for the value's :meth:`__format__` method. A + :exc:`TypeError` exception is raised if the method is not found or if either + the *format_spec* or the return value are not strings. +.. _func-frozenset: .. function:: frozenset([iterable]) :noindex: @@ -482,10 +553,10 @@ are always available. They are listed here in alphabetical order. .. function:: hasattr(object, name) - The arguments are an object and a string. The result is ``True`` if the string - is the name of one of the object's attributes, ``False`` if not. (This is - implemented by calling ``getattr(object, name)`` and seeing whether it raises an - exception or not.) + The arguments are an object and a string. The result is ``True`` if the + string is the name of one of the object's attributes, ``False`` if not. (This + is implemented by calling ``getattr(object, name)`` and seeing whether it + raises an :exc:`AttributeError` or not.) .. function:: hash(object) @@ -567,7 +638,8 @@ are always available. They are listed here in alphabetical order. .. function:: isinstance(object, classinfo) Return true if the *object* argument is an instance of the *classinfo* - argument, or of a (direct or indirect) subclass thereof. If *object* is not + argument, or of a (direct, indirect or :term:`virtual <abstract base + class>`) subclass thereof. If *object* is not an object of the given type, the function always returns false. If *classinfo* is not a class (type object), it may be a tuple of type objects, or may recursively contain other such tuples (other sequence types are not @@ -577,7 +649,8 @@ are always available. They are listed here in alphabetical order. .. function:: issubclass(class, classinfo) - Return true if *class* is a subclass (direct or indirect) of *classinfo*. A + Return true if *class* is a subclass (direct, indirect or :term:`virtual + <abstract base class>`) of *classinfo*. A class is considered a subclass of itself. *classinfo* may be a tuple of class objects, in which case every entry in *classinfo* will be checked. In any other case, a :exc:`TypeError` exception is raised. @@ -599,10 +672,10 @@ are always available. They are listed here in alphabetical order. One useful application of the second form of :func:`iter` is to read lines of a file until a certain line is reached. The following example reads a file - until ``"STOP"`` is reached: :: + until the :meth:`readline` method returns an empty string:: - with open("mydata.txt") as fp: - for line in iter(fp.readline, "STOP"): + with open('mydata.txt') as fp: + for line in iter(fp.readline, ''): process_line(line) @@ -653,7 +726,13 @@ are always available. They are listed here in alphabetical order. The optional keyword-only *key* argument specifies a one-argument ordering function like that used for :meth:`list.sort`. + If multiple items are maximal, the function returns the first one + encountered. This is consistent with other sort-stability preserving tools + such as ``sorted(iterable, key=keyfunc, reverse=True)[0]`` and + ``heapq.nlargest(1, iterable, key=keyfunc)``. + +.. _func-memoryview: .. function:: memoryview(obj) :noindex: @@ -670,6 +749,10 @@ are always available. They are listed here in alphabetical order. The optional keyword-only *key* argument specifies a one-argument ordering function like that used for :meth:`list.sort`. + If multiple items are minimal, the function returns the first one + encountered. This is consistent with other sort-stability preserving tools + such as ``sorted(iterable, key=keyfunc)[0]`` and ``heapq.nsmallest(1, + iterable, key=keyfunc)``. .. function:: next(iterator[, default]) @@ -745,7 +828,7 @@ are always available. They are listed here in alphabetical order. .. note:: Python doesn't depend on the underlying operating system's notion of text - files; all the the processing is done by Python itself, and is therefore + files; all the processing is done by Python itself, and is therefore platform-independent. *buffering* is an optional integer used to set the buffering policy. Pass 0 @@ -833,7 +916,7 @@ are always available. They are listed here in alphabetical order. .. XXX works for bytes too, but should it? .. function:: ord(c) - Given a string representing one Uncicode character, return an integer + Given a string representing one Unicode character, return an integer representing the Unicode code point of that character. For example, ``ord('a')`` returns the integer ``97`` and ``ord('\u2020')`` returns ``8224``. This is the inverse of :func:`chr`. @@ -871,7 +954,9 @@ are always available. They are listed here in alphabetical order. *end*. The *file* argument must be an object with a ``write(string)`` method; if it - is not present or ``None``, :data:`sys.stdout` will be used. + is not present or ``None``, :data:`sys.stdout` will be used. Output buffering + is determined by *file*. Use ``file.flush()`` to ensure, for instance, + immediate appearance on a screen. .. function:: property(fget=None, fset=None, fdel=None, doc=None) @@ -971,6 +1056,35 @@ are always available. They are listed here in alphabetical order. >>> list(range(1, 0)) [] + Range objects implement the :class:`collections.Sequence` ABC, and provide + features such as containment tests, element index lookup, slicing and + support for negative indices (see :ref:`typesseq`): + + >>> r = range(0, 20, 2) + >>> r + range(0, 20, 2) + >>> 11 in r + False + >>> 10 in r + True + >>> r.index(10) + 5 + >>> r[5] + 10 + >>> r[:5] + range(0, 10, 2) + >>> r[-1] + 18 + + Ranges containing absolute values larger than :data:`sys.maxsize` are permitted + but some features (such as :func:`len`) will raise :exc:`OverflowError`. + + .. versionchanged:: 3.2 + Implement the Sequence ABC. + Support slicing and negative indices. + Test integers for membership in constant time instead of iterating + through all items. + .. function:: repr(object) @@ -1012,6 +1126,8 @@ are always available. They are listed here in alphabetical order. can't be represented exactly as a float. See :ref:`tut-fp-issues` for more information. + +.. _func-set: .. function:: set([iterable]) :noindex: @@ -1050,14 +1166,14 @@ are always available. They are listed here in alphabetical order. Has two optional arguments which must be specified as keyword arguments. *key* specifies a function of one argument that is used to extract a comparison - key from each list element: ``key=str.lower``. The default value is ``None``. + key from each list element: ``key=str.lower``. The default value is ``None`` + (compare the elements directly). *reverse* is a boolean value. If set to ``True``, then the list elements are sorted as if each comparison were reversed. - To convert an old-style *cmp* function to a *key* function, see the - `CmpToKey recipe in the ASPN cookbook - <http://code.activestate.com/recipes/576653/>`_\. + Use :func:`functools.cmp_to_key` to convert an old-style *cmp* function to a + *key* function. For sorting examples and a brief sorting tutorial, see `Sorting HowTo <http://wiki.python.org/moin/HowTo/Sorting/>`_\. @@ -1079,8 +1195,9 @@ are always available. They are listed here in alphabetical order. It can be called either on the class (such as ``C.f()``) or on an instance (such as ``C().f()``). The instance is ignored except for its class. - Static methods in Python are similar to those found in Java or C++. For a more - advanced concept, see :func:`classmethod` in this section. + Static methods in Python are similar to those found in Java or C++. Also see + :func:`classmethod` for a variant that is useful for creating alternate class + constructors. For more information on static methods, consult the documentation on the standard type hierarchy in :ref:`types`. @@ -1180,6 +1297,10 @@ are always available. They are listed here in alphabetical order. references. The zero argument form automatically searches the stack frame for the class (``__class__``) and the first argument. + For practical suggestions on how to design cooperative classes using + :func:`super`, see `guide to using super() + <http://rhettinger.wordpress.com/2011/05/26/super-considered-super/>`_. + .. function:: tuple([iterable]) @@ -1248,10 +1369,10 @@ are always available. They are listed here in alphabetical order. def zip(*iterables): # zip('ABCD', 'xy') --> Ax By sentinel = object() - iterables = [iter(it) for it in iterables] - while iterables: + iterators = [iter(it) for it in iterables] + while iterators: result = [] - for it in iterables: + for it in iterators: elem = next(it, sentinel) if elem is sentinel: return @@ -1288,7 +1409,7 @@ are always available. They are listed here in alphabetical order. .. note:: This is an advanced function that is not needed in everyday Python - programming. + programming, unlike :func:`importlib.import_module`. This function is invoked by the :keyword:`import` statement. It can be replaced (by importing the :mod:`builtins` module and assigning to @@ -1338,15 +1459,8 @@ are always available. They are listed here in alphabetical order. names. If you simply want to import a module (potentially within a package) by name, - you can call :func:`__import__` and then look it up in :data:`sys.modules`:: - - >>> import sys - >>> name = 'foo.bar.baz' - >>> __import__(name) - <module 'foo' from ...> - >>> baz = sys.modules[name] - >>> baz - <module 'foo.bar.baz' from ...> + use :func:`importlib.import_module`. + .. rubric:: Footnotes diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst index 570f4d26a5..04743d31fa 100644 --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -1,13 +1,16 @@ -:mod:`functools` --- Higher order functions and operations on callable objects +:mod:`functools` --- Higher-order functions and operations on callable objects ============================================================================== .. module:: functools - :synopsis: Higher order functions and operations on callable objects. + :synopsis: Higher-order functions and operations on callable objects. .. moduleauthor:: Peter Harris <scav@blueyonder.co.uk> .. moduleauthor:: Raymond Hettinger <python@rcn.com> .. moduleauthor:: Nick Coghlan <ncoghlan@gmail.com> .. sectionauthor:: Peter Harris <scav@blueyonder.co.uk> +**Source code:** :source:`Lib/functools.py` + +-------------- The :mod:`functools` module is for higher-order functions: functions that act on or return other functions. In general, any callable object can be treated as a @@ -15,6 +18,123 @@ function for the purposes of this module. The :mod:`functools` module defines the following functions: +.. function:: cmp_to_key(func) + + Transform an old-style comparison function to a key function. Used with + tools that accept key functions (such as :func:`sorted`, :func:`min`, + :func:`max`, :func:`heapq.nlargest`, :func:`heapq.nsmallest`, + :func:`itertools.groupby`). This function is primarily used as a transition + tool for programs being converted from Python 2 which supported the use of + comparison functions. + + A comparison function is any callable that accept two arguments, compares them, + and returns a negative number for less-than, zero for equality, or a positive + number for greater-than. A key function is a callable that accepts one + argument and returns another value indicating the position in the desired + collation sequence. + + Example:: + + sorted(iterable, key=cmp_to_key(locale.strcoll)) # locale-aware sort order + + .. versionadded:: 3.2 + + +.. decorator:: lru_cache(maxsize=100) + + Decorator to wrap a function with a memoizing callable that saves up to the + *maxsize* most recent calls. It can save time when an expensive or I/O bound + function is periodically called with the same arguments. + + Since a dictionary is used to cache results, the positional and keyword + arguments to the function must be hashable. + + If *maxsize* is set to None, the LRU feature is disabled and the cache + can grow without bound. + + To help measure the effectiveness of the cache and tune the *maxsize* + parameter, the wrapped function is instrumented with a :func:`cache_info` + function that returns a :term:`named tuple` showing *hits*, *misses*, + *maxsize* and *currsize*. In a multi-threaded environment, the hits + and misses are approximate. + + The decorator also provides a :func:`cache_clear` function for clearing or + invalidating the cache. + + The original underlying function is accessible through the + :attr:`__wrapped__` attribute. This is useful for introspection, for + bypassing the cache, or for rewrapping the function with a different cache. + + An `LRU (least recently used) cache + <http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used>`_ works + best when more recent calls are the best predictors of upcoming calls (for + example, the most popular articles on a news server tend to change daily). + The cache's size limit assures that the cache does not grow without bound on + long-running processes such as web servers. + + Example of an LRU cache for static web content:: + + @lru_cache(maxsize=20) + def get_pep(num): + 'Retrieve text of a Python Enhancement Proposal' + resource = 'http://www.python.org/dev/peps/pep-%04d/' % num + try: + with urllib.request.urlopen(resource) as s: + return s.read() + except urllib.error.HTTPError: + return 'Not Found' + + >>> for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991: + ... pep = get_pep(n) + ... print(n, len(pep)) + + >>> print(get_pep.cache_info()) + CacheInfo(hits=3, misses=8, maxsize=20, currsize=8) + + Example of efficiently computing + `Fibonacci numbers <http://en.wikipedia.org/wiki/Fibonacci_number>`_ + using a cache to implement a + `dynamic programming <http://en.wikipedia.org/wiki/Dynamic_programming>`_ + technique:: + + @lru_cache(maxsize=None) + def fib(n): + if n < 2: + return n + return fib(n-1) + fib(n-2) + + >>> print([fib(n) for n in range(16)]) + [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610] + + >>> print(fib.cache_info()) + CacheInfo(hits=28, misses=16, maxsize=None, currsize=16) + + .. versionadded:: 3.2 + +.. decorator:: total_ordering + + Given a class defining one or more rich comparison ordering methods, this + class decorator supplies the rest. This simplifies the effort involved + in specifying all of the possible rich comparison operations: + + The class must define one of :meth:`__lt__`, :meth:`__le__`, + :meth:`__gt__`, or :meth:`__ge__`. + In addition, the class should supply an :meth:`__eq__` method. + + For example:: + + @total_ordering + class Student: + def __eq__(self, other): + return ((self.lastname.lower(), self.firstname.lower()) == + (other.lastname.lower(), other.firstname.lower())) + def __lt__(self, other): + return ((self.lastname.lower(), self.firstname.lower()) < + (other.lastname.lower(), other.firstname.lower())) + + .. versionadded:: 3.2 + + .. function:: partial(func, *args, **keywords) Return a new :class:`partial` object which when called will behave like *func* @@ -70,14 +190,34 @@ The :mod:`functools` module defines the following functions: documentation string) and *WRAPPER_UPDATES* (which updates the wrapper function's *__dict__*, i.e. the instance dictionary). + To allow access to the original function for introspection and other purposes + (e.g. bypassing a caching decorator such as :func:`lru_cache`), this function + automatically adds a __wrapped__ attribute to the wrapper that refers to + the original function. + The main intended use for this function is in :term:`decorator` functions which wrap the decorated function and return the wrapper. If the wrapper function is not updated, the metadata of the returned function will reflect the wrapper definition rather than the original function definition, which is typically less than helpful. + :func:`update_wrapper` may be used with callables other than functions. Any + attributes named in *assigned* or *updated* that are missing from the object + being wrapped are ignored (i.e. this function will not attempt to set them + on the wrapper function). :exc:`AttributeError` is still raised if the + wrapper function itself is missing any attributes named in *updated*. + + .. versionadded:: 3.2 + Automatic addition of the ``__wrapped__`` attribute. + + .. versionadded:: 3.2 + Copying of the ``__annotations__`` attribute by default. + + .. versionchanged:: 3.2 + Missing attributes no longer trigger an :exc:`AttributeError`. + -.. function:: wraps(wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES) +.. decorator:: wraps(wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES) This is a convenience function for invoking ``partial(update_wrapper, wrapped=wrapped, assigned=assigned, updated=updated)`` as a function decorator diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index 34aba6515f..0281bb761f 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -43,7 +43,7 @@ The :mod:`gc` module provides the following functions: :exc:`ValueError` is raised if the generation number is invalid. The number of unreachable objects found is returned. - The free lists maintained for a number of builtin types are cleared + The free lists maintained for a number of built-in types are cleared whenever a full collection or collection of the highest generation (2) is run. Not all items in some free lists may be freed due to the particular implementation, in particular :class:`float`. @@ -174,8 +174,15 @@ value but should not rebind it): with :meth:`__del__` methods, and *garbage* can be examined in that case to verify that no such cycles are being created. - If :const:`DEBUG_SAVEALL` is set, then all unreachable objects will be added to - this list rather than freed. + If :const:`DEBUG_SAVEALL` is set, then all unreachable objects will be added + to this list rather than freed. + + .. versionchanged:: 3.2 + If this list is non-empty at interpreter shutdown, a + :exc:`ResourceWarning` is emitted, which is silent by default. If + :const:`DEBUG_UNCOLLECTABLE` is set, in addition all uncollectable objects + are printed. + The following constants are provided for use with :func:`set_debug`: @@ -194,9 +201,12 @@ The following constants are provided for use with :func:`set_debug`: .. data:: DEBUG_UNCOLLECTABLE Print information of uncollectable objects found (objects which are not - reachable but cannot be freed by the collector). These objects will be added to - the ``garbage`` list. + reachable but cannot be freed by the collector). These objects will be added + to the ``garbage`` list. + .. versionchanged:: 3.2 + Also print the contents of the :data:`garbage` list at interpreter + shutdown, if it isn't empty. .. data:: DEBUG_SAVEALL diff --git a/Doc/library/getopt.rst b/Doc/library/getopt.rst index 6a95142b44..b6ab3df022 100644 --- a/Doc/library/getopt.rst +++ b/Doc/library/getopt.rst @@ -1,20 +1,27 @@ -:mod:`getopt` --- Parser for command line options -================================================= +:mod:`getopt` --- C-style parser for command line options +========================================================= .. module:: getopt :synopsis: Portable parser for command line options; support both short and long option names. +**Source code:** :source:`Lib/getopt.py` + +-------------- + +.. note:: + The :mod:`getopt` module is a parser for command line options whose API is + designed to be familiar to users of the C :c:func:`getopt` function. Users who + are unfamiliar with the C :c:func:`getopt` function or who would like to write + less code and get better help and error messages should consider using the + :mod:`argparse` module instead. This module helps scripts to parse the command line arguments in ``sys.argv``. -It supports the same conventions as the Unix :cfunc:`getopt` function (including +It supports the same conventions as the Unix :c:func:`getopt` function (including the special meanings of arguments of the form '``-``' and '``--``'). Long options similar to those supported by GNU software may be used as well via an optional third argument. -A more convenient, flexible, and powerful alternative is the -:mod:`optparse` module. - This module provides two functions and an exception: @@ -25,11 +32,11 @@ exception: be parsed, without the leading reference to the running program. Typically, this means ``sys.argv[1:]``. *shortopts* is the string of option letters that the script wants to recognize, with options that require an argument followed by a - colon (``':'``; i.e., the same format that Unix :cfunc:`getopt` uses). + colon (``':'``; i.e., the same format that Unix :c:func:`getopt` uses). .. note:: - Unlike GNU :cfunc:`getopt`, after a non-option argument, all further + Unlike GNU :c:func:`getopt`, after a non-option argument, all further arguments are considered also non-options. This is similar to the way non-GNU Unix systems work. @@ -136,9 +143,21 @@ In a script, typical usage is something like this:: if __name__ == "__main__": main() +Note that an equivalent command line interface could be produced with less code +and more informative help and error messages by using the :mod:`argparse` module:: + + import argparse + + if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('-o', '--output') + parser.add_argument('-v', dest='verbose', action='store_true') + args = parser.parse_args() + # ... do something with args.output ... + # ... do something with args.verbose .. .. seealso:: - Module :mod:`optparse` - More object-oriented command line option parsing. + Module :mod:`argparse` + Alternative command line option and argument parsing library. diff --git a/Doc/library/gettext.rst b/Doc/library/gettext.rst index 9e1528ba62..0fa022c4b8 100644 --- a/Doc/library/gettext.rst +++ b/Doc/library/gettext.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Barry A. Warsaw <barry@zope.com> .. sectionauthor:: Barry A. Warsaw <barry@zope.com> +**Source code:** :source:`Lib/gettext.py` + +-------------- The :mod:`gettext` module provides internationalization (I18N) and localization (L10N) services for your Python modules and applications. It supports both the @@ -260,7 +263,7 @@ are the methods of :class:`NullTranslations`: .. method:: lngettext(singular, plural, n) - If a fallback has been set, forward :meth:`ngettext` to the fallback. + If a fallback has been set, forward :meth:`lngettext` to the fallback. Otherwise, return the translated message. Overridden in derived classes. @@ -641,8 +644,8 @@ implementations, and valuable experience to the creation of this module: .. [#] See the footnote for :func:`bindtextdomain` above. .. [#] François Pinard has written a program called :program:`xpot` which does a - similar job. It is available as part of his :program:`po-utils` package at http - ://po-utils.progiciels-bpi.ca/. + similar job. It is available as part of his `po-utils package + <http://po-utils.progiciels-bpi.ca/>`_. .. [#] :program:`msgfmt.py` is binary compatible with GNU :program:`msgfmt` except that it provides a simpler, all-Python implementation. With this and diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst index 3e0322d1c9..3d31c116c8 100644 --- a/Doc/library/glob.rst +++ b/Doc/library/glob.rst @@ -7,6 +7,10 @@ .. index:: single: filenames; pathname expansion +**Source code:** :source:`Lib/glob.py` + +-------------- + The :mod:`glob` module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell. No tilde expansion is done, but ``*``, ``?``, and character ranges expressed with ``[]`` will be correctly diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst index fdd859049c..9422ea9943 100644 --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -4,6 +4,10 @@ .. module:: gzip :synopsis: Interfaces for gzip compression and decompression using file objects. +**Source code:** :source:`Lib/gzip.py` + +-------------- + This module provides a simple interface to compress and decompress files just like the GNU programs :program:`gzip` and :program:`gunzip` would. @@ -17,18 +21,15 @@ Note that additional file formats which can be decompressed by the :program:`gzip` and :program:`gunzip` programs, such as those produced by :program:`compress` and :program:`pack`, are not supported by this module. -For other archive formats, see the :mod:`bz2`, :mod:`zipfile`, and -:mod:`tarfile` modules. - The module defines the following items: .. class:: GzipFile(filename=None, mode=None, compresslevel=9, fileobj=None, mtime=None) - Constructor for the :class:`GzipFile` class, which simulates most of the methods - of a :term:`file object`, with the exception of the :meth:`readinto` and - :meth:`truncate` methods. At least one of *fileobj* and *filename* must be - given a non-trivial value. + Constructor for the :class:`GzipFile` class, which simulates most of the + methods of a :term:`file object`, with the exception of the :meth:`truncate` + method. At least one of *fileobj* and *filename* must be given a non-trivial + value. The new class instance is based on *fileobj*, which can be a regular file, a :class:`StringIO` object, or any other object which simulates a file. It @@ -43,9 +44,11 @@ The module defines the following items: The *mode* argument can be any of ``'r'``, ``'rb'``, ``'a'``, ``'ab'``, ``'w'``, or ``'wb'``, depending on whether the file will be read or written. The default - is the mode of *fileobj* if discernible; otherwise, the default is ``'rb'``. If - not given, the 'b' flag will be added to the mode to ensure the file is opened - in binary mode for cross-platform portability. + is the mode of *fileobj* if discernible; otherwise, the default is ``'rb'``. + + Note that the file is always opened in binary mode; text mode is not + supported. If you need to read a compressed file in text mode, wrap your + :class:`GzipFile` with an :class:`io.TextIOWrapper`. The *compresslevel* argument is an integer from ``1`` to ``9`` controlling the level of compression; ``1`` is fastest and produces the least compression, and @@ -57,20 +60,39 @@ The module defines the following items: time is used. This module ignores the timestamp when decompressing; however, some programs, such as :program:`gunzip`\ , make use of it. The format of the timestamp is the same as that of the return value of - ``time.time()`` and of the ``st_mtime`` member of the object returned + ``time.time()`` and of the ``st_mtime`` attribute of the object returned by ``os.stat()``. Calling a :class:`GzipFile` object's :meth:`close` method does not close *fileobj*, since you might wish to append more material after the compressed - data. This also allows you to pass a :class:`StringIO` object opened for + data. This also allows you to pass a :class:`io.BytesIO` object opened for writing as *fileobj*, and retrieve the resulting memory buffer using the - :class:`StringIO` object's :meth:`getvalue` method. + :class:`io.BytesIO` object's :meth:`~io.BytesIO.getvalue` method. + + :class:`GzipFile` supports the :class:`io.BufferedIOBase` interface, + including iteration and the :keyword:`with` statement. Only the + :meth:`read1` and :meth:`truncate` methods aren't implemented. - :class:`GzipFile` supports the :keyword:`with` statement. + :class:`GzipFile` also provides the following method: + + .. method:: peek([n]) + + Read *n* uncompressed bytes without advancing the file position. + At most one single read on the compressed stream is done to satisfy + the call. The number of bytes returned may be more or less than + requested. + + .. versionadded:: 3.2 .. versionchanged:: 3.1 Support for the :keyword:`with` statement was added. + .. versionchanged:: 3.2 + Support for zero-padded files was added. + + .. versionchanged:: 3.2 + Support for unseekable files was added. + .. function:: open(filename, mode='rb', compresslevel=9) @@ -78,6 +100,21 @@ The module defines the following items: The *filename* argument is required; *mode* defaults to ``'rb'`` and *compresslevel* defaults to ``9``. +.. function:: compress(data, compresslevel=9) + + Compress the *data*, returning a :class:`bytes` object containing + the compressed data. *compresslevel* has the same meaning as in + the :class:`GzipFile` constructor above. + + .. versionadded:: 3.2 + +.. function:: decompress(data) + + Decompress the *data*, returning a :class:`bytes` object containing the + uncompressed data. + + .. versionadded:: 3.2 + .. _gzip-usage-examples: @@ -104,6 +141,11 @@ Example of how to GZIP compress an existing file:: with gzip.open('/home/joe/file.txt.gz', 'wb') as f_out: f_out.writelines(f_in) +Example of how to GZIP compress a binary string:: + + import gzip + s_in = b"Lots of content here" + s_out = gzip.compress(s_in) .. seealso:: diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index b73d7536a5..bc8ab2ca6d 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -11,6 +11,10 @@ single: message digest, MD5 single: secure hash algorithm, SHA1, SHA224, SHA256, SHA384, SHA512 +**Source code:** :source:`Lib/hashlib.py` + +-------------- + This module implements a common interface to many different secure hash and message digest algorithms. Included are the FIPS secure hash algorithms SHA1, SHA224, SHA256, SHA384, and SHA512 (defined in FIPS 180-2) as well as RSA's MD5 @@ -70,10 +74,13 @@ More condensed: >>> hashlib.sha224(b"Nobody inspects the spammish repetition").hexdigest() 'a4337bc45a8fc544c03f52dc550cd6e1e87021bc896588bd79e901e2' -A generic :func:`new` constructor that takes the string name of the desired -algorithm as its first parameter also exists to allow access to the above listed -hashes as well as any other algorithms that your OpenSSL library may offer. The -named constructors are much faster than :func:`new` and should be preferred. +.. function:: new(name[, data]) + + Is a generic constructor that takes the string name of the desired + algorithm as its first parameter. It also exists to allow access to the + above listed hashes as well as any other algorithms that your OpenSSL + library may offer. The named constructors are much faster than :func:`new` + and should be preferred. Using :func:`new` with an algorithm provided by OpenSSL: @@ -82,6 +89,25 @@ Using :func:`new` with an algorithm provided by OpenSSL: >>> h.hexdigest() 'cc4a5ce1b3df48aec5d22d1f16b894a0b894eccc' +Hashlib provides the following constant attributes: + +.. data:: algorithms_guaranteed + + Contains the names of the hash algorithms guaranteed to be supported + by this module on all platforms. + + .. versionadded:: 3.2 + +.. data:: algorithms_available + + Contains the names of the hash algorithms that are available + in the running Python interpreter. These names will be recognized + when passed to :func:`new`. :attr:`algorithms_guaranteed` + will always be a subset. Duplicate algorithms with different + name formats may appear in this set (thanks to OpenSSL). + + .. versionadded:: 3.2 + The following values are provided as constant attributes of the hash objects returned by the constructors: diff --git a/Doc/library/heapq.rst b/Doc/library/heapq.rst index 7735365f79..768dfdc67b 100644 --- a/Doc/library/heapq.rst +++ b/Doc/library/heapq.rst @@ -8,6 +8,10 @@ .. sectionauthor:: François Pinard .. sectionauthor:: Raymond Hettinger +**Source code:** :source:`Lib/heapq.py` + +-------------- + This module provides an implementation of the heap queue algorithm, also known as the priority queue algorithm. @@ -169,36 +173,36 @@ changes to its priority or removing it entirely. Finding a task can be done with a dictionary pointing to an entry in the queue. Removing the entry or changing its priority is more difficult because it would -break the heap structure invariants. So, a possible solution is to mark an -entry as invalid and optionally add a new entry with the revised priority:: - - pq = [] # the priority queue list - counter = itertools.count(1) # unique sequence count - task_finder = {} # mapping of tasks to entries - INVALID = 0 # mark an entry as deleted - - def add_task(priority, task, count=None): - if count is None: - count = next(counter) +break the heap structure invariants. So, a possible solution is to mark the +entry as removed and add a new entry with the revised priority:: + + pq = [] # list of entries arranged in a heap + entry_finder = {} # mapping of tasks to entries + REMOVED = '<removed-task>' # placeholder for a removed task + counter = itertools.count() # unique sequence count + + def add_task(task, priority=0): + 'Add a new task or update the priority of an existing task' + if task in entry_finder: + remove_task(task) + count = next(counter) entry = [priority, count, task] - task_finder[task] = entry + entry_finder[task] = entry heappush(pq, entry) - def get_top_priority(): - while True: + def remove_task(task): + 'Mark an existing task as REMOVED. Raise KeyError if not found.' + entry = entry_finder.pop(task) + entry[-1] = REMOVED + + def pop_task(): + 'Remove and return the lowest priority task. Raise KeyError if empty.' + while pq: priority, count, task = heappop(pq) - del task_finder[task] - if count is not INVALID: + if task is not REMOVED: + del entry_finder[task] return task - - def delete_task(task): - entry = task_finder[task] - entry[1] = INVALID - - def reprioritize(priority, task): - entry = task_finder[task] - add_task(priority, task, entry[1]) - entry[1] = INVALID + raise KeyError('pop from an empty priority queue') Theory diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst index b2bd98dfc7..eff27241bc 100644 --- a/Doc/library/hmac.rst +++ b/Doc/library/hmac.rst @@ -7,6 +7,9 @@ .. moduleauthor:: Gerhard Häring <ghaering@users.sourceforge.net> .. sectionauthor:: Gerhard Häring <ghaering@users.sourceforge.net> +**Source code:** :source:`Lib/hmac.py` + +-------------- This module implements the HMAC algorithm as described by :rfc:`2104`. @@ -21,14 +24,14 @@ This module implements the HMAC algorithm as described by :rfc:`2104`. An HMAC object has the following methods: -.. method:: hmac.update(msg) +.. method:: HMAC.update(msg) Update the hmac object with the bytes object *msg*. Repeated calls are equivalent to a single call with the concatenation of all the arguments: ``m.update(a); m.update(b)`` is equivalent to ``m.update(a + b)``. -.. method:: hmac.digest() +.. method:: HMAC.digest() Return the digest of the bytes passed to the :meth:`update` method so far. This bytes object will be the same length as the *digest_size* of the digest @@ -36,14 +39,14 @@ An HMAC object has the following methods: bytes. -.. method:: hmac.hexdigest() +.. method:: HMAC.hexdigest() Like :meth:`digest` except the digest is returned as a string twice the length containing only hexadecimal digits. This may be used to exchange the value safely in email or other non-binary environments. -.. method:: hmac.copy() +.. method:: HMAC.copy() Return a copy ("clone") of the hmac object. This can be used to efficiently compute the digests of strings that share a common initial substring. diff --git a/Doc/library/html.entities.rst b/Doc/library/html.entities.rst index aa67bae7a3..b8b4aa8f91 100644 --- a/Doc/library/html.entities.rst +++ b/Doc/library/html.entities.rst @@ -5,10 +5,13 @@ :synopsis: Definitions of HTML general entities. .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/html/entities.py` + +-------------- This module defines three dictionaries, ``name2codepoint``, ``codepoint2name``, and ``entitydefs``. ``entitydefs`` is used to provide the :attr:`entitydefs` -member of the :class:`html.parser.HTMLParser` class. The definition provided +attribute of the :class:`html.parser.HTMLParser` class. The definition provided here contains all the entities defined by XHTML 1.0 that can be handled using simple textual substitution in the Latin-1 character set (ISO-8859-1). diff --git a/Doc/library/html.parser.rst b/Doc/library/html.parser.rst index 1fa11a2ef0..f3c36ec886 100644 --- a/Doc/library/html.parser.rst +++ b/Doc/library/html.parser.rst @@ -9,45 +9,92 @@ single: HTML single: XHTML +**Source code:** :source:`Lib/html/parser.py` + +-------------- + This module defines a class :class:`HTMLParser` which serves as the basis for parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. -.. class:: HTMLParser() +.. class:: HTMLParser(strict=True) - The :class:`HTMLParser` class is instantiated without arguments. + Create a parser instance. If *strict* is ``True`` (the default), invalid + HTML results in :exc:`~html.parser.HTMLParseError` exceptions [#]_. If + *strict* is ``False``, the parser uses heuristics to make a best guess at + the intention of any invalid HTML it encounters, similar to the way most + browsers do. Using ``strict=False`` is advised. - An :class:`HTMLParser` instance is fed HTML data and calls handler functions when tags - begin and end. The :class:`HTMLParser` class is meant to be overridden by the - user to provide a desired behavior. + An :class:`.HTMLParser` instance is fed HTML data and calls handler methods + when start tags, end tags, text, comments, and other markup elements are + encountered. The user should subclass :class:`.HTMLParser` and override its + methods to implement the desired behavior. This parser does not check that end tags match start tags or call the end-tag handler for elements which are closed implicitly by closing an outer element. + .. versionchanged:: 3.2 *strict* keyword added + An exception is defined as well: .. exception:: HTMLParseError Exception raised by the :class:`HTMLParser` class when it encounters an error - while parsing. This exception provides three attributes: :attr:`msg` is a brief - message explaining the error, :attr:`lineno` is the number of the line on which - the broken construct was detected, and :attr:`offset` is the number of - characters into the line at which the construct starts. + while parsing and *strict* is ``True``. This exception provides three + attributes: :attr:`msg` is a brief message explaining the error, + :attr:`lineno` is the number of the line on which the broken construct was + detected, and :attr:`offset` is the number of characters into the line at + which the construct starts. -:class:`HTMLParser` instances have the following methods: +Example HTML Parser Application +------------------------------- -.. method:: HTMLParser.reset() +As a basic example, below is a simple HTML parser that uses the +:class:`HTMLParser` class to print out start tags, end tags, and data +as they are encountered:: + + from html.parser import HTMLParser + + class MyHTMLParser(HTMLParser): + def handle_starttag(self, tag, attrs): + print("Encountered a start tag:", tag) + def handle_endtag(self, tag): + print("Encountered an end tag :", tag) + def handle_data(self, data): + print("Encountered some data :", data) + + parser = MyHTMLParser(strict=False) + parser.feed('<html><head><title>Test</title></head>' + '<body><h1>Parse me!</h1></body></html>') + +The output will then be:: + + Encountered a start tag: html + Encountered a start tag: head + Encountered a start tag: title + Encountered some data : Test + Encountered an end tag : title + Encountered an end tag : head + Encountered a start tag: body + Encountered a start tag: h1 + Encountered some data : Parse me! + Encountered an end tag : h1 + Encountered an end tag : body + Encountered an end tag : html - Reset the instance. Loses all unprocessed data. This is called implicitly at - instantiation time. + +:class:`.HTMLParser` Methods +---------------------------- + +:class:`HTMLParser` instances have the following methods: .. method:: HTMLParser.feed(data) Feed some text to the parser. It is processed insofar as it consists of complete elements; incomplete data is buffered until more data is fed or - :meth:`close` is called. + :meth:`close` is called. *data* must be :class:`str`. .. method:: HTMLParser.close() @@ -58,6 +105,12 @@ An exception is defined as well: the :class:`HTMLParser` base class method :meth:`close`. +.. method:: HTMLParser.reset() + + Reset the instance. Loses all unprocessed data. This is called implicitly at + instantiation time. + + .. method:: HTMLParser.getpos() Return current line number and offset. @@ -71,81 +124,83 @@ An exception is defined as well: attributes can be preserved, etc.). +The following methods are called when data or markup elements are encountered +and they are meant to be overridden in a subclass. The base class +implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`): + + .. method:: HTMLParser.handle_starttag(tag, attrs) - This method is called to handle the start of a tag. It is intended to be - overridden by a derived class; the base class implementation does nothing. + This method is called to handle the start of a tag (e.g. ``<div id="main">``). The *tag* argument is the name of the tag converted to lower case. The *attrs* argument is a list of ``(name, value)`` pairs containing the attributes found inside the tag's ``<>`` brackets. The *name* will be translated to lower case, and quotes in the *value* have been removed, and character and entity references - have been replaced. For instance, for the tag ``<A - HREF="http://www.cwi.nl/">``, this method would be called as - ``handle_starttag('a', [('href', 'http://www.cwi.nl/')])``. + have been replaced. + + For instance, for the tag ``<A HREF="http://www.cwi.nl/">``, this method + would be called as ``handle_starttag('a', [('href', 'http://www.cwi.nl/')])``. All entity references from :mod:`html.entities` are replaced in the attribute values. -.. method:: HTMLParser.handle_startendtag(tag, attrs) +.. method:: HTMLParser.handle_endtag(tag) - Similar to :meth:`handle_starttag`, but called when the parser encounters an - XHTML-style empty tag (``<a .../>``). This method may be overridden by - subclasses which require this particular lexical information; the default - implementation simple calls :meth:`handle_starttag` and :meth:`handle_endtag`. + This method is called to handle the end tag of an element (e.g. ``</div>``). + The *tag* argument is the name of the tag converted to lower case. -.. method:: HTMLParser.handle_endtag(tag) - This method is called to handle the end tag of an element. It is intended to be - overridden by a derived class; the base class implementation does nothing. The - *tag* argument is the name of the tag converted to lower case. +.. method:: HTMLParser.handle_startendtag(tag, attrs) + + Similar to :meth:`handle_starttag`, but called when the parser encounters an + XHTML-style empty tag (``<img ... />``). This method may be overridden by + subclasses which require this particular lexical information; the default + implementation simply calls :meth:`handle_starttag` and :meth:`handle_endtag`. .. method:: HTMLParser.handle_data(data) - This method is called to process arbitrary data. It is intended to be - overridden by a derived class; the base class implementation does nothing. + This method is called to process arbitrary data (e.g. text nodes and the + content of ``<script>...</script>`` and ``<style>...</style>``). -.. method:: HTMLParser.handle_charref(name) +.. method:: HTMLParser.handle_entityref(name) - This method is called to process a character reference of the form ``&#ref;``. - It is intended to be overridden by a derived class; the base class - implementation does nothing. + This method is called to process a named character reference of the form + ``&name;`` (e.g. ``>``), where *name* is a general entity reference + (e.g. ``'gt'``). -.. method:: HTMLParser.handle_entityref(name) +.. method:: HTMLParser.handle_charref(name) - This method is called to process a general entity reference of the form - ``&name;`` where *name* is an general entity reference. It is intended to be - overridden by a derived class; the base class implementation does nothing. + This method is called to process decimal and hexadecimal numeric character + references of the form ``&#NNN;`` and ``&#xNNN;``. For example, the decimal + equivalent for ``>`` is ``>``, whereas the hexadecimal is ``>``; + in this case the method will receive ``'62'`` or ``'x3E'``. .. method:: HTMLParser.handle_comment(data) - This method is called when a comment is encountered. The *comment* argument is - a string containing the text between the ``--`` and ``--`` delimiters, but not - the delimiters themselves. For example, the comment ``<!--text-->`` will cause - this method to be called with the argument ``'text'``. It is intended to be - overridden by a derived class; the base class implementation does nothing. + This method is called when a comment is encountered (e.g. ``<!--comment-->``). + For example, the comment ``<!-- comment -->`` will cause this method to be + called with the argument ``' comment '``. -.. method:: HTMLParser.handle_decl(decl) + The content of Internet Explorer conditional comments (condcoms) will also be + sent to this method, so, for ``<!--[if IE 9]>IE9-specific content<![endif]-->``, + this method will receive ``'[if IE 9]>IE-specific content<![endif]'``. - Method called when an SGML ``doctype`` declaration is read by the parser. - The *decl* parameter will be the entire contents of the declaration inside - the ``<!...>`` markup. It is intended to be overridden by a derived class; - the base class implementation does nothing. +.. method:: HTMLParser.handle_decl(decl) -.. method:: HTMLParser.unknown_decl(data) + This method is called to handle an HTML doctype declaration (e.g. + ``<!DOCTYPE html>``). - Method called when an unrecognized SGML declaration is read by the parser. - The *data* parameter will be the entire contents of the declaration inside - the ``<!...>`` markup. It is sometimes useful to be overridden by a - derived class; the base class implementation raises an :exc:`HTMLParseError`. + The *decl* parameter will be the entire contents of the declaration inside + the ``<!...>`` markup (e.g. ``'DOCTYPE html'``). .. method:: HTMLParser.handle_pi(data) @@ -163,31 +218,126 @@ An exception is defined as well: cause the ``'?'`` to be included in *data*. -.. _htmlparser-example: - -Example HTML Parser Application -------------------------------- +.. method:: HTMLParser.unknown_decl(data) -As a basic example, below is a very basic HTML parser that uses the -:class:`HTMLParser` class to print out tags as they are encountered:: + This method is called when an unrecognized declaration is read by the parser. - >>> from html.parser import HTMLParser + The *data* parameter will be the entire contents of the declaration inside + the ``<![...]>`` markup. It is sometimes useful to be overridden by a + derived class. The base class implementation raises an :exc:`HTMLParseError` + when *strict* is ``True``. + + +.. _htmlparser-examples: + +Examples +-------- + +The following class implements a parser that will be used to illustrate more +examples:: + + from html.parser import HTMLParser + from html.entities import name2codepoint + + class MyHTMLParser(HTMLParser): + def handle_starttag(self, tag, attrs): + print("Start tag:", tag) + for attr in attrs: + print(" attr:", attr) + def handle_endtag(self, tag): + print("End tag :", tag) + def handle_data(self, data): + print("Data :", data) + def handle_comment(self, data): + print("Comment :", data) + def handle_entityref(self, name): + c = chr(name2codepoint[name]) + print("Named ent:", c) + def handle_charref(self, name): + if name.startswith('x'): + c = chr(int(name[1:], 16)) + else: + c = chr(int(name)) + print("Num ent :", c) + def handle_decl(self, data): + print("Decl :", data) + + parser = MyHTMLParser(strict=False) + +Parsing a doctype:: + + >>> parser.feed('<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" ' + ... '"http://www.w3.org/TR/html4/strict.dtd">') + Decl : DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd" + +Parsing an element with a few attributes and a title:: + + >>> parser.feed('<img src="python-logo.png" alt="The Python logo">') + Start tag: img + attr: ('src', 'python-logo.png') + attr: ('alt', 'The Python logo') >>> - >>> class MyHTMLParser(HTMLParser): - ... def handle_starttag(self, tag, attrs): - ... print("Encountered a {} start tag".format(tag)) - ... def handle_endtag(self, tag): - ... print("Encountered a {} end tag".format(tag)) - ... - >>> page = """<html><h1>Title</h1><p>I'm a paragraph!</p></html>""" + >>> parser.feed('<h1>Python</h1>') + Start tag: h1 + Data : Python + End tag : h1 + +The content of ``script`` and ``style`` elements is returned as is, without +further parsing:: + + >>> parser.feed('<style type="text/css">#python { color: green }</style>') + Start tag: style + attr: ('type', 'text/css') + Data : #python { color: green } + End tag : style >>> - >>> myparser = MyHTMLParser() - >>> myparser.feed(page) - Encountered a html start tag - Encountered a h1 start tag - Encountered a h1 end tag - Encountered a p start tag - Encountered a p end tag - Encountered a html end tag + >>> parser.feed('<script type="text/javascript">' + ... 'alert("<strong>hello!</strong>");</script>') + Start tag: script + attr: ('type', 'text/javascript') + Data : alert("<strong>hello!</strong>"); + End tag : script + +Parsing comments:: + + >>> parser.feed('<!-- a comment -->' + ... '<!--[if IE 9]>IE-specific content<![endif]-->') + Comment : a comment + Comment : [if IE 9]>IE-specific content<![endif] +Parsing named and numeric character references and converting them to the +correct char (note: these 3 references are all equivalent to ``'>'``):: + >>> parser.feed('>>>') + Named ent: > + Num ent : > + Num ent : > + +Feeding incomplete chunks to :meth:`~HTMLParser.feed` works, but +:meth:`~HTMLParser.handle_data` might be called more than once:: + + >>> for chunk in ['<sp', 'an>buff', 'ered ', 'text</s', 'pan>']: + ... parser.feed(chunk) + ... + Start tag: span + Data : buff + Data : ered + Data : text + End tag : span + +Parsing invalid HTML (e.g. unquoted attributes) also works:: + + >>> parser.feed('<p><a class=link href=#main>tag soup</p ></a>') + Start tag: p + Start tag: a + attr: ('class', 'link') + attr: ('href', '#main') + Data : tag soup + End tag : p + End tag : a + +.. rubric:: Footnotes + +.. [#] For backward compatibility reasons *strict* mode does not raise + exceptions for all non-compliant HTML. That is, some invalid HTML + is tolerated even in *strict* mode. diff --git a/Doc/library/html.rst b/Doc/library/html.rst new file mode 100644 index 0000000000..0063db60cd --- /dev/null +++ b/Doc/library/html.rst @@ -0,0 +1,21 @@ +:mod:`html` --- HyperText Markup Language support +================================================= + +.. module:: html + :synopsis: Helpers for manipulating HTML. + +.. versionadded:: 3.2 + +**Source code:** :source:`Lib/html/__init__.py` + +-------------- + +This module defines utilities to manipulate HTML. + +.. function:: escape(s, quote=True) + + Convert the characters ``&``, ``<`` and ``>`` in string *s* to HTML-safe + sequences. Use this if you need to display text that might contain such + characters in HTML. If the optional flag *quote* is true, the characters + (``"``) and (``'``) are also translated; this helps for inclusion in an HTML + attribute value delimited by quotes, as in ``<a href="...">``. diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index cbe4f05663..52fbe573c0 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -11,30 +11,33 @@ .. index:: module: urllib.request +**Source code:** :source:`Lib/http/client.py` + +-------------- + This module defines classes which implement the client side of the HTTP and HTTPS protocols. It is normally not used directly --- the module :mod:`urllib.request` uses it to handle URLs that use HTTP and HTTPS. .. note:: - HTTPS support is only available if the :mod:`socket` module was compiled with - SSL support. + HTTPS support is only available if Python was compiled with SSL support + (through the :mod:`ssl` module). The module provides the following classes: -.. class:: HTTPConnection(host, port=None, strict=None[, timeout]) +.. class:: HTTPConnection(host, port=None[, strict[, timeout[, source_address]]]) An :class:`HTTPConnection` instance represents one transaction with an HTTP server. It should be instantiated passing it a host and optional port number. If no port number is passed, the port is extracted from the host string if it has the form ``host:port``, else the default HTTP port (80) is - used. When True, the optional parameter *strict* (which defaults to a false - value) causes ``BadStatusLine`` to - be raised if the status line can't be parsed as a valid HTTP/1.0 or 1.1 - status line. If the optional *timeout* parameter is given, blocking + used. If the optional *timeout* parameter is given, blocking operations (like connection attempts) will timeout after that many seconds (if it is not given, the global default timeout setting is used). + The optional *source_address* parameter may be a tuple of a (host, port) + to use as the source address the HTTP connection is made from. For example, the following calls all create instances that connect to the server at the same host and port:: @@ -44,24 +47,58 @@ The module provides the following classes: >>> h3 = http.client.HTTPConnection('www.cwi.nl', 80) >>> h3 = http.client.HTTPConnection('www.cwi.nl', 80, timeout=10) + .. versionchanged:: 3.2 + *source_address* was added. + + .. versionchanged:: 3.2 + The *strict* parameter is deprecated. HTTP 0.9-style "Simple Responses" + are not supported anymore. -.. class:: HTTPSConnection(host, port=None, key_file=None, cert_file=None, strict=None[, timeout]) + +.. class:: HTTPSConnection(host, port=None, key_file=None, cert_file=None[, strict[, timeout[, source_address]]], *, context=None, check_hostname=None) A subclass of :class:`HTTPConnection` that uses SSL for communication with - secure servers. Default port is ``443``. *key_file* is the name of a PEM - formatted file that contains your private key, and *cert_file* is a PEM - formatted certificate chain file; both can be used for authenticating - yourself against the server. + secure servers. Default port is ``443``. If *context* is specified, it + must be a :class:`ssl.SSLContext` instance describing the various SSL + options. If *context* is specified and has a :attr:`~ssl.SSLContext.verify_mode` + of either :data:`~ssl.CERT_OPTIONAL` or :data:`~ssl.CERT_REQUIRED`, then + by default *host* is matched against the host name(s) allowed by the + server's certificate. If you want to change that behaviour, you can + explicitly set *check_hostname* to False. + + *key_file* and *cert_file* are deprecated, please use + :meth:`ssl.SSLContext.load_cert_chain` instead. + + If you access arbitrary hosts on the Internet, it is recommended to + require certificate checking and feed the *context* with a set of + trusted CA certificates:: - .. warning:: - This does not do any verification of the server's certificate. + context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context.verify_mode = ssl.CERT_REQUIRED + context.load_verify_locations('/etc/pki/tls/certs/ca-bundle.crt') + h = client.HTTPSConnection('svn.python.org', 443, context=context) + .. versionchanged:: 3.2 + *source_address*, *context* and *check_hostname* were added. -.. class:: HTTPResponse(sock, debuglevel=0, strict=0, method=None, url=None) + .. versionchanged:: 3.2 + This class now supports HTTPS virtual hosts if possible (that is, + if :data:`ssl.HAS_SNI` is true). + + .. versionchanged:: 3.2 + The *strict* parameter is deprecated. HTTP 0.9-style "Simple Responses" + are not supported anymore. + + +.. class:: HTTPResponse(sock, debuglevel=0[, strict], method=None, url=None) Class whose instances are returned upon successful connection. Not instantiated directly by user. + .. versionchanged:: 3.2 + The *strict* parameter is deprecated. HTTP 0.9-style "Simple Responses" + are not supported anymore. + The following exceptions are raised as appropriate: @@ -360,14 +397,18 @@ HTTPConnection Objects string. The *body* may also be an open :term:`file object`, in which case the - contents of the file is sent; this file object should support - ``fileno()`` and ``read()`` methods. The header Content-Length is - automatically set to the length of the file as reported by - stat. + contents of the file is sent; this file object should support ``fileno()`` + and ``read()`` methods. The header Content-Length is automatically set to + the length of the file as reported by stat. The *body* argument may also be + an iterable and Content-Length header should be explicitly provided when the + body is an iterable. The *headers* argument should be a mapping of extra HTTP headers to send with the request. + .. versionadded:: 3.2 + *body* can now be an iterable. + .. method:: HTTPConnection.getresponse() Should be called after a request is sent to get the response from the server. @@ -389,6 +430,17 @@ HTTPConnection Objects .. versionadded:: 3.1 +.. method:: HTTPConnection.set_tunnel(host, port=None, headers=None) + + Set the host and the port for HTTP Connect Tunnelling. Normally used when it + is required to a HTTPS Connection through a proxy server. + + The headers argument should be a mapping of extra HTTP headers to send + with the CONNECT request. + + .. versionadded:: 3.2 + + .. method:: HTTPConnection.connect() Connect to the server specified when the object was created. @@ -420,10 +472,13 @@ also send your request step by step, by using the four functions below. an argument. -.. method:: HTTPConnection.endheaders() - - Send a blank line to the server, signalling the end of the headers. +.. method:: HTTPConnection.endheaders(message_body=None) + Send a blank line to the server, signalling the end of the headers. The + optional *message_body* argument can be used to pass a message body + associated with the request. The message body will be sent in the same + packet as the message headers if it is string, otherwise it is sent in a + separate packet. .. method:: HTTPConnection.send(data) @@ -491,6 +546,9 @@ statement. A debugging hook. If :attr:`debuglevel` is greater than zero, messages will be printed to stdout as the response is read and parsed. +.. attribute:: HTTPResponse.closed + + Is True if the stream is closed. Examples -------- @@ -503,7 +561,15 @@ Here is an example session that uses the ``GET`` method:: >>> r1 = conn.getresponse() >>> print(r1.status, r1.reason) 200 OK - >>> data1 = r1.read() + >>> data1 = r1.read() # This will return entire content. + >>> # The following example demonstrates reading data in chunks. + >>> conn.request("GET", "/index.html") + >>> r1 = conn.getresponse() + >>> while not r1.closed: + ... print(r1.read(200)) # 200 bytes + b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"... + ... + >>> # Example of an invalid request >>> conn.request("GET", "/parrot.spam") >>> r2 = conn.getresponse() >>> print(r2.status, r2.reason) @@ -511,9 +577,8 @@ Here is an example session that uses the ``GET`` method:: >>> data2 = r2.read() >>> conn.close() -Here is an example session that uses ``HEAD`` method. Note that ``HEAD`` method -never returns any data. :: - +Here is an example session that uses the ``HEAD`` method. Note that the +``HEAD`` method never returns any data. :: >>> import http.client >>> conn = http.client.HTTPConnection("www.python.org") @@ -530,15 +595,17 @@ never returns any data. :: Here is an example session that shows how to ``POST`` requests:: >>> import http.client, urllib.parse - >>> params = urllib.parse.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) + >>> params = urllib.parse.urlencode({'@number': 12524, '@type': 'issue', '@action': 'show'}) >>> headers = {"Content-type": "application/x-www-form-urlencoded", ... "Accept": "text/plain"} - >>> conn = http.client.HTTPConnection("musi-cal.mojam.com:80") - >>> conn.request("POST", "/cgi-bin/query", params, headers) + >>> conn = http.client.HTTPConnection("bugs.python.org") + >>> conn.request("POST", "", params, headers) >>> response = conn.getresponse() >>> print(response.status, response.reason) - 200 OK + 302 Found >>> data = response.read() + >>> data + b'Redirecting to <a href="http://bugs.python.org/issue12524">http://bugs.python.org/issue12524</a>' >>> conn.close() diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst index 74d8d1626a..97714968f2 100644 --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -6,6 +6,9 @@ .. moduleauthor:: John J. Lee <jjl@pobox.com> .. sectionauthor:: John J. Lee <jjl@pobox.com> +**Source code:** :source:`Lib/http/cookiejar.py` + +-------------- The :mod:`http.cookiejar` module defines classes for automatic handling of HTTP cookies. It is useful for accessing web sites that require small pieces of data diff --git a/Doc/library/http.cookies.rst b/Doc/library/http.cookies.rst index 472ddcf5b7..6baf28ec98 100644 --- a/Doc/library/http.cookies.rst +++ b/Doc/library/http.cookies.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Timothy O'Malley <timo@alum.mit.edu> .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/http/cookies.py` + +-------------- The :mod:`http.cookies` module defines classes for abstracting the concept of cookies, an HTTP state management mechanism. It supports both simple string-only @@ -149,7 +152,7 @@ Morsel Objects .. method:: Morsel.set(key, value, coded_value) - Set the *key*, *value* and *coded_value* members. + Set the *key*, *value* and *coded_value* attributes. .. method:: Morsel.isReservedKey(K) diff --git a/Doc/library/http.server.rst b/Doc/library/http.server.rst index 1ca1620813..e3a3a10dd1 100644 --- a/Doc/library/http.server.rst +++ b/Doc/library/http.server.rst @@ -11,6 +11,10 @@ single: URL single: httpd +**Source code:** :source:`Lib/http/server.py` + +-------------- + This module defines classes for implementing HTTP servers (Web servers). One class, :class:`HTTPServer`, is a :class:`socketserver.TCPServer` subclass. @@ -155,6 +159,17 @@ of which this module provides three different variants: This method will parse and dispatch the request to the appropriate :meth:`do_\*` method. You should never need to override it. + .. method:: handle_expect_100() + + When a HTTP/1.1 compliant server receives a ``Expect: 100-continue`` + request header it responds back with a ``100 Continue`` followed by ``200 + OK`` headers. + This method can be overridden to raise an error if the server does not + want the client to continue. For e.g. server can chose to send ``417 + Expectation Failed`` as a response header and ``return False``. + + .. versionadded:: 3.2 + .. method:: send_error(code, message=None) Sends and logs a complete error reply to the client. The numeric *code* @@ -171,13 +186,29 @@ of which this module provides three different variants: .. method:: send_header(keyword, value) - Writes a specific HTTP header to the output stream. *keyword* should - specify the header keyword, with *value* specifying its value. + Stores the HTTP header to an internal buffer which will be written to the + output stream when :meth:`end_headers` method is invoked. + *keyword* should specify the header keyword, with *value* + specifying its value. + + .. versionchanged:: 3.2 Storing the headers in an internal buffer + + + .. method:: send_response_only(code, message=None) + + Sends the reponse header only, used for the purposes when ``100 + Continue`` response is sent by the server to the client. The headers not + buffered and sent directly the output stream.If the *message* is not + specified, the HTTP message corresponding the response *code* is sent. + + .. versionadded:: 3.2 .. method:: end_headers() - Sends a blank line, indicating the end of the HTTP headers in the - response. + Write the buffered HTTP headers to the output stream and send a blank + line, indicating the end of the HTTP headers in the response. + + .. versionchanged:: 3.2 Writing the buffered headers to the output stream. .. method:: log_request(code='-', size='-') diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index 04088ac5aa..3f45c95a8a 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -16,6 +16,10 @@ pair: IMAP4_SSL; protocol pair: IMAP4_stream; protocol +**Source code:** :source:`Lib/imaplib.py` + +-------------- + This module defines three classes, :class:`IMAP4`, :class:`IMAP4_SSL` and :class:`IMAP4_stream`, which encapsulate a connection to an IMAP4 server and implement a large subset of the IMAP4rev1 client protocol as defined in @@ -56,6 +60,7 @@ Three exceptions are defined as attributes of the :class:`IMAP4` class: write permission, and the mailbox will need to be re-opened to re-obtain write permission. + There's also a subclass for secure connections: @@ -68,6 +73,7 @@ There's also a subclass for secure connections: and *certfile* are also optional - they can contain a PEM formatted private key and certificate chain file for the SSL connection. + The second subclass allows for connections created by a child process: @@ -83,9 +89,9 @@ The following utility functions are defined: .. function:: Internaldate2tuple(datestr) - Converts an IMAP4 INTERNALDATE string to Coordinated Universal Time. Returns a - :mod:`time` module tuple. - + Parse an IMAP4 ``INTERNALDATE`` string and return corresponding local + time. The return value is a :class:`time.struct_time` tuple or + None if the string has wrong format. .. function:: Int2AP(num) @@ -100,9 +106,13 @@ The following utility functions are defined: .. function:: Time2Internaldate(date_time) - Converts a :mod:`time` module tuple to an IMAP4 ``INTERNALDATE`` representation. - Returns a string in the form: ``"DD-Mmm-YYYY HH:MM:SS +HHMM"`` (including - double-quotes). + Convert *date_time* to an IMAP4 ``INTERNALDATE`` representation. The + return value is a string in the form: ``"DD-Mmm-YYYY HH:MM:SS + +HHMM"`` (including double-quotes). The *date_time* argument can be a + number (int or float) representing seconds since epoch (as returned + by :func:`time.time`), a 9-tuple representing local time (as returned by + :func:`time.localtime`), or a double-quoted string. In the last case, it + is assumed to already be in the correct format. Note that IMAP4 message numbers change as the mailbox changes; in particular, after an ``EXPUNGE`` command performs deletions the remaining messages are @@ -408,6 +418,15 @@ An :class:`IMAP4` instance has the following methods: This is an ``IMAP4rev1`` extension command. +.. method:: IMAP4.starttls(ssl_context=None) + + Send a ``STARTTLS`` command. The *ssl_context* argument is optional + and should be a :class:`ssl.SSLContext` object. This will enable + encryption on the IMAP connection. + + .. versionadded:: 3.2 + + .. method:: IMAP4.status(mailbox, names) Request named status conditions for *mailbox*. diff --git a/Doc/library/imghdr.rst b/Doc/library/imghdr.rst index 0c0722df17..32ec9cfc2f 100644 --- a/Doc/library/imghdr.rst +++ b/Doc/library/imghdr.rst @@ -4,6 +4,9 @@ .. module:: imghdr :synopsis: Determine the type of image contained in a file or byte stream. +**Source code:** :source:`Lib/imghdr.py` + +-------------- The :mod:`imghdr` module determines the type of image contained in a file or byte stream. diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst index 2d83893d33..6e9845ed00 100644 --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -190,8 +190,43 @@ This module provides an interface to the mechanisms used to implement the continue to use the old class definition. The same is true for derived classes. -The following constants with integer values, defined in this module, are used to -indicate the search result of :func:`find_module`. +The following functions are conveniences for handling :pep:`3147` byte-compiled +file paths. + +.. versionadded:: 3.2 + +.. function:: cache_from_source(path, debug_override=None) + + Return the :pep:`3147` path to the byte-compiled file associated with the + source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return + value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. + The ``cpython-32`` string comes from the current magic tag (see + :func:`get_tag`). The returned path will end in ``.pyc`` when + ``__debug__`` is True or ``.pyo`` for an optimized Python + (i.e. ``__debug__`` is False). By passing in True or False for + *debug_override* you can override the system's value for ``__debug__`` for + extension selection. + + *path* need not exist. + + +.. function:: source_from_cache(path) + + Given the *path* to a :pep:`3147` file name, return the associated source code + file path. For example, if *path* is + ``/foo/bar/__pycache__/baz.cpython-32.pyc`` the returned path would be + ``/foo/bar/baz.py``. *path* need not exist, however if it does not conform + to :pep:`3147` format, a ``ValueError`` is raised. + + +.. function:: get_tag() + + Return the :pep:`3147` magic tag string matching this version of Python's + magic number, as returned by :func:`get_magic`. + + +The following constants with integer values, defined in this module, are used +to indicate the search result of :func:`find_module`. .. data:: PY_SOURCE @@ -273,10 +308,3 @@ in that version, since :func:`find_module` has been extended and # Since we may exit via an exception, close fp explicitly. if fp: fp.close() - -.. index:: module: knee - -A more complete example that implements hierarchical module names and includes a -:func:`reload` function can be found in the module :mod:`knee`. The :mod:`knee` -module can be found in :file:`Demo/imputil/` in the Python source distribution. - diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index cf13ba3033..c9f742ad03 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -18,12 +18,12 @@ implementation of the :keyword:`import` statement (and thus, by extension, the :func:`__import__` function) in Python source code. This provides an implementation of :keyword:`import` which is portable to any Python interpreter. This also provides a reference implementation which is easier to -comprehend than one in a programming language other than Python. +comprehend than one implemented in a programming language other than Python. -Two, the components to implement :keyword:`import` can be exposed in this +Two, the components to implement :keyword:`import` are exposed in this package, making it easier for users to create their own custom objects (known generically as an :term:`importer`) to participate in the import process. -Details on providing custom importers can be found in :pep:`302`. +Details on custom importers can be found in :pep:`302`. .. seealso:: @@ -32,12 +32,11 @@ Details on providing custom importers can be found in :pep:`302`. `Packages specification <http://www.python.org/doc/essays/packages.html>`__ Original specification of packages. Some semantics have changed since - the writing of this document (e.g. redirecting based on :keyword:`None` + the writing of this document (e.g. redirecting based on ``None`` in :data:`sys.modules`). The :func:`.__import__` function - The built-in function for which the :keyword:`import` statement is - syntactic sugar. + The :keyword:`import` statement is syntactic sugar for this function. :pep:`235` Import on Case-Insensitive Platforms @@ -46,7 +45,7 @@ Details on providing custom importers can be found in :pep:`302`. Defining Python Source Code Encodings :pep:`302` - New Import Hooks. + New Import Hooks :pep:`328` Imports: Multi-Line and Absolute/Relative @@ -57,14 +56,16 @@ Details on providing custom importers can be found in :pep:`302`. :pep:`3120` Using UTF-8 as the Default Source Encoding + :pep:`3147` + PYC Repository Directories + Functions --------- .. function:: __import__(name, globals={}, locals={}, fromlist=list(), level=0) - An implementation of the built-in :func:`__import__` function. See the - built-in function's documentation for usage instructions. + An implementation of the built-in :func:`__import__` function. .. function:: import_module(name, package=None) @@ -108,7 +109,7 @@ are also provided to help in implementing the core ABCs. module. If the :term:`finder` is found on :data:`sys.meta_path` and the module to be searched for is a subpackage or module then *path* will be the value of :attr:`__path__` from the parent package. If a loader - cannot be found, :keyword:`None` is returned. + cannot be found, ``None`` is returned. .. class:: Loader @@ -184,14 +185,14 @@ are also provided to help in implementing the core ABCs. .. method:: get_code(fullname) An abstract method to return the :class:`code` object for a module. - :keyword:`None` is returned if the module does not have a code object + ``None`` is returned if the module does not have a code object (e.g. built-in module). :exc:`ImportError` is raised if loader cannot find the requested module. .. method:: get_source(fullname) An abstract method to return the source of a module. It is returned as - a text string with universal newlines. Returns :keyword:`None` if no + a text string with universal newlines. Returns ``None`` if no source is available (e.g. a built-in module). Raises :exc:`ImportError` if the loader cannot find the module specified. @@ -202,21 +203,133 @@ are also provided to help in implementing the core ABCs. :term:`loader` cannot find the module. +.. class:: ExecutionLoader + + An abstract base class which inherits from :class:`InspectLoader` that, + when implemented, helps a module to be executed as a script. The ABC + represents an optional :pep:`302` protocol. + + .. method:: get_filename(fullname) + + An abstract method that is to return the value of :attr:`__file__` for + the specified module. If no path is available, :exc:`ImportError` is + raised. + + If source code is available, then the method should return the path to + the source file, regardless of whether a bytecode was used to load the + module. + + +.. class:: SourceLoader + + An abstract base class for implementing source (and optionally bytecode) + file loading. The class inherits from both :class:`ResourceLoader` and + :class:`ExecutionLoader`, requiring the implementation of: + + * :meth:`ResourceLoader.get_data` + * :meth:`ExecutionLoader.get_filename` + Should only return the path to the source file; sourceless + loading is not supported. + + The abstract methods defined by this class are to add optional bytecode + file support. Not implementing these optional methods causes the loader to + only work with source code. Implementing the methods allows the loader to + work with source *and* bytecode files; it does not allow for *sourceless* + loading where only bytecode is provided. Bytecode files are an + optimization to speed up loading by removing the parsing step of Python's + compiler, and so no bytecode-specific API is exposed. + + .. method:: path_mtime(self, path) + + Optional abstract method which returns the modification time for the + specified path. + + .. method:: set_data(self, path, data) + + Optional abstract method which writes the specified bytes to a file + path. Any intermediate directories which do not exist are to be created + automatically. + + When writing to the path fails because the path is read-only + (:attr:`errno.EACCES`), do not propagate the exception. + + .. method:: get_code(self, fullname) + + Concrete implementation of :meth:`InspectLoader.get_code`. + + .. method:: load_module(self, fullname) + + Concrete implementation of :meth:`Loader.load_module`. + + .. method:: get_source(self, fullname) + + Concrete implementation of :meth:`InspectLoader.get_source`. + + .. method:: is_package(self, fullname) + + Concrete implementation of :meth:`InspectLoader.is_package`. A module + is determined to be a package if its file path is a file named + ``__init__`` when the file extension is removed. + + .. class:: PyLoader - An abstract base class inheriting from :class:`importlib.abc.InspectLoader` - and :class:`importlib.abc.ResourceLoader` designed to ease the loading of + An abstract base class inheriting from + :class:`ExecutionLoader` and + :class:`ResourceLoader` designed to ease the loading of Python source modules (bytecode is not handled; see - :class:`importlib.abc.PyPycLoader` for a source/bytecode ABC). A subclass + :class:`SourceLoader` for a source/bytecode ABC). A subclass implementing this ABC will only need to worry about exposing how the source code is stored; all other details for loading Python source code will be handled by the concrete implementations of key methods. + .. deprecated:: 3.2 + This class has been deprecated in favor of :class:`SourceLoader` and is + slated for removal in Python 3.4. See below for how to create a + subclass that is compatible with Python 3.1 onwards. + + If compatibility with Python 3.1 is required, then use the following idiom + to implement a subclass that will work with Python 3.1 onwards (make sure + to implement :meth:`ExecutionLoader.get_filename`):: + + try: + from importlib.abc import SourceLoader + except ImportError: + from importlib.abc import PyLoader as SourceLoader + + + class CustomLoader(SourceLoader): + def get_filename(self, fullname): + """Return the path to the source file.""" + # Implement ... + + def source_path(self, fullname): + """Implement source_path in terms of get_filename.""" + try: + return self.get_filename(fullname) + except ImportError: + return None + + def is_package(self, fullname): + """Implement is_package by looking for an __init__ file + name as returned by get_filename.""" + filename = os.path.basename(self.get_filename(fullname)) + return os.path.splitext(filename)[0] == '__init__' + + .. method:: source_path(fullname) An abstract method that returns the path to the source code for a - module. Should return :keyword:`None` if there is no source code. - :exc:`ImportError` if the module cannot be found. + module. Should return ``None`` if there is no source code. + Raises :exc:`ImportError` if the loader knows it cannot handle the + module. + + .. method:: get_filename(fullname) + + A concrete implementation of + :meth:`importlib.abc.ExecutionLoader.get_filename` that + relies on :meth:`source_path`. If :meth:`source_path` returns + ``None``, then :exc:`ImportError` is raised. .. method:: load_module(fullname) @@ -231,43 +344,62 @@ are also provided to help in implementing the core ABCs. A concrete implementation of :meth:`importlib.abc.InspectLoader.get_code` that creates code objects from Python source code, by requesting the source code (using - :meth:`source_path` and :meth:`get_data`), converting it to standard - newlines, and compiling it with the built-in :func:`compile` function. + :meth:`source_path` and :meth:`get_data`) and compiling it with the + built-in :func:`compile` function. .. method:: get_source(fullname) A concrete implementation of :meth:`importlib.abc.InspectLoader.get_source`. Uses - :meth:`importlib.abc.ResourceLoader.get_data` and :meth:`source_path` to - get the source code. It tries to guess the source encoding using + :meth:`importlib.abc.ResourceLoader.get_data` and :meth:`source_path` + to get the source code. It tries to guess the source encoding using :func:`tokenize.detect_encoding`. .. class:: PyPycLoader - An abstract base class inheriting from :class:`importlib.abc.PyLoader`. + An abstract base class inheriting from :class:`PyLoader`. This ABC is meant to help in creating loaders that support both Python source and bytecode. + .. deprecated:: 3.2 + This class has been deprecated in favor of :class:`SourceLoader` and to + properly support :pep:`3147`. If compatibility is required with + Python 3.1, implement both :class:`SourceLoader` and :class:`PyLoader`; + instructions on how to do so are included in the documentation for + :class:`PyLoader`. Do note that this solution will not support + sourceless/bytecode-only loading; only source *and* bytecode loading. + .. method:: source_mtime(fullname) An abstract method which returns the modification time for the source code of the specified module. The modification time should be an - integer. If there is no source code, return :keyword:`None`. If the + integer. If there is no source code, return ``None``. If the module cannot be found then :exc:`ImportError` is raised. .. method:: bytecode_path(fullname) An abstract method which returns the path to the bytecode for the - specified module, if it exists. It returns :keyword:`None` + specified module, if it exists. It returns ``None`` if no bytecode exists (yet). - Raises :exc:`ImportError` if the module is not found. + Raises :exc:`ImportError` if the loader knows it cannot handle the + module. + + .. method:: get_filename(fullname) + + A concrete implementation of + :meth:`ExecutionLoader.get_filename` that relies on + :meth:`PyLoader.source_path` and :meth:`bytecode_path`. + If :meth:`source_path` returns a path, then that value is returned. + Else if :meth:`bytecode_path` returns a path, that path will be + returned. If a path is not available from both methods, + :exc:`ImportError` is raised. .. method:: write_bytecode(fullname, bytecode) An abstract method which has the loader write *bytecode* for future - use. If the bytecode is written, return :keyword:`True`. Return - :keyword:`False` if the bytecode could not be written. This method + use. If the bytecode is written, return ``True``. Return + ``False`` if the bytecode could not be written. This method should not be called if :data:`sys.dont_write_bytecode` is true. The *bytecode* argument should be a bytes string or bytes array. @@ -311,7 +443,7 @@ find and load modules. terms of :data:`sys.path`. No implicit path hooks are assumed for simplification of the class and its semantics. - Only class method are defined by this class to alleviate the need for + Only class methods are defined by this class to alleviate the need for instantiation. .. classmethod:: find_module(fullname, path=None) @@ -325,7 +457,7 @@ find and load modules. :data:`sys.path_importer_cache`, then :data:`sys.path_hooks` is searched for a finder for the path entry and, if found, is stored in :data:`sys.path_importer_cache` along with being queried about the - module. If no finder is ever found then :keyword:`None` is returned. + module. If no finder is ever found then ``None`` is returned. :mod:`importlib.util` -- Utility code for importers @@ -337,7 +469,7 @@ find and load modules. This module contains the various objects that help in the construction of an :term:`importer`. -.. function:: module_for_loader(method) +.. decorator:: module_for_loader A :term:`decorator` for a :term:`loader` method, to handle selecting the proper @@ -362,7 +494,7 @@ an :term:`importer`. Use of this decorator handles all the details of which module object a loader should initialize as specified by :pep:`302`. -.. function:: set_loader(fxn) +.. decorator:: set_loader A :term:`decorator` for a :term:`loader` method, to set the :attr:`__loader__` @@ -370,11 +502,11 @@ an :term:`importer`. does nothing. It is assumed that the first positional argument to the wrapped method is what :attr:`__loader__` should be set to. -.. function:: set_package(fxn) +.. decorator:: set_package A :term:`decorator` for a :term:`loader` to set the :attr:`__package__` attribute on the module returned by the loader. If :attr:`__package__` is - set and has a value other than :keyword:`None` it will not be changed. + set and has a value other than ``None`` it will not be changed. Note that the module returned by the loader is what has the attribute set on and not the module found in :data:`sys.modules`. @@ -384,100 +516,3 @@ an :term:`importer`. attribute to be used at the global level of the module during initialization. - -Example -------- - -Below is an example meta path importer that uses a dict for back-end storage -for source code. While not an optimal solution -- manipulations of -:attr:`__path__` on packages does not influence import -- it does illustrate -what little is required to implement an importer. - -.. testcode:: - - """An importer where source is stored in a dict.""" - from importlib import abc - - - class DictImporter(abc.Finder, abc.PyLoader): - - """A meta path importer that stores source code in a dict. - - The keys are the module names -- packages must end in ``.__init__``. - The values must be something that can be passed to 'bytes'. - - """ - - def __init__(self, memory): - """Store the dict.""" - self.memory = memory - - def contains(self, name): - """See if a module or package is in the dict.""" - if name in self.memory: - return name - package_name = '{}.__init__'.format(name) - if package_name in self.memory: - return package_name - return False - - __contains__ = contains # Convenience. - - def find_module(self, fullname, path=None): - """Find the module in the dict.""" - if fullname in self: - return self - return None - - def source_path(self, fullname): - """Return the module name if the module is in the dict.""" - if not fullname in self: - raise ImportError - return fullname - - def get_data(self, path): - """Return the bytes for the source. - - The value found in the dict is passed through 'bytes' before being - returned. - - """ - name = self.contains(path) - if not name: - raise IOError - return bytes(self.memory[name]) - - def is_package(self, fullname): - """Tell if module is a package based on whether the dict contains the - name with ``.__init__`` appended to it.""" - if fullname not in self: - raise ImportError - if fullname in self.memory: - return False - # If name is in this importer but not as it is then it must end in - # ``__init__``. - else: - return True - -.. testcode:: - :hide: - - import importlib - import sys - - - # Build the dict; keys of name, value of __package__. - names = {'_top_level': '', '_pkg.__init__': '_pkg', '_pkg.mod': '_pkg'} - source = {name: "name = {!r}".format(name).encode() for name in names} - - # Register the meta path importer. - importer = DictImporter(source) - sys.meta_path.append(importer) - - # Sanity check. - for name in names: - module = importlib.import_module(name) - assert module.__name__ == name - assert getattr(module, 'name') == name - assert module.__loader__ is importer - assert module.__package__ == names[name] diff --git a/Doc/library/index.rst b/Doc/library/index.rst index aa582dea1a..9ac688c8c9 100644 --- a/Doc/library/index.rst +++ b/Doc/library/index.rst @@ -43,13 +43,13 @@ the `Python Package Index <http://pypi.python.org/pypi>`_. intro.rst functions.rst constants.rst - objects.rst stdtypes.rst exceptions.rst strings.rst datatypes.rst numeric.rst + functional.rst filesys.rst persistence.rst archiving.rst diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 4def286c21..d127ce8cfc 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Ka-Ping Yee <ping@lfw.org> .. sectionauthor:: Ka-Ping Yee <ping@lfw.org> +**Source code:** :source:`Lib/inspect.py` + +-------------- The :mod:`inspect` module provides several useful functions to help get information about live objects such as modules, classes, methods, functions, @@ -87,7 +90,7 @@ attributes: | frame | f_back | next outer frame object | | | | (this frame's caller) | +-----------+-----------------+---------------------------+ -| | f_builtins | built-in namespace seen | +| | f_builtins | builtins namespace seen | | | | by this frame | +-----------+-----------------+---------------------------+ | | f_code | code object being | @@ -295,7 +298,7 @@ attributes: .. impl-detail:: getsets are attributes defined in extension modules via - :ctype:`PyGetSetDef` structures. For Python implementations without such + :c:type:`PyGetSetDef` structures. For Python implementations without such types, this method will always return ``False``. @@ -306,7 +309,7 @@ attributes: .. impl-detail:: Member descriptors are attributes defined in extension modules via - :ctype:`PyMemberDef` structures. For Python implementations without such + :c:type:`PyMemberDef` structures. For Python implementations without such types, this method will always return ``False``. @@ -451,6 +454,32 @@ Classes and functions metatype is in use, cls will be the first element of the tuple. +.. function:: getcallargs(func[, *args][, **kwds]) + + Bind the *args* and *kwds* to the argument names of the Python function or + method *func*, as if it was called with them. For bound methods, bind also the + first argument (typically named ``self``) to the associated instance. A dict + is returned, mapping the argument names (including the names of the ``*`` and + ``**`` arguments, if any) to their values from *args* and *kwds*. In case of + invoking *func* incorrectly, i.e. whenever ``func(*args, **kwds)`` would raise + an exception because of incompatible signature, an exception of the same type + and the same or similar message is raised. For example:: + + >>> from inspect import getcallargs + >>> def f(a, b=1, *pos, **named): + ... pass + >>> getcallargs(f, 1, 2, 3) + {'a': 1, 'named': {}, 'b': 2, 'pos': (3,)} + >>> getcallargs(f, a=2, x=4) + {'a': 2, 'named': {'x': 4}, 'b': 1, 'pos': ()} + >>> getcallargs(f) + Traceback (most recent call last): + ... + TypeError: f() takes at least 1 argument (0 given) + + .. versionadded:: 3.2 + + .. _inspect-stack: The interpreter stack @@ -536,3 +565,81 @@ line. entry in the list represents the caller; the last entry represents where the exception was raised. + +Fetching attributes statically +------------------------------ + +Both :func:`getattr` and :func:`hasattr` can trigger code execution when +fetching or checking for the existence of attributes. Descriptors, like +properties, will be invoked and :meth:`__getattr__` and :meth:`__getattribute__` +may be called. + +For cases where you want passive introspection, like documentation tools, this +can be inconvenient. :func:`getattr_static` has the same signature as :func:`getattr` +but avoids executing code when it fetches attributes. + +.. function:: getattr_static(obj, attr, default=None) + + Retrieve attributes without triggering dynamic lookup via the + descriptor protocol, :meth:`__getattr__` or :meth:`__getattribute__`. + + Note: this function may not be able to retrieve all attributes + that getattr can fetch (like dynamically created attributes) + and may find attributes that getattr can't (like descriptors + that raise AttributeError). It can also return descriptors objects + instead of instance members. + + If the instance :attr:`__dict__` is shadowed by another member (for example a + property) then this function will be unable to find instance members. + + .. versionadded:: 3.2 + +:func:`getattr_static` does not resolve descriptors, for example slot descriptors or +getset descriptors on objects implemented in C. The descriptor object +is returned instead of the underlying attribute. + +You can handle these with code like the following. Note that +for arbitrary getset descriptors invoking these may trigger +code execution:: + + # example code for resolving the builtin descriptor types + class _foo: + __slots__ = ['foo'] + + slot_descriptor = type(_foo.foo) + getset_descriptor = type(type(open(__file__)).name) + wrapper_descriptor = type(str.__dict__['__add__']) + descriptor_types = (slot_descriptor, getset_descriptor, wrapper_descriptor) + + result = getattr_static(some_object, 'foo') + if type(result) in descriptor_types: + try: + result = result.__get__() + except AttributeError: + # descriptors can raise AttributeError to + # indicate there is no underlying value + # in which case the descriptor itself will + # have to do + pass + + +Current State of a Generator +---------------------------- + +When implementing coroutine schedulers and for other advanced uses of +generators, it is useful to determine whether a generator is currently +executing, is waiting to start or resume or execution, or has already +terminated. :func:`getgeneratorstate` allows the current state of a +generator to be determined easily. + +.. function:: getgeneratorstate(generator) + + Get current state of a generator-iterator. + + Possible states are: + * GEN_CREATED: Waiting to start execution. + * GEN_RUNNING: Currently being executed by the interpreter. + * GEN_SUSPENDED: Currently suspended at a yield expression. + * GEN_CLOSED: Execution has completed. + + .. versionadded:: 3.2 diff --git a/Doc/library/io.rst b/Doc/library/io.rst index bbe5112427..becc4a20b3 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -217,7 +217,7 @@ I/O Base Classes :class:`IOBase` object can be iterated over yielding the lines in a stream. Lines are defined slightly differently depending on whether the stream is a binary stream (yielding bytes), or a text stream (yielding character - strings). See :meth:`readline` below. + strings). See :meth:`~IOBase.readline` below. IOBase is also a context manager and therefore supports the :keyword:`with` statement. In this example, *file* is closed after the @@ -391,8 +391,8 @@ I/O Base Classes :class:`RawIOBase` implementation, but wrap one, like :class:`BufferedWriter` and :class:`BufferedReader` do. - :class:`BufferedIOBase` provides or overrides these members in addition to - those from :class:`IOBase`: + :class:`BufferedIOBase` provides or overrides these methods and attribute in + addition to those from :class:`IOBase`: .. attribute:: raw @@ -513,6 +513,24 @@ than raw I/O does. :class:`BytesIO` provides or overrides these methods in addition to those from :class:`BufferedIOBase` and :class:`IOBase`: + .. method:: getbuffer() + + Return a readable and writable view over the contents of the buffer + without copying them. Also, mutating the view will transparently + update the contents of the buffer:: + + >>> b = io.BytesIO(b"abcdef") + >>> view = b.getbuffer() + >>> view[2:4] = b"56" + >>> b.getvalue() + b'ab56ef' + + .. note:: + As long as the view exists, the :class:`BytesIO` object cannot be + resized. + + .. versionadded:: 3.2 + .. method:: getvalue() Return ``bytes`` containing the entire contents of the buffer. @@ -589,25 +607,6 @@ than raw I/O does. if the buffer needs to be written out but the raw stream blocks. -.. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE) - - A buffered I/O object giving a combined, higher-level access to two - sequential :class:`RawIOBase` objects: one readable, the other writeable. - It is useful for pairs of unidirectional communication channels - (pipes, for instance). It inherits :class:`BufferedIOBase`. - - *reader* and *writer* are :class:`RawIOBase` objects that are readable and - writeable respectively. If the *buffer_size* is omitted it defaults to - :data:`DEFAULT_BUFFER_SIZE`. - - A fourth argument, *max_buffer_size*, is supported, but unused and - deprecated. - - :class:`BufferedRWPair` implements all of :class:`BufferedIOBase`\'s methods - except for :meth:`~BufferedIOBase.detach`, which raises - :exc:`UnsupportedOperation`. - - .. class:: BufferedRandom(raw, buffer_size=DEFAULT_BUFFER_SIZE) A buffered interface to random access streams. It inherits @@ -624,6 +623,29 @@ than raw I/O does. :class:`BufferedWriter` can do. +.. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE) + + A buffered I/O object combining two unidirectional :class:`RawIOBase` + objects -- one readable, the other writeable -- into a single bidirectional + endpoint. It inherits :class:`BufferedIOBase`. + + *reader* and *writer* are :class:`RawIOBase` objects that are readable and + writeable respectively. If the *buffer_size* is omitted it defaults to + :data:`DEFAULT_BUFFER_SIZE`. + + A fourth argument, *max_buffer_size*, is supported, but unused and + deprecated. + + :class:`BufferedRWPair` implements all of :class:`BufferedIOBase`\'s methods + except for :meth:`~BufferedIOBase.detach`, which raises + :exc:`UnsupportedOperation`. + + .. warning:: + :class:`BufferedRWPair` does not attempt to synchronize accesses to + its underlying raw streams. You should not pass it the same object + as reader and writer; use :class:`BufferedRandom` instead. + + Text I/O ^^^^^^^^ @@ -682,6 +704,32 @@ Text I/O Read until newline or EOF and return a single ``str``. If the stream is already at EOF, an empty string is returned. + .. method:: seek(offset, whence=SEEK_SET) + + Change the stream position to the given *offset*. Behaviour depends + on the *whence* parameter: + + * :data:`SEEK_SET` or ``0``: seek from the start of the stream + (the default); *offset* must either be a number returned by + :meth:`TextIOBase.tell`, or zero. Any other *offset* value + produces undefined behaviour. + * :data:`SEEK_CUR` or ``1``: "seek" to the current position; + *offset* must be zero, which is a no-operation (all other values + are unsupported). + * :data:`SEEK_END` or ``2``: seek to the end of the stream; + *offset* must be zero (all other values are unsupported). + + Return the new absolute position as an opaque number. + + .. versionadded:: 3.1 + The ``SEEK_*`` constants. + + .. method:: tell() + + Return the current stream position as an opaque number. The number + does not usually represent a number of bytes in the underlying + binary storage. + .. method:: write(s) Write the string *s* to the stream and return the number of characters @@ -767,37 +815,33 @@ Text I/O inherits :class:`codecs.IncrementalDecoder`. -Advanced topics ---------------- - -Here we will discuss several advanced topics pertaining to the concrete -I/O implementations described above. - Performance -^^^^^^^^^^^ +----------- + +This section discusses the performance of the provided concrete I/O +implementations. Binary I/O -"""""""""" - -By reading and writing only large chunks of data even when the user asks -for a single byte, buffered I/O is designed to hide any inefficiency in -calling and executing the operating system's unbuffered I/O routines. The -gain will vary very much depending on the OS and the kind of I/O which is -performed (for example, on some contemporary OSes such as Linux, unbuffered -disk I/O can be as fast as buffered I/O). The bottom line, however, is -that buffered I/O will offer you predictable performance regardless of the -platform and the backing device. Therefore, it is most always preferable to -use buffered I/O rather than unbuffered I/O. +^^^^^^^^^^ + +By reading and writing only large chunks of data even when the user asks for a +single byte, buffered I/O hides any inefficiency in calling and executing the +operating system's unbuffered I/O routines. The gain depends on the OS and the +kind of I/O which is performed. For example, on some modern OSes such as Linux, +unbuffered disk I/O can be as fast as buffered I/O. The bottom line, however, +is that buffered I/O offers predictable performance regardless of the platform +and the backing device. Therefore, it is most always preferable to use buffered +I/O rather than unbuffered I/O for binary datal Text I/O -"""""""" +^^^^^^^^ Text I/O over a binary storage (such as a file) is significantly slower than -binary I/O over the same storage, because it implies conversions from -unicode to binary data using a character codec. This can become noticeable -if you handle huge amounts of text data (for example very large log files). -Also, :meth:`TextIOWrapper.tell` and :meth:`TextIOWrapper.seek` are both -quite slow due to the reconstruction algorithm used. +binary I/O over the same storage, because it requires conversions between +unicode and binary data using a character codec. This can become noticeable +handling huge amounts of text data like large log files. Also, +:meth:`TextIOWrapper.tell` and :meth:`TextIOWrapper.seek` are both quite slow +due to the reconstruction algorithm used. :class:`StringIO`, however, is a native in-memory unicode container and will exhibit similar speed to :class:`BytesIO`. @@ -805,9 +849,8 @@ exhibit similar speed to :class:`BytesIO`. Multi-threading ^^^^^^^^^^^^^^^ -:class:`FileIO` objects are thread-safe to the extent that the operating -system calls (such as ``read(2)`` under Unix) they are wrapping are thread-safe -too. +:class:`FileIO` objects are thread-safe to the extent that the operating system +calls (such as ``read(2)`` under Unix) they wrap are thread-safe too. Binary buffered objects (instances of :class:`BufferedReader`, :class:`BufferedWriter`, :class:`BufferedRandom` and :class:`BufferedRWPair`) @@ -822,12 +865,13 @@ Reentrancy Binary buffered objects (instances of :class:`BufferedReader`, :class:`BufferedWriter`, :class:`BufferedRandom` and :class:`BufferedRWPair`) are not reentrant. While reentrant calls will not happen in normal situations, -they can arise if you are doing I/O in a :mod:`signal` handler. If it is -attempted to enter a buffered object again while already being accessed -*from the same thread*, then a :exc:`RuntimeError` is raised. - -The above implicitly extends to text files, since the :func:`open()` -function will wrap a buffered object inside a :class:`TextIOWrapper`. This -includes standard streams and therefore affects the built-in function -:func:`print()` as well. +they can arise from doing I/O in a :mod:`signal` handler. If a thread tries to +renter a buffered object which it is already accessing, a :exc:`RuntimeError` is +raised. Note this doesn't prohibit a different thread from entering the +buffered object. + +The above implicitly extends to text files, since the :func:`open()` function +will wrap a buffered object inside a :class:`TextIOWrapper`. This includes +standard streams and therefore affects the built-in function :func:`print()` as +well. diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index add209153e..d1d1188106 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -46,6 +46,7 @@ Iterator Arguments Results ==================== ============================ ================================================= ============================================================= Iterator Arguments Results Example ==================== ============================ ================================================= ============================================================= +:func:`accumulate` p p0, p0+p1, p0+p1+p2, ... ``accumulate([1,2,3,4,5]) --> 1 3 6 10 15`` :func:`chain` p, q, ... p0, p1, ... plast, q0, q1, ... ``chain('ABC', 'DEF') --> A B C D E F`` :func:`compress` data, selectors (d[0] if s[0]), (d[1] if s[1]), ... ``compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F`` :func:`dropwhile` pred, seq seq[n], seq[n+1], starting when pred fails ``dropwhile(lambda x: x<5, [1,4,6,4,1]) --> 6 4 1`` @@ -83,6 +84,22 @@ The following module functions all construct and return iterators. Some provide streams of infinite length, so they should only be accessed by functions or loops that truncate the stream. +.. function:: accumulate(iterable) + + Make an iterator that returns accumulated sums. Elements may be any addable + type including :class:`Decimal` or :class:`Fraction`. Equivalent to:: + + def accumulate(iterable): + 'Return running totals' + # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 + it = iter(iterable) + total = next(it) + yield total + for element in it: + total = total + element + yield total + + .. versionadded:: 3.2 .. function:: chain(*iterables) @@ -402,7 +419,7 @@ loops that truncate the stream. if r > n: return indices = list(range(n)) - cycles = range(n, n-r, -1) + cycles = list(range(n, n-r, -1)) yield tuple(pool[i] for i in indices[:r]) while n: for i in reversed(range(r)): @@ -479,6 +496,11 @@ loops that truncate the stream. for i in range(times): yield object + A common use for *repeat* is to supply a stream of constant values to *map* + or *zip*:: + + >>> list(map(pow, range(10), repeat(2))) + [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] .. function:: starmap(function, iterable) @@ -540,16 +562,25 @@ loops that truncate the stream. iterables are of uneven length, missing values are filled-in with *fillvalue*. Iteration continues until the longest iterable is exhausted. Equivalent to:: - def zip_longest(*args, fillvalue=None): + class ZipExhausted(Exception): + pass + + def zip_longest(*args, **kwds): # zip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D- - def sentinel(counter = ([fillvalue]*(len(args)-1)).pop): - yield counter() # yields the fillvalue, or raises IndexError + fillvalue = kwds.get('fillvalue') + counter = len(args) - 1 + def sentinel(): + nonlocal counter + if not counter: + raise ZipExhausted + counter -= 1 + yield fillvalue fillers = repeat(fillvalue) - iters = [chain(it, sentinel(), fillers) for it in args] + iterators = [chain(it, sentinel(), fillers) for it in args] try: - for tup in zip(*iters): - yield tup - except IndexError: + while iterators: + yield tuple(map(next, iterators)) + except ZipExhausted: pass If one of the iterables is potentially infinite, then the :func:`zip_longest` @@ -560,8 +591,8 @@ loops that truncate the stream. .. _itertools-recipes: -Recipes -------- +Itertools Recipes +----------------- This section shows recipes for creating an extended toolset using the existing itertools as building blocks. @@ -653,6 +684,12 @@ which incur interpreter overhead. pending -= 1 nexts = cycle(islice(nexts, pending)) + def partition(pred, iterable): + 'Use a predicate to partition entries into false entries and true entries' + # partition(is_odd, range(10)) --> 0 2 4 6 8 and 1 3 5 7 9 + t1, t2 = tee(iterable) + return filterfalse(pred, t1), filter(pred, t2) + def powerset(iterable): "powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)" s = list(iterable) diff --git a/Doc/library/json.rst b/Doc/library/json.rst index 48acaf19d6..a791259831 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -6,7 +6,7 @@ .. moduleauthor:: Bob Ippolito <bob@redivi.com> .. sectionauthor:: Bob Ippolito <bob@redivi.com> -JSON (JavaScript Object Notation) <http://json.org> is a subset of JavaScript +`JSON (JavaScript Object Notation) <http://json.org>`_ is a subset of JavaScript syntax (ECMA-262 3rd edition) used as a lightweight data interchange format. :mod:`json` exposes an API familiar to users of the standard library @@ -125,6 +125,10 @@ Basic Usage :class:`bytes` objects. Therefore, ``fp.write()`` must support :class:`str` input. + If *ensure_ascii* is ``True`` (the default), the output is guaranteed to + have all incoming non-ASCII characters escaped. If *ensure_ascii* is + ``False``, these characters will be output as-is. + If *check_circular* is ``False`` (default: ``True``), then the circular reference check for container types will be skipped and a circular reference will result in an :exc:`OverflowError` (or worse). @@ -134,10 +138,12 @@ Basic Usage ``inf``, ``-inf``) in strict compliance of the JSON specification, instead of using the JavaScript equivalents (``NaN``, ``Infinity``, ``-Infinity``). - If *indent* is a non-negative integer, then JSON array elements and object - members will be pretty-printed with that indent level. An indent level of 0, - or negative, will only insert newlines. ``None`` (the default) selects the - most compact representation. + If *indent* is a non-negative integer or string, then JSON array elements and + object members will be pretty-printed with that indent level. An indent level + of 0, negative, or ``""`` will only insert newlines. ``None`` (the default) + selects the most compact representation. Using a positive integer indent + indents that many spaces per level. If *indent* is a string (such at '\t'), + that string is used to indent each level. If *separators* is an ``(item_separator, dict_separator)`` tuple, then it will be used instead of the default ``(', ', ': ')`` separators. ``(',', diff --git a/Doc/library/keyword.rst b/Doc/library/keyword.rst index a4bfa501bf..173db23544 100644 --- a/Doc/library/keyword.rst +++ b/Doc/library/keyword.rst @@ -4,6 +4,9 @@ .. module:: keyword :synopsis: Test whether a string is a keyword in Python. +**Source code:** :source:`Lib/keyword.py` + +-------------- This module allows a Python program to determine if a string is a keyword. @@ -18,4 +21,3 @@ This module allows a Python program to determine if a string is a keyword. Sequence containing all the keywords defined for the interpreter. If any keywords are defined to only be active when particular :mod:`__future__` statements are in effect, these will be included as well. - diff --git a/Doc/library/language.rst b/Doc/library/language.rst index 3d907a0652..1eac32e459 100644 --- a/Doc/library/language.rst +++ b/Doc/library/language.rst @@ -26,4 +26,3 @@ These modules include: compileall.rst dis.rst pickletools.rst - distutils.rst diff --git a/Doc/library/linecache.rst b/Doc/library/linecache.rst index 688e297307..dacf8aa002 100644 --- a/Doc/library/linecache.rst +++ b/Doc/library/linecache.rst @@ -5,6 +5,9 @@ :synopsis: This module provides random access to individual lines from text files. .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/linecache.py` + +-------------- The :mod:`linecache` module allows one to get any line from any file, while attempting to optimize internally, using a cache, the common case where many diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst index 5b2941eba5..3a49e98173 100644 --- a/Doc/library/locale.rst +++ b/Doc/library/locale.rst @@ -22,19 +22,19 @@ The :mod:`locale` module defines the following exception and functions: .. exception:: Error - Exception raised when :func:`setlocale` fails. + Exception raised when the locale passed to :func:`setlocale` is not + recognized. .. function:: setlocale(category, locale=None) - If *locale* is specified, it may be a string, a tuple of the form ``(language - code, encoding)``, or ``None``. If it is a tuple, it is converted to a string - using the locale aliasing engine. If *locale* is given and not ``None``, - :func:`setlocale` modifies the locale setting for the *category*. The available - categories are listed in the data description below. The value is the name of a - locale. An empty string specifies the user's default settings. If the - modification of the locale fails, the exception :exc:`Error` is raised. If - successful, the new locale setting is returned. + If *locale* is given and not ``None``, :func:`setlocale` modifies the locale + setting for the *category*. The available categories are listed in the data + description below. *locale* may be a string, or an iterable of two strings + (language code and encoding). If it's an iterable, it's converted to a locale + name using the locale aliasing engine. An empty string specifies the user's + default settings. If the modification of the locale fails, the exception + :exc:`Error` is raised. If successful, the new locale setting is returned. If *locale* is omitted or ``None``, the current setting for *category* is returned. @@ -215,7 +215,7 @@ The :mod:`locale` module defines the following exception and functions: .. note:: - The expression is in the syntax suitable for the :cfunc:`regex` function + The expression is in the syntax suitable for the :c:func:`regex` function from the C library, which might differ from the syntax used in :mod:`re`. .. data:: NOEXPR @@ -535,7 +535,7 @@ catalogs, and the C library's search algorithms for locating message catalogs. Python applications should normally find no need to invoke these functions, and should use :mod:`gettext` instead. A known exception to this rule are applications that link with additional C libraries which internally invoke -:cfunc:`gettext` or :func:`dcgettext`. For these applications, it may be +:c:func:`gettext` or :func:`dcgettext`. For these applications, it may be necessary to bind the text domain, so that the libraries can properly locate their message catalogs. diff --git a/Doc/library/logging.config.rst b/Doc/library/logging.config.rst new file mode 100644 index 0000000000..b2dd71e65a --- /dev/null +++ b/Doc/library/logging.config.rst @@ -0,0 +1,705 @@ +:mod:`logging.config` --- Logging configuration +=============================================== + +.. module:: logging.config + :synopsis: Configuration of the logging module. + + +.. moduleauthor:: Vinay Sajip <vinay_sajip@red-dove.com> +.. sectionauthor:: Vinay Sajip <vinay_sajip@red-dove.com> + +.. sidebar:: Important + + This page contains only reference information. For tutorials, + please see + + * :ref:`Basic Tutorial <logging-basic-tutorial>` + * :ref:`Advanced Tutorial <logging-advanced-tutorial>` + * :ref:`Logging Cookbook <logging-cookbook>` + +This section describes the API for configuring the logging module. + +.. _logging-config-api: + +Configuration functions +^^^^^^^^^^^^^^^^^^^^^^^ + +The following functions configure the logging module. They are located in the +:mod:`logging.config` module. Their use is optional --- you can configure the +logging module using these functions or by making calls to the main API (defined +in :mod:`logging` itself) and defining handlers which are declared either in +:mod:`logging` or :mod:`logging.handlers`. + +.. function:: dictConfig(config) + + Takes the logging configuration from a dictionary. The contents of + this dictionary are described in :ref:`logging-config-dictschema` + below. + + If an error is encountered during configuration, this function will + raise a :exc:`ValueError`, :exc:`TypeError`, :exc:`AttributeError` + or :exc:`ImportError` with a suitably descriptive message. The + following is a (possibly incomplete) list of conditions which will + raise an error: + + * A ``level`` which is not a string or which is a string not + corresponding to an actual logging level. + * A ``propagate`` value which is not a boolean. + * An id which does not have a corresponding destination. + * A non-existent handler id found during an incremental call. + * An invalid logger name. + * Inability to resolve to an internal or external object. + + Parsing is performed by the :class:`DictConfigurator` class, whose + constructor is passed the dictionary used for configuration, and + has a :meth:`configure` method. The :mod:`logging.config` module + has a callable attribute :attr:`dictConfigClass` + which is initially set to :class:`DictConfigurator`. + You can replace the value of :attr:`dictConfigClass` with a + suitable implementation of your own. + + :func:`dictConfig` calls :attr:`dictConfigClass` passing + the specified dictionary, and then calls the :meth:`configure` method on + the returned object to put the configuration into effect:: + + def dictConfig(config): + dictConfigClass(config).configure() + + For example, a subclass of :class:`DictConfigurator` could call + ``DictConfigurator.__init__()`` in its own :meth:`__init__()`, then + set up custom prefixes which would be usable in the subsequent + :meth:`configure` call. :attr:`dictConfigClass` would be bound to + this new subclass, and then :func:`dictConfig` could be called exactly as + in the default, uncustomized state. + + .. versionadded:: 3.2 + +.. function:: fileConfig(fname, defaults=None, disable_existing_loggers=True) + + Reads the logging configuration from a :mod:`configparser`\-format file + named *fname*. This function can be called several times from an + application, allowing an end user to select from various pre-canned + configurations (if the developer provides a mechanism to present the choices + and load the chosen configuration). + + :param defaults: Defaults to be passed to the ConfigParser can be specified + in this argument. + + :param disable_existing_loggers: If specified as ``False``, loggers which + exist when this call is made are left + alone. The default is ``True`` because this + enables old behaviour in a backward- + compatible way. This behaviour is to + disable any existing loggers unless they or + their ancestors are explicitly named in the + logging configuration. + + +.. function:: listen(port=DEFAULT_LOGGING_CONFIG_PORT) + + Starts up a socket server on the specified port, and listens for new + configurations. If no port is specified, the module's default + :const:`DEFAULT_LOGGING_CONFIG_PORT` is used. Logging configurations will be + sent as a file suitable for processing by :func:`fileConfig`. Returns a + :class:`Thread` instance on which you can call :meth:`start` to start the + server, and which you can :meth:`join` when appropriate. To stop the server, + call :func:`stopListening`. + + To send a configuration to the socket, read in the configuration file and + send it to the socket as a string of bytes preceded by a four-byte length + string packed in binary using ``struct.pack('>L', n)``. + + +.. function:: stopListening() + + Stops the listening server which was created with a call to :func:`listen`. + This is typically called before calling :meth:`join` on the return value from + :func:`listen`. + + +.. _logging-config-dictschema: + +Configuration dictionary schema +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Describing a logging configuration requires listing the various +objects to create and the connections between them; for example, you +may create a handler named 'console' and then say that the logger +named 'startup' will send its messages to the 'console' handler. +These objects aren't limited to those provided by the :mod:`logging` +module because you might write your own formatter or handler class. +The parameters to these classes may also need to include external +objects such as ``sys.stderr``. The syntax for describing these +objects and connections is defined in :ref:`logging-config-dict-connections` +below. + +Dictionary Schema Details +""""""""""""""""""""""""" + +The dictionary passed to :func:`dictConfig` must contain the following +keys: + +* *version* - to be set to an integer value representing the schema + version. The only valid value at present is 1, but having this key + allows the schema to evolve while still preserving backwards + compatibility. + +All other keys are optional, but if present they will be interpreted +as described below. In all cases below where a 'configuring dict' is +mentioned, it will be checked for the special ``'()'`` key to see if a +custom instantiation is required. If so, the mechanism described in +:ref:`logging-config-dict-userdef` below is used to create an instance; +otherwise, the context is used to determine what to instantiate. + +* *formatters* - the corresponding value will be a dict in which each + key is a formatter id and each value is a dict describing how to + configure the corresponding Formatter instance. + + The configuring dict is searched for keys ``format`` and ``datefmt`` + (with defaults of ``None``) and these are used to construct a + :class:`logging.Formatter` instance. + +* *filters* - the corresponding value will be a dict in which each key + is a filter id and each value is a dict describing how to configure + the corresponding Filter instance. + + The configuring dict is searched for the key ``name`` (defaulting to the + empty string) and this is used to construct a :class:`logging.Filter` + instance. + +* *handlers* - the corresponding value will be a dict in which each + key is a handler id and each value is a dict describing how to + configure the corresponding Handler instance. + + The configuring dict is searched for the following keys: + + * ``class`` (mandatory). This is the fully qualified name of the + handler class. + + * ``level`` (optional). The level of the handler. + + * ``formatter`` (optional). The id of the formatter for this + handler. + + * ``filters`` (optional). A list of ids of the filters for this + handler. + + All *other* keys are passed through as keyword arguments to the + handler's constructor. For example, given the snippet:: + + handlers: + console: + class : logging.StreamHandler + formatter: brief + level : INFO + filters: [allow_foo] + stream : ext://sys.stdout + file: + class : logging.handlers.RotatingFileHandler + formatter: precise + filename: logconfig.log + maxBytes: 1024 + backupCount: 3 + + the handler with id ``console`` is instantiated as a + :class:`logging.StreamHandler`, using ``sys.stdout`` as the underlying + stream. The handler with id ``file`` is instantiated as a + :class:`logging.handlers.RotatingFileHandler` with the keyword arguments + ``filename='logconfig.log', maxBytes=1024, backupCount=3``. + +* *loggers* - the corresponding value will be a dict in which each key + is a logger name and each value is a dict describing how to + configure the corresponding Logger instance. + + The configuring dict is searched for the following keys: + + * ``level`` (optional). The level of the logger. + + * ``propagate`` (optional). The propagation setting of the logger. + + * ``filters`` (optional). A list of ids of the filters for this + logger. + + * ``handlers`` (optional). A list of ids of the handlers for this + logger. + + The specified loggers will be configured according to the level, + propagation, filters and handlers specified. + +* *root* - this will be the configuration for the root logger. + Processing of the configuration will be as for any logger, except + that the ``propagate`` setting will not be applicable. + +* *incremental* - whether the configuration is to be interpreted as + incremental to the existing configuration. This value defaults to + ``False``, which means that the specified configuration replaces the + existing configuration with the same semantics as used by the + existing :func:`fileConfig` API. + + If the specified value is ``True``, the configuration is processed + as described in the section on :ref:`logging-config-dict-incremental`. + +* *disable_existing_loggers* - whether any existing loggers are to be + disabled. This setting mirrors the parameter of the same name in + :func:`fileConfig`. If absent, this parameter defaults to ``True``. + This value is ignored if *incremental* is ``True``. + +.. _logging-config-dict-incremental: + +Incremental Configuration +""""""""""""""""""""""""" + +It is difficult to provide complete flexibility for incremental +configuration. For example, because objects such as filters +and formatters are anonymous, once a configuration is set up, it is +not possible to refer to such anonymous objects when augmenting a +configuration. + +Furthermore, there is not a compelling case for arbitrarily altering +the object graph of loggers, handlers, filters, formatters at +run-time, once a configuration is set up; the verbosity of loggers and +handlers can be controlled just by setting levels (and, in the case of +loggers, propagation flags). Changing the object graph arbitrarily in +a safe way is problematic in a multi-threaded environment; while not +impossible, the benefits are not worth the complexity it adds to the +implementation. + +Thus, when the ``incremental`` key of a configuration dict is present +and is ``True``, the system will completely ignore any ``formatters`` and +``filters`` entries, and process only the ``level`` +settings in the ``handlers`` entries, and the ``level`` and +``propagate`` settings in the ``loggers`` and ``root`` entries. + +Using a value in the configuration dict lets configurations to be sent +over the wire as pickled dicts to a socket listener. Thus, the logging +verbosity of a long-running application can be altered over time with +no need to stop and restart the application. + +.. _logging-config-dict-connections: + +Object connections +"""""""""""""""""" + +The schema describes a set of logging objects - loggers, +handlers, formatters, filters - which are connected to each other in +an object graph. Thus, the schema needs to represent connections +between the objects. For example, say that, once configured, a +particular logger has attached to it a particular handler. For the +purposes of this discussion, we can say that the logger represents the +source, and the handler the destination, of a connection between the +two. Of course in the configured objects this is represented by the +logger holding a reference to the handler. In the configuration dict, +this is done by giving each destination object an id which identifies +it unambiguously, and then using the id in the source object's +configuration to indicate that a connection exists between the source +and the destination object with that id. + +So, for example, consider the following YAML snippet:: + + formatters: + brief: + # configuration for formatter with id 'brief' goes here + precise: + # configuration for formatter with id 'precise' goes here + handlers: + h1: #This is an id + # configuration of handler with id 'h1' goes here + formatter: brief + h2: #This is another id + # configuration of handler with id 'h2' goes here + formatter: precise + loggers: + foo.bar.baz: + # other configuration for logger 'foo.bar.baz' + handlers: [h1, h2] + +(Note: YAML used here because it's a little more readable than the +equivalent Python source form for the dictionary.) + +The ids for loggers are the logger names which would be used +programmatically to obtain a reference to those loggers, e.g. +``foo.bar.baz``. The ids for Formatters and Filters can be any string +value (such as ``brief``, ``precise`` above) and they are transient, +in that they are only meaningful for processing the configuration +dictionary and used to determine connections between objects, and are +not persisted anywhere when the configuration call is complete. + +The above snippet indicates that logger named ``foo.bar.baz`` should +have two handlers attached to it, which are described by the handler +ids ``h1`` and ``h2``. The formatter for ``h1`` is that described by id +``brief``, and the formatter for ``h2`` is that described by id +``precise``. + + +.. _logging-config-dict-userdef: + +User-defined objects +"""""""""""""""""""" + +The schema supports user-defined objects for handlers, filters and +formatters. (Loggers do not need to have different types for +different instances, so there is no support in this configuration +schema for user-defined logger classes.) + +Objects to be configured are described by dictionaries +which detail their configuration. In some places, the logging system +will be able to infer from the context how an object is to be +instantiated, but when a user-defined object is to be instantiated, +the system will not know how to do this. In order to provide complete +flexibility for user-defined object instantiation, the user needs +to provide a 'factory' - a callable which is called with a +configuration dictionary and which returns the instantiated object. +This is signalled by an absolute import path to the factory being +made available under the special key ``'()'``. Here's a concrete +example:: + + formatters: + brief: + format: '%(message)s' + default: + format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s' + datefmt: '%Y-%m-%d %H:%M:%S' + custom: + (): my.package.customFormatterFactory + bar: baz + spam: 99.9 + answer: 42 + +The above YAML snippet defines three formatters. The first, with id +``brief``, is a standard :class:`logging.Formatter` instance with the +specified format string. The second, with id ``default``, has a +longer format and also defines the time format explicitly, and will +result in a :class:`logging.Formatter` initialized with those two format +strings. Shown in Python source form, the ``brief`` and ``default`` +formatters have configuration sub-dictionaries:: + + { + 'format' : '%(message)s' + } + +and:: + + { + 'format' : '%(asctime)s %(levelname)-8s %(name)-15s %(message)s', + 'datefmt' : '%Y-%m-%d %H:%M:%S' + } + +respectively, and as these dictionaries do not contain the special key +``'()'``, the instantiation is inferred from the context: as a result, +standard :class:`logging.Formatter` instances are created. The +configuration sub-dictionary for the third formatter, with id +``custom``, is:: + + { + '()' : 'my.package.customFormatterFactory', + 'bar' : 'baz', + 'spam' : 99.9, + 'answer' : 42 + } + +and this contains the special key ``'()'``, which means that +user-defined instantiation is wanted. In this case, the specified +factory callable will be used. If it is an actual callable it will be +used directly - otherwise, if you specify a string (as in the example) +the actual callable will be located using normal import mechanisms. +The callable will be called with the **remaining** items in the +configuration sub-dictionary as keyword arguments. In the above +example, the formatter with id ``custom`` will be assumed to be +returned by the call:: + + my.package.customFormatterFactory(bar='baz', spam=99.9, answer=42) + +The key ``'()'`` has been used as the special key because it is not a +valid keyword parameter name, and so will not clash with the names of +the keyword arguments used in the call. The ``'()'`` also serves as a +mnemonic that the corresponding value is a callable. + + +.. _logging-config-dict-externalobj: + +Access to external objects +"""""""""""""""""""""""""" + +There are times where a configuration needs to refer to objects +external to the configuration, for example ``sys.stderr``. If the +configuration dict is constructed using Python code, this is +straightforward, but a problem arises when the configuration is +provided via a text file (e.g. JSON, YAML). In a text file, there is +no standard way to distinguish ``sys.stderr`` from the literal string +``'sys.stderr'``. To facilitate this distinction, the configuration +system looks for certain special prefixes in string values and +treat them specially. For example, if the literal string +``'ext://sys.stderr'`` is provided as a value in the configuration, +then the ``ext://`` will be stripped off and the remainder of the +value processed using normal import mechanisms. + +The handling of such prefixes is done in a way analogous to protocol +handling: there is a generic mechanism to look for prefixes which +match the regular expression ``^(?P<prefix>[a-z]+)://(?P<suffix>.*)$`` +whereby, if the ``prefix`` is recognised, the ``suffix`` is processed +in a prefix-dependent manner and the result of the processing replaces +the string value. If the prefix is not recognised, then the string +value will be left as-is. + + +.. _logging-config-dict-internalobj: + +Access to internal objects +"""""""""""""""""""""""""" + +As well as external objects, there is sometimes also a need to refer +to objects in the configuration. This will be done implicitly by the +configuration system for things that it knows about. For example, the +string value ``'DEBUG'`` for a ``level`` in a logger or handler will +automatically be converted to the value ``logging.DEBUG``, and the +``handlers``, ``filters`` and ``formatter`` entries will take an +object id and resolve to the appropriate destination object. + +However, a more generic mechanism is needed for user-defined +objects which are not known to the :mod:`logging` module. For +example, consider :class:`logging.handlers.MemoryHandler`, which takes +a ``target`` argument which is another handler to delegate to. Since +the system already knows about this class, then in the configuration, +the given ``target`` just needs to be the object id of the relevant +target handler, and the system will resolve to the handler from the +id. If, however, a user defines a ``my.package.MyHandler`` which has +an ``alternate`` handler, the configuration system would not know that +the ``alternate`` referred to a handler. To cater for this, a generic +resolution system allows the user to specify:: + + handlers: + file: + # configuration of file handler goes here + + custom: + (): my.package.MyHandler + alternate: cfg://handlers.file + +The literal string ``'cfg://handlers.file'`` will be resolved in an +analogous way to strings with the ``ext://`` prefix, but looking +in the configuration itself rather than the import namespace. The +mechanism allows access by dot or by index, in a similar way to +that provided by ``str.format``. Thus, given the following snippet:: + + handlers: + email: + class: logging.handlers.SMTPHandler + mailhost: localhost + fromaddr: my_app@domain.tld + toaddrs: + - support_team@domain.tld + - dev_team@domain.tld + subject: Houston, we have a problem. + +in the configuration, the string ``'cfg://handlers'`` would resolve to +the dict with key ``handlers``, the string ``'cfg://handlers.email`` +would resolve to the dict with key ``email`` in the ``handlers`` dict, +and so on. The string ``'cfg://handlers.email.toaddrs[1]`` would +resolve to ``'dev_team.domain.tld'`` and the string +``'cfg://handlers.email.toaddrs[0]'`` would resolve to the value +``'support_team@domain.tld'``. The ``subject`` value could be accessed +using either ``'cfg://handlers.email.subject'`` or, equivalently, +``'cfg://handlers.email[subject]'``. The latter form only needs to be +used if the key contains spaces or non-alphanumeric characters. If an +index value consists only of decimal digits, access will be attempted +using the corresponding integer value, falling back to the string +value if needed. + +Given a string ``cfg://handlers.myhandler.mykey.123``, this will +resolve to ``config_dict['handlers']['myhandler']['mykey']['123']``. +If the string is specified as ``cfg://handlers.myhandler.mykey[123]``, +the system will attempt to retrieve the value from +``config_dict['handlers']['myhandler']['mykey'][123]``, and fall back +to ``config_dict['handlers']['myhandler']['mykey']['123']`` if that +fails. + + +.. _logging-import-resolution: + +Import resolution and custom importers +"""""""""""""""""""""""""""""""""""""" + +Import resolution, by default, uses the builtin :func:`__import__` function +to do its importing. You may want to replace this with your own importing +mechanism: if so, you can replace the :attr:`importer` attribute of the +:class:`DictConfigurator` or its superclass, the +:class:`BaseConfigurator` class. However, you need to be +careful because of the way functions are accessed from classes via +descriptors. If you are using a Python callable to do your imports, and you +want to define it at class level rather than instance level, you need to wrap +it with :func:`staticmethod`. For example:: + + from importlib import import_module + from logging.config import BaseConfigurator + + BaseConfigurator.importer = staticmethod(import_module) + +You don't need to wrap with :func:`staticmethod` if you're setting the import +callable on a configurator *instance*. + + +.. _logging-config-fileformat: + +Configuration file format +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The configuration file format understood by :func:`fileConfig` is based on +:mod:`configparser` functionality. The file must contain sections called +``[loggers]``, ``[handlers]`` and ``[formatters]`` which identify by name the +entities of each type which are defined in the file. For each such entity, there +is a separate section which identifies how that entity is configured. Thus, for +a logger named ``log01`` in the ``[loggers]`` section, the relevant +configuration details are held in a section ``[logger_log01]``. Similarly, a +handler called ``hand01`` in the ``[handlers]`` section will have its +configuration held in a section called ``[handler_hand01]``, while a formatter +called ``form01`` in the ``[formatters]`` section will have its configuration +specified in a section called ``[formatter_form01]``. The root logger +configuration must be specified in a section called ``[logger_root]``. + +Examples of these sections in the file are given below. :: + + [loggers] + keys=root,log02,log03,log04,log05,log06,log07 + + [handlers] + keys=hand01,hand02,hand03,hand04,hand05,hand06,hand07,hand08,hand09 + + [formatters] + keys=form01,form02,form03,form04,form05,form06,form07,form08,form09 + +The root logger must specify a level and a list of handlers. An example of a +root logger section is given below. :: + + [logger_root] + level=NOTSET + handlers=hand01 + +The ``level`` entry can be one of ``DEBUG, INFO, WARNING, ERROR, CRITICAL`` or +``NOTSET``. For the root logger only, ``NOTSET`` means that all messages will be +logged. Level values are :func:`eval`\ uated in the context of the ``logging`` +package's namespace. + +The ``handlers`` entry is a comma-separated list of handler names, which must +appear in the ``[handlers]`` section. These names must appear in the +``[handlers]`` section and have corresponding sections in the configuration +file. + +For loggers other than the root logger, some additional information is required. +This is illustrated by the following example. :: + + [logger_parser] + level=DEBUG + handlers=hand01 + propagate=1 + qualname=compiler.parser + +The ``level`` and ``handlers`` entries are interpreted as for the root logger, +except that if a non-root logger's level is specified as ``NOTSET``, the system +consults loggers higher up the hierarchy to determine the effective level of the +logger. The ``propagate`` entry is set to 1 to indicate that messages must +propagate to handlers higher up the logger hierarchy from this logger, or 0 to +indicate that messages are **not** propagated to handlers up the hierarchy. The +``qualname`` entry is the hierarchical channel name of the logger, that is to +say the name used by the application to get the logger. + +Sections which specify handler configuration are exemplified by the following. +:: + + [handler_hand01] + class=StreamHandler + level=NOTSET + formatter=form01 + args=(sys.stdout,) + +The ``class`` entry indicates the handler's class (as determined by :func:`eval` +in the ``logging`` package's namespace). The ``level`` is interpreted as for +loggers, and ``NOTSET`` is taken to mean 'log everything'. + +The ``formatter`` entry indicates the key name of the formatter for this +handler. If blank, a default formatter (``logging._defaultFormatter``) is used. +If a name is specified, it must appear in the ``[formatters]`` section and have +a corresponding section in the configuration file. + +The ``args`` entry, when :func:`eval`\ uated in the context of the ``logging`` +package's namespace, is the list of arguments to the constructor for the handler +class. Refer to the constructors for the relevant handlers, or to the examples +below, to see how typical entries are constructed. :: + + [handler_hand02] + class=FileHandler + level=DEBUG + formatter=form02 + args=('python.log', 'w') + + [handler_hand03] + class=handlers.SocketHandler + level=INFO + formatter=form03 + args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT) + + [handler_hand04] + class=handlers.DatagramHandler + level=WARN + formatter=form04 + args=('localhost', handlers.DEFAULT_UDP_LOGGING_PORT) + + [handler_hand05] + class=handlers.SysLogHandler + level=ERROR + formatter=form05 + args=(('localhost', handlers.SYSLOG_UDP_PORT), handlers.SysLogHandler.LOG_USER) + + [handler_hand06] + class=handlers.NTEventLogHandler + level=CRITICAL + formatter=form06 + args=('Python Application', '', 'Application') + + [handler_hand07] + class=handlers.SMTPHandler + level=WARN + formatter=form07 + args=('localhost', 'from@abc', ['user1@abc', 'user2@xyz'], 'Logger Subject') + + [handler_hand08] + class=handlers.MemoryHandler + level=NOTSET + formatter=form08 + target= + args=(10, ERROR) + + [handler_hand09] + class=handlers.HTTPHandler + level=NOTSET + formatter=form09 + args=('localhost:9022', '/log', 'GET') + +Sections which specify formatter configuration are typified by the following. :: + + [formatter_form01] + format=F1 %(asctime)s %(levelname)s %(message)s + datefmt= + class=logging.Formatter + +The ``format`` entry is the overall format string, and the ``datefmt`` entry is +the :func:`strftime`\ -compatible date/time format string. If empty, the +package substitutes ISO8601 format date/times, which is almost equivalent to +specifying the date format string ``'%Y-%m-%d %H:%M:%S'``. The ISO8601 format +also specifies milliseconds, which are appended to the result of using the above +format string, with a comma separator. An example time in ISO8601 format is +``2003-01-23 00:29:50,411``. + +The ``class`` entry is optional. It indicates the name of the formatter's class +(as a dotted module and class name.) This option is useful for instantiating a +:class:`Formatter` subclass. Subclasses of :class:`Formatter` can present +exception tracebacks in an expanded or condensed format. + +.. seealso:: + + Module :mod:`logging` + API reference for the logging module. + + Module :mod:`logging.handlers` + Useful handlers included with the logging module. + + diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst new file mode 100644 index 0000000000..c4dd438f5b --- /dev/null +++ b/Doc/library/logging.handlers.rst @@ -0,0 +1,871 @@ +:mod:`logging.handlers` --- Logging handlers +============================================ + +.. module:: logging.handlers + :synopsis: Handlers for the logging module. + + +.. moduleauthor:: Vinay Sajip <vinay_sajip@red-dove.com> +.. sectionauthor:: Vinay Sajip <vinay_sajip@red-dove.com> + +.. sidebar:: Important + + This page contains only reference information. For tutorials, + please see + + * :ref:`Basic Tutorial <logging-basic-tutorial>` + * :ref:`Advanced Tutorial <logging-advanced-tutorial>` + * :ref:`Logging Cookbook <logging-cookbook>` + +.. currentmodule:: logging + +The following useful handlers are provided in the package. Note that three of +the handlers (:class:`StreamHandler`, :class:`FileHandler` and +:class:`NullHandler`) are actually defined in the :mod:`logging` module itself, +but have been documented here along with the other handlers. + +.. _stream-handler: + +StreamHandler +^^^^^^^^^^^^^ + +The :class:`StreamHandler` class, located in the core :mod:`logging` package, +sends logging output to streams such as *sys.stdout*, *sys.stderr* or any +file-like object (or, more precisely, any object which supports :meth:`write` +and :meth:`flush` methods). + + +.. class:: StreamHandler(stream=None) + + Returns a new instance of the :class:`StreamHandler` class. If *stream* is + specified, the instance will use it for logging output; otherwise, *sys.stderr* + will be used. + + + .. method:: emit(record) + + If a formatter is specified, it is used to format the record. The record + is then written to the stream with a terminator. If exception information + is present, it is formatted using :func:`traceback.print_exception` and + appended to the stream. + + + .. method:: flush() + + Flushes the stream by calling its :meth:`flush` method. Note that the + :meth:`close` method is inherited from :class:`Handler` and so does + no output, so an explicit :meth:`flush` call may be needed at times. + +.. versionchanged:: 3.2 + The ``StreamHandler`` class now has a ``terminator`` attribute, default + value ``'\n'``, which is used as the terminator when writing a formatted + record to a stream. If you don't want this newline termination, you can + set the handler instance's ``terminator`` attribute to the empty string. + In earlier versions, the terminator was hardcoded as ``'\n'``. + +.. _file-handler: + +FileHandler +^^^^^^^^^^^ + +The :class:`FileHandler` class, located in the core :mod:`logging` package, +sends logging output to a disk file. It inherits the output functionality from +:class:`StreamHandler`. + + +.. class:: FileHandler(filename, mode='a', encoding=None, delay=False) + + Returns a new instance of the :class:`FileHandler` class. The specified file is + opened and used as the stream for logging. If *mode* is not specified, + :const:`'a'` is used. If *encoding* is not *None*, it is used to open the file + with that encoding. If *delay* is true, then file opening is deferred until the + first call to :meth:`emit`. By default, the file grows indefinitely. + + + .. method:: close() + + Closes the file. + + + .. method:: emit(record) + + Outputs the record to the file. + + +.. _null-handler: + +NullHandler +^^^^^^^^^^^ + +.. versionadded:: 3.1 + +The :class:`NullHandler` class, located in the core :mod:`logging` package, +does not do any formatting or output. It is essentially a 'no-op' handler +for use by library developers. + +.. class:: NullHandler() + + Returns a new instance of the :class:`NullHandler` class. + + .. method:: emit(record) + + This method does nothing. + + .. method:: handle(record) + + This method does nothing. + + .. method:: createLock() + + This method returns ``None`` for the lock, since there is no + underlying I/O to which access needs to be serialized. + + +See :ref:`library-config` for more information on how to use +:class:`NullHandler`. + +.. _watched-file-handler: + +WatchedFileHandler +^^^^^^^^^^^^^^^^^^ + +.. currentmodule:: logging.handlers + +The :class:`WatchedFileHandler` class, located in the :mod:`logging.handlers` +module, is a :class:`FileHandler` which watches the file it is logging to. If +the file changes, it is closed and reopened using the file name. + +A file change can happen because of usage of programs such as *newsyslog* and +*logrotate* which perform log file rotation. This handler, intended for use +under Unix/Linux, watches the file to see if it has changed since the last emit. +(A file is deemed to have changed if its device or inode have changed.) If the +file has changed, the old file stream is closed, and the file opened to get a +new stream. + +This handler is not appropriate for use under Windows, because under Windows +open log files cannot be moved or renamed - logging opens the files with +exclusive locks - and so there is no need for such a handler. Furthermore, +*ST_INO* is not supported under Windows; :func:`stat` always returns zero for +this value. + + +.. class:: WatchedFileHandler(filename[,mode[, encoding[, delay]]]) + + Returns a new instance of the :class:`WatchedFileHandler` class. The specified + file is opened and used as the stream for logging. If *mode* is not specified, + :const:`'a'` is used. If *encoding* is not *None*, it is used to open the file + with that encoding. If *delay* is true, then file opening is deferred until the + first call to :meth:`emit`. By default, the file grows indefinitely. + + + .. method:: emit(record) + + Outputs the record to the file, but first checks to see if the file has + changed. If it has, the existing stream is flushed and closed and the + file opened again, before outputting the record to the file. + +.. _rotating-file-handler: + +RotatingFileHandler +^^^^^^^^^^^^^^^^^^^ + +The :class:`RotatingFileHandler` class, located in the :mod:`logging.handlers` +module, supports rotation of disk log files. + + +.. class:: RotatingFileHandler(filename, mode='a', maxBytes=0, backupCount=0, encoding=None, delay=0) + + Returns a new instance of the :class:`RotatingFileHandler` class. The specified + file is opened and used as the stream for logging. If *mode* is not specified, + ``'a'`` is used. If *encoding* is not *None*, it is used to open the file + with that encoding. If *delay* is true, then file opening is deferred until the + first call to :meth:`emit`. By default, the file grows indefinitely. + + You can use the *maxBytes* and *backupCount* values to allow the file to + :dfn:`rollover` at a predetermined size. When the size is about to be exceeded, + the file is closed and a new file is silently opened for output. Rollover occurs + whenever the current log file is nearly *maxBytes* in length; if *maxBytes* is + zero, rollover never occurs. If *backupCount* is non-zero, the system will save + old log files by appending the extensions '.1', '.2' etc., to the filename. For + example, with a *backupCount* of 5 and a base file name of :file:`app.log`, you + would get :file:`app.log`, :file:`app.log.1`, :file:`app.log.2`, up to + :file:`app.log.5`. The file being written to is always :file:`app.log`. When + this file is filled, it is closed and renamed to :file:`app.log.1`, and if files + :file:`app.log.1`, :file:`app.log.2`, etc. exist, then they are renamed to + :file:`app.log.2`, :file:`app.log.3` etc. respectively. + + + .. method:: doRollover() + + Does a rollover, as described above. + + + .. method:: emit(record) + + Outputs the record to the file, catering for rollover as described + previously. + +.. _timed-rotating-file-handler: + +TimedRotatingFileHandler +^^^^^^^^^^^^^^^^^^^^^^^^ + +The :class:`TimedRotatingFileHandler` class, located in the +:mod:`logging.handlers` module, supports rotation of disk log files at certain +timed intervals. + + +.. class:: TimedRotatingFileHandler(filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False) + + Returns a new instance of the :class:`TimedRotatingFileHandler` class. The + specified file is opened and used as the stream for logging. On rotating it also + sets the filename suffix. Rotating happens based on the product of *when* and + *interval*. + + You can use the *when* to specify the type of *interval*. The list of possible + values is below. Note that they are not case sensitive. + + +----------------+-----------------------+ + | Value | Type of interval | + +================+=======================+ + | ``'S'`` | Seconds | + +----------------+-----------------------+ + | ``'M'`` | Minutes | + +----------------+-----------------------+ + | ``'H'`` | Hours | + +----------------+-----------------------+ + | ``'D'`` | Days | + +----------------+-----------------------+ + | ``'W'`` | Week day (0=Monday) | + +----------------+-----------------------+ + | ``'midnight'`` | Roll over at midnight | + +----------------+-----------------------+ + + The system will save old log files by appending extensions to the filename. + The extensions are date-and-time based, using the strftime format + ``%Y-%m-%d_%H-%M-%S`` or a leading portion thereof, depending on the + rollover interval. + + When computing the next rollover time for the first time (when the handler + is created), the last modification time of an existing log file, or else + the current time, is used to compute when the next rotation will occur. + + If the *utc* argument is true, times in UTC will be used; otherwise + local time is used. + + If *backupCount* is nonzero, at most *backupCount* files + will be kept, and if more would be created when rollover occurs, the oldest + one is deleted. The deletion logic uses the interval to determine which + files to delete, so changing the interval may leave old files lying around. + + If *delay* is true, then file opening is deferred until the first call to + :meth:`emit`. + + + .. method:: doRollover() + + Does a rollover, as described above. + + + .. method:: emit(record) + + Outputs the record to the file, catering for rollover as described above. + + +.. _socket-handler: + +SocketHandler +^^^^^^^^^^^^^ + +The :class:`SocketHandler` class, located in the :mod:`logging.handlers` module, +sends logging output to a network socket. The base class uses a TCP socket. + + +.. class:: SocketHandler(host, port) + + Returns a new instance of the :class:`SocketHandler` class intended to + communicate with a remote machine whose address is given by *host* and *port*. + + + .. method:: close() + + Closes the socket. + + + .. method:: emit() + + Pickles the record's attribute dictionary and writes it to the socket in + binary format. If there is an error with the socket, silently drops the + packet. If the connection was previously lost, re-establishes the + connection. To unpickle the record at the receiving end into a + :class:`LogRecord`, use the :func:`makeLogRecord` function. + + + .. method:: handleError() + + Handles an error which has occurred during :meth:`emit`. The most likely + cause is a lost connection. Closes the socket so that we can retry on the + next event. + + + .. method:: makeSocket() + + This is a factory method which allows subclasses to define the precise + type of socket they want. The default implementation creates a TCP socket + (:const:`socket.SOCK_STREAM`). + + + .. method:: makePickle(record) + + Pickles the record's attribute dictionary in binary format with a length + prefix, and returns it ready for transmission across the socket. + + Note that pickles aren't completely secure. If you are concerned about + security, you may want to override this method to implement a more secure + mechanism. For example, you can sign pickles using HMAC and then verify + them on the receiving end, or alternatively you can disable unpickling of + global objects on the receiving end. + + + .. method:: send(packet) + + Send a pickled string *packet* to the socket. This function allows for + partial sends which can happen when the network is busy. + + + .. method:: createSocket() + + Tries to create a socket; on failure, uses an exponential back-off + algorithm. On intial failure, the handler will drop the message it was + trying to send. When subsequent messages are handled by the same + instance, it will not try connecting until some time has passed. The + default parameters are such that the initial delay is one second, and if + after that delay the connection still can't be made, the handler will + double the delay each time up to a maximum of 30 seconds. + + This behaviour is controlled by the following handler attributes: + + * ``retryStart`` (initial delay, defaulting to 1.0 seconds). + * ``retryFactor`` (multiplier, defaulting to 2.0). + * ``retryMax`` (maximum delay, defaulting to 30.0 seconds). + + This means that if the remote listener starts up *after* the handler has + been used, you could lose messages (since the handler won't even attempt + a connection until the delay has elapsed, but just silently drop messages + during the delay period). + + +.. _datagram-handler: + +DatagramHandler +^^^^^^^^^^^^^^^ + +The :class:`DatagramHandler` class, located in the :mod:`logging.handlers` +module, inherits from :class:`SocketHandler` to support sending logging messages +over UDP sockets. + + +.. class:: DatagramHandler(host, port) + + Returns a new instance of the :class:`DatagramHandler` class intended to + communicate with a remote machine whose address is given by *host* and *port*. + + + .. method:: emit() + + Pickles the record's attribute dictionary and writes it to the socket in + binary format. If there is an error with the socket, silently drops the + packet. To unpickle the record at the receiving end into a + :class:`LogRecord`, use the :func:`makeLogRecord` function. + + + .. method:: makeSocket() + + The factory method of :class:`SocketHandler` is here overridden to create + a UDP socket (:const:`socket.SOCK_DGRAM`). + + + .. method:: send(s) + + Send a pickled string to a socket. + + +.. _syslog-handler: + +SysLogHandler +^^^^^^^^^^^^^ + +The :class:`SysLogHandler` class, located in the :mod:`logging.handlers` module, +supports sending logging messages to a remote or local Unix syslog. + + +.. class:: SysLogHandler(address=('localhost', SYSLOG_UDP_PORT), facility=LOG_USER, socktype=socket.SOCK_DGRAM) + + Returns a new instance of the :class:`SysLogHandler` class intended to + communicate with a remote Unix machine whose address is given by *address* in + the form of a ``(host, port)`` tuple. If *address* is not specified, + ``('localhost', 514)`` is used. The address is used to open a socket. An + alternative to providing a ``(host, port)`` tuple is providing an address as a + string, for example '/dev/log'. In this case, a Unix domain socket is used to + send the message to the syslog. If *facility* is not specified, + :const:`LOG_USER` is used. The type of socket opened depends on the + *socktype* argument, which defaults to :const:`socket.SOCK_DGRAM` and thus + opens a UDP socket. To open a TCP socket (for use with the newer syslog + daemons such as rsyslog), specify a value of :const:`socket.SOCK_STREAM`. + + Note that if your server is not listening on UDP port 514, + :class:`SysLogHandler` may appear not to work. In that case, check what + address you should be using for a domain socket - it's system dependent. + For example, on Linux it's usually '/dev/log' but on OS/X it's + '/var/run/syslog'. You'll need to check your platform and use the + appropriate address (you may need to do this check at runtime if your + application needs to run on several platforms). On Windows, you pretty + much have to use the UDP option. + + .. versionchanged:: 3.2 + *socktype* was added. + + + .. method:: close() + + Closes the socket to the remote host. + + + .. method:: emit(record) + + The record is formatted, and then sent to the syslog server. If exception + information is present, it is *not* sent to the server. + + .. versionchanged:: 3.2.1 + (See: :issue:`12168`.) In earlier versions, the message sent to the + syslog daemons was always terminated with a NUL byte, because early + versions of these daemons expected a NUL terminated message - even + though it's not in the relevant specification (RF 5424). More recent + versions of these daemons don't expect the NUL byte but strip it off + if it's there, and even more recent daemons (which adhere more closely + to RFC 5424) pass the NUL byte on as part of the message. + + To enable easier handling of syslog messages in the face of all these + differing daemon behaviours, the appending of the NUL byte has been + made configurable, through the use of a class-level attribute, + ``append_nul``. This defaults to ``True`` (preserving the existing + behaviour) but can be set to ``False`` on a ``SysLogHandler`` instance + in order for that instance to *not* append the NUL terminator. + + .. method:: encodePriority(facility, priority) + + Encodes the facility and priority into an integer. You can pass in strings + or integers - if strings are passed, internal mapping dictionaries are + used to convert them to integers. + + The symbolic ``LOG_`` values are defined in :class:`SysLogHandler` and + mirror the values defined in the ``sys/syslog.h`` header file. + + **Priorities** + + +--------------------------+---------------+ + | Name (string) | Symbolic value| + +==========================+===============+ + | ``alert`` | LOG_ALERT | + +--------------------------+---------------+ + | ``crit`` or ``critical`` | LOG_CRIT | + +--------------------------+---------------+ + | ``debug`` | LOG_DEBUG | + +--------------------------+---------------+ + | ``emerg`` or ``panic`` | LOG_EMERG | + +--------------------------+---------------+ + | ``err`` or ``error`` | LOG_ERR | + +--------------------------+---------------+ + | ``info`` | LOG_INFO | + +--------------------------+---------------+ + | ``notice`` | LOG_NOTICE | + +--------------------------+---------------+ + | ``warn`` or ``warning`` | LOG_WARNING | + +--------------------------+---------------+ + + **Facilities** + + +---------------+---------------+ + | Name (string) | Symbolic value| + +===============+===============+ + | ``auth`` | LOG_AUTH | + +---------------+---------------+ + | ``authpriv`` | LOG_AUTHPRIV | + +---------------+---------------+ + | ``cron`` | LOG_CRON | + +---------------+---------------+ + | ``daemon`` | LOG_DAEMON | + +---------------+---------------+ + | ``ftp`` | LOG_FTP | + +---------------+---------------+ + | ``kern`` | LOG_KERN | + +---------------+---------------+ + | ``lpr`` | LOG_LPR | + +---------------+---------------+ + | ``mail`` | LOG_MAIL | + +---------------+---------------+ + | ``news`` | LOG_NEWS | + +---------------+---------------+ + | ``syslog`` | LOG_SYSLOG | + +---------------+---------------+ + | ``user`` | LOG_USER | + +---------------+---------------+ + | ``uucp`` | LOG_UUCP | + +---------------+---------------+ + | ``local0`` | LOG_LOCAL0 | + +---------------+---------------+ + | ``local1`` | LOG_LOCAL1 | + +---------------+---------------+ + | ``local2`` | LOG_LOCAL2 | + +---------------+---------------+ + | ``local3`` | LOG_LOCAL3 | + +---------------+---------------+ + | ``local4`` | LOG_LOCAL4 | + +---------------+---------------+ + | ``local5`` | LOG_LOCAL5 | + +---------------+---------------+ + | ``local6`` | LOG_LOCAL6 | + +---------------+---------------+ + | ``local7`` | LOG_LOCAL7 | + +---------------+---------------+ + + .. method:: mapPriority(levelname) + + Maps a logging level name to a syslog priority name. + You may need to override this if you are using custom levels, or + if the default algorithm is not suitable for your needs. The + default algorithm maps ``DEBUG``, ``INFO``, ``WARNING``, ``ERROR`` and + ``CRITICAL`` to the equivalent syslog names, and all other level + names to 'warning'. + +.. _nt-eventlog-handler: + +NTEventLogHandler +^^^^^^^^^^^^^^^^^ + +The :class:`NTEventLogHandler` class, located in the :mod:`logging.handlers` +module, supports sending logging messages to a local Windows NT, Windows 2000 or +Windows XP event log. Before you can use it, you need Mark Hammond's Win32 +extensions for Python installed. + + +.. class:: NTEventLogHandler(appname, dllname=None, logtype='Application') + + Returns a new instance of the :class:`NTEventLogHandler` class. The *appname* is + used to define the application name as it appears in the event log. An + appropriate registry entry is created using this name. The *dllname* should give + the fully qualified pathname of a .dll or .exe which contains message + definitions to hold in the log (if not specified, ``'win32service.pyd'`` is used + - this is installed with the Win32 extensions and contains some basic + placeholder message definitions. Note that use of these placeholders will make + your event logs big, as the entire message source is held in the log. If you + want slimmer logs, you have to pass in the name of your own .dll or .exe which + contains the message definitions you want to use in the event log). The + *logtype* is one of ``'Application'``, ``'System'`` or ``'Security'``, and + defaults to ``'Application'``. + + + .. method:: close() + + At this point, you can remove the application name from the registry as a + source of event log entries. However, if you do this, you will not be able + to see the events as you intended in the Event Log Viewer - it needs to be + able to access the registry to get the .dll name. The current version does + not do this. + + + .. method:: emit(record) + + Determines the message ID, event category and event type, and then logs + the message in the NT event log. + + + .. method:: getEventCategory(record) + + Returns the event category for the record. Override this if you want to + specify your own categories. This version returns 0. + + + .. method:: getEventType(record) + + Returns the event type for the record. Override this if you want to + specify your own types. This version does a mapping using the handler's + typemap attribute, which is set up in :meth:`__init__` to a dictionary + which contains mappings for :const:`DEBUG`, :const:`INFO`, + :const:`WARNING`, :const:`ERROR` and :const:`CRITICAL`. If you are using + your own levels, you will either need to override this method or place a + suitable dictionary in the handler's *typemap* attribute. + + + .. method:: getMessageID(record) + + Returns the message ID for the record. If you are using your own messages, + you could do this by having the *msg* passed to the logger being an ID + rather than a format string. Then, in here, you could use a dictionary + lookup to get the message ID. This version returns 1, which is the base + message ID in :file:`win32service.pyd`. + +.. _smtp-handler: + +SMTPHandler +^^^^^^^^^^^ + +The :class:`SMTPHandler` class, located in the :mod:`logging.handlers` module, +supports sending logging messages to an email address via SMTP. + + +.. class:: SMTPHandler(mailhost, fromaddr, toaddrs, subject, credentials=None, secure=None) + + Returns a new instance of the :class:`SMTPHandler` class. The instance is + initialized with the from and to addresses and subject line of the email. The + *toaddrs* should be a list of strings. To specify a non-standard SMTP port, use + the (host, port) tuple format for the *mailhost* argument. If you use a string, + the standard SMTP port is used. If your SMTP server requires authentication, you + can specify a (username, password) tuple for the *credentials* argument. + + To specify the use of a secure protocol (TLS), pass in a tuple to the + *secure* argument. This will only be used when authentication credentials are + supplied. The tuple should be either an empty tuple, or a single-value tuple + with the name of a keyfile, or a 2-value tuple with the names of the keyfile + and certificate file. (This tuple is passed to the + :meth:`smtplib.SMTP.starttls` method.) + + .. method:: emit(record) + + Formats the record and sends it to the specified addressees. + + + .. method:: getSubject(record) + + If you want to specify a subject line which is record-dependent, override + this method. + +.. _memory-handler: + +MemoryHandler +^^^^^^^^^^^^^ + +The :class:`MemoryHandler` class, located in the :mod:`logging.handlers` module, +supports buffering of logging records in memory, periodically flushing them to a +:dfn:`target` handler. Flushing occurs whenever the buffer is full, or when an +event of a certain severity or greater is seen. + +:class:`MemoryHandler` is a subclass of the more general +:class:`BufferingHandler`, which is an abstract class. This buffers logging +records in memory. Whenever each record is added to the buffer, a check is made +by calling :meth:`shouldFlush` to see if the buffer should be flushed. If it +should, then :meth:`flush` is expected to do the needful. + + +.. class:: BufferingHandler(capacity) + + Initializes the handler with a buffer of the specified capacity. + + + .. method:: emit(record) + + Appends the record to the buffer. If :meth:`shouldFlush` returns true, + calls :meth:`flush` to process the buffer. + + + .. method:: flush() + + You can override this to implement custom flushing behavior. This version + just zaps the buffer to empty. + + + .. method:: shouldFlush(record) + + Returns true if the buffer is up to capacity. This method can be + overridden to implement custom flushing strategies. + + +.. class:: MemoryHandler(capacity, flushLevel=ERROR, target=None) + + Returns a new instance of the :class:`MemoryHandler` class. The instance is + initialized with a buffer size of *capacity*. If *flushLevel* is not specified, + :const:`ERROR` is used. If no *target* is specified, the target will need to be + set using :meth:`setTarget` before this handler does anything useful. + + + .. method:: close() + + Calls :meth:`flush`, sets the target to :const:`None` and clears the + buffer. + + + .. method:: flush() + + For a :class:`MemoryHandler`, flushing means just sending the buffered + records to the target, if there is one. The buffer is also cleared when + this happens. Override if you want different behavior. + + + .. method:: setTarget(target) + + Sets the target handler for this handler. + + + .. method:: shouldFlush(record) + + Checks for buffer full or a record at the *flushLevel* or higher. + + +.. _http-handler: + +HTTPHandler +^^^^^^^^^^^ + +The :class:`HTTPHandler` class, located in the :mod:`logging.handlers` module, +supports sending logging messages to a Web server, using either ``GET`` or +``POST`` semantics. + + +.. class:: HTTPHandler(host, url, method='GET', secure=False, credentials=None) + + Returns a new instance of the :class:`HTTPHandler` class. The *host* can be + of the form ``host:port``, should you need to use a specific port number. + If no *method* is specified, ``GET`` is used. If *secure* is True, an HTTPS + connection will be used. If *credentials* is specified, it should be a + 2-tuple consisting of userid and password, which will be placed in an HTTP + 'Authorization' header using Basic authentication. If you specify + credentials, you should also specify secure=True so that your userid and + password are not passed in cleartext across the wire. + + + .. method:: emit(record) + + Sends the record to the Web server as a percent-encoded dictionary. + + +.. _queue-handler: + + +QueueHandler +^^^^^^^^^^^^ + +.. versionadded:: 3.2 + +The :class:`QueueHandler` class, located in the :mod:`logging.handlers` module, +supports sending logging messages to a queue, such as those implemented in the +:mod:`queue` or :mod:`multiprocessing` modules. + +Along with the :class:`QueueListener` class, :class:`QueueHandler` can be used +to let handlers do their work on a separate thread from the one which does the +logging. This is important in Web applications and also other service +applications where threads servicing clients need to respond as quickly as +possible, while any potentially slow operations (such as sending an email via +:class:`SMTPHandler`) are done on a separate thread. + +.. class:: QueueHandler(queue) + + Returns a new instance of the :class:`QueueHandler` class. The instance is + initialized with the queue to send messages to. The queue can be any queue- + like object; it's used as-is by the :meth:`enqueue` method, which needs + to know how to send messages to it. + + + .. method:: emit(record) + + Enqueues the result of preparing the LogRecord. + + .. method:: prepare(record) + + Prepares a record for queuing. The object returned by this + method is enqueued. + + The base implementation formats the record to merge the message + and arguments, and removes unpickleable items from the record + in-place. + + You might want to override this method if you want to convert + the record to a dict or JSON string, or send a modified copy + of the record while leaving the original intact. + + .. method:: enqueue(record) + + Enqueues the record on the queue using ``put_nowait()``; you may + want to override this if you want to use blocking behaviour, or a + timeout, or a customised queue implementation. + + + +.. _queue-listener: + +QueueListener +^^^^^^^^^^^^^ + +.. versionadded:: 3.2 + +The :class:`QueueListener` class, located in the :mod:`logging.handlers` +module, supports receiving logging messages from a queue, such as those +implemented in the :mod:`queue` or :mod:`multiprocessing` modules. The +messages are received from a queue in an internal thread and passed, on +the same thread, to one or more handlers for processing. While +:class:`QueueListener` is not itself a handler, it is documented here +because it works hand-in-hand with :class:`QueueHandler`. + +Along with the :class:`QueueHandler` class, :class:`QueueListener` can be used +to let handlers do their work on a separate thread from the one which does the +logging. This is important in Web applications and also other service +applications where threads servicing clients need to respond as quickly as +possible, while any potentially slow operations (such as sending an email via +:class:`SMTPHandler`) are done on a separate thread. + +.. class:: QueueListener(queue, *handlers) + + Returns a new instance of the :class:`QueueListener` class. The instance is + initialized with the queue to send messages to and a list of handlers which + will handle entries placed on the queue. The queue can be any queue- + like object; it's passed as-is to the :meth:`dequeue` method, which needs + to know how to get messages from it. + + .. method:: dequeue(block) + + Dequeues a record and return it, optionally blocking. + + The base implementation uses ``get()``. You may want to override this + method if you want to use timeouts or work with custom queue + implementations. + + .. method:: prepare(record) + + Prepare a record for handling. + + This implementation just returns the passed-in record. You may want to + override this method if you need to do any custom marshalling or + manipulation of the record before passing it to the handlers. + + .. method:: handle(record) + + Handle a record. + + This just loops through the handlers offering them the record + to handle. The actual object passed to the handlers is that which + is returned from :meth:`prepare`. + + .. method:: start() + + Starts the listener. + + This starts up a background thread to monitor the queue for + LogRecords to process. + + .. method:: stop() + + Stops the listener. + + This asks the thread to terminate, and then waits for it to do so. + Note that if you don't call this before your application exits, there + may be some records still left on the queue, which won't be processed. + + +.. seealso:: + + Module :mod:`logging` + API reference for the logging module. + + Module :mod:`logging.config` + Configuration API for the logging module. + + diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 46b643f8e9..c429c85985 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -2,7 +2,7 @@ ============================================== .. module:: logging - :synopsis: Flexible error logging system for applications. + :synopsis: Flexible event logging system for applications. .. moduleauthor:: Vinay Sajip <vinay_sajip@red-dove.com> @@ -11,833 +11,38 @@ .. index:: pair: Errors; logging -This module defines functions and classes which implement a flexible error -logging system for applications. +.. sidebar:: Important -Logging is performed by calling methods on instances of the :class:`Logger` -class (hereafter called :dfn:`loggers`). Each instance has a name, and they are -conceptually arranged in a namespace hierarchy using dots (periods) as -separators. For example, a logger named "scan" is the parent of loggers -"scan.text", "scan.html" and "scan.pdf". Logger names can be anything you want, -and indicate the area of an application in which a logged message originates. + This page contains the API reference information. For tutorial + information and discussion of more advanced topics, see -Logged messages also have levels of importance associated with them. The default -levels provided are :const:`DEBUG`, :const:`INFO`, :const:`WARNING`, -:const:`ERROR` and :const:`CRITICAL`. As a convenience, you indicate the -importance of a logged message by calling an appropriate method of -:class:`Logger`. The methods are :meth:`debug`, :meth:`info`, :meth:`warning`, -:meth:`error` and :meth:`critical`, which mirror the default levels. You are not -constrained to use these levels: you can specify your own and use a more general -:class:`Logger` method, :meth:`log`, which takes an explicit level argument. + * :ref:`Basic Tutorial <logging-basic-tutorial>` + * :ref:`Advanced Tutorial <logging-advanced-tutorial>` + * :ref:`Logging Cookbook <logging-cookbook>` -Logging tutorial ----------------- +This module defines functions and classes which implement a flexible event +logging system for applications and libraries. The key benefit of having the logging API provided by a standard library module is that all Python modules can participate in logging, so your application log -can include messages from third-party modules. +can include your own messages integrated with messages from third-party +modules. -It is, of course, possible to log messages with different verbosity levels or to -different destinations. Support for writing log messages to files, HTTP -GET/POST locations, email via SMTP, generic sockets, or OS-specific logging -mechanisms are all supported by the standard module. You can also create your -own log destination class if you have special requirements not met by any of the -built-in classes. +The module provides a lot of functionality and flexibility. If you are +unfamiliar with logging, the best way to get to grips with it is to see the +tutorials (see the links on the right). -Simple examples -^^^^^^^^^^^^^^^ - -.. sectionauthor:: Doug Hellmann -.. (see <http://blog.doughellmann.com/2007/05/pymotw-logging.html>) +The basic classes defined by the module, together with their functions, are +listed below. -Most applications are probably going to want to log to a file, so let's start -with that case. Using the :func:`basicConfig` function, we can set up the -default handler so that debug messages are written to a file (in the example, -we assume that you have the appropriate permissions to create a file called -*example.log* in the current directory):: - - import logging - LOG_FILENAME = 'example.log' - logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG) - - logging.debug('This message should go to the log file') - -And now if we open the file and look at what we have, we should find the log -message:: - - DEBUG:root:This message should go to the log file - -If you run the script repeatedly, the additional log messages are appended to -the file. To create a new file each time, you can pass a *filemode* argument to -:func:`basicConfig` with a value of ``'w'``. Rather than managing the file size -yourself, though, it is simpler to use a :class:`RotatingFileHandler`:: - - import glob - import logging - import logging.handlers - - LOG_FILENAME = 'logging_rotatingfile_example.out' - - # Set up a specific logger with our desired output level - my_logger = logging.getLogger('MyLogger') - my_logger.setLevel(logging.DEBUG) - - # Add the log message handler to the logger - handler = logging.handlers.RotatingFileHandler( - LOG_FILENAME, maxBytes=20, backupCount=5) +* Loggers expose the interface that application code directly uses. +* Handlers send the log records (created by loggers) to the appropriate + destination. +* Filters provide a finer grained facility for determining which log records + to output. +* Formatters specify the layout of log records in the final output. - my_logger.addHandler(handler) - - # Log some messages - for i in range(20): - my_logger.debug('i = %d' % i) - - # See what files are created - logfiles = glob.glob('%s*' % LOG_FILENAME) - - for filename in logfiles: - print(filename) - -The result should be 6 separate files, each with part of the log history for the -application:: - - logging_rotatingfile_example.out - logging_rotatingfile_example.out.1 - logging_rotatingfile_example.out.2 - logging_rotatingfile_example.out.3 - logging_rotatingfile_example.out.4 - logging_rotatingfile_example.out.5 - -The most current file is always :file:`logging_rotatingfile_example.out`, -and each time it reaches the size limit it is renamed with the suffix -``.1``. Each of the existing backup files is renamed to increment the suffix -(``.1`` becomes ``.2``, etc.) and the ``.6`` file is erased. - -Obviously this example sets the log length much much too small as an extreme -example. You would want to set *maxBytes* to an appropriate value. - -Another useful feature of the logging API is the ability to produce different -messages at different log levels. This allows you to instrument your code with -debug messages, for example, but turning the log level down so that those debug -messages are not written for your production system. The default levels are -``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG`` and ``NOTSET``. - -The logger, handler, and log message call each specify a level. The log message -is only emitted if the handler and logger are configured to emit messages of -that level or lower. For example, if a message is ``CRITICAL``, and the logger -is set to ``ERROR``, the message is emitted. If a message is a ``WARNING``, and -the logger is set to produce only ``ERROR``\s, the message is not emitted:: - - import logging - import sys - - LEVELS = {'debug': logging.DEBUG, - 'info': logging.INFO, - 'warning': logging.WARNING, - 'error': logging.ERROR, - 'critical': logging.CRITICAL} - - if len(sys.argv) > 1: - level_name = sys.argv[1] - level = LEVELS.get(level_name, logging.NOTSET) - logging.basicConfig(level=level) - - logging.debug('This is a debug message') - logging.info('This is an info message') - logging.warning('This is a warning message') - logging.error('This is an error message') - logging.critical('This is a critical error message') - -Run the script with an argument like 'debug' or 'warning' to see which messages -show up at different levels:: - - $ python logging_level_example.py debug - DEBUG:root:This is a debug message - INFO:root:This is an info message - WARNING:root:This is a warning message - ERROR:root:This is an error message - CRITICAL:root:This is a critical error message - - $ python logging_level_example.py info - INFO:root:This is an info message - WARNING:root:This is a warning message - ERROR:root:This is an error message - CRITICAL:root:This is a critical error message - -You will notice that these log messages all have ``root`` embedded in them. The -logging module supports a hierarchy of loggers with different names. An easy -way to tell where a specific log message comes from is to use a separate logger -object for each of your modules. Each new logger "inherits" the configuration -of its parent, and log messages sent to a logger include the name of that -logger. Optionally, each logger can be configured differently, so that messages -from different modules are handled in different ways. Let's look at a simple -example of how to log from different modules so it is easy to trace the source -of the message:: - - import logging - - logging.basicConfig(level=logging.WARNING) - - logger1 = logging.getLogger('package1.module1') - logger2 = logging.getLogger('package2.module2') - - logger1.warning('This message comes from one module') - logger2.warning('And this message comes from another module') - -And the output:: - - $ python logging_modules_example.py - WARNING:package1.module1:This message comes from one module - WARNING:package2.module2:And this message comes from another module - -There are many more options for configuring logging, including different log -message formatting options, having messages delivered to multiple destinations, -and changing the configuration of a long-running application on the fly using a -socket interface. All of these options are covered in depth in the library -module documentation. - -Loggers -^^^^^^^ - -The logging library takes a modular approach and offers the several categories -of components: loggers, handlers, filters, and formatters. Loggers expose the -interface that application code directly uses. Handlers send the log records to -the appropriate destination. Filters provide a finer grained facility for -determining which log records to send on to a handler. Formatters specify the -layout of the resultant log record. - -:class:`Logger` objects have a threefold job. First, they expose several -methods to application code so that applications can log messages at runtime. -Second, logger objects determine which log messages to act upon based upon -severity (the default filtering facility) or filter objects. Third, logger -objects pass along relevant log messages to all interested log handlers. - -The most widely used methods on logger objects fall into two categories: -configuration and message sending. - -* :meth:`Logger.setLevel` specifies the lowest-severity log message a logger - will handle, where debug is the lowest built-in severity level and critical is - the highest built-in severity. For example, if the severity level is info, - the logger will handle only info, warning, error, and critical messages and - will ignore debug messages. - -* :meth:`Logger.addFilter` and :meth:`Logger.removeFilter` add and remove filter - objects from the logger object. This tutorial does not address filters. - -With the logger object configured, the following methods create log messages: - -* :meth:`Logger.debug`, :meth:`Logger.info`, :meth:`Logger.warning`, - :meth:`Logger.error`, and :meth:`Logger.critical` all create log records with - a message and a level that corresponds to their respective method names. The - message is actually a format string, which may contain the standard string - substitution syntax of :const:`%s`, :const:`%d`, :const:`%f`, and so on. The - rest of their arguments is a list of objects that correspond with the - substitution fields in the message. With regard to :const:`**kwargs`, the - logging methods care only about a keyword of :const:`exc_info` and use it to - determine whether to log exception information. - -* :meth:`Logger.exception` creates a log message similar to - :meth:`Logger.error`. The difference is that :meth:`Logger.exception` dumps a - stack trace along with it. Call this method only from an exception handler. - -* :meth:`Logger.log` takes a log level as an explicit argument. This is a - little more verbose for logging messages than using the log level convenience - methods listed above, but this is how to log at custom log levels. - -:func:`getLogger` returns a reference to a logger instance with the specified -name if it is provided, or ``root`` if not. The names are period-separated -hierarchical structures. Multiple calls to :func:`getLogger` with the same name -will return a reference to the same logger object. Loggers that are further -down in the hierarchical list are children of loggers higher up in the list. -For example, given a logger with a name of ``foo``, loggers with names of -``foo.bar``, ``foo.bar.baz``, and ``foo.bam`` are all descendants of ``foo``. -Child loggers propagate messages up to the handlers associated with their -ancestor loggers. Because of this, it is unnecessary to define and configure -handlers for all the loggers an application uses. It is sufficient to -configure handlers for a top-level logger and create child loggers as needed. - - -Handlers -^^^^^^^^ - -:class:`Handler` objects are responsible for dispatching the appropriate log -messages (based on the log messages' severity) to the handler's specified -destination. Logger objects can add zero or more handler objects to themselves -with an :func:`addHandler` method. As an example scenario, an application may -want to send all log messages to a log file, all log messages of error or higher -to stdout, and all messages of critical to an email address. This scenario -requires three individual handlers where each handler is responsible for sending -messages of a specific severity to a specific location. - -The standard library includes quite a few handler types; this tutorial uses only -:class:`StreamHandler` and :class:`FileHandler` in its examples. - -There are very few methods in a handler for application developers to concern -themselves with. The only handler methods that seem relevant for application -developers who are using the built-in handler objects (that is, not creating -custom handlers) are the following configuration methods: - -* The :meth:`Handler.setLevel` method, just as in logger objects, specifies the - lowest severity that will be dispatched to the appropriate destination. Why - are there two :func:`setLevel` methods? The level set in the logger - determines which severity of messages it will pass to its handlers. The level - set in each handler determines which messages that handler will send on. - -* :func:`setFormatter` selects a Formatter object for this handler to use. - -* :func:`addFilter` and :func:`removeFilter` respectively configure and - deconfigure filter objects on handlers. - -Application code should not directly instantiate and use instances of -:class:`Handler`. Instead, the :class:`Handler` class is a base class that -defines the interface that all handlers should have and establishes some -default behavior that child classes can use (or override). - - -Formatters -^^^^^^^^^^ - -Formatter objects configure the final order, structure, and contents of the log -message. Unlike the base :class:`logging.Handler` class, application code may -instantiate formatter classes, although you could likely subclass the formatter -if your application needs special behavior. The constructor takes two optional -arguments: a message format string and a date format string. If there is no -message format string, the default is to use the raw message. If there is no -date format string, the default date format is:: - - %Y-%m-%d %H:%M:%S - -with the milliseconds tacked on at the end. - -The message format string uses ``%(<dictionary key>)s`` styled string -substitution; the possible keys are documented in :ref:`formatter-objects`. - -The following message format string will log the time in a human-readable -format, the severity of the message, and the contents of the message, in that -order:: - - "%(asctime)s - %(levelname)s - %(message)s" - -Formatters use a user-configurable function to convert the creation time of a -record to a tuple. By default, :func:`time.localtime` is used; to change this -for a particular formatter instance, set the ``converter`` attribute of the -instance to a function with the same signature as :func:`time.localtime` or -:func:`time.gmtime`. To change it for all formatters, for example if you want -all logging times to be shown in GMT, set the ``converter`` attribute in the -Formatter class (to ``time.gmtime`` for GMT display). - - -Configuring Logging -^^^^^^^^^^^^^^^^^^^ - -Programmers can configure logging either by creating loggers, handlers, and -formatters explicitly in a main module with the configuration methods listed -above (using Python code), or by creating a logging config file. The following -code is an example of configuring a very simple logger, a console handler, and a -simple formatter in a Python module:: - - import logging - - # create logger - logger = logging.getLogger("simple_example") - logger.setLevel(logging.DEBUG) - # create console handler and set level to debug - ch = logging.StreamHandler() - ch.setLevel(logging.DEBUG) - # create formatter - formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") - # add formatter to ch - ch.setFormatter(formatter) - # add ch to logger - logger.addHandler(ch) - - # "application" code - logger.debug("debug message") - logger.info("info message") - logger.warn("warn message") - logger.error("error message") - logger.critical("critical message") - -Running this module from the command line produces the following output:: - - $ python simple_logging_module.py - 2005-03-19 15:10:26,618 - simple_example - DEBUG - debug message - 2005-03-19 15:10:26,620 - simple_example - INFO - info message - 2005-03-19 15:10:26,695 - simple_example - WARNING - warn message - 2005-03-19 15:10:26,697 - simple_example - ERROR - error message - 2005-03-19 15:10:26,773 - simple_example - CRITICAL - critical message - -The following Python module creates a logger, handler, and formatter nearly -identical to those in the example listed above, with the only difference being -the names of the objects:: - - import logging - import logging.config - - logging.config.fileConfig("logging.conf") - - # create logger - logger = logging.getLogger("simpleExample") - - # "application" code - logger.debug("debug message") - logger.info("info message") - logger.warn("warn message") - logger.error("error message") - logger.critical("critical message") - -Here is the logging.conf file:: - - [loggers] - keys=root,simpleExample - - [handlers] - keys=consoleHandler - - [formatters] - keys=simpleFormatter - - [logger_root] - level=DEBUG - handlers=consoleHandler - - [logger_simpleExample] - level=DEBUG - handlers=consoleHandler - qualname=simpleExample - propagate=0 - - [handler_consoleHandler] - class=StreamHandler - level=DEBUG - formatter=simpleFormatter - args=(sys.stdout,) - - [formatter_simpleFormatter] - format=%(asctime)s - %(name)s - %(levelname)s - %(message)s - datefmt= - -The output is nearly identical to that of the non-config-file-based example:: - - $ python simple_logging_config.py - 2005-03-19 15:38:55,977 - simpleExample - DEBUG - debug message - 2005-03-19 15:38:55,979 - simpleExample - INFO - info message - 2005-03-19 15:38:56,054 - simpleExample - WARNING - warn message - 2005-03-19 15:38:56,055 - simpleExample - ERROR - error message - 2005-03-19 15:38:56,130 - simpleExample - CRITICAL - critical message - -You can see that the config file approach has a few advantages over the Python -code approach, mainly separation of configuration and code and the ability of -noncoders to easily modify the logging properties. - -.. _library-config: - -Configuring Logging for a Library -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When developing a library which uses logging, some consideration needs to be -given to its configuration. If the using application does not use logging, and -library code makes logging calls, then a one-off message "No handlers could be -found for logger X.Y.Z" is printed to the console. This message is intended -to catch mistakes in logging configuration, but will confuse an application -developer who is not aware of logging by the library. - -In addition to documenting how a library uses logging, a good way to configure -library logging so that it does not cause a spurious message is to add a -handler which does nothing. This avoids the message being printed, since a -handler will be found: it just doesn't produce any output. If the library user -configures logging for application use, presumably that configuration will add -some handlers, and if levels are suitably configured then logging calls made -in library code will send output to those handlers, as normal. - -A do-nothing handler can be simply defined as follows:: - - import logging - - class NullHandler(logging.Handler): - def emit(self, record): - pass - -An instance of this handler should be added to the top-level logger of the -logging namespace used by the library. If all logging by a library *foo* is -done using loggers with names matching "foo.x.y", then the code:: - - import logging - - h = NullHandler() - logging.getLogger("foo").addHandler(h) - -should have the desired effect. If an organisation produces a number of -libraries, then the logger name specified can be "orgname.foo" rather than -just "foo". - -.. versionadded:: 3.1 - The :class:`NullHandler` class. - - -Logging Levels --------------- - -The numeric values of logging levels are given in the following table. These are -primarily of interest if you want to define your own levels, and need them to -have specific values relative to the predefined levels. If you define a level -with the same numeric value, it overwrites the predefined value; the predefined -name is lost. - -+--------------+---------------+ -| Level | Numeric value | -+==============+===============+ -| ``CRITICAL`` | 50 | -+--------------+---------------+ -| ``ERROR`` | 40 | -+--------------+---------------+ -| ``WARNING`` | 30 | -+--------------+---------------+ -| ``INFO`` | 20 | -+--------------+---------------+ -| ``DEBUG`` | 10 | -+--------------+---------------+ -| ``NOTSET`` | 0 | -+--------------+---------------+ - -Levels can also be associated with loggers, being set either by the developer or -through loading a saved logging configuration. When a logging method is called -on a logger, the logger compares its own level with the level associated with -the method call. If the logger's level is higher than the method call's, no -logging message is actually generated. This is the basic mechanism controlling -the verbosity of logging output. - -Logging messages are encoded as instances of the :class:`LogRecord` class. When -a logger decides to actually log an event, a :class:`LogRecord` instance is -created from the logging message. - -Logging messages are subjected to a dispatch mechanism through the use of -:dfn:`handlers`, which are instances of subclasses of the :class:`Handler` -class. Handlers are responsible for ensuring that a logged message (in the form -of a :class:`LogRecord`) ends up in a particular location (or set of locations) -which is useful for the target audience for that message (such as end users, -support desk staff, system administrators, developers). Handlers are passed -:class:`LogRecord` instances intended for particular destinations. Each logger -can have zero, one or more handlers associated with it (via the -:meth:`addHandler` method of :class:`Logger`). In addition to any handlers -directly associated with a logger, *all handlers associated with all ancestors -of the logger* are called to dispatch the message (unless the *propagate* flag -for a logger is set to a false value, at which point the passing to ancestor -handlers stops). - -Just as for loggers, handlers can have levels associated with them. A handler's -level acts as a filter in the same way as a logger's level does. If a handler -decides to actually dispatch an event, the :meth:`emit` method is used to send -the message to its destination. Most user-defined subclasses of :class:`Handler` -will need to override this :meth:`emit`. - -Useful Handlers ---------------- - -In addition to the base :class:`Handler` class, many useful subclasses are -provided: - -#. :class:`StreamHandler` instances send messages to streams (file-like - objects). - -#. :class:`FileHandler` instances send messages to disk files. - -.. module:: logging.handlers - -#. :class:`BaseRotatingHandler` is the base class for handlers that - rotate log files at a certain point. It is not meant to be instantiated - directly. Instead, use :class:`RotatingFileHandler` or - :class:`TimedRotatingFileHandler`. - -#. :class:`RotatingFileHandler` instances send messages to disk - files, with support for maximum log file sizes and log file rotation. - -#. :class:`TimedRotatingFileHandler` instances send messages to - disk files, rotating the log file at certain timed intervals. - -#. :class:`SocketHandler` instances send messages to TCP/IP - sockets. - -#. :class:`DatagramHandler` instances send messages to UDP - sockets. - -#. :class:`SMTPHandler` instances send messages to a designated - email address. - -#. :class:`SysLogHandler` instances send messages to a Unix - syslog daemon, possibly on a remote machine. - -#. :class:`NTEventLogHandler` instances send messages to a - Windows NT/2000/XP event log. - -#. :class:`MemoryHandler` instances send messages to a buffer - in memory, which is flushed whenever specific criteria are met. - -#. :class:`HTTPHandler` instances send messages to an HTTP - server using either ``GET`` or ``POST`` semantics. - -#. :class:`WatchedFileHandler` instances watch the file they are - logging to. If the file changes, it is closed and reopened using the file - name. This handler is only useful on Unix-like systems; Windows does not - support the underlying mechanism used. - -.. currentmodule:: logging - -#. :class:`NullHandler` instances do nothing with error messages. They are used - by library developers who want to use logging, but want to avoid the "No - handlers could be found for logger XXX" message which can be displayed if - the library user has not configured logging. See :ref:`library-config` for - more information. - -.. versionadded:: 3.1 - The :class:`NullHandler` class. - -The :class:`NullHandler`, :class:`StreamHandler` and :class:`FileHandler` -classes are defined in the core logging package. The other handlers are -defined in a sub- module, :mod:`logging.handlers`. (There is also another -sub-module, :mod:`logging.config`, for configuration functionality.) - -Logged messages are formatted for presentation through instances of the -:class:`Formatter` class. They are initialized with a format string suitable for -use with the % operator and a dictionary. - -For formatting multiple messages in a batch, instances of -:class:`BufferingFormatter` can be used. In addition to the format string (which -is applied to each message in the batch), there is provision for header and -trailer format strings. - -When filtering based on logger level and/or handler level is not enough, -instances of :class:`Filter` can be added to both :class:`Logger` and -:class:`Handler` instances (through their :meth:`addFilter` method). Before -deciding to process a message further, both loggers and handlers consult all -their filters for permission. If any filter returns a false value, the message -is not processed further. - -The basic :class:`Filter` functionality allows filtering by specific logger -name. If this feature is used, messages sent to the named logger and its -children are allowed through the filter, and all others dropped. - -Module-Level Functions ----------------------- - -In addition to the classes described above, there are a number of module- level -functions. - - -.. function:: getLogger(name=None) - - Return a logger with the specified name or, if name is ``None``, return a - logger which is the root logger of the hierarchy. If specified, the name is - typically a dot-separated hierarchical name like *"a"*, *"a.b"* or *"a.b.c.d"*. - Choice of these names is entirely up to the developer who is using logging. - - All calls to this function with a given name return the same logger instance. - This means that logger instances never need to be passed between different parts - of an application. - - -.. function:: getLoggerClass() - - Return either the standard :class:`Logger` class, or the last class passed to - :func:`setLoggerClass`. This function may be called from within a new class - definition, to ensure that installing a customised :class:`Logger` class will - not undo customisations already applied by other code. For example:: - - class MyLogger(logging.getLoggerClass()): - # ... override behaviour here - - -.. function:: debug(msg, *args, **kwargs) - - Logs a message with level :const:`DEBUG` on the root logger. The *msg* is the - message format string, and the *args* are the arguments which are merged into - *msg* using the string formatting operator. (Note that this means that you can - use keywords in the format string, together with a single dictionary argument.) - - There are two keyword arguments in *kwargs* which are inspected: *exc_info* - which, if it does not evaluate as false, causes exception information to be - added to the logging message. If an exception tuple (in the format returned by - :func:`sys.exc_info`) is provided, it is used; otherwise, :func:`sys.exc_info` - is called to get the exception information. - - The other optional keyword argument is *extra* which can be used to pass a - dictionary which is used to populate the __dict__ of the LogRecord created for - the logging event with user-defined attributes. These custom attributes can then - be used as you like. For example, they could be incorporated into logged - messages. For example:: - - FORMAT = "%(asctime)-15s %(clientip)s %(user)-8s %(message)s" - logging.basicConfig(format=FORMAT) - d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} - logging.warning("Protocol problem: %s", "connection reset", extra=d) - - would print something like:: - - 2006-02-08 22:20:02,165 192.168.0.1 fbloggs Protocol problem: connection reset - - The keys in the dictionary passed in *extra* should not clash with the keys used - by the logging system. (See the :class:`Formatter` documentation for more - information on which keys are used by the logging system.) - - If you choose to use these attributes in logged messages, you need to exercise - some care. In the above example, for instance, the :class:`Formatter` has been - set up with a format string which expects 'clientip' and 'user' in the attribute - dictionary of the LogRecord. If these are missing, the message will not be - logged because a string formatting exception will occur. So in this case, you - always need to pass the *extra* dictionary with these keys. - - While this might be annoying, this feature is intended for use in specialized - circumstances, such as multi-threaded servers where the same code executes in - many contexts, and interesting conditions which arise are dependent on this - context (such as remote client IP address and authenticated user name, in the - above example). In such circumstances, it is likely that specialized - :class:`Formatter`\ s would be used with particular :class:`Handler`\ s. - - -.. function:: info(msg, *args, **kwargs) - - Logs a message with level :const:`INFO` on the root logger. The arguments are - interpreted as for :func:`debug`. - - -.. function:: warning(msg, *args, **kwargs) - - Logs a message with level :const:`WARNING` on the root logger. The arguments are - interpreted as for :func:`debug`. - - -.. function:: error(msg, *args, **kwargs) - - Logs a message with level :const:`ERROR` on the root logger. The arguments are - interpreted as for :func:`debug`. - - -.. function:: critical(msg, *args, **kwargs) - - Logs a message with level :const:`CRITICAL` on the root logger. The arguments - are interpreted as for :func:`debug`. - - -.. function:: exception(msg, *args) - - Logs a message with level :const:`ERROR` on the root logger. The arguments are - interpreted as for :func:`debug`. Exception info is added to the logging - message. This function should only be called from an exception handler. - - -.. function:: log(level, msg, *args, **kwargs) - - Logs a message with level *level* on the root logger. The other arguments are - interpreted as for :func:`debug`. - - -.. function:: disable(lvl) - - Provides an overriding level *lvl* for all loggers which takes precedence over - the logger's own level. When the need arises to temporarily throttle logging - output down across the whole application, this function can be useful. Its - effect is to disable all logging calls of severity *lvl* and below, so that - if you call it with a value of INFO, then all INFO and DEBUG events would be - discarded, whereas those of severity WARNING and above would be processed - according to the logger's effective level. - - -.. function:: addLevelName(lvl, levelName) - - Associates level *lvl* with text *levelName* in an internal dictionary, which is - used to map numeric levels to a textual representation, for example when a - :class:`Formatter` formats a message. This function can also be used to define - your own levels. The only constraints are that all levels used must be - registered using this function, levels should be positive integers and they - should increase in increasing order of severity. - - -.. function:: getLevelName(lvl) - - Returns the textual representation of logging level *lvl*. If the level is one - of the predefined levels :const:`CRITICAL`, :const:`ERROR`, :const:`WARNING`, - :const:`INFO` or :const:`DEBUG` then you get the corresponding string. If you - have associated levels with names using :func:`addLevelName` then the name you - have associated with *lvl* is returned. If a numeric value corresponding to one - of the defined levels is passed in, the corresponding string representation is - returned. Otherwise, the string "Level %s" % lvl is returned. - - -.. function:: makeLogRecord(attrdict) - - Creates and returns a new :class:`LogRecord` instance whose attributes are - defined by *attrdict*. This function is useful for taking a pickled - :class:`LogRecord` attribute dictionary, sent over a socket, and reconstituting - it as a :class:`LogRecord` instance at the receiving end. - - -.. function:: basicConfig(**kwargs) - - Does basic configuration for the logging system by creating a - :class:`StreamHandler` with a default :class:`Formatter` and adding it to the - root logger. The functions :func:`debug`, :func:`info`, :func:`warning`, - :func:`error` and :func:`critical` will call :func:`basicConfig` automatically - if no handlers are defined for the root logger. - - This function does nothing if the root logger already has handlers - configured for it. - - The following keyword arguments are supported. - - +--------------+---------------------------------------------+ - | Format | Description | - +==============+=============================================+ - | ``filename`` | Specifies that a FileHandler be created, | - | | using the specified filename, rather than a | - | | StreamHandler. | - +--------------+---------------------------------------------+ - | ``filemode`` | Specifies the mode to open the file, if | - | | filename is specified (if filemode is | - | | unspecified, it defaults to 'a'). | - +--------------+---------------------------------------------+ - | ``format`` | Use the specified format string for the | - | | handler. | - +--------------+---------------------------------------------+ - | ``datefmt`` | Use the specified date/time format. | - +--------------+---------------------------------------------+ - | ``level`` | Set the root logger level to the specified | - | | level. | - +--------------+---------------------------------------------+ - | ``stream`` | Use the specified stream to initialize the | - | | StreamHandler. Note that this argument is | - | | incompatible with 'filename' - if both are | - | | present, 'stream' is ignored. | - +--------------+---------------------------------------------+ - - -.. function:: shutdown() - - Informs the logging system to perform an orderly shutdown by flushing and - closing all handlers. This should be called at application exit and no - further use of the logging system should be made after this call. - - -.. function:: setLoggerClass(klass) - - Tells the logging system to use the class *klass* when instantiating a logger. - The class should define :meth:`__init__` such that only a name argument is - required, and the :meth:`__init__` should call :meth:`Logger.__init__`. This - function is typically called before any loggers are instantiated by applications - which need to use custom logger behavior. - - -.. seealso:: - - :pep:`282` - A Logging System - The proposal which described this feature for inclusion in the Python standard - library. - - `Original Python logging package <http://www.red-dove.com/python_logging.html>`_ - This is the original source for the :mod:`logging` package. The version of the - package available from this site is suitable for use with Python 1.5.2, 2.1.x - and 2.2.x, which do not include the :mod:`logging` package in the standard - library. .. _logger: @@ -848,12 +53,19 @@ Loggers have the following attributes and methods. Note that Loggers are never instantiated directly, but always through the module-level function ``logging.getLogger(name)``. +.. class:: Logger .. attribute:: Logger.propagate - If this evaluates to false, logging messages are not passed by this logger or by - its child loggers to the handlers of higher level (ancestor) loggers. The - constructor sets this attribute to 1. + If this evaluates to true, logging messages are passed by this logger and by + its child loggers to the handlers of higher level (ancestor) loggers. + Messages are passed directly to the ancestor loggers' handlers - neither the + level nor filters of the ancestor loggers in question are considered. + + If this evaluates to false, logging messages are not passed to the handlers + of ancestor loggers. + + The constructor sets this attribute to ``True``. .. method:: Logger.setLevel(lvl) @@ -864,7 +76,7 @@ instantiated directly, but always through the module-level function the root logger, or delegation to the parent when the logger is a non-root logger). Note that the root logger is created with level :const:`WARNING`. - The term "delegation to the parent" means that if a logger has a level of + The term 'delegation to the parent' means that if a logger has a level of NOTSET, its chain of ancestor loggers is traversed until either an ancestor with a level other than NOTSET is found, or the root is reached. @@ -875,6 +87,11 @@ instantiated directly, but always through the module-level function If the root is reached, and it has a level of NOTSET, then all messages will be processed. Otherwise, the root's level will be used as the effective level. + .. versionchanged:: 3.2 + The *lvl* parameter now accepts a string representation of the + level such as 'INFO' as an alternative to the integer constants + such as :const:`INFO`. + .. method:: Logger.isEnabledFor(lvl) @@ -892,6 +109,16 @@ instantiated directly, but always through the module-level function :const:`NOTSET` is found, and that value is returned. +.. method:: Logger.getChild(suffix) + + Returns a logger which is a descendant to this logger, as determined by the suffix. + Thus, ``logging.getLogger('abc').getChild('def.ghi')`` would return the same + logger as would be returned by ``logging.getLogger('abc.def.ghi')``. This is a + convenience method, useful when the parent logger is named using e.g. ``__name__`` + rather than a literal string. + + .. versionadded:: 3.2 + .. method:: Logger.debug(msg, *args, **kwargs) @@ -900,23 +127,41 @@ instantiated directly, but always through the module-level function *msg* using the string formatting operator. (Note that this means that you can use keywords in the format string, together with a single dictionary argument.) - There are two keyword arguments in *kwargs* which are inspected: *exc_info* + There are three keyword arguments in *kwargs* which are inspected: *exc_info* which, if it does not evaluate as false, causes exception information to be added to the logging message. If an exception tuple (in the format returned by :func:`sys.exc_info`) is provided, it is used; otherwise, :func:`sys.exc_info` is called to get the exception information. - The other optional keyword argument is *extra* which can be used to pass a + The second optional keyword argument is *stack_info*, which defaults to + False. If specified as True, stack information is added to the logging + message, including the actual logging call. Note that this is not the same + stack information as that displayed through specifying *exc_info*: The + former is stack frames from the bottom of the stack up to the logging call + in the current thread, whereas the latter is information about stack frames + which have been unwound, following an exception, while searching for + exception handlers. + + You can specify *stack_info* independently of *exc_info*, e.g. to just show + how you got to a certain point in your code, even when no exceptions were + raised. The stack frames are printed following a header line which says:: + + Stack (most recent call last): + + This mimics the ``Traceback (most recent call last):`` which is used when + displaying exception frames. + + The third keyword argument is *extra* which can be used to pass a dictionary which is used to populate the __dict__ of the LogRecord created for the logging event with user-defined attributes. These custom attributes can then be used as you like. For example, they could be incorporated into logged messages. For example:: - FORMAT = "%(asctime)-15s %(clientip)s %(user)-8s %(message)s" + FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' logging.basicConfig(format=FORMAT) d = { 'clientip' : '192.168.0.1', 'user' : 'fbloggs' } - logger = logging.getLogger("tcpserver") - logger.warning("Protocol problem: %s", "connection reset", extra=d) + logger = logging.getLogger('tcpserver') + logger.warning('Protocol problem: %s', 'connection reset', extra=d) would print something like :: @@ -940,6 +185,9 @@ instantiated directly, but always through the module-level function above example). In such circumstances, it is likely that specialized :class:`Formatter`\ s would be used with particular :class:`Handler`\ s. + .. versionadded:: 3.2 + The *stack_info* parameter was added. + .. method:: Logger.info(msg, *args, **kwargs) @@ -1004,10 +252,11 @@ instantiated directly, but always through the module-level function Removes the specified handler *hdlr* from this logger. -.. method:: Logger.findCaller() +.. method:: Logger.findCaller(stack_info=False) Finds the caller's source filename and line number. Returns the filename, line - number and function name as a 3-element tuple. + number, function name and stack information as a 4-element tuple. The stack + information is returned as *None* unless *stack_info* is *True*. .. method:: Logger.handle(record) @@ -1018,630 +267,22 @@ instantiated directly, but always through the module-level function Logger-level filtering is applied using :meth:`~Logger.filter`. -.. method:: Logger.makeRecord(name, lvl, fn, lno, msg, args, exc_info, func=None, extra=None) +.. method:: Logger.makeRecord(name, lvl, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None) This is a factory method which can be overridden in subclasses to create specialized :class:`LogRecord` instances. +.. method:: Logger.hasHandlers() -.. _minimal-example: - -Basic example -------------- - -The :mod:`logging` package provides a lot of flexibility, and its configuration -can appear daunting. This section demonstrates that simple use of the logging -package is possible. - -The simplest example shows logging to the console:: - - import logging - - logging.debug('A debug message') - logging.info('Some information') - logging.warning('A shot across the bows') - -If you run the above script, you'll see this:: - - WARNING:root:A shot across the bows - -Because no particular logger was specified, the system used the root logger. The -debug and info messages didn't appear because by default, the root logger is -configured to only handle messages with a severity of WARNING or above. The -message format is also a configuration default, as is the output destination of -the messages - ``sys.stderr``. The severity level, the message format and -destination can be easily changed, as shown in the example below:: - - import logging - - logging.basicConfig(level=logging.DEBUG, - format='%(asctime)s %(levelname)s %(message)s', - filename='myapp.log', - filemode='w') - logging.debug('A debug message') - logging.info('Some information') - logging.warning('A shot across the bows') + Checks to see if this logger has any handlers configured. This is done by + looking for handlers in this logger and its parents in the logger hierarchy. + Returns True if a handler was found, else False. The method stops searching + up the hierarchy whenever a logger with the 'propagate' attribute set to + False is found - that will be the last logger which is checked for the + existence of handlers. -The :meth:`basicConfig` method is used to change the configuration defaults, -which results in output (written to ``myapp.log``) which should look -something like the following:: - - 2004-07-02 13:00:08,743 DEBUG A debug message - 2004-07-02 13:00:08,743 INFO Some information - 2004-07-02 13:00:08,743 WARNING A shot across the bows - -This time, all messages with a severity of DEBUG or above were handled, and the -format of the messages was also changed, and output went to the specified file -rather than the console. - -.. XXX logging should probably be updated for new string formatting! - -Formatting uses the old Python string formatting - see section -:ref:`old-string-formatting`. The format string takes the following common -specifiers. For a complete list of specifiers, consult the :class:`Formatter` -documentation. - -+-------------------+-----------------------------------------------+ -| Format | Description | -+===================+===============================================+ -| ``%(name)s`` | Name of the logger (logging channel). | -+-------------------+-----------------------------------------------+ -| ``%(levelname)s`` | Text logging level for the message | -| | (``'DEBUG'``, ``'INFO'``, ``'WARNING'``, | -| | ``'ERROR'``, ``'CRITICAL'``). | -+-------------------+-----------------------------------------------+ -| ``%(asctime)s`` | Human-readable time when the | -| | :class:`LogRecord` was created. By default | -| | this is of the form "2003-07-08 16:49:45,896" | -| | (the numbers after the comma are millisecond | -| | portion of the time). | -+-------------------+-----------------------------------------------+ -| ``%(message)s`` | The logged message. | -+-------------------+-----------------------------------------------+ - -To change the date/time format, you can pass an additional keyword parameter, -*datefmt*, as in the following:: - - import logging - - logging.basicConfig(level=logging.DEBUG, - format='%(asctime)s %(levelname)-8s %(message)s', - datefmt='%a, %d %b %Y %H:%M:%S', - filename='/temp/myapp.log', - filemode='w') - logging.debug('A debug message') - logging.info('Some information') - logging.warning('A shot across the bows') - -which would result in output like :: + .. versionadded:: 3.2 - Fri, 02 Jul 2004 13:06:18 DEBUG A debug message - Fri, 02 Jul 2004 13:06:18 INFO Some information - Fri, 02 Jul 2004 13:06:18 WARNING A shot across the bows - -The date format string follows the requirements of :func:`strftime` - see the -documentation for the :mod:`time` module. - -If, instead of sending logging output to the console or a file, you'd rather use -a file-like object which you have created separately, you can pass it to -:func:`basicConfig` using the *stream* keyword argument. Note that if both -*stream* and *filename* keyword arguments are passed, the *stream* argument is -ignored. - -Of course, you can put variable information in your output. To do this, simply -have the message be a format string and pass in additional arguments containing -the variable information, as in the following example:: - - import logging - - logging.basicConfig(level=logging.DEBUG, - format='%(asctime)s %(levelname)-8s %(message)s', - datefmt='%a, %d %b %Y %H:%M:%S', - filename='/temp/myapp.log', - filemode='w') - logging.error('Pack my box with %d dozen %s', 5, 'liquor jugs') - -which would result in :: - - Wed, 21 Jul 2004 15:35:16 ERROR Pack my box with 5 dozen liquor jugs - - -.. _multiple-destinations: - -Logging to multiple destinations --------------------------------- - -Let's say you want to log to console and file with different message formats and -in differing circumstances. Say you want to log messages with levels of DEBUG -and higher to file, and those messages at level INFO and higher to the console. -Let's also assume that the file should contain timestamps, but the console -messages should not. Here's how you can achieve this:: - - import logging - - # set up logging to file - see previous section for more details - logging.basicConfig(level=logging.DEBUG, - format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', - datefmt='%m-%d %H:%M', - filename='/temp/myapp.log', - filemode='w') - # define a Handler which writes INFO messages or higher to the sys.stderr - console = logging.StreamHandler() - console.setLevel(logging.INFO) - # set a format which is simpler for console use - formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s') - # tell the handler to use this format - console.setFormatter(formatter) - # add the handler to the root logger - logging.getLogger('').addHandler(console) - - # Now, we can log to the root logger, or any other logger. First the root... - logging.info('Jackdaws love my big sphinx of quartz.') - - # Now, define a couple of other loggers which might represent areas in your - # application: - - logger1 = logging.getLogger('myapp.area1') - logger2 = logging.getLogger('myapp.area2') - - logger1.debug('Quick zephyrs blow, vexing daft Jim.') - logger1.info('How quickly daft jumping zebras vex.') - logger2.warning('Jail zesty vixen who grabbed pay from quack.') - logger2.error('The five boxing wizards jump quickly.') - -When you run this, on the console you will see :: - - root : INFO Jackdaws love my big sphinx of quartz. - myapp.area1 : INFO How quickly daft jumping zebras vex. - myapp.area2 : WARNING Jail zesty vixen who grabbed pay from quack. - myapp.area2 : ERROR The five boxing wizards jump quickly. - -and in the file you will see something like :: - - 10-22 22:19 root INFO Jackdaws love my big sphinx of quartz. - 10-22 22:19 myapp.area1 DEBUG Quick zephyrs blow, vexing daft Jim. - 10-22 22:19 myapp.area1 INFO How quickly daft jumping zebras vex. - 10-22 22:19 myapp.area2 WARNING Jail zesty vixen who grabbed pay from quack. - 10-22 22:19 myapp.area2 ERROR The five boxing wizards jump quickly. - -As you can see, the DEBUG message only shows up in the file. The other messages -are sent to both destinations. - -This example uses console and file handlers, but you can use any number and -combination of handlers you choose. - -.. _logging-exceptions: - -Exceptions raised during logging --------------------------------- - -The logging package is designed to swallow exceptions which occur while logging -in production. This is so that errors which occur while handling logging events -- such as logging misconfiguration, network or other similar errors - do not -cause the application using logging to terminate prematurely. - -:class:`SystemExit` and :class:`KeyboardInterrupt` exceptions are never -swallowed. Other exceptions which occur during the :meth:`emit` method of a -:class:`Handler` subclass are passed to its :meth:`handleError` method. - -The default implementation of :meth:`handleError` in :class:`Handler` checks -to see if a module-level variable, :data:`raiseExceptions`, is set. If set, a -traceback is printed to :data:`sys.stderr`. If not set, the exception is swallowed. - -**Note:** The default value of :data:`raiseExceptions` is ``True``. This is because -during development, you typically want to be notified of any exceptions that -occur. It's advised that you set :data:`raiseExceptions` to ``False`` for production -usage. - -.. _context-info: - -Adding contextual information to your logging output ----------------------------------------------------- - -Sometimes you want logging output to contain contextual information in -addition to the parameters passed to the logging call. For example, in a -networked application, it may be desirable to log client-specific information -in the log (e.g. remote client's username, or IP address). Although you could -use the *extra* parameter to achieve this, it's not always convenient to pass -the information in this way. While it might be tempting to create -:class:`Logger` instances on a per-connection basis, this is not a good idea -because these instances are not garbage collected. While this is not a problem -in practice, when the number of :class:`Logger` instances is dependent on the -level of granularity you want to use in logging an application, it could -be hard to manage if the number of :class:`Logger` instances becomes -effectively unbounded. - - -Using LoggerAdapters to impart contextual information -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -An easy way in which you can pass contextual information to be output along -with logging event information is to use the :class:`LoggerAdapter` class. -This class is designed to look like a :class:`Logger`, so that you can call -:meth:`debug`, :meth:`info`, :meth:`warning`, :meth:`error`, -:meth:`exception`, :meth:`critical` and :meth:`log`. These methods have the -same signatures as their counterparts in :class:`Logger`, so you can use the -two types of instances interchangeably. - -When you create an instance of :class:`LoggerAdapter`, you pass it a -:class:`Logger` instance and a dict-like object which contains your contextual -information. When you call one of the logging methods on an instance of -:class:`LoggerAdapter`, it delegates the call to the underlying instance of -:class:`Logger` passed to its constructor, and arranges to pass the contextual -information in the delegated call. Here's a snippet from the code of -:class:`LoggerAdapter`:: - - def debug(self, msg, *args, **kwargs): - """ - Delegate a debug call to the underlying logger, after adding - contextual information from this adapter instance. - """ - msg, kwargs = self.process(msg, kwargs) - self.logger.debug(msg, *args, **kwargs) - -The :meth:`process` method of :class:`LoggerAdapter` is where the contextual -information is added to the logging output. It's passed the message and -keyword arguments of the logging call, and it passes back (potentially) -modified versions of these to use in the call to the underlying logger. The -default implementation of this method leaves the message alone, but inserts -an "extra" key in the keyword argument whose value is the dict-like object -passed to the constructor. Of course, if you had passed an "extra" keyword -argument in the call to the adapter, it will be silently overwritten. - -The advantage of using "extra" is that the values in the dict-like object are -merged into the :class:`LogRecord` instance's __dict__, allowing you to use -customized strings with your :class:`Formatter` instances which know about -the keys of the dict-like object. If you need a different method, e.g. if you -want to prepend or append the contextual information to the message string, -you just need to subclass :class:`LoggerAdapter` and override :meth:`process` -to do what you need. Here's an example script which uses this class, which -also illustrates what dict-like behaviour is needed from an arbitrary -"dict-like" object for use in the constructor:: - - import logging - - class ConnInfo: - """ - An example class which shows how an arbitrary class can be used as - the 'extra' context information repository passed to a LoggerAdapter. - """ - - def __getitem__(self, name): - """ - To allow this instance to look like a dict. - """ - from random import choice - if name == "ip": - result = choice(["127.0.0.1", "192.168.0.1"]) - elif name == "user": - result = choice(["jim", "fred", "sheila"]) - else: - result = self.__dict__.get(name, "?") - return result - - def __iter__(self): - """ - To allow iteration over keys, which will be merged into - the LogRecord dict before formatting and output. - """ - keys = ["ip", "user"] - keys.extend(self.__dict__.keys()) - return keys.__iter__() - - if __name__ == "__main__": - from random import choice - levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL) - a1 = logging.LoggerAdapter(logging.getLogger("a.b.c"), - { "ip" : "123.231.231.123", "user" : "sheila" }) - logging.basicConfig(level=logging.DEBUG, - format="%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s") - a1.debug("A debug message") - a1.info("An info message with %s", "some parameters") - a2 = logging.LoggerAdapter(logging.getLogger("d.e.f"), ConnInfo()) - for x in range(10): - lvl = choice(levels) - lvlname = logging.getLevelName(lvl) - a2.log(lvl, "A message at %s level with %d %s", lvlname, 2, "parameters") - -When this script is run, the output should look something like this:: - - 2008-01-18 14:49:54,023 a.b.c DEBUG IP: 123.231.231.123 User: sheila A debug message - 2008-01-18 14:49:54,023 a.b.c INFO IP: 123.231.231.123 User: sheila An info message with some parameters - 2008-01-18 14:49:54,023 d.e.f CRITICAL IP: 192.168.0.1 User: jim A message at CRITICAL level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f INFO IP: 192.168.0.1 User: jim A message at INFO level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f WARNING IP: 192.168.0.1 User: sheila A message at WARNING level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f ERROR IP: 127.0.0.1 User: fred A message at ERROR level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f ERROR IP: 127.0.0.1 User: sheila A message at ERROR level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f WARNING IP: 192.168.0.1 User: sheila A message at WARNING level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f WARNING IP: 192.168.0.1 User: jim A message at WARNING level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f INFO IP: 192.168.0.1 User: fred A message at INFO level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f WARNING IP: 192.168.0.1 User: sheila A message at WARNING level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f WARNING IP: 127.0.0.1 User: jim A message at WARNING level with 2 parameters - - -Using Filters to impart contextual information -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -You can also add contextual information to log output using a user-defined -:class:`Filter`. ``Filter`` instances are allowed to modify the ``LogRecords`` -passed to them, including adding additional attributes which can then be output -using a suitable format string, or if needed a custom :class:`Formatter`. - -For example in a web application, the request being processed (or at least, -the interesting parts of it) can be stored in a threadlocal -(:class:`threading.local`) variable, and then accessed from a ``Filter`` to -add, say, information from the request - say, the remote IP address and remote -user's username - to the ``LogRecord``, using the attribute names 'ip' and -'user' as in the ``LoggerAdapter`` example above. In that case, the same format -string can be used to get similar output to that shown above. Here's an example -script:: - - import logging - from random import choice - - class ContextFilter(logging.Filter): - """ - This is a filter which injects contextual information into the log. - - Rather than use actual contextual information, we just use random - data in this demo. - """ - - USERS = ['jim', 'fred', 'sheila'] - IPS = ['123.231.231.123', '127.0.0.1', '192.168.0.1'] - - def filter(self, record): - - record.ip = choice(ContextFilter.IPS) - record.user = choice(ContextFilter.USERS) - return True - - if __name__ == "__main__": - levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL) - a1 = logging.LoggerAdapter(logging.getLogger("a.b.c"), - { "ip" : "123.231.231.123", "user" : "sheila" }) - logging.basicConfig(level=logging.DEBUG, - format="%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s") - a1 = logging.getLogger("a.b.c") - a2 = logging.getLogger("d.e.f") - - f = ContextFilter() - a1.addFilter(f) - a2.addFilter(f) - a1.debug("A debug message") - a1.info("An info message with %s", "some parameters") - for x in range(10): - lvl = choice(levels) - lvlname = logging.getLevelName(lvl) - a2.log(lvl, "A message at %s level with %d %s", lvlname, 2, "parameters") - -which, when run, produces something like:: - - 2010-09-06 22:38:15,292 a.b.c DEBUG IP: 123.231.231.123 User: fred A debug message - 2010-09-06 22:38:15,300 a.b.c INFO IP: 192.168.0.1 User: sheila An info message with some parameters - 2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL level with 2 parameters - 2010-09-06 22:38:15,300 d.e.f ERROR IP: 127.0.0.1 User: jim A message at ERROR level with 2 parameters - 2010-09-06 22:38:15,300 d.e.f DEBUG IP: 127.0.0.1 User: sheila A message at DEBUG level with 2 parameters - 2010-09-06 22:38:15,300 d.e.f ERROR IP: 123.231.231.123 User: fred A message at ERROR level with 2 parameters - 2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 192.168.0.1 User: jim A message at CRITICAL level with 2 parameters - 2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL level with 2 parameters - 2010-09-06 22:38:15,300 d.e.f DEBUG IP: 192.168.0.1 User: jim A message at DEBUG level with 2 parameters - 2010-09-06 22:38:15,301 d.e.f ERROR IP: 127.0.0.1 User: sheila A message at ERROR level with 2 parameters - 2010-09-06 22:38:15,301 d.e.f DEBUG IP: 123.231.231.123 User: fred A message at DEBUG level with 2 parameters - 2010-09-06 22:38:15,301 d.e.f INFO IP: 123.231.231.123 User: fred A message at INFO level with 2 parameters - - -.. _multiple-processes: - -Logging to a single file from multiple processes ------------------------------------------------- - -Although logging is thread-safe, and logging to a single file from multiple -threads in a single process *is* supported, logging to a single file from -*multiple processes* is *not* supported, because there is no standard way to -serialize access to a single file across multiple processes in Python. If you -need to log to a single file from multiple processes, one way of doing this is -to have all the processes log to a :class:`SocketHandler`, and have a separate -process which implements a socket server which reads from the socket and logs -to file. (If you prefer, you can dedicate one thread in one of the existing -processes to perform this function.) The following section documents this -approach in more detail and includes a working socket receiver which can be -used as a starting point for you to adapt in your own applications. - -If you are using a recent version of Python which includes the -:mod:`multiprocessing` module, you could write your own handler which uses the -:class:`Lock` class from this module to serialize access to the file from -your processes. The existing :class:`FileHandler` and subclasses do not make -use of :mod:`multiprocessing` at present, though they may do so in the future. -Note that at present, the :mod:`multiprocessing` module does not provide -working lock functionality on all platforms (see -http://bugs.python.org/issue3770). - - -.. _network-logging: - -Sending and receiving logging events across a network ------------------------------------------------------ - -Let's say you want to send logging events across a network, and handle them at -the receiving end. A simple way of doing this is attaching a -:class:`SocketHandler` instance to the root logger at the sending end:: - - import logging, logging.handlers - - rootLogger = logging.getLogger('') - rootLogger.setLevel(logging.DEBUG) - socketHandler = logging.handlers.SocketHandler('localhost', - logging.handlers.DEFAULT_TCP_LOGGING_PORT) - # don't bother with a formatter, since a socket handler sends the event as - # an unformatted pickle - rootLogger.addHandler(socketHandler) - - # Now, we can log to the root logger, or any other logger. First the root... - logging.info('Jackdaws love my big sphinx of quartz.') - - # Now, define a couple of other loggers which might represent areas in your - # application: - - logger1 = logging.getLogger('myapp.area1') - logger2 = logging.getLogger('myapp.area2') - - logger1.debug('Quick zephyrs blow, vexing daft Jim.') - logger1.info('How quickly daft jumping zebras vex.') - logger2.warning('Jail zesty vixen who grabbed pay from quack.') - logger2.error('The five boxing wizards jump quickly.') - -At the receiving end, you can set up a receiver using the :mod:`socketserver` -module. Here is a basic working example:: - - import pickle - import logging - import logging.handlers - import socketserver - import struct - - - class LogRecordStreamHandler(socketserver.StreamRequestHandler): - """Handler for a streaming logging request. - - This basically logs the record using whatever logging policy is - configured locally. - """ - - def handle(self): - """ - Handle multiple requests - each expected to be a 4-byte length, - followed by the LogRecord in pickle format. Logs the record - according to whatever policy is configured locally. - """ - while True: - chunk = self.connection.recv(4) - if len(chunk) < 4: - break - slen = struct.unpack(">L", chunk)[0] - chunk = self.connection.recv(slen) - while len(chunk) < slen: - chunk = chunk + self.connection.recv(slen - len(chunk)) - obj = self.unPickle(chunk) - record = logging.makeLogRecord(obj) - self.handleLogRecord(record) - - def unPickle(self, data): - return pickle.loads(data) - - def handleLogRecord(self, record): - # if a name is specified, we use the named logger rather than the one - # implied by the record. - if self.server.logname is not None: - name = self.server.logname - else: - name = record.name - logger = logging.getLogger(name) - # N.B. EVERY record gets logged. This is because Logger.handle - # is normally called AFTER logger-level filtering. If you want - # to do filtering, do it at the client end to save wasting - # cycles and network bandwidth! - logger.handle(record) - - class LogRecordSocketReceiver(socketserver.ThreadingTCPServer): - """simple TCP socket-based logging receiver suitable for testing. - """ - - allow_reuse_address = 1 - - def __init__(self, host='localhost', - port=logging.handlers.DEFAULT_TCP_LOGGING_PORT, - handler=LogRecordStreamHandler): - socketserver.ThreadingTCPServer.__init__(self, (host, port), handler) - self.abort = 0 - self.timeout = 1 - self.logname = None - - def serve_until_stopped(self): - import select - abort = 0 - while not abort: - rd, wr, ex = select.select([self.socket.fileno()], - [], [], - self.timeout) - if rd: - self.handle_request() - abort = self.abort - - def main(): - logging.basicConfig( - format="%(relativeCreated)5d %(name)-15s %(levelname)-8s %(message)s") - tcpserver = LogRecordSocketReceiver() - print("About to start TCP server...") - tcpserver.serve_until_stopped() - - if __name__ == "__main__": - main() - -First run the server, and then the client. On the client side, nothing is -printed on the console; on the server side, you should see something like:: - - About to start TCP server... - 59 root INFO Jackdaws love my big sphinx of quartz. - 59 myapp.area1 DEBUG Quick zephyrs blow, vexing daft Jim. - 69 myapp.area1 INFO How quickly daft jumping zebras vex. - 69 myapp.area2 WARNING Jail zesty vixen who grabbed pay from quack. - 69 myapp.area2 ERROR The five boxing wizards jump quickly. - -Note that there are some security issues with pickle in some scenarios. If -these affect you, you can use an alternative serialization scheme by overriding -the :meth:`makePickle` method and implementing your alternative there, as -well as adapting the above script to use your alternative serialization. - -.. _arbitrary-object-messages: - -Using arbitrary objects as messages ------------------------------------ - -In the preceding sections and examples, it has been assumed that the message -passed when logging the event is a string. However, this is not the only -possibility. You can pass an arbitrary object as a message, and its -:meth:`__str__` method will be called when the logging system needs to convert -it to a string representation. In fact, if you want to, you can avoid -computing a string representation altogether - for example, the -:class:`SocketHandler` emits an event by pickling it and sending it over the -wire. - -Optimization ------------- - -Formatting of message arguments is deferred until it cannot be avoided. -However, computing the arguments passed to the logging method can also be -expensive, and you may want to avoid doing it if the logger will just throw -away your event. To decide what to do, you can call the :meth:`isEnabledFor` -method which takes a level argument and returns true if the event would be -created by the Logger for that level of call. You can write code like this:: - - if logger.isEnabledFor(logging.DEBUG): - logger.debug("Message with %s, %s", expensive_func1(), - expensive_func2()) - -so that if the logger's threshold is set above ``DEBUG``, the calls to -:func:`expensive_func1` and :func:`expensive_func2` are never made. - -There are other optimizations which can be made for specific applications which -need more precise control over what logging information is collected. Here's a -list of things you can do to avoid processing during logging which you don't -need: - -+-----------------------------------------------+----------------------------------------+ -| What you don't want to collect | How to avoid collecting it | -+===============================================+========================================+ -| Information about where calls were made from. | Set ``logging._srcfile`` to ``None``. | -+-----------------------------------------------+----------------------------------------+ -| Threading information. | Set ``logging.logThreads`` to ``0``. | -+-----------------------------------------------+----------------------------------------+ -| Process information. | Set ``logging.logProcesses`` to ``0``. | -+-----------------------------------------------+----------------------------------------+ - -Also note that the core logging module only includes the basic handlers. If -you don't import :mod:`logging.handlers` and :mod:`logging.config`, they won't -take up any memory. .. _handler: @@ -1683,6 +324,11 @@ subclasses. However, the :meth:`__init__` method in subclasses needs to call severe than *lvl* will be ignored. When a handler is created, the level is set to :const:`NOTSET` (which causes all messages to be processed). + .. versionchanged:: 3.2 + The *lvl* parameter now accepts a string representation of the + level such as 'INFO' as an alternative to the integer constants + such as :const:`INFO`. + .. method:: Handler.setFormatter(form) @@ -1729,12 +375,14 @@ subclasses. However, the :meth:`__init__` method in subclasses needs to call .. method:: Handler.handleError(record) This method should be called from handlers when an exception is encountered - during an :meth:`emit` call. By default it does nothing, which means that - exceptions get silently ignored. This is what is mostly wanted for a logging - system - most users will not care about errors in the logging system, they are - more interested in application errors. You could, however, replace this with a - custom handler if you wish. The specified record is the one which was being - processed when the exception occurred. + during an :meth:`emit` call. If the module-level attribute + ``raiseExceptions`` is ``False``, exceptions get silently ignored. This is + what is mostly wanted for a logging system - most users will not care about + errors in the logging system, they are more interested in application + errors. You could, however, replace this with a custom handler if you wish. + The specified record is the one which was being processed when the exception + occurred. (The default value of ``raiseExceptions`` is ``True``, as that is + more useful during development). .. method:: Handler.format(record) @@ -1749,643 +397,7 @@ subclasses. However, the :meth:`__init__` method in subclasses needs to call is intended to be implemented by subclasses and so raises a :exc:`NotImplementedError`. - -.. _stream-handler: - -StreamHandler -^^^^^^^^^^^^^ - -The :class:`StreamHandler` class, located in the core :mod:`logging` package, -sends logging output to streams such as *sys.stdout*, *sys.stderr* or any -file-like object (or, more precisely, any object which supports :meth:`write` -and :meth:`flush` methods). - - -.. currentmodule:: logging - -.. class:: StreamHandler(stream=None) - - Returns a new instance of the :class:`StreamHandler` class. If *stream* is - specified, the instance will use it for logging output; otherwise, *sys.stderr* - will be used. - - - .. method:: emit(record) - - If a formatter is specified, it is used to format the record. The record - is then written to the stream with a trailing newline. If exception - information is present, it is formatted using - :func:`traceback.print_exception` and appended to the stream. - - - .. method:: flush() - - Flushes the stream by calling its :meth:`flush` method. Note that the - :meth:`close` method is inherited from :class:`Handler` and so does - no output, so an explicit :meth:`flush` call may be needed at times. - - -.. _file-handler: - -FileHandler -^^^^^^^^^^^ - -The :class:`FileHandler` class, located in the core :mod:`logging` package, -sends logging output to a disk file. It inherits the output functionality from -:class:`StreamHandler`. - - -.. class:: FileHandler(filename, mode='a', encoding=None, delay=0) - - Returns a new instance of the :class:`FileHandler` class. The specified file is - opened and used as the stream for logging. If *mode* is not specified, - :const:`'a'` is used. If *encoding* is not *None*, it is used to open the file - with that encoding. If *delay* is true, then file opening is deferred until the - first call to :meth:`emit`. By default, the file grows indefinitely. - - - .. method:: close() - - Closes the file. - - - .. method:: emit(record) - - Outputs the record to the file. - - -.. _null-handler: - -NullHandler -^^^^^^^^^^^ - -.. versionadded:: 3.1 - -The :class:`NullHandler` class, located in the core :mod:`logging` package, -does not do any formatting or output. It is essentially a "no-op" handler -for use by library developers. - -.. class:: NullHandler() - - Returns a new instance of the :class:`NullHandler` class. - - .. method:: emit(record) - - This method does nothing. - -See :ref:`library-config` for more information on how to use -:class:`NullHandler`. - -.. _watched-file-handler: - -WatchedFileHandler -^^^^^^^^^^^^^^^^^^ - -.. currentmodule:: logging.handlers - -The :class:`WatchedFileHandler` class, located in the :mod:`logging.handlers` -module, is a :class:`FileHandler` which watches the file it is logging to. If -the file changes, it is closed and reopened using the file name. - -A file change can happen because of usage of programs such as *newsyslog* and -*logrotate* which perform log file rotation. This handler, intended for use -under Unix/Linux, watches the file to see if it has changed since the last emit. -(A file is deemed to have changed if its device or inode have changed.) If the -file has changed, the old file stream is closed, and the file opened to get a -new stream. - -This handler is not appropriate for use under Windows, because under Windows -open log files cannot be moved or renamed - logging opens the files with -exclusive locks - and so there is no need for such a handler. Furthermore, -*ST_INO* is not supported under Windows; :func:`stat` always returns zero for -this value. - - -.. class:: WatchedFileHandler(filename[,mode[, encoding[, delay]]]) - - Returns a new instance of the :class:`WatchedFileHandler` class. The specified - file is opened and used as the stream for logging. If *mode* is not specified, - :const:`'a'` is used. If *encoding* is not *None*, it is used to open the file - with that encoding. If *delay* is true, then file opening is deferred until the - first call to :meth:`emit`. By default, the file grows indefinitely. - - - .. method:: emit(record) - - Outputs the record to the file, but first checks to see if the file has - changed. If it has, the existing stream is flushed and closed and the - file opened again, before outputting the record to the file. - -.. _rotating-file-handler: - -RotatingFileHandler -^^^^^^^^^^^^^^^^^^^ - -The :class:`RotatingFileHandler` class, located in the :mod:`logging.handlers` -module, supports rotation of disk log files. - - -.. class:: RotatingFileHandler(filename, mode='a', maxBytes=0, backupCount=0, encoding=None, delay=0) - - Returns a new instance of the :class:`RotatingFileHandler` class. The specified - file is opened and used as the stream for logging. If *mode* is not specified, - ``'a'`` is used. If *encoding* is not *None*, it is used to open the file - with that encoding. If *delay* is true, then file opening is deferred until the - first call to :meth:`emit`. By default, the file grows indefinitely. - - You can use the *maxBytes* and *backupCount* values to allow the file to - :dfn:`rollover` at a predetermined size. When the size is about to be exceeded, - the file is closed and a new file is silently opened for output. Rollover occurs - whenever the current log file is nearly *maxBytes* in length; if *maxBytes* is - zero, rollover never occurs. If *backupCount* is non-zero, the system will save - old log files by appending the extensions ".1", ".2" etc., to the filename. For - example, with a *backupCount* of 5 and a base file name of :file:`app.log`, you - would get :file:`app.log`, :file:`app.log.1`, :file:`app.log.2`, up to - :file:`app.log.5`. The file being written to is always :file:`app.log`. When - this file is filled, it is closed and renamed to :file:`app.log.1`, and if files - :file:`app.log.1`, :file:`app.log.2`, etc. exist, then they are renamed to - :file:`app.log.2`, :file:`app.log.3` etc. respectively. - - - .. method:: doRollover() - - Does a rollover, as described above. - - - .. method:: emit(record) - - Outputs the record to the file, catering for rollover as described - previously. - -.. _timed-rotating-file-handler: - -TimedRotatingFileHandler -^^^^^^^^^^^^^^^^^^^^^^^^ - -The :class:`TimedRotatingFileHandler` class, located in the -:mod:`logging.handlers` module, supports rotation of disk log files at certain -timed intervals. - - -.. class:: TimedRotatingFileHandler(filename, when='h', interval=1, backupCount=0, encoding=None, delay=0, utc=False) - - Returns a new instance of the :class:`TimedRotatingFileHandler` class. The - specified file is opened and used as the stream for logging. On rotating it also - sets the filename suffix. Rotating happens based on the product of *when* and - *interval*. - - You can use the *when* to specify the type of *interval*. The list of possible - values is below. Note that they are not case sensitive. - - +----------------+-----------------------+ - | Value | Type of interval | - +================+=======================+ - | ``'S'`` | Seconds | - +----------------+-----------------------+ - | ``'M'`` | Minutes | - +----------------+-----------------------+ - | ``'H'`` | Hours | - +----------------+-----------------------+ - | ``'D'`` | Days | - +----------------+-----------------------+ - | ``'W'`` | Week day (0=Monday) | - +----------------+-----------------------+ - | ``'midnight'`` | Roll over at midnight | - +----------------+-----------------------+ - - The system will save old log files by appending extensions to the filename. - The extensions are date-and-time based, using the strftime format - ``%Y-%m-%d_%H-%M-%S`` or a leading portion thereof, depending on the - rollover interval. - - When computing the next rollover time for the first time (when the handler - is created), the last modification time of an existing log file, or else - the current time, is used to compute when the next rotation will occur. - - If the *utc* argument is true, times in UTC will be used; otherwise - local time is used. - - If *backupCount* is nonzero, at most *backupCount* files - will be kept, and if more would be created when rollover occurs, the oldest - one is deleted. The deletion logic uses the interval to determine which - files to delete, so changing the interval may leave old files lying around. - - If *delay* is true, then file opening is deferred until the first call to - :meth:`emit`. - - - .. method:: doRollover() - - Does a rollover, as described above. - - - .. method:: emit(record) - - Outputs the record to the file, catering for rollover as described above. - - -.. _socket-handler: - -SocketHandler -^^^^^^^^^^^^^ - -The :class:`SocketHandler` class, located in the :mod:`logging.handlers` module, -sends logging output to a network socket. The base class uses a TCP socket. - - -.. class:: SocketHandler(host, port) - - Returns a new instance of the :class:`SocketHandler` class intended to - communicate with a remote machine whose address is given by *host* and *port*. - - - .. method:: close() - - Closes the socket. - - - .. method:: emit() - - Pickles the record's attribute dictionary and writes it to the socket in - binary format. If there is an error with the socket, silently drops the - packet. If the connection was previously lost, re-establishes the - connection. To unpickle the record at the receiving end into a - :class:`LogRecord`, use the :func:`makeLogRecord` function. - - - .. method:: handleError() - - Handles an error which has occurred during :meth:`emit`. The most likely - cause is a lost connection. Closes the socket so that we can retry on the - next event. - - - .. method:: makeSocket() - - This is a factory method which allows subclasses to define the precise - type of socket they want. The default implementation creates a TCP socket - (:const:`socket.SOCK_STREAM`). - - - .. method:: makePickle(record) - - Pickles the record's attribute dictionary in binary format with a length - prefix, and returns it ready for transmission across the socket. - - Note that pickles aren't completely secure. If you are concerned about - security, you may want to override this method to implement a more secure - mechanism. For example, you can sign pickles using HMAC and then verify - them on the receiving end, or alternatively you can disable unpickling of - global objects on the receiving end. - - .. method:: send(packet) - - Send a pickled string *packet* to the socket. This function allows for - partial sends which can happen when the network is busy. - - -.. _datagram-handler: - -DatagramHandler -^^^^^^^^^^^^^^^ - -The :class:`DatagramHandler` class, located in the :mod:`logging.handlers` -module, inherits from :class:`SocketHandler` to support sending logging messages -over UDP sockets. - - -.. class:: DatagramHandler(host, port) - - Returns a new instance of the :class:`DatagramHandler` class intended to - communicate with a remote machine whose address is given by *host* and *port*. - - - .. method:: emit() - - Pickles the record's attribute dictionary and writes it to the socket in - binary format. If there is an error with the socket, silently drops the - packet. To unpickle the record at the receiving end into a - :class:`LogRecord`, use the :func:`makeLogRecord` function. - - - .. method:: makeSocket() - - The factory method of :class:`SocketHandler` is here overridden to create - a UDP socket (:const:`socket.SOCK_DGRAM`). - - - .. method:: send(s) - - Send a pickled string to a socket. - - -.. _syslog-handler: - -SysLogHandler -^^^^^^^^^^^^^ - -The :class:`SysLogHandler` class, located in the :mod:`logging.handlers` module, -supports sending logging messages to a remote or local Unix syslog. - - -.. class:: SysLogHandler(address=('localhost', SYSLOG_UDP_PORT), facility=LOG_USER) - - Returns a new instance of the :class:`SysLogHandler` class intended to - communicate with a remote Unix machine whose address is given by *address* in - the form of a ``(host, port)`` tuple. If *address* is not specified, - ``('localhost', 514)`` is used. The address is used to open a UDP socket. An - alternative to providing a ``(host, port)`` tuple is providing an address as a - string, for example "/dev/log". In this case, a Unix domain socket is used to - send the message to the syslog. If *facility* is not specified, - :const:`LOG_USER` is used. - - - .. method:: close() - - Closes the socket to the remote host. - - - .. method:: emit(record) - - The record is formatted, and then sent to the syslog server. If exception - information is present, it is *not* sent to the server. - - - .. method:: encodePriority(facility, priority) - - Encodes the facility and priority into an integer. You can pass in strings - or integers - if strings are passed, internal mapping dictionaries are - used to convert them to integers. - - The symbolic ``LOG_`` values are defined in :class:`SysLogHandler` and - mirror the values defined in the ``sys/syslog.h`` header file. - - **Priorities** - - +--------------------------+---------------+ - | Name (string) | Symbolic value| - +==========================+===============+ - | ``alert`` | LOG_ALERT | - +--------------------------+---------------+ - | ``crit`` or ``critical`` | LOG_CRIT | - +--------------------------+---------------+ - | ``debug`` | LOG_DEBUG | - +--------------------------+---------------+ - | ``emerg`` or ``panic`` | LOG_EMERG | - +--------------------------+---------------+ - | ``err`` or ``error`` | LOG_ERR | - +--------------------------+---------------+ - | ``info`` | LOG_INFO | - +--------------------------+---------------+ - | ``notice`` | LOG_NOTICE | - +--------------------------+---------------+ - | ``warn`` or ``warning`` | LOG_WARNING | - +--------------------------+---------------+ - - **Facilities** - - +---------------+---------------+ - | Name (string) | Symbolic value| - +===============+===============+ - | ``auth`` | LOG_AUTH | - +---------------+---------------+ - | ``authpriv`` | LOG_AUTHPRIV | - +---------------+---------------+ - | ``cron`` | LOG_CRON | - +---------------+---------------+ - | ``daemon`` | LOG_DAEMON | - +---------------+---------------+ - | ``ftp`` | LOG_FTP | - +---------------+---------------+ - | ``kern`` | LOG_KERN | - +---------------+---------------+ - | ``lpr`` | LOG_LPR | - +---------------+---------------+ - | ``mail`` | LOG_MAIL | - +---------------+---------------+ - | ``news`` | LOG_NEWS | - +---------------+---------------+ - | ``syslog`` | LOG_SYSLOG | - +---------------+---------------+ - | ``user`` | LOG_USER | - +---------------+---------------+ - | ``uucp`` | LOG_UUCP | - +---------------+---------------+ - | ``local0`` | LOG_LOCAL0 | - +---------------+---------------+ - | ``local1`` | LOG_LOCAL1 | - +---------------+---------------+ - | ``local2`` | LOG_LOCAL2 | - +---------------+---------------+ - | ``local3`` | LOG_LOCAL3 | - +---------------+---------------+ - | ``local4`` | LOG_LOCAL4 | - +---------------+---------------+ - | ``local5`` | LOG_LOCAL5 | - +---------------+---------------+ - | ``local6`` | LOG_LOCAL6 | - +---------------+---------------+ - | ``local7`` | LOG_LOCAL7 | - +---------------+---------------+ - - .. method:: mapPriority(levelname) - - Maps a logging level name to a syslog priority name. - You may need to override this if you are using custom levels, or - if the default algorithm is not suitable for your needs. The - default algorithm maps ``DEBUG``, ``INFO``, ``WARNING``, ``ERROR`` and - ``CRITICAL`` to the equivalent syslog names, and all other level - names to "warning". - -.. _nt-eventlog-handler: - -NTEventLogHandler -^^^^^^^^^^^^^^^^^ - -The :class:`NTEventLogHandler` class, located in the :mod:`logging.handlers` -module, supports sending logging messages to a local Windows NT, Windows 2000 or -Windows XP event log. Before you can use it, you need Mark Hammond's Win32 -extensions for Python installed. - - -.. class:: NTEventLogHandler(appname, dllname=None, logtype='Application') - - Returns a new instance of the :class:`NTEventLogHandler` class. The *appname* is - used to define the application name as it appears in the event log. An - appropriate registry entry is created using this name. The *dllname* should give - the fully qualified pathname of a .dll or .exe which contains message - definitions to hold in the log (if not specified, ``'win32service.pyd'`` is used - - this is installed with the Win32 extensions and contains some basic - placeholder message definitions. Note that use of these placeholders will make - your event logs big, as the entire message source is held in the log. If you - want slimmer logs, you have to pass in the name of your own .dll or .exe which - contains the message definitions you want to use in the event log). The - *logtype* is one of ``'Application'``, ``'System'`` or ``'Security'``, and - defaults to ``'Application'``. - - - .. method:: close() - - At this point, you can remove the application name from the registry as a - source of event log entries. However, if you do this, you will not be able - to see the events as you intended in the Event Log Viewer - it needs to be - able to access the registry to get the .dll name. The current version does - not do this. - - - .. method:: emit(record) - - Determines the message ID, event category and event type, and then logs - the message in the NT event log. - - - .. method:: getEventCategory(record) - - Returns the event category for the record. Override this if you want to - specify your own categories. This version returns 0. - - - .. method:: getEventType(record) - - Returns the event type for the record. Override this if you want to - specify your own types. This version does a mapping using the handler's - typemap attribute, which is set up in :meth:`__init__` to a dictionary - which contains mappings for :const:`DEBUG`, :const:`INFO`, - :const:`WARNING`, :const:`ERROR` and :const:`CRITICAL`. If you are using - your own levels, you will either need to override this method or place a - suitable dictionary in the handler's *typemap* attribute. - - - .. method:: getMessageID(record) - - Returns the message ID for the record. If you are using your own messages, - you could do this by having the *msg* passed to the logger being an ID - rather than a format string. Then, in here, you could use a dictionary - lookup to get the message ID. This version returns 1, which is the base - message ID in :file:`win32service.pyd`. - -.. _smtp-handler: - -SMTPHandler -^^^^^^^^^^^ - -The :class:`SMTPHandler` class, located in the :mod:`logging.handlers` module, -supports sending logging messages to an email address via SMTP. - - -.. class:: SMTPHandler(mailhost, fromaddr, toaddrs, subject, credentials=None) - - Returns a new instance of the :class:`SMTPHandler` class. The instance is - initialized with the from and to addresses and subject line of the email. The - *toaddrs* should be a list of strings. To specify a non-standard SMTP port, use - the (host, port) tuple format for the *mailhost* argument. If you use a string, - the standard SMTP port is used. If your SMTP server requires authentication, you - can specify a (username, password) tuple for the *credentials* argument. - - - .. method:: emit(record) - - Formats the record and sends it to the specified addressees. - - - .. method:: getSubject(record) - - If you want to specify a subject line which is record-dependent, override - this method. - -.. _memory-handler: - -MemoryHandler -^^^^^^^^^^^^^ - -The :class:`MemoryHandler` class, located in the :mod:`logging.handlers` module, -supports buffering of logging records in memory, periodically flushing them to a -:dfn:`target` handler. Flushing occurs whenever the buffer is full, or when an -event of a certain severity or greater is seen. - -:class:`MemoryHandler` is a subclass of the more general -:class:`BufferingHandler`, which is an abstract class. This buffers logging -records in memory. Whenever each record is added to the buffer, a check is made -by calling :meth:`shouldFlush` to see if the buffer should be flushed. If it -should, then :meth:`flush` is expected to do the needful. - - -.. class:: BufferingHandler(capacity) - - Initializes the handler with a buffer of the specified capacity. - - - .. method:: emit(record) - - Appends the record to the buffer. If :meth:`shouldFlush` returns true, - calls :meth:`flush` to process the buffer. - - - .. method:: flush() - - You can override this to implement custom flushing behavior. This version - just zaps the buffer to empty. - - - .. method:: shouldFlush(record) - - Returns true if the buffer is up to capacity. This method can be - overridden to implement custom flushing strategies. - - -.. class:: MemoryHandler(capacity, flushLevel=ERROR, target=None) - - Returns a new instance of the :class:`MemoryHandler` class. The instance is - initialized with a buffer size of *capacity*. If *flushLevel* is not specified, - :const:`ERROR` is used. If no *target* is specified, the target will need to be - set using :meth:`setTarget` before this handler does anything useful. - - - .. method:: close() - - Calls :meth:`flush`, sets the target to :const:`None` and clears the - buffer. - - - .. method:: flush() - - For a :class:`MemoryHandler`, flushing means just sending the buffered - records to the target, if there is one. Override if you want different - behavior. - - - .. method:: setTarget(target) - - Sets the target handler for this handler. - - - .. method:: shouldFlush(record) - - Checks for buffer full or a record at the *flushLevel* or higher. - - -.. _http-handler: - -HTTPHandler -^^^^^^^^^^^ - -The :class:`HTTPHandler` class, located in the :mod:`logging.handlers` module, -supports sending logging messages to a Web server, using either ``GET`` or -``POST`` semantics. - - -.. class:: HTTPHandler(host, url, method='GET') - - Returns a new instance of the :class:`HTTPHandler` class. The instance is - initialized with a host address, url and HTTP method. The *host* can be of the - form ``host:port``, should you need to use a specific port number. If no - *method* is specified, ``GET`` is used. - - - .. method:: emit(record) - - Sends the record to the Web server as a percent-encoded dictionary. - +For a list of handlers included as standard, see :mod:`logging.handlers`. .. _formatter-objects: @@ -2394,7 +406,7 @@ Formatter Objects .. currentmodule:: logging -:class:`Formatter`\ s have the following attributes and methods. They are +:class:`Formatter` objects have the following attributes and methods. They are responsible for converting a :class:`LogRecord` to (usually) a string which can be interpreted by either a human or an external system. The base :class:`Formatter` allows a formatting string to be specified. If none is @@ -2407,64 +419,11 @@ into a :class:`LogRecord`'s *message* attribute. This format string contains standard Python %-style mapping keys. See section :ref:`old-string-formatting` for more information on string formatting. -Currently, the useful mapping keys in a :class:`LogRecord` are: - -+-------------------------+-----------------------------------------------+ -| Format | Description | -+=========================+===============================================+ -| ``%(name)s`` | Name of the logger (logging channel). | -+-------------------------+-----------------------------------------------+ -| ``%(levelno)s`` | Numeric logging level for the message | -| | (:const:`DEBUG`, :const:`INFO`, | -| | :const:`WARNING`, :const:`ERROR`, | -| | :const:`CRITICAL`). | -+-------------------------+-----------------------------------------------+ -| ``%(levelname)s`` | Text logging level for the message | -| | (``'DEBUG'``, ``'INFO'``, ``'WARNING'``, | -| | ``'ERROR'``, ``'CRITICAL'``). | -+-------------------------+-----------------------------------------------+ -| ``%(pathname)s`` | Full pathname of the source file where the | -| | logging call was issued (if available). | -+-------------------------+-----------------------------------------------+ -| ``%(filename)s`` | Filename portion of pathname. | -+-------------------------+-----------------------------------------------+ -| ``%(module)s`` | Module (name portion of filename). | -+-------------------------+-----------------------------------------------+ -| ``%(funcName)s`` | Name of function containing the logging call. | -+-------------------------+-----------------------------------------------+ -| ``%(lineno)d`` | Source line number where the logging call was | -| | issued (if available). | -+-------------------------+-----------------------------------------------+ -| ``%(created)f`` | Time when the :class:`LogRecord` was created | -| | (as returned by :func:`time.time`). | -+-------------------------+-----------------------------------------------+ -| ``%(relativeCreated)d`` | Time in milliseconds when the LogRecord was | -| | created, relative to the time the logging | -| | module was loaded. | -+-------------------------+-----------------------------------------------+ -| ``%(asctime)s`` | Human-readable time when the | -| | :class:`LogRecord` was created. By default | -| | this is of the form "2003-07-08 16:49:45,896" | -| | (the numbers after the comma are millisecond | -| | portion of the time). | -+-------------------------+-----------------------------------------------+ -| ``%(msecs)d`` | Millisecond portion of the time when the | -| | :class:`LogRecord` was created. | -+-------------------------+-----------------------------------------------+ -| ``%(thread)d`` | Thread ID (if available). | -+-------------------------+-----------------------------------------------+ -| ``%(threadName)s`` | Thread name (if available). | -+-------------------------+-----------------------------------------------+ -| ``%(process)d`` | Process ID (if available). | -+-------------------------+-----------------------------------------------+ -| ``%(processName)s`` | Process name (if available). | -+-------------------------+-----------------------------------------------+ -| ``%(message)s`` | The logged message, computed as ``msg % | -| | args``. | -+-------------------------+-----------------------------------------------+ - - -.. class:: Formatter(fmt=None, datefmt=None) +The useful mapping keys in a :class:`LogRecord` are given in the section on +:ref:`logrecord-attributes`. + + +.. class:: Formatter(fmt=None, datefmt=None, style='%') Returns a new instance of the :class:`Formatter` class. The instance is initialized with a format string for the message as a whole, as well as a @@ -2472,6 +431,14 @@ Currently, the useful mapping keys in a :class:`LogRecord` are: specified, ``'%(message)s'`` is used. If no *datefmt* is specified, the ISO8601 date format is used. + The *style* parameter can be one of '%', '{' or '$' and determines how + the format string will be merged with its data: using one of %-formatting, + :meth:`str.format` or :class:`string.Template`. + + .. versionchanged:: 3.2 + The *style* parameter was added. + + .. method:: format(record) The record's attribute dictionary is used as the operand to a string @@ -2490,6 +457,9 @@ Currently, the useful mapping keys in a :class:`LogRecord` are: formatter to handle the event doesn't use the cached value but recalculates it afresh. + If stack information is available, it's appended after the exception + information, using :meth:`formatStack` to transform it if necessary. + .. method:: formatTime(record, datefmt=None) @@ -2501,6 +471,13 @@ Currently, the useful mapping keys in a :class:`LogRecord` are: record. Otherwise, the ISO8601 format is used. The resulting string is returned. + This function uses a user-configurable function to convert the creation + time to a tuple. By default, :func:`time.localtime` is used; to change + this for a particular formatter instance, set the ``converter`` attribute + to a function with the same signature as :func:`time.localtime` or + :func:`time.gmtime`. To change it for all formatters, for example if you + want all logging times to be shown in GMT, set the ``converter`` + attribute in the ``Formatter`` class. .. method:: formatException(exc_info) @@ -2509,17 +486,23 @@ Currently, the useful mapping keys in a :class:`LogRecord` are: just uses :func:`traceback.print_exception`. The resulting string is returned. + .. method:: formatStack(stack_info) + + Formats the specified stack information (a string as returned by + :func:`traceback.print_stack`, but with the last newline removed) as a + string. This default implementation just returns the input value. + .. _filter: Filter Objects -------------- -:class:`Filter`\ s can be used by :class:`Handler`\ s and :class:`Logger`\ s for -more sophisticated filtering than is provided by levels. The base filter class -only allows events which are below a certain point in the logger hierarchy. For -example, a filter initialized with "A.B" will allow events logged by loggers -"A.B", "A.B.C", "A.B.C.D", "A.B.D" etc. but not "A.BB", "B.A.B" etc. If -initialized with the empty string, all events are passed. +``Filters`` can be used by ``Handlers`` and ``Loggers`` for more sophisticated +filtering than is provided by levels. The base filter class only allows events +which are below a certain point in the logger hierarchy. For example, a filter +initialized with 'A.B' will allow events logged by loggers 'A.B', 'A.B.C', +'A.B.C.D', 'A.B.D' etc. but not 'A.BB', 'B.A.B' etc. If initialized with the +empty string, all events are passed. .. class:: Filter(name='') @@ -2542,6 +525,28 @@ etc.) This means that events which have been generated by descendant loggers will not be filtered by a logger's filter setting, unless the filter has also been applied to those descendant loggers. +You don't actually need to subclass ``Filter``: you can pass any instance +which has a ``filter`` method with the same semantics. + +.. versionchanged:: 3.2 + You don't need to create specialized ``Filter`` classes, or use other + classes with a ``filter`` method: you can use a function (or other + callable) as a filter. The filtering logic will check to see if the filter + object has a ``filter`` attribute: if it does, it's assumed to be a + ``Filter`` and its :meth:`~Filter.filter` method is called. Otherwise, it's + assumed to be a callable and called with the record as the single + parameter. The returned value should conform to that returned by + :meth:`~Filter.filter`. + +Although filters are used primarily to filter records based on more +sophisticated criteria than levels, they get to see every record which is +processed by the handler or logger they're attached to: this can be useful if +you want to do things like counting how many records were processed by a +particular logger or handler, or adding, changing or removing attributes in +the LogRecord being processed. Obviously changing the LogRecord needs to be +done with some care, but it does allow the injection of contextual information +into logs (see :ref:`filters-contextual`). + .. _log-record: LogRecord Objects @@ -2553,7 +558,7 @@ every time something is logged, and can be created manually via wire). -.. class:: LogRecord(name, lvl, pathname, lineno, msg, args, exc_info, func=None) +.. class:: LogRecord(name, level, pathname, lineno, msg, args, exc_info, func=None, sinfo=None) Contains all the information pertinent to the event being logged. @@ -2561,49 +566,156 @@ wire). are combined using ``msg % args`` to create the :attr:`message` field of the record. - .. attribute:: args - - Tuple of arguments to be used in formatting :attr:`msg`. - - .. attribute:: exc_info - - Exception tuple (à la `sys.exc_info`) or `None` if no exception - information is available. - - .. attribute:: func - - Name of the function of origin (i.e. in which the logging call was made). - - .. attribute:: lineno - - Line number in the source file of origin. - - .. attribute:: lvl - - Numeric logging level. - - .. attribute:: message - - Bound to the result of :meth:`getMessage` when - :meth:`Formatter.format(record)<Formatter.format>` is invoked. - - .. attribute:: msg - - User-supplied :ref:`format string<string-formatting>` or arbitrary object - (see :ref:`arbitrary-object-messages`) used in :meth:`getMessage`. - - .. attribute:: name - - Name of the logger that emitted the record. - - .. attribute:: pathname - - Absolute pathname of the source file of origin. + :param name: The name of the logger used to log the event represented by + this LogRecord. + :param level: The numeric level of the logging event (one of DEBUG, INFO etc.) + Note that this is converted to *two* attributes of the LogRecord: + ``levelno`` for the numeric value and ``levelname`` for the + corresponding level name. + :param pathname: The full pathname of the source file where the logging call + was made. + :param lineno: The line number in the source file where the logging call was + made. + :param msg: The event description message, possibly a format string with + placeholders for variable data. + :param args: Variable data to merge into the *msg* argument to obtain the + event description. + :param exc_info: An exception tuple with the current exception information, + or *None* if no exception information is available. + :param func: The name of the function or method from which the logging call + was invoked. + :param sinfo: A text string representing stack information from the base of + the stack in the current thread, up to the logging call. .. method:: getMessage() Returns the message for this :class:`LogRecord` instance after merging any - user-supplied arguments with the message. + user-supplied arguments with the message. If the user-supplied message + argument to the logging call is not a string, :func:`str` is called on it to + convert it to a string. This allows use of user-defined classes as + messages, whose ``__str__`` method can return the actual format string to + be used. + + .. versionchanged:: 3.2 + The creation of a ``LogRecord`` has been made more configurable by + providing a factory which is used to create the record. The factory can be + set using :func:`getLogRecordFactory` and :func:`setLogRecordFactory` + (see this for the factory's signature). + + This functionality can be used to inject your own values into a + LogRecord at creation time. You can use the following pattern:: + + old_factory = logging.getLogRecordFactory() + + def record_factory(*args, **kwargs): + record = old_factory(*args, **kwargs) + record.custom_attribute = 0xdecafbad + return record + + logging.setLogRecordFactory(record_factory) + + With this pattern, multiple factories could be chained, and as long + as they don't overwrite each other's attributes or unintentionally + overwrite the standard attributes listed above, there should be no + surprises. + + +.. _logrecord-attributes: + +LogRecord attributes +-------------------- + +The LogRecord has a number of attributes, most of which are derived from the +parameters to the constructor. (Note that the names do not always correspond +exactly between the LogRecord constructor parameters and the LogRecord +attributes.) These attributes can be used to merge data from the record into +the format string. The following table lists (in alphabetical order) the +attribute names, their meanings and the corresponding placeholder in a %-style +format string. + +If you are using {}-formatting (:func:`str.format`), you can use +``{attrname}`` as the placeholder in the format string. If you are using +$-formatting (:class:`string.Template`), use the form ``${attrname}``. In +both cases, of course, replace ``attrname`` with the actual attribute name +you want to use. + +In the case of {}-formatting, you can specify formatting flags by placing them +after the attribute name, separated from it with a colon. For example: a +placeholder of ``{msecs:03d}`` would format a millisecond value of ``4`` as +``004``. Refer to the :meth:`str.format` documentation for full details on +the options available to you. + ++----------------+-------------------------+-----------------------------------------------+ +| Attribute name | Format | Description | ++================+=========================+===============================================+ +| args | You shouldn't need to | The tuple of arguments merged into ``msg`` to | +| | format this yourself. | produce ``message``. | ++----------------+-------------------------+-----------------------------------------------+ +| asctime | ``%(asctime)s`` | Human-readable time when the | +| | | :class:`LogRecord` was created. By default | +| | | this is of the form '2003-07-08 16:49:45,896' | +| | | (the numbers after the comma are millisecond | +| | | portion of the time). | ++----------------+-------------------------+-----------------------------------------------+ +| created | ``%(created)f`` | Time when the :class:`LogRecord` was created | +| | | (as returned by :func:`time.time`). | ++----------------+-------------------------+-----------------------------------------------+ +| exc_info | You shouldn't need to | Exception tuple (à la ``sys.exc_info``) or, | +| | format this yourself. | if no exception has occurred, *None*. | ++----------------+-------------------------+-----------------------------------------------+ +| filename | ``%(filename)s`` | Filename portion of ``pathname``. | ++----------------+-------------------------+-----------------------------------------------+ +| funcName | ``%(funcName)s`` | Name of function containing the logging call. | ++----------------+-------------------------+-----------------------------------------------+ +| levelname | ``%(levelname)s`` | Text logging level for the message | +| | | (``'DEBUG'``, ``'INFO'``, ``'WARNING'``, | +| | | ``'ERROR'``, ``'CRITICAL'``). | ++----------------+-------------------------+-----------------------------------------------+ +| levelno | ``%(levelno)s`` | Numeric logging level for the message | +| | | (:const:`DEBUG`, :const:`INFO`, | +| | | :const:`WARNING`, :const:`ERROR`, | +| | | :const:`CRITICAL`). | ++----------------+-------------------------+-----------------------------------------------+ +| lineno | ``%(lineno)d`` | Source line number where the logging call was | +| | | issued (if available). | ++----------------+-------------------------+-----------------------------------------------+ +| module | ``%(module)s`` | Module (name portion of ``filename``). | ++----------------+-------------------------+-----------------------------------------------+ +| msecs | ``%(msecs)d`` | Millisecond portion of the time when the | +| | | :class:`LogRecord` was created. | ++----------------+-------------------------+-----------------------------------------------+ +| message | ``%(message)s`` | The logged message, computed as ``msg % | +| | | args``. This is set when | +| | | :meth:`Formatter.format` is invoked. | ++----------------+-------------------------+-----------------------------------------------+ +| msg | You shouldn't need to | The format string passed in the original | +| | format this yourself. | logging call. Merged with ``args`` to | +| | | produce ``message``, or an arbitrary object | +| | | (see :ref:`arbitrary-object-messages`). | ++----------------+-------------------------+-----------------------------------------------+ +| name | ``%(name)s`` | Name of the logger used to log the call. | ++----------------+-------------------------+-----------------------------------------------+ +| pathname | ``%(pathname)s`` | Full pathname of the source file where the | +| | | logging call was issued (if available). | ++----------------+-------------------------+-----------------------------------------------+ +| process | ``%(process)d`` | Process ID (if available). | ++----------------+-------------------------+-----------------------------------------------+ +| processName | ``%(processName)s`` | Process name (if available). | ++----------------+-------------------------+-----------------------------------------------+ +| relativeCreated| ``%(relativeCreated)d`` | Time in milliseconds when the LogRecord was | +| | | created, relative to the time the logging | +| | | module was loaded. | ++----------------+-------------------------+-----------------------------------------------+ +| stack_info | You shouldn't need to | Stack frame information (where available) | +| | format this yourself. | from the bottom of the stack in the current | +| | | thread, up to and including the stack frame | +| | | of the logging call which resulted in the | +| | | creation of this record. | ++----------------+-------------------------+-----------------------------------------------+ +| thread | ``%(thread)d`` | Thread ID (if available). | ++----------------+-------------------------+-----------------------------------------------+ +| threadName | ``%(threadName)s`` | Thread name (if available). | ++----------------+-------------------------+-----------------------------------------------+ .. _logger-adapter: @@ -2615,7 +727,6 @@ LoggerAdapter Objects information into logging calls. For a usage example , see the section on :ref:`adding contextual information to your logging output <context-info>`. - .. class:: LoggerAdapter(logger, extra) Returns an instance of :class:`LoggerAdapter` initialized with an @@ -2629,14 +740,18 @@ information into logging calls. For a usage example , see the section on 'extra'. The return value is a (*msg*, *kwargs*) tuple which has the (possibly modified) versions of the arguments passed in. -In addition to the above, :class:`LoggerAdapter` supports all the logging +In addition to the above, :class:`LoggerAdapter` supports the following methods of :class:`Logger`, i.e. :meth:`debug`, :meth:`info`, :meth:`warning`, -:meth:`error`, :meth:`exception`, :meth:`critical` and :meth:`log`. These -methods have the same signatures as their counterparts in :class:`Logger`, so -you can use the two types of instances interchangeably. +:meth:`error`, :meth:`exception`, :meth:`critical`, :meth:`log`, +:meth:`isEnabledFor`, :meth:`getEffectiveLevel`, :meth:`setLevel`, +:meth:`hasHandlers`. These methods have the same signatures as their +counterparts in :class:`Logger`, so you can use the two types of instances +interchangeably. - The :meth:`isEnabledFor` method was added to :class:`LoggerAdapter`. This - method delegates to the underlying logger. +.. versionchanged:: 3.2 + The :meth:`isEnabledFor`, :meth:`getEffectiveLevel`, :meth:`setLevel` and + :meth:`hasHandlers` methods were added to :class:`LoggerAdapter`. These + methods delegate to the underlying logger. Thread Safety @@ -2652,401 +767,338 @@ module, you may not be able to use logging from within such handlers. This is because lock implementations in the :mod:`threading` module are not always re-entrant, and so cannot be invoked from such signal handlers. -Configuration -------------- +Module-Level Functions +---------------------- -.. _logging-config-api: +In addition to the classes described above, there are a number of module- level +functions. -Configuration functions -^^^^^^^^^^^^^^^^^^^^^^^ -The following functions configure the logging module. They are located in the -:mod:`logging.config` module. Their use is optional --- you can configure the -logging module using these functions or by making calls to the main API (defined -in :mod:`logging` itself) and defining handlers which are declared either in -:mod:`logging` or :mod:`logging.handlers`. +.. function:: getLogger(name=None) + Return a logger with the specified name or, if name is ``None``, return a + logger which is the root logger of the hierarchy. If specified, the name is + typically a dot-separated hierarchical name like *'a'*, *'a.b'* or *'a.b.c.d'*. + Choice of these names is entirely up to the developer who is using logging. -.. function:: fileConfig(fname, defaults=None, disable_existing_loggers=True) + All calls to this function with a given name return the same logger instance. + This means that logger instances never need to be passed between different parts + of an application. - Reads the logging configuration from a :mod:`configparser`\-format file named - *fname*. This function can be called several times from an application, - allowing an end user the ability to select from various pre-canned - configurations (if the developer provides a mechanism to present the choices - and load the chosen configuration). Defaults to be passed to the ConfigParser - can be specified in the *defaults* argument. - If *disable_existing_loggers* is true, any existing loggers that are not - children of named loggers will be disabled. +.. function:: getLoggerClass() + Return either the standard :class:`Logger` class, or the last class passed to + :func:`setLoggerClass`. This function may be called from within a new class + definition, to ensure that installing a customised :class:`Logger` class will + not undo customisations already applied by other code. For example:: -.. function:: listen(port=DEFAULT_LOGGING_CONFIG_PORT) + class MyLogger(logging.getLoggerClass()): + # ... override behaviour here - Starts up a socket server on the specified port, and listens for new - configurations. If no port is specified, the module's default - :const:`DEFAULT_LOGGING_CONFIG_PORT` is used. Logging configurations will be - sent as a file suitable for processing by :func:`fileConfig`. Returns a - :class:`Thread` instance on which you can call :meth:`start` to start the - server, and which you can :meth:`join` when appropriate. To stop the server, - call :func:`stopListening`. - To send a configuration to the socket, read in the configuration file and - send it to the socket as a string of bytes preceded by a four-byte length - string packed in binary using ``struct.pack('>L', n)``. +.. function:: getLogRecordFactory() + Return a callable which is used to create a :class:`LogRecord`. -.. function:: stopListening() + .. versionadded:: 3.2 + This function has been provided, along with :func:`setLogRecordFactory`, + to allow developers more control over how the :class:`LogRecord` + representing a logging event is constructed. - Stops the listening server which was created with a call to :func:`listen`. - This is typically called before calling :meth:`join` on the return value from - :func:`listen`. + See :func:`setLogRecordFactory` for more information about the how the + factory is called. +.. function:: debug(msg, *args, **kwargs) -.. _logging-config-fileformat: + Logs a message with level :const:`DEBUG` on the root logger. The *msg* is the + message format string, and the *args* are the arguments which are merged into + *msg* using the string formatting operator. (Note that this means that you can + use keywords in the format string, together with a single dictionary argument.) -Configuration file format -^^^^^^^^^^^^^^^^^^^^^^^^^ + There are three keyword arguments in *kwargs* which are inspected: *exc_info* + which, if it does not evaluate as false, causes exception information to be + added to the logging message. If an exception tuple (in the format returned by + :func:`sys.exc_info`) is provided, it is used; otherwise, :func:`sys.exc_info` + is called to get the exception information. -The configuration file format understood by :func:`fileConfig` is based on -:mod:`configparser` functionality. The file must contain sections called -``[loggers]``, ``[handlers]`` and ``[formatters]`` which identify by name the -entities of each type which are defined in the file. For each such entity, there -is a separate section which identifies how that entity is configured. Thus, for -a logger named ``log01`` in the ``[loggers]`` section, the relevant -configuration details are held in a section ``[logger_log01]``. Similarly, a -handler called ``hand01`` in the ``[handlers]`` section will have its -configuration held in a section called ``[handler_hand01]``, while a formatter -called ``form01`` in the ``[formatters]`` section will have its configuration -specified in a section called ``[formatter_form01]``. The root logger -configuration must be specified in a section called ``[logger_root]``. + The second optional keyword argument is *stack_info*, which defaults to + False. If specified as True, stack information is added to the logging + message, including the actual logging call. Note that this is not the same + stack information as that displayed through specifying *exc_info*: The + former is stack frames from the bottom of the stack up to the logging call + in the current thread, whereas the latter is information about stack frames + which have been unwound, following an exception, while searching for + exception handlers. + + You can specify *stack_info* independently of *exc_info*, e.g. to just show + how you got to a certain point in your code, even when no exceptions were + raised. The stack frames are printed following a header line which says:: + + Stack (most recent call last): + + This mimics the ``Traceback (most recent call last):`` which is used when + displaying exception frames. + + The third optional keyword argument is *extra* which can be used to pass a + dictionary which is used to populate the __dict__ of the LogRecord created for + the logging event with user-defined attributes. These custom attributes can then + be used as you like. For example, they could be incorporated into logged + messages. For example:: + + FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' + logging.basicConfig(format=FORMAT) + d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} + logging.warning('Protocol problem: %s', 'connection reset', extra=d) + + would print something like:: + + 2006-02-08 22:20:02,165 192.168.0.1 fbloggs Protocol problem: connection reset + + The keys in the dictionary passed in *extra* should not clash with the keys used + by the logging system. (See the :class:`Formatter` documentation for more + information on which keys are used by the logging system.) + + If you choose to use these attributes in logged messages, you need to exercise + some care. In the above example, for instance, the :class:`Formatter` has been + set up with a format string which expects 'clientip' and 'user' in the attribute + dictionary of the LogRecord. If these are missing, the message will not be + logged because a string formatting exception will occur. So in this case, you + always need to pass the *extra* dictionary with these keys. + + While this might be annoying, this feature is intended for use in specialized + circumstances, such as multi-threaded servers where the same code executes in + many contexts, and interesting conditions which arise are dependent on this + context (such as remote client IP address and authenticated user name, in the + above example). In such circumstances, it is likely that specialized + :class:`Formatter`\ s would be used with particular :class:`Handler`\ s. + + .. versionadded:: 3.2 + The *stack_info* parameter was added. + +.. function:: info(msg, *args, **kwargs) + + Logs a message with level :const:`INFO` on the root logger. The arguments are + interpreted as for :func:`debug`. + + +.. function:: warning(msg, *args, **kwargs) + + Logs a message with level :const:`WARNING` on the root logger. The arguments are + interpreted as for :func:`debug`. + + +.. function:: error(msg, *args, **kwargs) + + Logs a message with level :const:`ERROR` on the root logger. The arguments are + interpreted as for :func:`debug`. + + +.. function:: critical(msg, *args, **kwargs) + + Logs a message with level :const:`CRITICAL` on the root logger. The arguments + are interpreted as for :func:`debug`. + + +.. function:: exception(msg, *args) + + Logs a message with level :const:`ERROR` on the root logger. The arguments are + interpreted as for :func:`debug`. Exception info is added to the logging + message. This function should only be called from an exception handler. + +.. function:: log(level, msg, *args, **kwargs) + + Logs a message with level *level* on the root logger. The other arguments are + interpreted as for :func:`debug`. + + PLEASE NOTE: The above module-level functions which delegate to the root + logger should *not* be used in threads, in versions of Python earlier than + 2.7.1 and 3.2, unless at least one handler has been added to the root + logger *before* the threads are started. These convenience functions call + :func:`basicConfig` to ensure that at least one handler is available; in + earlier versions of Python, this can (under rare circumstances) lead to + handlers being added multiple times to the root logger, which can in turn + lead to multiple messages for the same event. + +.. function:: disable(lvl) + + Provides an overriding level *lvl* for all loggers which takes precedence over + the logger's own level. When the need arises to temporarily throttle logging + output down across the whole application, this function can be useful. Its + effect is to disable all logging calls of severity *lvl* and below, so that + if you call it with a value of INFO, then all INFO and DEBUG events would be + discarded, whereas those of severity WARNING and above would be processed + according to the logger's effective level. + + +.. function:: addLevelName(lvl, levelName) + + Associates level *lvl* with text *levelName* in an internal dictionary, which is + used to map numeric levels to a textual representation, for example when a + :class:`Formatter` formats a message. This function can also be used to define + your own levels. The only constraints are that all levels used must be + registered using this function, levels should be positive integers and they + should increase in increasing order of severity. + + NOTE: If you are thinking of defining your own levels, please see the section + on :ref:`custom-levels`. + +.. function:: getLevelName(lvl) + + Returns the textual representation of logging level *lvl*. If the level is one + of the predefined levels :const:`CRITICAL`, :const:`ERROR`, :const:`WARNING`, + :const:`INFO` or :const:`DEBUG` then you get the corresponding string. If you + have associated levels with names using :func:`addLevelName` then the name you + have associated with *lvl* is returned. If a numeric value corresponding to one + of the defined levels is passed in, the corresponding string representation is + returned. Otherwise, the string 'Level %s' % lvl is returned. -Examples of these sections in the file are given below. :: - - [loggers] - keys=root,log02,log03,log04,log05,log06,log07 - - [handlers] - keys=hand01,hand02,hand03,hand04,hand05,hand06,hand07,hand08,hand09 - - [formatters] - keys=form01,form02,form03,form04,form05,form06,form07,form08,form09 - -The root logger must specify a level and a list of handlers. An example of a -root logger section is given below. :: - - [logger_root] - level=NOTSET - handlers=hand01 - -The ``level`` entry can be one of ``DEBUG, INFO, WARNING, ERROR, CRITICAL`` or -``NOTSET``. For the root logger only, ``NOTSET`` means that all messages will be -logged. Level values are :func:`eval`\ uated in the context of the ``logging`` -package's namespace. - -The ``handlers`` entry is a comma-separated list of handler names, which must -appear in the ``[handlers]`` section. These names must appear in the -``[handlers]`` section and have corresponding sections in the configuration -file. - -For loggers other than the root logger, some additional information is required. -This is illustrated by the following example. :: - - [logger_parser] - level=DEBUG - handlers=hand01 - propagate=1 - qualname=compiler.parser - -The ``level`` and ``handlers`` entries are interpreted as for the root logger, -except that if a non-root logger's level is specified as ``NOTSET``, the system -consults loggers higher up the hierarchy to determine the effective level of the -logger. The ``propagate`` entry is set to 1 to indicate that messages must -propagate to handlers higher up the logger hierarchy from this logger, or 0 to -indicate that messages are **not** propagated to handlers up the hierarchy. The -``qualname`` entry is the hierarchical channel name of the logger, that is to -say the name used by the application to get the logger. - -Sections which specify handler configuration are exemplified by the following. -:: - - [handler_hand01] - class=StreamHandler - level=NOTSET - formatter=form01 - args=(sys.stdout,) - -The ``class`` entry indicates the handler's class (as determined by :func:`eval` -in the ``logging`` package's namespace). The ``level`` is interpreted as for -loggers, and ``NOTSET`` is taken to mean "log everything". - -The ``formatter`` entry indicates the key name of the formatter for this -handler. If blank, a default formatter (``logging._defaultFormatter``) is used. -If a name is specified, it must appear in the ``[formatters]`` section and have -a corresponding section in the configuration file. - -The ``args`` entry, when :func:`eval`\ uated in the context of the ``logging`` -package's namespace, is the list of arguments to the constructor for the handler -class. Refer to the constructors for the relevant handlers, or to the examples -below, to see how typical entries are constructed. :: - - [handler_hand02] - class=FileHandler - level=DEBUG - formatter=form02 - args=('python.log', 'w') - - [handler_hand03] - class=handlers.SocketHandler - level=INFO - formatter=form03 - args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT) - - [handler_hand04] - class=handlers.DatagramHandler - level=WARN - formatter=form04 - args=('localhost', handlers.DEFAULT_UDP_LOGGING_PORT) - - [handler_hand05] - class=handlers.SysLogHandler - level=ERROR - formatter=form05 - args=(('localhost', handlers.SYSLOG_UDP_PORT), handlers.SysLogHandler.LOG_USER) - - [handler_hand06] - class=handlers.NTEventLogHandler - level=CRITICAL - formatter=form06 - args=('Python Application', '', 'Application') - - [handler_hand07] - class=handlers.SMTPHandler - level=WARN - formatter=form07 - args=('localhost', 'from@abc', ['user1@abc', 'user2@xyz'], 'Logger Subject') - - [handler_hand08] - class=handlers.MemoryHandler - level=NOTSET - formatter=form08 - target= - args=(10, ERROR) - - [handler_hand09] - class=handlers.HTTPHandler - level=NOTSET - formatter=form09 - args=('localhost:9022', '/log', 'GET') - -Sections which specify formatter configuration are typified by the following. :: - - [formatter_form01] - format=F1 %(asctime)s %(levelname)s %(message)s - datefmt= - class=logging.Formatter - -The ``format`` entry is the overall format string, and the ``datefmt`` entry is -the :func:`strftime`\ -compatible date/time format string. If empty, the -package substitutes ISO8601 format date/times, which is almost equivalent to -specifying the date format string ``"%Y-%m-%d %H:%M:%S"``. The ISO8601 format -also specifies milliseconds, which are appended to the result of using the above -format string, with a comma separator. An example time in ISO8601 format is -``2003-01-23 00:29:50,411``. - -The ``class`` entry is optional. It indicates the name of the formatter's class -(as a dotted module and class name.) This option is useful for instantiating a -:class:`Formatter` subclass. Subclasses of :class:`Formatter` can present -exception tracebacks in an expanded or condensed format. - - -Configuration server example -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Here is an example of a module using the logging configuration server:: - - import logging - import logging.config - import time - import os - - # read initial config file - logging.config.fileConfig("logging.conf") - - # create and start listener on port 9999 - t = logging.config.listen(9999) - t.start() - - logger = logging.getLogger("simpleExample") - - try: - # loop through logging calls to see the difference - # new configurations make, until Ctrl+C is pressed - while True: - logger.debug("debug message") - logger.info("info message") - logger.warn("warn message") - logger.error("error message") - logger.critical("critical message") - time.sleep(5) - except KeyboardInterrupt: - # cleanup - logging.config.stopListening() - t.join() - -And here is a script that takes a filename and sends that file to the server, -properly preceded with the binary-encoded length, as the new logging -configuration:: - - #!/usr/bin/env python - import socket, sys, struct - - data_to_send = open(sys.argv[1], "r").read() - - HOST = 'localhost' - PORT = 9999 - s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - print("connecting...") - s.connect((HOST, PORT)) - print("sending config...") - s.send(struct.pack(">L", len(data_to_send))) - s.send(data_to_send) - s.close() - print("complete") - - -More examples -------------- -Multiple handlers and formatters -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Loggers are plain Python objects. The :func:`addHandler` method has no minimum -or maximum quota for the number of handlers you may add. Sometimes it will be -beneficial for an application to log all messages of all severities to a text -file while simultaneously logging errors or above to the console. To set this -up, simply configure the appropriate handlers. The logging calls in the -application code will remain unchanged. Here is a slight modification to the -previous simple module-based configuration example:: - - import logging - - logger = logging.getLogger("simple_example") - logger.setLevel(logging.DEBUG) - # create file handler which logs even debug messages - fh = logging.FileHandler("spam.log") - fh.setLevel(logging.DEBUG) - # create console handler with a higher log level - ch = logging.StreamHandler() - ch.setLevel(logging.ERROR) - # create formatter and add it to the handlers - formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") - ch.setFormatter(formatter) - fh.setFormatter(formatter) - # add the handlers to logger - logger.addHandler(ch) - logger.addHandler(fh) - - # "application" code - logger.debug("debug message") - logger.info("info message") - logger.warn("warn message") - logger.error("error message") - logger.critical("critical message") - -Notice that the "application" code does not care about multiple handlers. All -that changed was the addition and configuration of a new handler named *fh*. - -The ability to create new handlers with higher- or lower-severity filters can be -very helpful when writing and testing an application. Instead of using many -``print`` statements for debugging, use ``logger.debug``: Unlike the print -statements, which you will have to delete or comment out later, the logger.debug -statements can remain intact in the source code and remain dormant until you -need them again. At that time, the only change that needs to happen is to -modify the severity level of the logger and/or handler to debug. - - -Using logging in multiple modules -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -It was mentioned above that multiple calls to -``logging.getLogger('someLogger')`` return a reference to the same logger -object. This is true not only within the same module, but also across modules -as long as it is in the same Python interpreter process. It is true for -references to the same object; additionally, application code can define and -configure a parent logger in one module and create (but not configure) a child -logger in a separate module, and all logger calls to the child will pass up to -the parent. Here is a main module:: - - import logging - import auxiliary_module - - # create logger with "spam_application" - logger = logging.getLogger("spam_application") - logger.setLevel(logging.DEBUG) - # create file handler which logs even debug messages - fh = logging.FileHandler("spam.log") - fh.setLevel(logging.DEBUG) - # create console handler with a higher log level - ch = logging.StreamHandler() - ch.setLevel(logging.ERROR) - # create formatter and add it to the handlers - formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") - fh.setFormatter(formatter) - ch.setFormatter(formatter) - # add the handlers to the logger - logger.addHandler(fh) - logger.addHandler(ch) - - logger.info("creating an instance of auxiliary_module.Auxiliary") - a = auxiliary_module.Auxiliary() - logger.info("created an instance of auxiliary_module.Auxiliary") - logger.info("calling auxiliary_module.Auxiliary.do_something") - a.do_something() - logger.info("finished auxiliary_module.Auxiliary.do_something") - logger.info("calling auxiliary_module.some_function()") - auxiliary_module.some_function() - logger.info("done with auxiliary_module.some_function()") - -Here is the auxiliary module:: - - import logging - - # create logger - module_logger = logging.getLogger("spam_application.auxiliary") - - class Auxiliary: - def __init__(self): - self.logger = logging.getLogger("spam_application.auxiliary.Auxiliary") - self.logger.info("creating an instance of Auxiliary") - def do_something(self): - self.logger.info("doing something") - a = 1 + 1 - self.logger.info("done doing something") - - def some_function(): - module_logger.info("received a call to \"some_function\"") - -The output looks like this:: - - 2005-03-23 23:47:11,663 - spam_application - INFO - - creating an instance of auxiliary_module.Auxiliary - 2005-03-23 23:47:11,665 - spam_application.auxiliary.Auxiliary - INFO - - creating an instance of Auxiliary - 2005-03-23 23:47:11,665 - spam_application - INFO - - created an instance of auxiliary_module.Auxiliary - 2005-03-23 23:47:11,668 - spam_application - INFO - - calling auxiliary_module.Auxiliary.do_something - 2005-03-23 23:47:11,668 - spam_application.auxiliary.Auxiliary - INFO - - doing something - 2005-03-23 23:47:11,669 - spam_application.auxiliary.Auxiliary - INFO - - done doing something - 2005-03-23 23:47:11,670 - spam_application - INFO - - finished auxiliary_module.Auxiliary.do_something - 2005-03-23 23:47:11,671 - spam_application - INFO - - calling auxiliary_module.some_function() - 2005-03-23 23:47:11,672 - spam_application.auxiliary - INFO - - received a call to "some_function" - 2005-03-23 23:47:11,673 - spam_application - INFO - - done with auxiliary_module.some_function() +.. function:: makeLogRecord(attrdict) + + Creates and returns a new :class:`LogRecord` instance whose attributes are + defined by *attrdict*. This function is useful for taking a pickled + :class:`LogRecord` attribute dictionary, sent over a socket, and reconstituting + it as a :class:`LogRecord` instance at the receiving end. + + +.. function:: basicConfig(**kwargs) + + Does basic configuration for the logging system by creating a + :class:`StreamHandler` with a default :class:`Formatter` and adding it to the + root logger. The functions :func:`debug`, :func:`info`, :func:`warning`, + :func:`error` and :func:`critical` will call :func:`basicConfig` automatically + if no handlers are defined for the root logger. + + This function does nothing if the root logger already has handlers + configured for it. + + PLEASE NOTE: This function should be called from the main thread + before other threads are started. In versions of Python prior to + 2.7.1 and 3.2, if this function is called from multiple threads, + it is possible (in rare circumstances) that a handler will be added + to the root logger more than once, leading to unexpected results + such as messages being duplicated in the log. + + The following keyword arguments are supported. + + +--------------+---------------------------------------------+ + | Format | Description | + +==============+=============================================+ + | ``filename`` | Specifies that a FileHandler be created, | + | | using the specified filename, rather than a | + | | StreamHandler. | + +--------------+---------------------------------------------+ + | ``filemode`` | Specifies the mode to open the file, if | + | | filename is specified (if filemode is | + | | unspecified, it defaults to 'a'). | + +--------------+---------------------------------------------+ + | ``format`` | Use the specified format string for the | + | | handler. | + +--------------+---------------------------------------------+ + | ``datefmt`` | Use the specified date/time format. | + +--------------+---------------------------------------------+ + | ``style`` | If ``format`` is specified, use this style | + | | for the format string. One of '%', '{' or | + | | '$' for %-formatting, :meth:`str.format` or | + | | :class:`string.Template` respectively, and | + | | defaulting to '%' if not specified. | + +--------------+---------------------------------------------+ + | ``level`` | Set the root logger level to the specified | + | | level. | + +--------------+---------------------------------------------+ + | ``stream`` | Use the specified stream to initialize the | + | | StreamHandler. Note that this argument is | + | | incompatible with 'filename' - if both are | + | | present, 'stream' is ignored. | + +--------------+---------------------------------------------+ + + .. versionchanged:: 3.2 + The ``style`` argument was added. + + +.. function:: shutdown() + + Informs the logging system to perform an orderly shutdown by flushing and + closing all handlers. This should be called at application exit and no + further use of the logging system should be made after this call. + + +.. function:: setLoggerClass(klass) + + Tells the logging system to use the class *klass* when instantiating a logger. + The class should define :meth:`__init__` such that only a name argument is + required, and the :meth:`__init__` should call :meth:`Logger.__init__`. This + function is typically called before any loggers are instantiated by applications + which need to use custom logger behavior. + + +.. function:: setLogRecordFactory(factory) + + Set a callable which is used to create a :class:`LogRecord`. + + :param factory: The factory callable to be used to instantiate a log record. + + .. versionadded:: 3.2 + This function has been provided, along with :func:`getLogRecordFactory`, to + allow developers more control over how the :class:`LogRecord` representing + a logging event is constructed. + + The factory has the following signature: + + ``factory(name, level, fn, lno, msg, args, exc_info, func=None, sinfo=None, **kwargs)`` + + :name: The logger name. + :level: The logging level (numeric). + :fn: The full pathname of the file where the logging call was made. + :lno: The line number in the file where the logging call was made. + :msg: The logging message. + :args: The arguments for the logging message. + :exc_info: An exception tuple, or None. + :func: The name of the function or method which invoked the logging + call. + :sinfo: A stack traceback such as is provided by + :func:`traceback.print_stack`, showing the call hierarchy. + :kwargs: Additional keyword arguments. + + +Integration with the warnings module +------------------------------------ + +The :func:`captureWarnings` function can be used to integrate :mod:`logging` +with the :mod:`warnings` module. + +.. function:: captureWarnings(capture) + + This function is used to turn the capture of warnings by logging on and + off. + + If *capture* is ``True``, warnings issued by the :mod:`warnings` module will + be redirected to the logging system. Specifically, a warning will be + formatted using :func:`warnings.formatwarning` and the resulting string + logged to a logger named ``'py.warnings'`` with a severity of ``'WARNING'``. + + If *capture* is ``False``, the redirection of warnings to the logging system + will stop, and warnings will be redirected to their original destinations + (i.e. those in effect before ``captureWarnings(True)`` was called). + + +.. seealso:: + + Module :mod:`logging.config` + Configuration API for the logging module. + + Module :mod:`logging.handlers` + Useful handlers included with the logging module. + + :pep:`282` - A Logging System + The proposal which described this feature for inclusion in the Python standard + library. + + `Original Python logging package <http://www.red-dove.com/python_logging.html>`_ + This is the original source for the :mod:`logging` package. The version of the + package available from this site is suitable for use with Python 1.5.2, 2.1.x + and 2.2.x, which do not include the :mod:`logging` package in the standard + library. diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst index 7409af5f15..83a590ee68 100644 --- a/Doc/library/mailbox.rst +++ b/Doc/library/mailbox.rst @@ -81,13 +81,16 @@ Maildir, mbox, MH, Babyl, and MMDF. it. Parameter *message* may be a :class:`Message` instance, an - :class:`email.Message.Message` instance, a string, or a file-like object - (which should be open in text mode). If *message* is an instance of the + :class:`email.Message.Message` instance, a string, a byte string, or a + file-like object (which should be open in binary mode). If *message* is + an instance of the appropriate format-specific :class:`Message` subclass (e.g., if it's an :class:`mboxMessage` instance and this is an :class:`mbox` instance), its format-specific information is used. Otherwise, reasonable defaults for format-specific information are used. + .. versionchanged:: 3.2 support for binary input + .. method:: remove(key) __delitem__(key) @@ -108,8 +111,9 @@ Maildir, mbox, MH, Babyl, and MMDF. :exc:`KeyError` exception if no message already corresponds to *key*. As with :meth:`add`, parameter *message* may be a :class:`Message` - instance, an :class:`email.Message.Message` instance, a string, or a - file-like object (which should be open in text mode). If *message* is an + instance, an :class:`email.Message.Message` instance, a string, a byte + string, or a file-like object (which should be open in binary mode). If + *message* is an instance of the appropriate format-specific :class:`Message` subclass (e.g., if it's an :class:`mboxMessage` instance and this is an :class:`mbox` instance), its format-specific information is @@ -171,24 +175,40 @@ Maildir, mbox, MH, Babyl, and MMDF. raise a :exc:`KeyError` exception if no such message exists. + .. method:: get_bytes(key) + + Return a byte representation of the message corresponding to *key*, or + raise a :exc:`KeyError` exception if no such message exists. + + .. versionadded:: 3.2 + + .. method:: get_string(key) Return a string representation of the message corresponding to *key*, or - raise a :exc:`KeyError` exception if no such message exists. + raise a :exc:`KeyError` exception if no such message exists. The + message is processed through :class:`email.message.Message` to + convert it to a 7bit clean representation. .. method:: get_file(key) Return a file-like representation of the message corresponding to *key*, - or raise a :exc:`KeyError` exception if no such message exists. The - file-like object behaves as if open in binary mode. This file should be + or raise a :exc:`KeyError` exception if no such message exists. The + file-like object behaves as if open in binary mode. This file should be closed once it is no longer needed. + .. versionchanged:: 3.2 + The file object really is a binary file; previously it was incorrectly + returned in text mode. Also, the file-like object now supports the + context manager protocol: you can use a :keyword:`with` statement to + automatically close it. + .. note:: Unlike other representations of messages, file-like representations are not necessarily independent of the :class:`Mailbox` instance that - created them or of the underlying mailbox. More specific documentation + created them or of the underlying mailbox. More specific documentation is provided by each subclass. @@ -452,7 +472,7 @@ Maildir, mbox, MH, Babyl, and MMDF. unlock() Three locking mechanisms are used---dot locking and, if available, the - :cfunc:`flock` and :cfunc:`lockf` system calls. + :c:func:`flock` and :c:func:`lockf` system calls. .. seealso:: @@ -566,7 +586,7 @@ Maildir, mbox, MH, Babyl, and MMDF. unlock() Three locking mechanisms are used---dot locking and, if available, the - :cfunc:`flock` and :cfunc:`lockf` system calls. For MH mailboxes, locking + :c:func:`flock` and :c:func:`lockf` system calls. For MH mailboxes, locking the mailbox means locking the :file:`.mh_sequences` file and, only for the duration of any operations that affect them, locking individual message files. @@ -664,7 +684,7 @@ Maildir, mbox, MH, Babyl, and MMDF. unlock() Three locking mechanisms are used---dot locking and, if available, the - :cfunc:`flock` and :cfunc:`lockf` system calls. + :c:func:`flock` and :c:func:`lockf` system calls. .. seealso:: @@ -715,7 +735,7 @@ Maildir, mbox, MH, Babyl, and MMDF. unlock() Three locking mechanisms are used---dot locking and, if available, the - :cfunc:`flock` and :cfunc:`lockf` system calls. + :c:func:`flock` and :c:func:`lockf` system calls. .. seealso:: @@ -742,9 +762,11 @@ Maildir, mbox, MH, Babyl, and MMDF. If *message* is omitted, the new instance is created in a default, empty state. If *message* is an :class:`email.Message.Message` instance, its contents are copied; furthermore, any format-specific information is converted insofar as - possible if *message* is a :class:`Message` instance. If *message* is a string + possible if *message* is a :class:`Message` instance. If *message* is a string, + a byte string, or a file, it should contain an :rfc:`2822`\ -compliant message, which is read - and parsed. + and parsed. Files should be open in binary mode, but text mode files + are accepted for backward compatibility. The format-specific state and behaviors offered by subclasses vary, but in general it is only the properties that are not specific to a particular @@ -758,7 +780,7 @@ Maildir, mbox, MH, Babyl, and MMDF. There is no requirement that :class:`Message` instances be used to represent messages retrieved using :class:`Mailbox` instances. In some situations, the time and memory required to generate :class:`Message` representations might - not not acceptable. For such situations, :class:`Mailbox` instances also + not be acceptable. For such situations, :class:`Mailbox` instances also offer string and file-like representations, and a custom message factory may be specified when a :class:`Mailbox` instance is initialized. diff --git a/Doc/library/mailcap.rst b/Doc/library/mailcap.rst index 0a0a79088e..4bb31bfc05 100644 --- a/Doc/library/mailcap.rst +++ b/Doc/library/mailcap.rst @@ -4,6 +4,9 @@ .. module:: mailcap :synopsis: Mailcap file handling. +**Source code:** :source:`Lib/mailcap.py` + +-------------- Mailcap files are used to configure how MIME-aware applications such as mail readers and Web browsers react to files with different MIME types. (The name diff --git a/Doc/library/markup.rst b/Doc/library/markup.rst index ae97b6964e..49794ef707 100644 --- a/Doc/library/markup.rst +++ b/Doc/library/markup.rst @@ -20,6 +20,7 @@ definition of the Python bindings for the DOM and SAX interfaces. .. toctree:: + html.rst html.parser.rst html.entities.rst pyexpat.rst diff --git a/Doc/library/math.rst b/Doc/library/math.rst index c760701ef1..98c5b334fc 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -97,15 +97,23 @@ Number-theoretic and representation functions <http://code.activestate.com/recipes/393090/>`_\. +.. function:: isfinite(x) + + Return ``True`` if *x* is neither an infinity nor a NaN, and + ``False`` otherwise. (Note that ``0.0`` *is* considered finite.) + + .. versionadded:: 3.2 + + .. function:: isinf(x) - Check if the float *x* is positive or negative infinity. + Return ``True`` if *x* is a positive or negative infinity, and + ``False`` otherwise. .. function:: isnan(x) - Check if the float *x* is a NaN (not a number). For more information - on NaNs, see the IEEE 754 standards. + Return ``True`` if *x* is a NaN (not a number), and ``False`` otherwise. .. function:: ldexp(x, i) @@ -146,6 +154,22 @@ Power and logarithmic functions Return ``e**x``. +.. function:: expm1(x) + + Return ``e**x - 1``. For small floats *x*, the subtraction in ``exp(x) - 1`` + can result in a `significant loss of precision + <http://en.wikipedia.org/wiki/Loss_of_significance>`_\; the :func:`expm1` + function provides a way to compute this quantity to full precision:: + + >>> from math import exp, expm1 + >>> exp(1e-5) - 1 # gives result accurate to 11 places + 1.0000050000069649e-05 + >>> expm1(1e-5) # result accurate to full precision + 1.0000050000166668e-05 + + .. versionadded:: 3.2 + + .. function:: log(x[, base]) With one argument, return the natural logarithm of *x* (to base *e*). @@ -245,6 +269,9 @@ Angular conversion Hyperbolic functions -------------------- +`Hyperbolic functions <http://en.wikipedia.org/wiki/Hyperbolic_function>`_ +are analogs of trigonometric functions that are based on hyperbolas +instead of circles. .. function:: acosh(x) @@ -276,6 +303,52 @@ Hyperbolic functions Return the hyperbolic tangent of *x*. +Special functions +----------------- + +.. function:: erf(x) + + Return the `error function <http://en.wikipedia.org/wiki/Error_function>`_ at + *x*. + + The :func:`erf` function can be used to compute traditional statistical + functions such as the `cumulative standard normal distribution + <http://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution_function>`_:: + + def phi(x): + 'Cumulative distribution function for the standard normal distribution' + return (1.0 + erf(x / sqrt(2.0))) / 2.0 + + .. versionadded:: 3.2 + + +.. function:: erfc(x) + + Return the complementary error function at *x*. The `complementary error + function <http://en.wikipedia.org/wiki/Error_function>`_ is defined as + ``1.0 - erf(x)``. It is used for large values of *x* where a subtraction + from one would cause a `loss of significance + <http://en.wikipedia.org/wiki/Loss_of_significance>`_\. + + .. versionadded:: 3.2 + + +.. function:: gamma(x) + + Return the `Gamma function <http://en.wikipedia.org/wiki/Gamma_function>`_ at + *x*. + + .. versionadded:: 3.2 + + +.. function:: lgamma(x) + + Return the natural logarithm of the absolute value of the Gamma + function at *x*. + + .. versionadded:: 3.2 + + Constants --------- diff --git a/Doc/library/mimetypes.rst b/Doc/library/mimetypes.rst index fe1437a85b..be11c0dce7 100644 --- a/Doc/library/mimetypes.rst +++ b/Doc/library/mimetypes.rst @@ -8,6 +8,10 @@ .. index:: pair: MIME; content type +**Source code:** :source:`Lib/mimetypes.py` + +-------------- + The :mod:`mimetypes` module converts between a filename or URL and the MIME type associated with the filename extension. Conversions are provided from filename to MIME type and from MIME type to filename extension; encodings are not @@ -26,23 +30,23 @@ the information :func:`init` sets up. .. index:: pair: MIME; headers - Guess the type of a file based on its filename or URL, given by *filename*. The + Guess the type of a file based on its filename or URL, given by *url*. The return value is a tuple ``(type, encoding)`` where *type* is ``None`` if the type can't be guessed (missing or unknown suffix) or a string of the form ``'type/subtype'``, usable for a MIME :mailheader:`content-type` header. *encoding* is ``None`` for no encoding or the name of the program used to encode (e.g. :program:`compress` or :program:`gzip`). The encoding is suitable for use - as a :mailheader:`Content-Encoding` header, *not* as a + as a :mailheader:`Content-Encoding` header, **not** as a :mailheader:`Content-Transfer-Encoding` header. The mappings are table driven. Encoding suffixes are case sensitive; type suffixes are first tried case sensitively, then case insensitively. - Optional *strict* is a flag specifying whether the list of known MIME types + The optional *strict* argument is a flag specifying whether the list of known MIME types is limited to only the official types `registered with IANA - <http://www.iana.org/assignments/media-types/>`_ are recognized. - When *strict* is true (the default), only the IANA types are supported; when - *strict* is false, some additional non-standard but commonly used MIME types + <http://www.iana.org/assignments/media-types/>`_. + When *strict* is ``True`` (the default), only the IANA types are supported; when + *strict* is ``False``, some additional non-standard but commonly used MIME types are also recognized. @@ -54,7 +58,7 @@ the information :func:`init` sets up. been associated with any particular data stream, but would be mapped to the MIME type *type* by :func:`guess_type`. - Optional *strict* has the same meaning as with the :func:`guess_type` function. + The optional *strict* argument has the same meaning as with the :func:`guess_type` function. .. function:: guess_extension(type, strict=True) @@ -62,11 +66,11 @@ the information :func:`init` sets up. Guess the extension for a file based on its MIME type, given by *type*. The return value is a string giving a filename extension, including the leading dot (``'.'``). The extension is not guaranteed to have been associated with any - particular data stream, but would be mapped to the MIME type *type* by + particular data stream, but would be mapped to the MIME type *type* by :func:`guess_type`. If no extension can be guessed for *type*, ``None`` is returned. - Optional *strict* has the same meaning as with the :func:`guess_type` function. + The optional *strict* argument has the same meaning as with the :func:`guess_type` function. Some additional functions and data items are available for controlling the behavior of the module. @@ -76,14 +80,18 @@ behavior of the module. Initialize the internal data structures. If given, *files* must be a sequence of file names which should be used to augment the default type map. If omitted, - the file names to use are taken from :const:`knownfiles`. Each file named in - *files* or :const:`knownfiles` takes precedence over those named before it. - Calling :func:`init` repeatedly is allowed. + the file names to use are taken from :const:`knownfiles`; on Windows, the + current registry settings are loaded. Each file named in *files* or + :const:`knownfiles` takes precedence over those named before it. Calling + :func:`init` repeatedly is allowed. + + .. versionchanged:: 3.2 + Previously, Windows registry settings were ignored. .. function:: read_mime_types(filename) - Load the type map given in the file *filename*, if it exists. The type map is + Load the type map given in the file *filename*, if it exists. The type map is returned as a dictionary mapping filename extensions, including the leading dot (``'.'``), to strings of the form ``'type/subtype'``. If the file *filename* does not exist or cannot be read, ``None`` is returned. @@ -91,18 +99,18 @@ behavior of the module. .. function:: add_type(type, ext, strict=True) - Add a mapping from the mimetype *type* to the extension *ext*. When the + Add a mapping from the MIME type *type* to the extension *ext*. When the extension is already known, the new type will replace the old one. When the type is already known the extension will be added to the list of known extensions. - When *strict* is True (the default), the mapping will added to the official MIME + When *strict* is ``True`` (the default), the mapping will added to the official MIME types, otherwise to the non-standard ones. .. data:: inited Flag indicating whether or not the global data structures have been initialized. - This is set to true by :func:`init`. + This is set to ``True`` by :func:`init`. .. data:: knownfiles @@ -137,22 +145,6 @@ behavior of the module. Dictionary mapping filename extensions to non-standard, but commonly found MIME types. -The :class:`MimeTypes` class may be useful for applications which may want more -than one MIME-type database: - - -.. class:: MimeTypes(filenames=(), strict=True) - - This class represents a MIME-types database. By default, it provides access to - the same database as the rest of this module. The initial database is a copy of - that provided by the module, and may be extended by loading additional - :file:`mime.types`\ -style files into the database using the :meth:`read` or - :meth:`readfp` methods. The mapping dictionaries may also be cleared before - loading additional data if the default data is not desired. - - The optional *filenames* parameter can be used to cause additional files to be - loaded "on top" of the default database. - An example usage of the module:: @@ -173,36 +165,53 @@ An example usage of the module:: MimeTypes Objects ----------------- -:class:`MimeTypes` instances provide an interface which is very like that of the +The :class:`MimeTypes` class may be useful for applications which may want more +than one MIME-type database; it provides an interface similar to the one of the :mod:`mimetypes` module. +.. class:: MimeTypes(filenames=(), strict=True) + + This class represents a MIME-types database. By default, it provides access to + the same database as the rest of this module. The initial database is a copy of + that provided by the module, and may be extended by loading additional + :file:`mime.types`\ -style files into the database using the :meth:`read` or + :meth:`readfp` methods. The mapping dictionaries may also be cleared before + loading additional data if the default data is not desired. + + The optional *filenames* parameter can be used to cause additional files to be + loaded "on top" of the default database. + + .. attribute:: MimeTypes.suffix_map Dictionary mapping suffixes to suffixes. This is used to allow recognition of encoded files for which the encoding and the type are indicated by the same extension. For example, the :file:`.tgz` extension is mapped to :file:`.tar.gz` to allow the encoding and type to be recognized separately. This is initially a - copy of the global ``suffix_map`` defined in the module. + copy of the global :data:`suffix_map` defined in the module. .. attribute:: MimeTypes.encodings_map Dictionary mapping filename extensions to encoding types. This is initially a - copy of the global ``encodings_map`` defined in the module. + copy of the global :data:`encodings_map` defined in the module. .. attribute:: MimeTypes.types_map - Dictionary mapping filename extensions to MIME types. This is initially a copy - of the global ``types_map`` defined in the module. + Tuple containing two dictionaries, mapping filename extensions to MIME types: + the first dictionary is for the non-standards types and the second one is for + the standard types. They are initialized by :data:`common_types` and + :data:`types_map`. -.. attribute:: MimeTypes.common_types +.. attribute:: MimeTypes.types_map_inv - Dictionary mapping filename extensions to non-standard, but commonly found MIME - types. This is initially a copy of the global ``common_types`` defined in the - module. + Tuple containing two dictionaries, mapping MIME types to a list of filename + extensions: the first dictionary is for the non-standards types and the + second one is for the standard types. They are initialized by + :data:`common_types` and :data:`types_map`. .. method:: MimeTypes.guess_extension(type, strict=True) @@ -217,14 +226,35 @@ MimeTypes Objects the object. -.. method:: MimeTypes.read(path) +.. method:: MimeTypes.guess_all_extensions(type, strict=True) + + Similar to the :func:`guess_all_extensions` function, using the tables stored + as part of the object. + - Load MIME information from a file named *path*. This uses :meth:`readfp` to +.. method:: MimeTypes.read(filename, strict=True) + + Load MIME information from a file named *filename*. This uses :meth:`readfp` to parse the file. + If *strict* is ``True``, information will be added to list of standard types, + else to the list of non-standard types. + -.. method:: MimeTypes.readfp(file) +.. method:: MimeTypes.readfp(fp, strict=True) - Load MIME type information from an open file. The file must have the format of + Load MIME type information from an open file *fp*. The file must have the format of the standard :file:`mime.types` files. + If *strict* is ``True``, information will be added to the list of standard + types, else to the list of non-standard types. + + +.. method:: MimeTypes.read_windows_registry(strict=True) + + Load MIME type information from the Windows registry. Availability: Windows. + + If *strict* is ``True``, information will be added to the list of standard + types, else to the list of non-standard types. + + .. versionadded:: 3.2 diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index e061088a69..5f0f004e0c 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -21,6 +21,12 @@ file object, use its :meth:`fileno` method to obtain the correct value for the :func:`os.open` function, which returns a file descriptor directly (the file still needs to be closed when done). +.. note:: + If you want to create a memory-mapping for a writable, buffered file, you + should :func:`~io.IOBase.flush` the file first. This is necessary to ensure + that local modifications to the buffers are actually available to the + mapping. + For both the Unix and Windows versions of the constructor, *access* may be specified as an optional keyword parameter. *access* accepts one of three values: :const:`ACCESS_READ`, :const:`ACCESS_WRITE`, or :const:`ACCESS_COPY` @@ -115,6 +121,18 @@ To map anonymous memory, -1 should be passed as the fileno along with the length map.close() + :class:`mmap` can also be used as a context manager in a :keyword:`with` + statement.:: + + import mmap + + with mmap.mmap(-1, 13) as map: + map.write("Hello world!") + + .. versionadded:: 3.2 + Context manager support. + + The next example demonstrates how to create an anonymous map and exchange data between the parent and child processes:: @@ -135,13 +153,19 @@ To map anonymous memory, -1 should be passed as the fileno along with the length Memory-mapped file objects support the following methods: - .. method:: close() Close the file. Subsequent calls to other methods of the object will result in an exception being raised. + .. attribute:: closed + + True if the file is closed. + + .. versionadded:: 3.2 + + .. method:: find(sub[, start[, end]]) Returns the lowest index in the object where the subsequence *sub* is @@ -235,7 +259,7 @@ To map anonymous memory, -1 should be passed as the fileno along with the length .. method:: write_byte(byte) - Write the the integer *byte* into memory at the current + Write the integer *byte* into memory at the current position of the file pointer; the file position is advanced by ``1``. If the mmap was created with :const:`ACCESS_READ`, then writing to it will raise a :exc:`TypeError` exception. diff --git a/Doc/library/modulefinder.rst b/Doc/library/modulefinder.rst index 41c603ce9b..97ace60179 100644 --- a/Doc/library/modulefinder.rst +++ b/Doc/library/modulefinder.rst @@ -7,6 +7,9 @@ .. module:: modulefinder :synopsis: Find modules used by a script. +**Source code:** :source:`Lib/modulefinder.py` + +-------------- This module provides a :class:`ModuleFinder` class that can be used to determine the set of modules imported by a script. ``modulefinder.py`` can also be run as diff --git a/Doc/library/msilib.rst b/Doc/library/msilib.rst index f27c3d803c..270f4ff9f2 100644 --- a/Doc/library/msilib.rst +++ b/Doc/library/msilib.rst @@ -42,7 +42,7 @@ structures. .. function:: UuidCreate() Return the string representation of a new unique identifier. This wraps the - Windows API functions :cfunc:`UuidCreate` and :cfunc:`UuidToString`. + Windows API functions :c:func:`UuidCreate` and :c:func:`UuidToString`. .. function:: OpenDatabase(path, persist) @@ -58,7 +58,7 @@ structures. .. function:: CreateRecord(count) - Return a new record object by calling :cfunc:`MSICreateRecord`. *count* is the + Return a new record object by calling :c:func:`MSICreateRecord`. *count* is the number of fields of the record. @@ -133,20 +133,20 @@ Database Objects .. method:: Database.OpenView(sql) - Return a view object, by calling :cfunc:`MSIDatabaseOpenView`. *sql* is the SQL + Return a view object, by calling :c:func:`MSIDatabaseOpenView`. *sql* is the SQL statement to execute. .. method:: Database.Commit() Commit the changes pending in the current transaction, by calling - :cfunc:`MSIDatabaseCommit`. + :c:func:`MSIDatabaseCommit`. .. method:: Database.GetSummaryInformation(count) Return a new summary information object, by calling - :cfunc:`MsiGetSummaryInformation`. *count* is the maximum number of updated + :c:func:`MsiGetSummaryInformation`. *count* is the maximum number of updated values. @@ -164,7 +164,7 @@ View Objects .. method:: View.Execute(params) - Execute the SQL query of the view, through :cfunc:`MSIViewExecute`. If + Execute the SQL query of the view, through :c:func:`MSIViewExecute`. If *params* is not ``None``, it is a record describing actual values of the parameter tokens in the query. @@ -172,18 +172,18 @@ View Objects .. method:: View.GetColumnInfo(kind) Return a record describing the columns of the view, through calling - :cfunc:`MsiViewGetColumnInfo`. *kind* can be either ``MSICOLINFO_NAMES`` or + :c:func:`MsiViewGetColumnInfo`. *kind* can be either ``MSICOLINFO_NAMES`` or ``MSICOLINFO_TYPES``. .. method:: View.Fetch() - Return a result record of the query, through calling :cfunc:`MsiViewFetch`. + Return a result record of the query, through calling :c:func:`MsiViewFetch`. .. method:: View.Modify(kind, data) - Modify the view, by calling :cfunc:`MsiViewModify`. *kind* can be one of + Modify the view, by calling :c:func:`MsiViewModify`. *kind* can be one of ``MSIMODIFY_SEEK``, ``MSIMODIFY_REFRESH``, ``MSIMODIFY_INSERT``, ``MSIMODIFY_UPDATE``, ``MSIMODIFY_ASSIGN``, ``MSIMODIFY_REPLACE``, ``MSIMODIFY_MERGE``, ``MSIMODIFY_DELETE``, ``MSIMODIFY_INSERT_TEMPORARY``, @@ -195,7 +195,7 @@ View Objects .. method:: View.Close() - Close the view, through :cfunc:`MsiViewClose`. + Close the view, through :c:func:`MsiViewClose`. .. seealso:: @@ -214,7 +214,7 @@ Summary Information Objects .. method:: SummaryInformation.GetProperty(field) - Return a property of the summary, through :cfunc:`MsiSummaryInfoGetProperty`. + Return a property of the summary, through :c:func:`MsiSummaryInfoGetProperty`. *field* is the name of the property, and can be one of the constants ``PID_CODEPAGE``, ``PID_TITLE``, ``PID_SUBJECT``, ``PID_AUTHOR``, ``PID_KEYWORDS``, ``PID_COMMENTS``, ``PID_TEMPLATE``, ``PID_LASTAUTHOR``, @@ -226,12 +226,12 @@ Summary Information Objects .. method:: SummaryInformation.GetPropertyCount() Return the number of summary properties, through - :cfunc:`MsiSummaryInfoGetPropertyCount`. + :c:func:`MsiSummaryInfoGetPropertyCount`. .. method:: SummaryInformation.SetProperty(field, value) - Set a property through :cfunc:`MsiSummaryInfoSetProperty`. *field* can have the + Set a property through :c:func:`MsiSummaryInfoSetProperty`. *field* can have the same values as in :meth:`GetProperty`, *value* is the new value of the property. Possible value types are integer and string. @@ -239,7 +239,7 @@ Summary Information Objects .. method:: SummaryInformation.Persist() Write the modified properties to the summary information stream, using - :cfunc:`MsiSummaryInfoPersist`. + :c:func:`MsiSummaryInfoPersist`. .. seealso:: @@ -258,7 +258,7 @@ Record Objects .. method:: Record.GetFieldCount() Return the number of fields of the record, through - :cfunc:`MsiRecordGetFieldCount`. + :c:func:`MsiRecordGetFieldCount`. .. method:: Record.GetInteger(field) @@ -275,25 +275,25 @@ Record Objects .. method:: Record.SetString(field, value) - Set *field* to *value* through :cfunc:`MsiRecordSetString`. *field* must be an + Set *field* to *value* through :c:func:`MsiRecordSetString`. *field* must be an integer; *value* a string. .. method:: Record.SetStream(field, value) Set *field* to the contents of the file named *value*, through - :cfunc:`MsiRecordSetStream`. *field* must be an integer; *value* a string. + :c:func:`MsiRecordSetStream`. *field* must be an integer; *value* a string. .. method:: Record.SetInteger(field, value) - Set *field* to *value* through :cfunc:`MsiRecordSetInteger`. Both *field* and + Set *field* to *value* through :c:func:`MsiRecordSetInteger`. Both *field* and *value* must be an integer. .. method:: Record.ClearData() - Set all fields of the record to 0, through :cfunc:`MsiRecordClearData`. + Set all fields of the record to 0, through :c:func:`MsiRecordClearData`. .. seealso:: diff --git a/Doc/library/msvcrt.rst b/Doc/library/msvcrt.rst index d968833251..889a0c5eb1 100644 --- a/Doc/library/msvcrt.rst +++ b/Doc/library/msvcrt.rst @@ -90,12 +90,12 @@ Console I/O .. function:: getch() - Read a keypress and return the resulting character. Nothing is echoed to the - console. This call will block if a keypress is not already available, but will - not wait for :kbd:`Enter` to be pressed. If the pressed key was a special - function key, this will return ``'\000'`` or ``'\xe0'``; the next call will - return the keycode. The :kbd:`Control-C` keypress cannot be read with this - function. + Read a keypress and return the resulting character as a byte string. + Nothing is echoed to the console. This call will block if a keypress + is not already available, but will not wait for :kbd:`Enter` to be + pressed. If the pressed key was a special function key, this will + return ``'\000'`` or ``'\xe0'``; the next call will return the keycode. + The :kbd:`Control-C` keypress cannot be read with this function. .. function:: getwch() @@ -116,7 +116,7 @@ Console I/O .. function:: putch(char) - Print the character *char* to the console without buffering. + Print the byte string *char* to the console without buffering. .. function:: putwch(unicode_char) @@ -126,8 +126,8 @@ Console I/O .. function:: ungetch(char) - Cause the character *char* to be "pushed back" into the console buffer; it will - be the next character read by :func:`getch` or :func:`getche`. + Cause the byte string *char* to be "pushed back" into the console buffer; + it will be the next character read by :func:`getch` or :func:`getche`. .. function:: ungetwch(unicode_char) @@ -143,5 +143,5 @@ Other Functions .. function:: heapmin() - Force the :cfunc:`malloc` heap to clean itself up and return unused blocks to + Force the :c:func:`malloc` heap to clean itself up and return unused blocks to the operating system. On failure, this raises :exc:`IOError`. diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index cb20a5c75c..7db2aed1bc 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -16,7 +16,7 @@ to this, the :mod:`multiprocessing` module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows. -.. warning:: +.. note:: Some of this package's functionality requires a functioning shared semaphore implementation on the host operating system. Without one, the @@ -120,7 +120,9 @@ processes: print(q.get()) # prints "[42, None, 'hello']" p.join() - Queues are thread and process safe. + Queues are thread and process safe, but note that they must never + be instantiated as a side effect of importing a module: this can lead + to a deadlock! (see :ref:`threaded-imports`) **Pipes** @@ -406,7 +408,7 @@ The :mod:`multiprocessing` package mostly replicates the API of the .. method:: terminate() Terminate the process. On Unix this is done using the ``SIGTERM`` signal; - on Windows :cfunc:`TerminateProcess` is used. Note that exit handlers and + on Windows :c:func:`TerminateProcess` is used. Note that exit handlers and finally clauses, etc., will not be executed. Note that descendant processes of the process will *not* be terminated -- @@ -420,9 +422,9 @@ The :mod:`multiprocessing` package mostly replicates the API of the acquired a lock or semaphore etc. then terminating it is liable to cause other processes to deadlock. - Note that the :meth:`start`, :meth:`join`, :meth:`is_alive` and - :attr:`exit_code` methods should only be called by the process that created - the process object. + Note that the :meth:`start`, :meth:`join`, :meth:`is_alive`, + :meth:`terminate` and :attr:`exit_code` methods should only be called by + the process that created the process object. Example usage of some of the methods of :class:`Process`: @@ -462,7 +464,7 @@ primitives like locks. For passing messages one can use :func:`Pipe` (for a connection between two processes) or a queue (which allows multiple producers and consumers). -The :class:`Queue` and :class:`JoinableQueue` types are multi-producer, +The :class:`Queue`, :class:`multiprocessing.queues.SimpleQueue` and :class:`JoinableQueue` types are multi-producer, multi-consumer FIFO queues modelled on the :class:`queue.Queue` class in the standard library. They differ in that :class:`Queue` lacks the :meth:`~queue.Queue.task_done` and :meth:`~queue.Queue.join` methods introduced @@ -470,7 +472,7 @@ into Python 2.5's :class:`queue.Queue` class. If you use :class:`JoinableQueue` then you **must** call :meth:`JoinableQueue.task_done` for each task removed from the queue or else the -semaphore used to count the number of unfinished tasks may eventually overflow +semaphore used to count the number of unfinished tasks may eventually overflow, raising an exception. Note that one can also create a shared queue by using a manager object -- see @@ -488,7 +490,7 @@ Note that one can also create a shared queue by using a manager object -- see If a process is killed using :meth:`Process.terminate` or :func:`os.kill` while it is trying to use a :class:`Queue`, then the data in the queue is - likely to become corrupted. This may cause any other processes to get an + likely to become corrupted. This may cause any other process to get an exception when it tries to use the queue later on. .. warning:: @@ -550,9 +552,9 @@ For an example of the usage of queues for interprocess communication see Return ``True`` if the queue is full, ``False`` otherwise. Because of multithreading/multiprocessing semantics, this is not reliable. - .. method:: put(item[, block[, timeout]]) + .. method:: put(obj[, block[, timeout]]) - Put item into the queue. If the optional argument *block* is ``True`` + Put obj into the queue. If the optional argument *block* is ``True`` (the default) and *timeout* is ``None`` (the default), block if necessary until a free slot is available. If *timeout* is a positive number, it blocks at most *timeout* seconds and raises the :exc:`queue.Full` exception if no @@ -561,9 +563,9 @@ For an example of the usage of queues for interprocess communication see available, else raise the :exc:`queue.Full` exception (*timeout* is ignored in that case). - .. method:: put_nowait(item) + .. method:: put_nowait(obj) - Equivalent to ``put(item, False)``. + Equivalent to ``put(obj, False)``. .. method:: get([block[, timeout]]) @@ -608,6 +610,23 @@ For an example of the usage of queues for interprocess communication see exits -- see :meth:`join_thread`. +.. class:: multiprocessing.queues.SimpleQueue() + + It is a simplified :class:`Queue` type, very close to a locked :class:`Pipe`. + + .. method:: empty() + + Return ``True`` if the queue is empty, ``False`` otherwise. + + .. method:: get() + + Remove and return an item from the queue. + + .. method:: put(item) + + Put *item* into the queue. + + .. class:: JoinableQueue([maxsize]) :class:`JoinableQueue`, a :class:`Queue` subclass, is a queue which @@ -690,7 +709,7 @@ Miscellaneous (By default :data:`sys.executable` is used). Embedders will probably need to do some thing like :: - setExecutable(os.path.join(sys.exec_prefix, 'pythonw.exe')) + set_executable(os.path.join(sys.exec_prefix, 'pythonw.exe')) before they can create child processes. (Windows only) @@ -709,7 +728,7 @@ Connection Objects Connection objects allow the sending and receiving of picklable objects or strings. They can be thought of as message oriented connected sockets. -Connection objects usually created using :func:`Pipe` -- see also +Connection objects are usually created using :func:`Pipe` -- see also :ref:`multiprocessing-listeners-clients`. .. class:: Connection @@ -725,12 +744,13 @@ Connection objects usually created using :func:`Pipe` -- see also .. method:: recv() Return an object sent from the other end of the connection using - :meth:`send`. Raises :exc:`EOFError` if there is nothing left to receive + :meth:`send`. Blocks until there its something to receive. Raises + :exc:`EOFError` if there is nothing left to receive and the other end was closed. .. method:: fileno() - Returns the file descriptor or handle used by the connection. + Return the file descriptor or handle used by the connection. .. method:: close() @@ -754,12 +774,13 @@ Connection objects usually created using :func:`Pipe` -- see also If *offset* is given then data is read from that position in *buffer*. If *size* is given then that many bytes will be read from buffer. Very large buffers (approximately 32 MB+, though it depends on the OS) may raise a - ValueError exception + :exc:`ValueError` exception .. method:: recv_bytes([maxlength]) Return a complete message of byte data sent from the other end of the - connection as a string. Raises :exc:`EOFError` if there is nothing left + connection as a string. Blocks until there is something to receive. + Raises :exc:`EOFError` if there is nothing left to receive and the other end has closed. If *maxlength* is specified and the message is longer than *maxlength* @@ -769,7 +790,8 @@ Connection objects usually created using :func:`Pipe` -- see also .. method:: recv_bytes_into(buffer[, offset]) Read into *buffer* a complete message of byte data sent from the other end - of the connection and return the number of bytes in the message. Raises + of the connection and return the number of bytes in the message. Blocks + until there is something to receive. Raises :exc:`EOFError` if there is nothing left to receive and the other end was closed. @@ -871,14 +893,6 @@ object -- see :ref:`multiprocessing-managers`. .. note:: - The :meth:`acquire` method of :class:`BoundedSemaphore`, :class:`Lock`, - :class:`RLock` and :class:`Semaphore` has a timeout parameter not supported - by the equivalents in :mod:`threading`. The signature is - ``acquire(block=True, timeout=None)`` with keyword parameters being - acceptable. If *block* is ``True`` and *timeout* is not ``None`` then it - specifies a timeout in seconds. If *block* is ``False`` then *timeout* is - ignored. - On Mac OS X, ``sem_timedwait`` is unsupported, so calling ``acquire()`` with a timeout will emulate that function's behavior using a sleeping loop. @@ -1130,7 +1144,7 @@ their parent process exits. The manager classes are defined in the Create a BaseManager object. - Once created one should call :meth:`start` or :meth:`serve_forever` to ensure + Once created one should call :meth:`start` or ``get_server().serve_forever()`` to ensure that the manager object refers to a started manager process. *address* is the address on which the manager process listens for new @@ -1146,10 +1160,6 @@ their parent process exits. The manager classes are defined in the Start a subprocess to start the manager. If *initializer* is not ``None`` then the subprocess will call ``initializer(*initargs)`` when it starts. - .. method:: serve_forever() - - Run the server in the current process. - .. method:: get_server() Returns a :class:`Server` object which represents the actual server under @@ -1331,7 +1341,7 @@ Customized managers >>>>>>>>>>>>>>>>>>> To create one's own manager, one creates a subclass of :class:`BaseManager` and -use the :meth:`~BaseManager.register` classmethod to register new types or +uses the :meth:`~BaseManager.register` classmethod to register new types or callables with the manager class. For example:: from multiprocessing.managers import BaseManager @@ -1496,7 +1506,7 @@ itself. This means, for example, that one shared object can contain a second: a new shared object -- see documentation for the *method_to_typeid* argument of :meth:`BaseManager.register`. - If an exception is raised by the call, then then is re-raised by + If an exception is raised by the call, then is re-raised by :meth:`_callmethod`. If some other exception is raised in the manager's process then this is converted into a :exc:`RemoteError` exception and is raised by :meth:`_callmethod`. @@ -1552,7 +1562,7 @@ Process Pools One can create a pool of processes which will carry out tasks submitted to it with the :class:`Pool` class. -.. class:: multiprocessing.Pool([processes[, initializer[, initargs]]]) +.. class:: multiprocessing.Pool([processes[, initializer[, initargs[, maxtasksperchild]]]]) A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and @@ -1563,39 +1573,69 @@ with the :class:`Pool` class. *initializer* is not ``None`` then each worker process will call ``initializer(*initargs)`` when it starts. + .. versionadded:: 3.2 + *maxtasksperchild* is the number of tasks a worker process can complete + before it will exit and be replaced with a fresh worker process, to enable + unused resources to be freed. The default *maxtasksperchild* is None, which + means worker processes will live as long as the pool. + + .. note:: + + Worker processes within a :class:`Pool` typically live for the complete + duration of the Pool's work queue. A frequent pattern found in other + systems (such as Apache, mod_wsgi, etc) to free resources held by + workers is to allow a worker within a pool to complete only a set + amount of work before being exiting, being cleaned up and a new + process spawned to replace the old one. The *maxtasksperchild* + argument to the :class:`Pool` exposes this ability to the end user. + .. method:: apply(func[, args[, kwds]]) Call *func* with arguments *args* and keyword arguments *kwds*. It blocks - till the result is ready. Given this blocks, :meth:`apply_async` is better - suited for performing work in parallel. Additionally, the passed in - function is only executed in one of the workers of the pool. + until the result is ready. Given this blocks, :meth:`apply_async` is + better suited for performing work in parallel. Additionally, *func* + is only executed in one of the workers of the pool. - .. method:: apply_async(func[, args[, kwds[, callback]]]) + .. method:: apply_async(func[, args[, kwds[, callback[, error_callback]]]]) A variant of the :meth:`apply` method which returns a result object. If *callback* is specified then it should be a callable which accepts a single argument. When the result becomes ready *callback* is applied to - it (unless the call failed). *callback* should complete immediately since - otherwise the thread which handles the results will get blocked. + it, that is unless the call failed, in which case the *error_callback* + is applied instead + + If *error_callback* is specified then it should be a callable which + accepts a single argument. If the target function fails, then + the *error_callback* is called with the exception instance. + + Callbacks should complete immediately since otherwise the thread which + handles the results will get blocked. .. method:: map(func, iterable[, chunksize]) A parallel equivalent of the :func:`map` built-in function (it supports only - one *iterable* argument though). It blocks till the result is ready. + one *iterable* argument though). It blocks until the result is ready. This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. The (approximate) size of these chunks can be specified by setting *chunksize* to a positive integer. - .. method:: map_async(func, iterable[, chunksize[, callback]]) + .. method:: map_async(func, iterable[, chunksize[, callback[, error_callback]]]) A variant of the :meth:`.map` method which returns a result object. If *callback* is specified then it should be a callable which accepts a single argument. When the result becomes ready *callback* is applied to - it (unless the call failed). *callback* should complete immediately since - otherwise the thread which handles the results will get blocked. + it, that is unless the call failed, in which case the *error_callback* + is applied instead + + If *error_callback* is specified then it should be a callable which + accepts a single argument. If the target function fails, then + the *error_callback* is called with the exception instance. + + Callbacks should complete immediately since otherwise the thread which + handles the results will get blocked. .. method:: imap(func, iterable[, chunksize]) @@ -1603,7 +1643,7 @@ with the :class:`Pool` class. The *chunksize* argument is the same as the one used by the :meth:`.map` method. For very long iterables using a large value for *chunksize* can - make make the job complete **much** faster than using the default value of + make the job complete **much** faster than using the default value of ``1``. Also if *chunksize* is ``1`` then the :meth:`!next` method of the iterator @@ -2032,7 +2072,7 @@ Better to inherit than pickle/unpickle On Windows many types from :mod:`multiprocessing` need to be picklable so that child processes can use them. However, one should generally avoid sending shared objects to other processes using pipes or queues. Instead - you should arrange the program so that a process which need access to a + you should arrange the program so that a process which needs access to a shared resource created elsewhere can inherit it from an ancestor process. Avoid terminating processes @@ -2111,7 +2151,7 @@ Explicitly pass resources to child processes for i in range(10): Process(target=f, args=(lock,)).start() -Beware replacing sys.stdin with a "file like object" +Beware of replacing :data:`sys.stdin` with a "file like object" :mod:`multiprocessing` originally unconditionally called:: @@ -2229,7 +2269,7 @@ Synchronization types like locks, conditions and queues: An example showing how to use queues to feed tasks to a collection of worker -process and collect the results: +processes and collect the results: .. literalinclude:: ../includes/mp_workers.py diff --git a/Doc/library/netrc.rst b/Doc/library/netrc.rst index 91990df029..3f38cbce7a 100644 --- a/Doc/library/netrc.rst +++ b/Doc/library/netrc.rst @@ -7,6 +7,9 @@ .. moduleauthor:: Eric S. Raymond <esr@snark.thyrsus.com> .. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com> +**Source code:** :source:`Lib/netrc.py` + +-------------- The :class:`netrc` class parses and encapsulates the netrc file format used by the Unix :program:`ftp` program and other FTP clients. diff --git a/Doc/library/nntplib.rst b/Doc/library/nntplib.rst index c3cbd2bc95..ef507e1ff9 100644 --- a/Doc/library/nntplib.rst +++ b/Doc/library/nntplib.rst @@ -10,101 +10,122 @@ pair: NNTP; protocol single: Network News Transfer Protocol +**Source code:** :source:`Lib/nntplib.py` + +-------------- + This module defines the class :class:`NNTP` which implements the client side of -the NNTP protocol. It can be used to implement a news reader or poster, or -automated news processors. For more information on NNTP (Network News Transfer -Protocol), see Internet :rfc:`977`. +the Network News Transfer Protocol. It can be used to implement a news reader +or poster, or automated news processors. It is compatible with :rfc:`3977` +as well as the older :rfc:`977` and :rfc:`2980`. Here are two small examples of how it can be used. To list some statistics about a newsgroup and print the subjects of the last 10 articles:: - >>> s = NNTP('news.gmane.org') + >>> s = nntplib.NNTP('news.gmane.org') >>> resp, count, first, last, name = s.group('gmane.comp.python.committers') >>> print('Group', name, 'has', count, 'articles, range', first, 'to', last) - Group gmane.comp.python.committers has 1071 articles, range 1 to 1071 - >>> resp, subs = s.xhdr('subject', first + '-' + last) - >>> for id, sub in subs[-10:]: print(id, sub) + Group gmane.comp.python.committers has 1096 articles, range 1 to 1096 + >>> resp, overviews = s.over((last - 9, last)) + >>> for id, over in overviews: + ... print(id, nntplib.decode_header(over['subject'])) ... - 1062 Re: Mercurial Status? - 1063 Re: [python-committers] (Windows) buildbots on 3.x - 1064 Re: Mercurial Status? - 1065 Re: Mercurial Status? - 1066 Python 2.6.6 status - 1067 Commit Privileges for Ask Solem - 1068 Re: Commit Privileges for Ask Solem - 1069 Re: Commit Privileges for Ask Solem - 1070 Re: Commit Privileges for Ask Solem - 1071 2.6.6 rc 2 + 1087 Re: Commit privileges for Łukasz Langa + 1088 Re: 3.2 alpha 2 freeze + 1089 Re: 3.2 alpha 2 freeze + 1090 Re: Commit privileges for Łukasz Langa + 1091 Re: Commit privileges for Łukasz Langa + 1092 Updated ssh key + 1093 Re: Updated ssh key + 1094 Re: Updated ssh key + 1095 Hello fellow committers! + 1096 Re: Hello fellow committers! >>> s.quit() '205 Bye!' -To post an article from a file (this assumes that the article has valid +To post an article from a binary file (this assumes that the article has valid headers, and that you have right to post on the particular newsgroup):: - >>> s = NNTP('news.gmane.org') - >>> f = open('/tmp/article') + >>> s = nntplib.NNTP('news.gmane.org') + >>> f = open('/tmp/article.txt', 'rb') >>> s.post(f) '240 Article posted successfully.' >>> s.quit() '205 Bye!' -The module itself defines the following items: +The module itself defines the following classes: -.. class:: NNTP(host[, port [, user[, password [, readermode][, usenetrc]]]]) +.. class:: NNTP(host, port=119, user=None, password=None, readermode=None, usenetrc=False, [timeout]) - Return a new instance of the :class:`NNTP` class, representing a connection - to the NNTP server running on host *host*, listening at port *port*. The - default *port* is 119. If the optional *user* and *password* are provided, - or if suitable credentials are present in :file:`/.netrc` and the optional - flag *usenetrc* is true (the default), the ``AUTHINFO USER`` and ``AUTHINFO - PASS`` commands are used to identify and authenticate the user to the server. - If the optional flag *readermode* is true, then a ``mode reader`` command is - sent before authentication is performed. Reader mode is sometimes necessary - if you are connecting to an NNTP server on the local machine and intend to - call reader-specific commands, such as ``group``. If you get unexpected + Return a new :class:`NNTP` object, representing a connection + to the NNTP server running on host *host*, listening at port *port*. + An optional *timeout* can be specified for the socket connection. + If the optional *user* and *password* are provided, or if suitable + credentials are present in :file:`/.netrc` and the optional flag *usenetrc* + is true, the ``AUTHINFO USER`` and ``AUTHINFO PASS`` commands are used + to identify and authenticate the user to the server. If the optional + flag *readermode* is true, then a ``mode reader`` command is sent before + authentication is performed. Reader mode is sometimes necessary if you are + connecting to an NNTP server on the local machine and intend to call + reader-specific commands, such as ``group``. If you get unexpected :exc:`NNTPPermanentError`\ s, you might need to set *readermode*. - *readermode* defaults to ``None``. *usenetrc* defaults to ``True``. + + .. versionchanged:: 3.2 + *usenetrc* is now False by default. + + +.. class:: NNTP_SSL(host, port=563, user=None, password=None, ssl_context=None, readermode=None, usenetrc=False, [timeout]) + + Return a new :class:`NNTP_SSL` object, representing an encrypted + connection to the NNTP server running on host *host*, listening at + port *port*. :class:`NNTP_SSL` objects have the same methods as + :class:`NNTP` objects. If *port* is omitted, port 563 (NNTPS) is used. + *ssl_context* is also optional, and is a :class:`~ssl.SSLContext` object. + All other parameters behave the same as for :class:`NNTP`. + + Note that SSL-on-563 is discouraged per :rfc:`4642`, in favor of + STARTTLS as described below. However, some servers only support the + former. + + .. versionadded:: 3.2 .. exception:: NNTPError - Derived from the standard exception :exc:`Exception`, this is the base class for - all exceptions raised by the :mod:`nntplib` module. + Derived from the standard exception :exc:`Exception`, this is the base + class for all exceptions raised by the :mod:`nntplib` module. Instances + of this class have the following attribute: + + .. attribute:: response + + The response of the server if available, as a :class:`str` object. .. exception:: NNTPReplyError - Exception raised when an unexpected reply is received from the server. For - backwards compatibility, the exception ``error_reply`` is equivalent to this - class. + Exception raised when an unexpected reply is received from the server. .. exception:: NNTPTemporaryError - Exception raised when an error code in the range 400--499 is received. For - backwards compatibility, the exception ``error_temp`` is equivalent to this - class. + Exception raised when a response code in the range 400--499 is received. .. exception:: NNTPPermanentError - Exception raised when an error code in the range 500--599 is received. For - backwards compatibility, the exception ``error_perm`` is equivalent to this - class. + Exception raised when a response code in the range 500--599 is received. .. exception:: NNTPProtocolError Exception raised when a reply is received from the server that does not begin - with a digit in the range 1--5. For backwards compatibility, the exception - ``error_proto`` is equivalent to this class. + with a digit in the range 1--5. .. exception:: NNTPDataError - Exception raised when there is some error in the response data. For backwards - compatibility, the exception ``error_data`` is equivalent to this class. + Exception raised when there is some error in the response data. .. _nntp-objects: @@ -112,10 +133,51 @@ The module itself defines the following items: NNTP Objects ------------ -NNTP instances have the following methods. The *response* that is returned as -the first item in the return tuple of almost all methods is the server's -response: a string beginning with a three-digit code. If the server's response -indicates an error, the method raises one of the above exceptions. +When connected, :class:`NNTP` and :class:`NNTP_SSL` objects support the +following methods and attributes. + +Attributes +^^^^^^^^^^ + +.. attribute:: NNTP.nntp_version + + An integer representing the version of the NNTP protocol supported by the + server. In practice, this should be ``2`` for servers advertising + :rfc:`3977` compliance and ``1`` for others. + + .. versionadded:: 3.2 + +.. attribute:: NNTP.nntp_implementation + + A string describing the software name and version of the NNTP server, + or :const:`None` if not advertised by the server. + + .. versionadded:: 3.2 + +Methods +^^^^^^^ + +The *response* that is returned as the first item in the return tuple of almost +all methods is the server's response: a string beginning with a three-digit +code. If the server's response indicates an error, the method raises one of +the above exceptions. + +Many of the following methods take an optional keyword-only argument *file*. +When the *file* argument is supplied, it must be either a :term:`file object` +opened for binary writing, or the name of an on-disk file to be written to. +The method will then write any data returned by the server (except for the +response line and the terminating dot) to the file; any list of lines, +tuples or objects that the method normally returns will be empty. + +.. versionchanged:: 3.2 + Many of the following methods have been reworked and fixed, which makes + them incompatible with their 3.1 counterparts. + + +.. method:: NNTP.quit() + + Send a ``QUIT`` command and close the connection. Once this method has been + called, no other methods of the NNTP object should be called. .. method:: NNTP.getwelcome() @@ -125,62 +187,114 @@ indicates an error, the method raises one of the above exceptions. that may be relevant to the user.) -.. method:: NNTP.set_debuglevel(level) +.. method:: NNTP.getcapabilities() - Set the instance's debugging level. This controls the amount of debugging - output printed. The default, ``0``, produces no debugging output. A value of - ``1`` produces a moderate amount of debugging output, generally a single line - per request or response. A value of ``2`` or higher produces the maximum amount - of debugging output, logging each line sent and received on the connection - (including message text). + Return the :rfc:`3977` capabilities advertised by the server, as a + :class:`dict` instance mapping capability names to (possibly empty) lists + of values. On legacy servers which don't understand the ``CAPABILITIES`` + command, an empty dictionary is returned instead. + + >>> s = NNTP('news.gmane.org') + >>> 'POST' in s.getcapabilities() + True + + .. versionadded:: 3.2 + + +.. method:: NNTP.login(user=None, password=None, usenetrc=True) + + Send ``AUTHINFO`` commands with the user name and password. If *user* + and *password* are None and *usenetrc* is True, credentials from + ``~/.netrc`` will be used if possible. + Unless intentionally delayed, login is normally performed during the + :class:`NNTP` object initialization and separately calling this function + is unnecessary. To force authentication to be delayed, you must not set + *user* or *password* when creating the object, and must set *usenetrc* to + False. -.. method:: NNTP.newgroups(date, time, [file]) + .. versionadded:: 3.2 - Send a ``NEWGROUPS`` command. The *date* argument should be a string of the - form ``'yymmdd'`` indicating the date, and *time* should be a string of the form - ``'hhmmss'`` indicating the time. Return a pair ``(response, groups)`` where - *groups* is a list of group names that are new since the given date and time. If - the *file* parameter is supplied, then the output of the ``NEWGROUPS`` command - is stored in a file. If *file* is a string, then the method will open a file - object with that name, write to it then close it. If *file* is a :term:`file - object`, then it will start calling :meth:`write` on it to store the lines of - the command output. If *file* is supplied, then the returned *list* is an empty list. +.. method:: NNTP.starttls(ssl_context=None) -.. method:: NNTP.newnews(group, date, time, [file]) + Send a ``STARTTLS`` command. The *ssl_context* argument is optional + and should be a :class:`ssl.SSLContext` object. This will enable + encryption on the NNTP connection. + + Note that this may not be done after authentication information has + been transmitted, and authentication occurs by default if possible during a + :class:`NNTP` object initialization. See :meth:`NNTP.login` for information + on suppressing this behavior. + + .. versionadded:: 3.2 + + +.. method:: NNTP.newgroups(date, *, file=None) + + Send a ``NEWGROUPS`` command. The *date* argument should be a + :class:`datetime.date` or :class:`datetime.datetime` object. + Return a pair ``(response, groups)`` where *groups* is a list representing + the groups that are new since the given *date*. If *file* is supplied, + though, then *groups* will be empty. + + >>> from datetime import date, timedelta + >>> resp, groups = s.newgroups(date.today() - timedelta(days=3)) + >>> len(groups) + 85 + >>> groups[0] + GroupInfo(group='gmane.network.tor.devel', last='4', first='1', flag='m') + + +.. method:: NNTP.newnews(group, date, *, file=None) Send a ``NEWNEWS`` command. Here, *group* is a group name or ``'*'``, and - *date* and *time* have the same meaning as for :meth:`newgroups`. Return a pair - ``(response, articles)`` where *articles* is a list of message ids. If the - *file* parameter is supplied, then the output of the ``NEWNEWS`` command is - stored in a file. If *file* is a string, then the method will open a file - object with that name, write to it then close it. If *file* is a :term:`file - object`, then it will start calling :meth:`write` on it to store the lines of the - command output. If *file* is supplied, then the returned *list* is an empty list. - - -.. method:: NNTP.list([file]) - - Send a ``LIST`` command. Return a pair ``(response, list)`` where *list* is a - list of tuples. Each tuple has the form ``(group, last, first, flag)``, where - *group* is a group name, *last* and *first* are the last and first article - numbers (as strings), and *flag* is ``'y'`` if posting is allowed, ``'n'`` if - not, and ``'m'`` if the newsgroup is moderated. (Note the ordering: *last*, - *first*.) If the *file* parameter is supplied, then the output of the ``LIST`` - command is stored in a file. If *file* is a string, then the method will open - a file with that name, write to it then close it. If *file* is a :term:`file - object`, then it will start calling :meth:`write` on it to store the lines of - the command output. If *file* is supplied, then the returned *list* is an empty - list. + *date* has the same meaning as for :meth:`newgroups`. Return a pair + ``(response, articles)`` where *articles* is a list of message ids. + + This command is frequently disabled by NNTP server administrators. + + +.. method:: NNTP.list(group_pattern=None, *, file=None) + + Send a ``LIST`` or ``LIST ACTIVE`` command. Return a pair + ``(response, list)`` where *list* is a list of tuples representing all + the groups available from this NNTP server, optionally matching the + pattern string *group_pattern*. Each tuple has the form + ``(group, last, first, flag)``, where *group* is a group name, *last* + and *first* are the last and first article numbers, and *flag* usually + takes one of these values: + + * ``y``: Local postings and articles from peers are allowed. + * ``m``: The group is moderated and all postings must be approved. + * ``n``: No local postings are allowed, only articles from peers. + * ``j``: Articles from peers are filed in the junk group instead. + * ``x``: No local postings, and articles from peers are ignored. + * ``=foo.bar``: Articles are filed in the ``foo.bar`` group instead. + + If *flag* has another value, then the status of the newsgroup should be + considered unknown. + + This command can return very large results, especially if *group_pattern* + is not specified. It is best to cache the results offline unless you + really need to refresh them. + + .. versionchanged:: 3.2 + *group_pattern* was added. .. method:: NNTP.descriptions(grouppattern) Send a ``LIST NEWSGROUPS`` command, where *grouppattern* is a wildmat string as - specified in RFC2980 (it's essentially the same as DOS or UNIX shell wildcard - strings). Return a pair ``(response, list)``, where *list* is a list of tuples - containing ``(name, title)``. + specified in :rfc:`3977` (it's essentially the same as DOS or UNIX shell wildcard + strings). Return a pair ``(response, descriptions)``, where *descriptions* + is a dictionary mapping group names to textual descriptions. + + >>> resp, descs = s.descriptions('gmane.comp.python.*') + >>> len(descs) + 295 + >>> descs.popitem() + ('gmane.comp.python.bio.general', 'BioPython discussion list (Moderated)') .. method:: NNTP.description(group) @@ -195,30 +309,75 @@ indicates an error, the method raises one of the above exceptions. .. method:: NNTP.group(name) - Send a ``GROUP`` command, where *name* is the group name. Return a tuple - ``(response, count, first, last, name)`` where *count* is the (estimated) number - of articles in the group, *first* is the first article number in the group, - *last* is the last article number in the group, and *name* is the group name. - The numbers are returned as strings. + Send a ``GROUP`` command, where *name* is the group name. The group is + selected as the current group, if it exists. Return a tuple + ``(response, count, first, last, name)`` where *count* is the (estimated) + number of articles in the group, *first* is the first article number in + the group, *last* is the last article number in the group, and *name* + is the group name. + + +.. method:: NNTP.over(message_spec, *, file=None) + + Send a ``OVER`` command, or a ``XOVER`` command on legacy servers. + *message_spec* can be either a string representing a message id, or + a ``(first, last)`` tuple of numbers indicating a range of articles in + the current group, or a ``(first, None)`` tuple indicating a range of + articles starting from *first* to the last article in the current group, + or :const:`None` to select the current article in the current group. + + Return a pair ``(response, overviews)``. *overviews* is a list of + ``(article_number, overview)`` tuples, one for each article selected + by *message_spec*. Each *overview* is a dictionary with the same number + of items, but this number depends on the server. These items are either + message headers (the key is then the lower-cased header name) or metadata + items (the key is then the metadata name prepended with ``":"``). The + following items are guaranteed to be present by the NNTP specification: + + * the ``subject``, ``from``, ``date``, ``message-id`` and ``references`` + headers + * the ``:bytes`` metadata: the number of bytes in the entire raw article + (including headers and body) + * the ``:lines`` metadata: the number of lines in the article body + + The value of each item is either a string, or :const:`None` if not present. + + It is advisable to use the :func:`decode_header` function on header + values when they may contain non-ASCII characters:: + + >>> _, _, first, last, _ = s.group('gmane.comp.python.devel') + >>> resp, overviews = s.over((last, last)) + >>> art_num, over = overviews[0] + >>> art_num + 117216 + >>> list(over.keys()) + ['xref', 'from', ':lines', ':bytes', 'references', 'date', 'message-id', 'subject'] + >>> over['from'] + '=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?= <martin@v.loewis.de>' + >>> nntplib.decode_header(over['from']) + '"Martin v. Löwis" <martin@v.loewis.de>' + + .. versionadded:: 3.2 -.. method:: NNTP.help([file]) +.. method:: NNTP.help(*, file=None) Send a ``HELP`` command. Return a pair ``(response, list)`` where *list* is a - list of help strings. If the *file* parameter is supplied, then the output of - the ``HELP`` command is stored in a file. If *file* is a string, then the - method will open a file with that name, write to it then close it. If *file* - is a :term:`file object`, then it will start calling :meth:`write` on it to store - the lines of the command output. If *file* is supplied, then the returned *list* - is an empty list. + list of help strings. -.. method:: NNTP.stat(id) +.. method:: NNTP.stat(message_spec=None) - Send a ``STAT`` command, where *id* is the message id (enclosed in ``'<'`` and - ``'>'``) or an article number (as a string). Return a triple ``(response, - number, id)`` where *number* is the article number (as a string) and *id* is the - message id (enclosed in ``'<'`` and ``'>'``). + Send a ``STAT`` command, where *message_spec* is either a message id + (enclosed in ``'<'`` and ``'>'``) or an article number in the current group. + If *message_spec* is omitted or :const:`None`, the current article in the + current group is considered. Return a triple ``(response, number, id)`` + where *number* is the article number and *id* is the message id. + + >>> _, _, first, last, _ = s.group('gmane.comp.python.devel') + >>> resp, number, message_id = s.stat(first) + >>> number, message_id + (9099, '<20030112190404.GE29873@epoch.metaslash.com>') .. method:: NNTP.next() @@ -231,28 +390,69 @@ indicates an error, the method raises one of the above exceptions. Send a ``LAST`` command. Return as for :meth:`stat`. -.. method:: NNTP.head(id) +.. method:: NNTP.article(message_spec=None, *, file=None) + + Send an ``ARTICLE`` command, where *message_spec* has the same meaning as + for :meth:`stat`. Return a tuple ``(response, info)`` where *info* + is a :class:`~collections.namedtuple` with three attributes *number*, + *message_id* and *lines* (in that order). *number* is the article number + in the group (or 0 if the information is not available), *message_id* the + message id as a string, and *lines* a list of lines (without terminating + newlines) comprising the raw message including headers and body. + + >>> resp, info = s.article('<20030112190404.GE29873@epoch.metaslash.com>') + >>> info.number + 0 + >>> info.message_id + '<20030112190404.GE29873@epoch.metaslash.com>' + >>> len(info.lines) + 65 + >>> info.lines[0] + b'Path: main.gmane.org!not-for-mail' + >>> info.lines[1] + b'From: Neal Norwitz <neal@metaslash.com>' + >>> info.lines[-3:] + [b'There is a patch for 2.3 as well as 2.2.', b'', b'Neal'] + + +.. method:: NNTP.head(message_spec=None, *, file=None) + + Same as :meth:`article()`, but sends a ``HEAD`` command. The *lines* + returned (or written to *file*) will only contain the message headers, not + the body. + - Send a ``HEAD`` command, where *id* has the same meaning as for :meth:`stat`. - Return a tuple ``(response, number, id, list)`` where the first three are the - same as for :meth:`stat`, and *list* is a list of the article's headers (an - uninterpreted list of lines, without trailing newlines). +.. method:: NNTP.body(message_spec=None, *, file=None) + Same as :meth:`article()`, but sends a ``BODY`` command. The *lines* + returned (or written to *file*) will only contain the message body, not the + headers. -.. method:: NNTP.body(id,[file]) - Send a ``BODY`` command, where *id* has the same meaning as for :meth:`stat`. - If the *file* parameter is supplied, then the body is stored in a file. If - *file* is a string, then the method will open a file with that name, write - to it then close it. If *file* is a :term:`file object`, then it will start - calling :meth:`write` on it to store the lines of the body. Return as for - :meth:`head`. If *file* is supplied, then the returned *list* is an empty list. +.. method:: NNTP.post(data) + Post an article using the ``POST`` command. The *data* argument is either + a :term:`file object` opened for binary reading, or any iterable of bytes + objects (representing raw lines of the article to be posted). It should + represent a well-formed news article, including the required headers. The + :meth:`post` method automatically escapes lines beginning with ``.`` and + appends the termination line. -.. method:: NNTP.article(id) + If the method succeeds, the server's response is returned. If the server + refuses posting, a :class:`NNTPReplyError` is raised. + + +.. method:: NNTP.ihave(message_id, data) + + Send an ``IHAVE`` command. *message_id* is the id of the message to send + to the server (enclosed in ``'<'`` and ``'>'``). The *data* parameter + and the return value are the same as for :meth:`post()`. + + +.. method:: NNTP.date() - Send an ``ARTICLE`` command, where *id* has the same meaning as for - :meth:`stat`. Return as for :meth:`head`. + Return a pair ``(response, date)``. *date* is a :class:`~datetime.datetime` + object containing the current date and time of the server. .. method:: NNTP.slave() @@ -260,10 +460,23 @@ indicates an error, the method raises one of the above exceptions. Send a ``SLAVE`` command. Return the server's *response*. -.. method:: NNTP.xhdr(header, string, [file]) +.. method:: NNTP.set_debuglevel(level) - Send an ``XHDR`` command. This command is not defined in the RFC but is a - common extension. The *header* argument is a header keyword, e.g. + Set the instance's debugging level. This controls the amount of debugging + output printed. The default, ``0``, produces no debugging output. A value of + ``1`` produces a moderate amount of debugging output, generally a single line + per request or response. A value of ``2`` or higher produces the maximum amount + of debugging output, logging each line sent and received on the connection + (including message text). + + +The following are optional NNTP extensions defined in :rfc:`2980`. Some of +them have been superseded by newer commands in :rfc:`3977`. + + +.. method:: NNTP.xhdr(header, string, *, file=None) + + Send an ``XHDR`` command. The *header* argument is a header keyword, e.g. ``'subject'``. The *string* argument should have the form ``'first-last'`` where *first* and *last* are the first and last article numbers to search. Return a pair ``(response, list)``, where *list* is a list of pairs ``(id, @@ -276,66 +489,55 @@ indicates an error, the method raises one of the above exceptions. returned *list* is an empty list. -.. method:: NNTP.post(file) - - Post an article using the ``POST`` command. The *file* argument is an open file - object which is read until EOF using its :meth:`readline` method. It should be - a well-formed news article, including the required headers. The :meth:`post` - method automatically escapes lines beginning with ``.``. +.. method:: NNTP.xover(start, end, *, file=None) + Send an ``XOVER`` command. *start* and *end* are article numbers + delimiting the range of articles to select. The return value is the + same of for :meth:`over()`. It is recommended to use :meth:`over()` + instead, since it will automatically use the newer ``OVER`` command + if available. -.. method:: NNTP.ihave(id, file) - - Send an ``IHAVE`` command. *id* is a message id (enclosed in ``'<'`` and - ``'>'``). If the response is not an error, treat *file* exactly as for the - :meth:`post` method. - - -.. method:: NNTP.date() - - Return a triple ``(response, date, time)``, containing the current date and time - in a form suitable for the :meth:`newnews` and :meth:`newgroups` methods. This - is an optional NNTP extension, and may not be supported by all servers. +.. method:: NNTP.xpath(id) -.. method:: NNTP.xgtitle(name, [file]) + Return a pair ``(resp, path)``, where *path* is the directory path to the + article with message ID *id*. Most of the time, this extension is not + enabled by NNTP server administrators. - Process an ``XGTITLE`` command, returning a pair ``(response, list)``, where - *list* is a list of tuples containing ``(name, title)``. If the *file* parameter - is supplied, then the output of the ``XGTITLE`` command is stored in a file. - If *file* is a string, then the method will open a file with that name, write - to it then close it. If *file* is a :term:`file object`, then it will start - calling :meth:`write` on it to store the lines of the command output. If *file* - is supplied, then the returned *list* is an empty list. This is an optional NNTP - extension, and may not be supported by all servers. - RFC2980 says "It is suggested that this extension be deprecated". Use - :meth:`descriptions` or :meth:`description` instead. +.. XXX deprecated: + .. method:: NNTP.xgtitle(name, *, file=None) -.. method:: NNTP.xover(start, end, [file]) + Process an ``XGTITLE`` command, returning a pair ``(response, list)``, where + *list* is a list of tuples containing ``(name, title)``. If the *file* parameter + is supplied, then the output of the ``XGTITLE`` command is stored in a file. + If *file* is a string, then the method will open a file with that name, write + to it then close it. If *file* is a :term:`file object`, then it will start + calling :meth:`write` on it to store the lines of the command output. If *file* + is supplied, then the returned *list* is an empty list. This is an optional NNTP + extension, and may not be supported by all servers. - Return a pair ``(resp, list)``. *list* is a list of tuples, one for each - article in the range delimited by the *start* and *end* article numbers. Each - tuple is of the form ``(article number, subject, poster, date, id, references, - size, lines)``. If the *file* parameter is supplied, then the output of the - ``XOVER`` command is stored in a file. If *file* is a string, then the method - will open a file with that name, write to it then close it. If *file* is a - :term:`file object`, then it will start calling :meth:`write` on it to store the - lines of the command output. If *file* is supplied, then the returned *list* is - an empty list. This is an optional NNTP extension, and may not be supported by - all servers. + RFC2980 says "It is suggested that this extension be deprecated". Use + :meth:`descriptions` or :meth:`description` instead. -.. method:: NNTP.xpath(id) +Utility functions +----------------- - Return a pair ``(resp, path)``, where *path* is the directory path to the - article with message ID *id*. This is an optional NNTP extension, and may not - be supported by all servers. +The module also defines the following utility function: -.. method:: NNTP.quit() +.. function:: decode_header(header_str) - Send a ``QUIT`` command and close the connection. Once this method has been - called, no other methods of the NNTP object should be called. + Decode a header value, un-escaping any escaped non-ASCII characters. + *header_str* must be a :class:`str` object. The unescaped value is + returned. Using this function is recommended to display some headers + in a human readable form:: + >>> decode_header("Some subject") + 'Some subject' + >>> decode_header("=?ISO-8859-15?Q?D=E9buter_en_Python?=") + 'Débuter en Python' + >>> decode_header("Re: =?UTF-8?B?cHJvYmzDqG1lIGRlIG1hdHJpY2U=?=") + 'Re: problème de matrice' diff --git a/Doc/library/numbers.rst b/Doc/library/numbers.rst index df8d52168c..ad33396f10 100644 --- a/Doc/library/numbers.rst +++ b/Doc/library/numbers.rst @@ -5,9 +5,9 @@ :synopsis: Numeric abstract base classes (Complex, Real, Integral, etc.). -The :mod:`numbers` module (:pep:`3141`) defines a hierarchy of numeric abstract -base classes which progressively define more operations. None of the types -defined in this module can be instantiated. +The :mod:`numbers` module (:pep:`3141`) defines a hierarchy of numeric +:term:`abstract base classes <abstract base class>` which progressively define +more operations. None of the types defined in this module can be instantiated. .. class:: Number diff --git a/Doc/library/numeric.rst b/Doc/library/numeric.rst index 641f58885c..ba22cb6882 100644 --- a/Doc/library/numeric.rst +++ b/Doc/library/numeric.rst @@ -23,6 +23,3 @@ The following modules are documented in this chapter: decimal.rst fractions.rst random.rst - itertools.rst - functools.rst - operator.rst diff --git a/Doc/library/objects.rst b/Doc/library/objects.rst deleted file mode 100644 index 1b7516117a..0000000000 --- a/Doc/library/objects.rst +++ /dev/null @@ -1,27 +0,0 @@ - -.. _builtin: - -**************** -Built-in Objects -**************** - -.. index:: - pair: built-in; types - pair: built-in; exceptions - pair: built-in; functions - pair: built-in; constants - single: symbol table - -Names for built-in exceptions and functions and a number of constants are found -in a separate symbol table. This table is searched last when the interpreter -looks up the meaning of a name, so local and global user-defined names can -override built-in names. Built-in types are described together here for easy -reference. - -The tables in this chapter document the priorities of operators by listing them -in order of ascending priority (within a table) and grouping operators that have -the same priority in the same box. Binary operators of the same priority group -from left to right. (Unary operators group from right to left, but there you -have no real choice.) See :ref:`operator-summary` for the complete picture on -operator priorities. - diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst index 43bdeef465..b03d9df208 100644 --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -9,14 +9,14 @@ .. testsetup:: import operator - from operator import itemgetter + from operator import itemgetter, iadd -The :mod:`operator` module exports a set of functions implemented in C -corresponding to the intrinsic operators of Python. For example, -``operator.add(x, y)`` is equivalent to the expression ``x+y``. The function -names are those used for special class methods; variants without leading and -trailing ``__`` are also provided for convenience. +The :mod:`operator` module exports a set of efficient functions corresponding to +the intrinsic operators of Python. For example, ``operator.add(x, y)`` is +equivalent to the expression ``x+y``. The function names are those used for +special class methods; variants without leading and trailing ``__`` are also +provided for convenience. The functions fall into categories that perform object comparisons, logical operations, mathematical operations and sequence operations. @@ -225,91 +225,6 @@ Operations which work with sequences (some of them with mappings too) include: Set the value of *a* at index *b* to *c*. - -Many operations have an "in-place" version. The following functions provide a -more primitive access to in-place operators than the usual syntax does; for -example, the :term:`statement` ``x += y`` is equivalent to -``x = operator.iadd(x, y)``. Another way to put it is to say that -``z = operator.iadd(x, y)`` is equivalent to the compound statement -``z = x; z += y``. - -.. function:: iadd(a, b) - __iadd__(a, b) - - ``a = iadd(a, b)`` is equivalent to ``a += b``. - - -.. function:: iand(a, b) - __iand__(a, b) - - ``a = iand(a, b)`` is equivalent to ``a &= b``. - - -.. function:: iconcat(a, b) - __iconcat__(a, b) - - ``a = iconcat(a, b)`` is equivalent to ``a += b`` for *a* and *b* sequences. - - -.. function:: ifloordiv(a, b) - __ifloordiv__(a, b) - - ``a = ifloordiv(a, b)`` is equivalent to ``a //= b``. - - -.. function:: ilshift(a, b) - __ilshift__(a, b) - - ``a = ilshift(a, b)`` is equivalent to ``a <<= b``. - - -.. function:: imod(a, b) - __imod__(a, b) - - ``a = imod(a, b)`` is equivalent to ``a %= b``. - - -.. function:: imul(a, b) - __imul__(a, b) - - ``a = imul(a, b)`` is equivalent to ``a *= b``. - - -.. function:: ior(a, b) - __ior__(a, b) - - ``a = ior(a, b)`` is equivalent to ``a |= b``. - - -.. function:: ipow(a, b) - __ipow__(a, b) - - ``a = ipow(a, b)`` is equivalent to ``a **= b``. - - -.. function:: irshift(a, b) - __irshift__(a, b) - - ``a = irshift(a, b)`` is equivalent to ``a >>= b``. - - -.. function:: isub(a, b) - __isub__(a, b) - - ``a = isub(a, b)`` is equivalent to ``a -= b``. - - -.. function:: itruediv(a, b) - __itruediv__(a, b) - - ``a = itruediv(a, b)`` is equivalent to ``a /= b``. - - -.. function:: ixor(a, b) - __ixor__(a, b) - - ``a = ixor(a, b)`` is equivalent to ``a ^= b``. - Example: Build a dictionary that maps the ordinals from ``0`` to ``255`` to their character equivalents. @@ -335,6 +250,8 @@ expect a function argument. b.date)``. Equivalent to:: def attrgetter(*items): + if any(not isinstance(item, str) for item in items): + raise TypeError('attribute name must be a string') if len(items) == 1: attr = items[0] def g(obj): @@ -461,8 +378,6 @@ Python syntax and the functions in the :mod:`operator` module. +-----------------------+-------------------------+---------------------------------------+ | Right Shift | ``a >> b`` | ``rshift(a, b)`` | +-----------------------+-------------------------+---------------------------------------+ -| Sequence Repetition | ``seq * i`` | ``repeat(seq, i)`` | -+-----------------------+-------------------------+---------------------------------------+ | Slice Assignment | ``seq[i:j] = values`` | ``setitem(seq, slice(i, j), values)`` | +-----------------------+-------------------------+---------------------------------------+ | Slice Deletion | ``del seq[i:j]`` | ``delitem(seq, slice(i, j))`` | @@ -488,3 +403,112 @@ Python syntax and the functions in the :mod:`operator` module. | Ordering | ``a > b`` | ``gt(a, b)`` | +-----------------------+-------------------------+---------------------------------------+ +Inplace Operators +================= + +Many operations have an "in-place" version. Listed below are functions +providing a more primitive access to in-place operators than the usual syntax +does; for example, the :term:`statement` ``x += y`` is equivalent to +``x = operator.iadd(x, y)``. Another way to put it is to say that +``z = operator.iadd(x, y)`` is equivalent to the compound statement +``z = x; z += y``. + +In those examples, note that when an in-place method is called, the computation +and assignment are performed in two separate steps. The in-place functions +listed below only do the first step, calling the in-place method. The second +step, assignment, is not handled. + +For immutable targets such as strings, numbers, and tuples, the updated +value is computed, but not assigned back to the input variable: + +>>> a = 'hello' +>>> iadd(a, ' world') +'hello world' +>>> a +'hello' + +For mutable targets such as lists and dictionaries, the inplace method +will perform the update, so no subsequent assignment is necessary: + +>>> s = ['h', 'e', 'l', 'l', 'o'] +>>> iadd(s, [' ', 'w', 'o', 'r', 'l', 'd']) +['h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd'] +>>> s +['h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd'] + +.. function:: iadd(a, b) + __iadd__(a, b) + + ``a = iadd(a, b)`` is equivalent to ``a += b``. + + +.. function:: iand(a, b) + __iand__(a, b) + + ``a = iand(a, b)`` is equivalent to ``a &= b``. + + +.. function:: iconcat(a, b) + __iconcat__(a, b) + + ``a = iconcat(a, b)`` is equivalent to ``a += b`` for *a* and *b* sequences. + + +.. function:: ifloordiv(a, b) + __ifloordiv__(a, b) + + ``a = ifloordiv(a, b)`` is equivalent to ``a //= b``. + + +.. function:: ilshift(a, b) + __ilshift__(a, b) + + ``a = ilshift(a, b)`` is equivalent to ``a <<= b``. + + +.. function:: imod(a, b) + __imod__(a, b) + + ``a = imod(a, b)`` is equivalent to ``a %= b``. + + +.. function:: imul(a, b) + __imul__(a, b) + + ``a = imul(a, b)`` is equivalent to ``a *= b``. + + +.. function:: ior(a, b) + __ior__(a, b) + + ``a = ior(a, b)`` is equivalent to ``a |= b``. + + +.. function:: ipow(a, b) + __ipow__(a, b) + + ``a = ipow(a, b)`` is equivalent to ``a **= b``. + + +.. function:: irshift(a, b) + __irshift__(a, b) + + ``a = irshift(a, b)`` is equivalent to ``a >>= b``. + + +.. function:: isub(a, b) + __isub__(a, b) + + ``a = isub(a, b)`` is equivalent to ``a -= b``. + + +.. function:: itruediv(a, b) + __itruediv__(a, b) + + ``a = itruediv(a, b)`` is equivalent to ``a /= b``. + + +.. function:: ixor(a, b) + __ixor__(a, b) + + ``a = ixor(a, b)`` is equivalent to ``a ^= b``. diff --git a/Doc/library/optparse.rst b/Doc/library/optparse.rst index 87ec4de8d9..add582efed 100644 --- a/Doc/library/optparse.rst +++ b/Doc/library/optparse.rst @@ -1,11 +1,19 @@ -:mod:`optparse` --- More powerful command line option parser -============================================================ +:mod:`optparse` --- Parser for command line options +=================================================== .. module:: optparse - :synopsis: More convenient, flexible, and powerful command-line parsing library. + :synopsis: Command-line option parsing library. + :deprecated: .. moduleauthor:: Greg Ward <gward@python.net> .. sectionauthor:: Greg Ward <gward@python.net> +.. deprecated:: 3.2 + The :mod:`optparse` module is deprecated and will not be developed further; + development will continue with the :mod:`argparse` module. + +**Source code:** :source:`Lib/optparse.py` + +-------------- :mod:`optparse` is a more convenient, flexible, and powerful library for parsing command-line options than the old :mod:`getopt` module. :mod:`optparse` uses a @@ -599,8 +607,8 @@ This would result in the following help output: -g Group option. -A bit more complete example might invole using more than one group: still -extendind the previous example:: +A bit more complete example might involve using more than one group: still +extending the previous example:: group = OptionGroup(parser, "Dangerous Options", "Caution: use these options at your own risk. " @@ -647,8 +655,9 @@ option groups is: .. method:: OptionParser.get_option_group(opt_str) - Return, if defined, the :class:`OptionGroup` that has the title or the long - description equals to *opt_str* + Return the :class:`OptionGroup` to which the short or long option + string *opt_str* (e.g. ``'-o'`` or ``'--option'``) belongs. If + there's no such :class:`OptionGroup`, return ``None``. .. _optparse-printing-version-string: diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index 28a7affa7a..214e27c601 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -190,10 +190,11 @@ applications should use string objects to access all files. path, all previous components (on Windows, including the previous drive letter, if there was one) are thrown away, and joining continues. The return value is the concatenation of *path1*, and optionally *path2*, etc., with exactly one - directory separator (``os.sep``) inserted between components, unless *path2* is - empty. Note that on Windows, since there is a current directory for each drive, - ``os.path.join("c:", "foo")`` represents a path relative to the current - directory on drive :file:`C:` (:file:`c:foo`), not :file:`c:\\foo`. + directory separator (``os.sep``) following each non-empty part except the last. + (This means that an empty last part will result in a path that ends with a + separator.) Note that on Windows, since there is a current directory for + each drive, ``os.path.join("c:", "foo")`` represents a path relative to the + current directory on drive :file:`C:` (:file:`c:foo`), not :file:`c:\\foo`. .. function:: normcase(path) @@ -201,6 +202,7 @@ applications should use string objects to access all files. Normalize the case of a pathname. On Unix and Mac OS X, this returns the path unchanged; on case-insensitive filesystems, it converts the path to lowercase. On Windows, it also converts forward slashes to backward slashes. + Raise a TypeError if the type of *path* is not ``str`` or ``bytes``. .. function:: normpath(path) @@ -220,7 +222,7 @@ applications should use string objects to access all files. links encountered in the path (if they are supported by the operating system). -.. function:: relpath(path[, start]) +.. function:: relpath(path, start=None) Return a relative filepath to *path* either from the current directory or from an optional *start* point. @@ -232,18 +234,27 @@ applications should use string objects to access all files. .. function:: samefile(path1, path2) - Return ``True`` if both pathname arguments refer to the same file or directory - (as indicated by device number and i-node number). Raise an exception if a - :func:`os.stat` call on either pathname fails. + Return ``True`` if both pathname arguments refer to the same file or directory. + On Unix, this is determined by the device number and i-node number and raises an + exception if a :func:`os.stat` call on either pathname fails. - Availability: Unix. + On Windows, two files are the same if they resolve to the same final path + name using the Windows API call GetFinalPathNameByHandle. This function + raises an exception if handles cannot be obtained to either file. + + Availability: Unix, Windows. + + .. versionchanged:: 3.2 + Added Windows support. .. function:: sameopenfile(fp1, fp2) Return ``True`` if the file descriptors *fp1* and *fp2* refer to the same file. - Availability: Unix. + Availability: Unix, Windows. + + .. versionchanged:: 3.2 Added Windows support. .. function:: samestat(stat1, stat2) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 187e44a032..be322a01fe 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -29,11 +29,6 @@ Notes on the availability of these functions: objects, and result in an object of the same type, if a path or file name is returned. -.. note:: - - If not separately noted, all functions that claim "Availability: Unix" are - supported on Mac OS X, which builds on a Unix core. - * An "Availability: Unix" note means that this function is commonly found on Unix systems. It does not make any claims about its existence on a specific operating system. @@ -61,6 +56,13 @@ Notes on the availability of these functions: names have currently been registered: ``'posix'``, ``'nt'``, ``'mac'``, ``'os2'``, ``'ce'``, ``'java'``. + .. seealso:: + :attr:`sys.platform` has a finer granularity. :func:`os.uname` gives + system-dependent version information. + + The :mod:`platform` module provides detailed checks for the + system's identity. + .. _os-filenames: @@ -109,6 +111,10 @@ process and user. to modify the environment as well as query the environment. :func:`putenv` will be called automatically when the mapping is modified. + On Unix, keys and values use :func:`sys.getfilesystemencoding` and + ``'surrogateescape'`` error handler. Use :data:`environb` if you would like + to use a different encoding. + .. note:: Calling :func:`putenv` directly does not change ``os.environ``, so it's better @@ -118,7 +124,7 @@ process and user. On some platforms, including FreeBSD and Mac OS X, setting ``environ`` may cause memory leaks. Refer to the system documentation for - :cfunc:`putenv`. + :c:func:`putenv`. If :func:`putenv` is not provided, a modified copy of this mapping may be passed to the appropriate process-creation functions to cause child processes @@ -130,6 +136,19 @@ process and user. one of the :meth:`pop` or :meth:`clear` methods is called. +.. data:: environb + + Bytes version of :data:`environ`: a mapping object representing the + environment as byte strings. :data:`environ` and :data:`environb` are + synchronized (modify :data:`environb` updates :data:`environ`, and vice + versa). + + :data:`environb` is only available if :data:`supports_bytes_environ` is + True. + + .. versionadded:: 3.2 + + .. function:: chdir(path) fchdir(fd) getcwd() @@ -138,6 +157,37 @@ process and user. These functions are described in :ref:`os-file-dir`. +.. function:: fsencode(filename) + + Encode *filename* to the filesystem encoding with ``'surrogateescape'`` + error handler, or ``'strict'`` on Windows; return :class:`bytes` unchanged. + + :func:`fsdecode` is the reverse function. + + .. versionadded:: 3.2 + + +.. function:: fsdecode(filename) + + Decode *filename* from the filesystem encoding with ``'surrogateescape'`` + error handler, or ``'strict'`` on Windows; return :class:`str` unchanged. + + :func:`fsencode` is the reverse function. + + .. versionadded:: 3.2 + + +.. function:: get_exec_path(env=None) + + Returns the list of directories that will be searched for a named + executable, similar to a shell, when launching a process. + *env*, when specified, should be an environment variable dictionary + to lookup the PATH in. + By default, when *env* is None, :data:`environ` is used. + + .. versionadded:: 3.2 + + .. function:: ctermid() Return the filename corresponding to the controlling terminal of the process. @@ -178,15 +228,26 @@ process and user. Availability: Unix. +.. function:: initgroups(username, gid) + + Call the system initgroups() to initialize the group access list with all of + the groups of which the specified username is a member, plus the specified + group id. + + Availability: Unix. + + .. versionadded:: 3.2 + + .. function:: getlogin() Return the name of the user logged in on the controlling terminal of the - process. For most purposes, it is more useful to use the environment variable - :envvar:`LOGNAME` to find out who the user is, or + process. For most purposes, it is more useful to use the environment variables + :envvar:`LOGNAME` or :envvar:`USERNAME` to find out who the user is, or ``pwd.getpwuid(os.getuid())[0]`` to get the login name of the currently effective user id. - Availability: Unix. + Availability: Unix, Windows. .. function:: getpgid(pid) @@ -218,10 +279,34 @@ process and user. .. index:: single: process; id of parent - Return the parent's process id. + Return the parent's process id. When the parent process has exited, on Unix + the id returned is the one of the init process (1), on Windows it is still + the same id, which may be already reused by another process. + + Availability: Unix, Windows + + .. versionchanged:: 3.2 + Added support for Windows. + +.. function:: getresuid() + + Return a tuple (ruid, euid, suid) denoting the current process's + real, effective, and saved user ids. + + Availability: Unix. + + .. versionadded:: 3.2 + + +.. function:: getresgid() + + Return a tuple (rgid, egid, sgid) denoting the current process's + real, effective, and saved group ids. Availability: Unix. + .. versionadded:: 3.2 + .. function:: getuid() @@ -232,19 +317,33 @@ process and user. Availability: Unix. -.. function:: getenv(varname[, value]) +.. function:: getenv(key, default=None) + + Return the value of the environment variable *key* if it exists, or + *default* if it doesn't. *key*, *default* and the result are str. - Return the value of the environment variable *varname* if it exists, or *value* - if it doesn't. *value* defaults to ``None``. + On Unix, keys and values are decoded with :func:`sys.getfilesystemencoding` + and ``'surrogateescape'`` error handler. Use :func:`os.getenvb` if you + would like to use a different encoding. Availability: most flavors of Unix, Windows. -.. function:: putenv(varname, value) +.. function:: getenvb(key, default=None) + + Return the value of the environment variable *key* if it exists, or + *default* if it doesn't. *key*, *default* and the result are bytes. + + Availability: most flavors of Unix. + + .. versionadded:: 3.2 + + +.. function:: putenv(key, value) .. index:: single: environment variables; setting - Set the environment variable named *varname* to the string *value*. Such + Set the environment variable named *key* to the string *value*. Such changes to the environment affect subprocesses started with :func:`os.system`, :func:`popen` or :func:`fork` and :func:`execv`. @@ -293,7 +392,7 @@ process and user. .. function:: setpgrp() - Call the system call :cfunc:`setpgrp` or :cfunc:`setpgrp(0, 0)` depending on + Call the system call :c:func:`setpgrp` or :c:func:`setpgrp(0, 0)` depending on which version is implemented (if any). See the Unix manual for the semantics. Availability: Unix. @@ -301,37 +400,55 @@ process and user. .. function:: setpgid(pid, pgrp) - Call the system call :cfunc:`setpgid` to set the process group id of the + Call the system call :c:func:`setpgid` to set the process group id of the process with id *pid* to the process group with id *pgrp*. See the Unix manual for the semantics. Availability: Unix. -.. function:: setreuid(ruid, euid) +.. function:: setregid(rgid, egid) - Set the current process's real and effective user ids. + Set the current process's real and effective group ids. Availability: Unix. -.. function:: setregid(rgid, egid) +.. function:: setresgid(rgid, egid, sgid) - Set the current process's real and effective group ids. + Set the current process's real, effective, and saved group ids. + + Availability: Unix. + + .. versionadded:: 3.2 + + +.. function:: setresuid(ruid, euid, suid) + + Set the current process's real, effective, and saved user ids. + + Availability: Unix. + + .. versionadded:: 3.2 + + +.. function:: setreuid(ruid, euid) + + Set the current process's real and effective user ids. Availability: Unix. .. function:: getsid(pid) - Call the system call :cfunc:`getsid`. See the Unix manual for the semantics. + Call the system call :c:func:`getsid`. See the Unix manual for the semantics. Availability: Unix. .. function:: setsid() - Call the system call :cfunc:`setsid`. See the Unix manual for the semantics. + Call the system call :c:func:`setsid`. See the Unix manual for the semantics. Availability: Unix. @@ -349,12 +466,20 @@ process and user. .. function:: strerror(code) Return the error message corresponding to the error code in *code*. - On platforms where :cfunc:`strerror` returns ``NULL`` when given an unknown + On platforms where :c:func:`strerror` returns ``NULL`` when given an unknown error number, :exc:`ValueError` is raised. Availability: Unix, Windows. +.. data:: supports_bytes_environ + + True if the native OS type of the environment is bytes (eg. False on + Windows). + + .. versionadded:: 3.2 + + .. function:: umask(mask) Set the current numeric umask and return the previous umask. @@ -378,11 +503,11 @@ process and user. Availability: recent flavors of Unix. -.. function:: unsetenv(varname) +.. function:: unsetenv(key) .. index:: single: environment variables; deleting - Unset (delete) the environment variable named *varname*. Such changes to the + Unset (delete) the environment variable named *key*. Such changes to the environment affect subprocesses started with :func:`os.system`, :func:`popen` or :func:`fork` and :func:`execv`. @@ -414,7 +539,7 @@ These functions create new :term:`file objects <file object>`. (See also :func:` ``'r'``, ``'w'``, or ``'a'``, otherwise a :exc:`ValueError` is raised. On Unix, when the *mode* argument starts with ``'a'``, the *O_APPEND* flag is - set on the file descriptor (which the :cfunc:`fdopen` implementation already + set on the file descriptor (which the :c:func:`fdopen` implementation already does on most platforms). Availability: Unix, Windows. @@ -550,7 +675,7 @@ as internal buffering of data. .. function:: fsync(fd) Force write of file with filedescriptor *fd* to disk. On Unix, this calls the - native :cfunc:`fsync` function; on Windows, the MS :cfunc:`_commit` function. + native :c:func:`fsync` function; on Windows, the MS :c:func:`_commit` function. If you're starting with a buffered Python :term:`file object` *f*, first do ``f.flush()``, and then do ``os.fsync(f.fileno())``, to ensure that all internal @@ -581,7 +706,7 @@ as internal buffering of data. by *how*: :const:`SEEK_SET` or ``0`` to set the position relative to the beginning of the file; :const:`SEEK_CUR` or ``1`` to set it relative to the current position; :const:`os.SEEK_END` or ``2`` to set it relative to the end of - the file. + the file. Return the new cursor position in bytes, starting from the beginning. Availability: Unix, Windows. @@ -612,7 +737,7 @@ as internal buffering of data. This function is intended for low-level I/O. For normal usage, use the built-in function :func:`open`, which returns a :term:`file object` with - :meth:`~file.read` and :meth:`~file.wprite` methods (and many more). To + :meth:`~file.read` and :meth:`~file.write` methods (and many more). To wrap a file descriptor in a file object, use :func:`fdopen`. @@ -785,7 +910,7 @@ Files and Directories try: fp = open("myfile") except IOError as e: - if e.errno == errno.EACCESS: + if e.errno == errno.EACCES: return "some default data" # Not a permission error. raise @@ -866,6 +991,8 @@ Files and Directories * :data:`stat.UF_APPEND` * :data:`stat.UF_OPAQUE` * :data:`stat.UF_NOUNLINK` + * :data:`stat.UF_COMPRESSED` + * :data:`stat.UF_HIDDEN` * :data:`stat.SF_ARCHIVED` * :data:`stat.SF_IMMUTABLE` * :data:`stat.SF_APPEND` @@ -954,13 +1081,16 @@ Files and Directories Create a hard link pointing to *source* named *link_name*. - Availability: Unix. + Availability: Unix, Windows. + .. versionchanged:: 3.2 + Added Windows support. -.. function:: listdir(path) + +.. function:: listdir(path='.') Return a list containing the names of the entries in the directory given by - *path*. The list is in arbitrary order. It does not include the special + *path* (default: ``'.'``). The list is in arbitrary order. It does not include the special entries ``'.'`` and ``'..'`` even if they are present in the directory. This function can be called with a bytes or string argument, and returns @@ -968,14 +1098,19 @@ Files and Directories Availability: Unix, Windows. + .. versionchanged:: 3.2 + The *path* parameter became optional. .. function:: lstat(path) - Perform the equivalent of an :cfunc:`lstat` system call on the given path. + Perform the equivalent of an :c:func:`lstat` system call on the given path. Similar to :func:`~os.stat`, but does not follow symbolic links. On platforms that do not support symbolic links, this is an alias for :func:`~os.stat`. + .. versionchanged:: 3.2 + Added support for Windows 6.0 (Vista) symbolic links. + .. function:: mkfifo(path[, mode]) @@ -992,28 +1127,27 @@ Files and Directories Availability: Unix. -.. function:: mknod(filename[, mode=0o600, device]) +.. function:: mknod(filename[, mode=0o600[, device]]) Create a filesystem node (file, device special file or named pipe) named - *filename*. *mode* specifies both the permissions to use and the type of node to - be created, being combined (bitwise OR) with one of ``stat.S_IFREG``, - ``stat.S_IFCHR``, ``stat.S_IFBLK``, - and ``stat.S_IFIFO`` (those constants are available in :mod:`stat`). - For ``stat.S_IFCHR`` and - ``stat.S_IFBLK``, *device* defines the newly created device special file (probably using + *filename*. *mode* specifies both the permissions to use and the type of node + to be created, being combined (bitwise OR) with one of ``stat.S_IFREG``, + ``stat.S_IFCHR``, ``stat.S_IFBLK``, and ``stat.S_IFIFO`` (those constants are + available in :mod:`stat`). For ``stat.S_IFCHR`` and ``stat.S_IFBLK``, + *device* defines the newly created device special file (probably using :func:`os.makedev`), otherwise it is ignored. .. function:: major(device) Extract the device major number from a raw device number (usually the - :attr:`st_dev` or :attr:`st_rdev` field from :ctype:`stat`). + :attr:`st_dev` or :attr:`st_rdev` field from :c:type:`stat`). .. function:: minor(device) Extract the device minor number from a raw device number (usually the - :attr:`st_dev` or :attr:`st_rdev` field from :ctype:`stat`). + :attr:`st_dev` or :attr:`st_rdev` field from :c:type:`stat`). .. function:: makedev(major, minor) @@ -1034,17 +1168,20 @@ Files and Directories Availability: Unix, Windows. -.. function:: makedirs(path, mode=0o777) +.. function:: makedirs(path, mode=0o777, exist_ok=False) .. index:: single: directory; creating single: UNC paths; and os.makedirs() Recursive directory creation function. Like :func:`mkdir`, but makes all - intermediate-level directories needed to contain the leaf directory. Raises - an :exc:`error` exception if the leaf directory already exists or cannot be - created. The default *mode* is ``0o777`` (octal). On some systems, *mode* - is ignored. Where it is used, the current umask value is first masked out. + intermediate-level directories needed to contain the leaf directory. If + the target directory with the same mode as specified already exists, + raises an :exc:`OSError` exception if *exist_ok* is False, otherwise no + exception is raised. If the directory cannot be created in other cases, + raises an :exc:`OSError` exception. The default *mode* is ``0o777`` (octal). + On some systems, *mode* is ignored. Where it is used, the current umask + value is first masked out. .. note:: @@ -1053,6 +1190,9 @@ Files and Directories This function handles UNC paths correctly. + .. versionadded:: 3.2 + The *exist_ok* parameter. + .. function:: pathconf(path, name) @@ -1091,7 +1231,10 @@ Files and Directories and the call may raise an UnicodeDecodeError. If the *path* is a bytes object, the result will be a bytes object. - Availability: Unix. + Availability: Unix, Windows + + .. versionchanged:: 3.2 + Added support for Windows 6.0 (Vista) symbolic links. .. function:: remove(path) @@ -1158,11 +1301,11 @@ Files and Directories .. function:: stat(path) - Perform the equivalent of a :cfunc:`stat` system call on the given path. + Perform the equivalent of a :c:func:`stat` system call on the given path. (This function follows symlinks; to stat a symlink use :func:`lstat`.) The return value is an object whose attributes correspond to the members - of the :ctype:`stat` structure, namely: + of the :c:type:`stat` structure, namely: * :attr:`st_mode` - protection bits, * :attr:`st_ino` - inode number, @@ -1198,15 +1341,16 @@ Files and Directories .. note:: - The exact meaning and resolution of the :attr:`st_atime`, :attr:`st_mtime`, and - :attr:`st_ctime` members depends on the operating system and the file system. - For example, on Windows systems using the FAT or FAT32 file systems, - :attr:`st_mtime` has 2-second resolution, and :attr:`st_atime` has only 1-day - resolution. See your operating system documentation for details. + The exact meaning and resolution of the :attr:`st_atime`, + :attr:`st_mtime`, and :attr:`st_ctime` attributes depend on the operating + system and the file system. For example, on Windows systems using the FAT + or FAT32 file systems, :attr:`st_mtime` has 2-second resolution, and + :attr:`st_atime` has only 1-day resolution. See your operating system + documentation for details. For backward compatibility, the return value of :func:`~os.stat` is also accessible as a tuple of at least 10 integers giving the most important (and portable) - members of the :ctype:`stat` structure, in the order :attr:`st_mode`, + members of the :c:type:`stat` structure, in the order :attr:`st_mode`, :attr:`st_ino`, :attr:`st_dev`, :attr:`st_nlink`, :attr:`st_uid`, :attr:`st_gid`, :attr:`st_size`, :attr:`st_atime`, :attr:`st_mtime`, :attr:`st_ctime`. More items may be added at the end by some implementations. @@ -1214,7 +1358,7 @@ Files and Directories .. index:: module: stat The standard module :mod:`stat` defines functions and constants that are useful - for extracting information from a :ctype:`stat` structure. (On Windows, some + for extracting information from a :c:type:`stat` structure. (On Windows, some items are filled with dummy values.) Example:: @@ -1222,9 +1366,11 @@ Files and Directories >>> import os >>> statinfo = os.stat('somefile.txt') >>> statinfo - (33188, 422511, 769, 1, 1032, 100, 926, 1105022698,1105022732, 1105022732) + posix.stat_result(st_mode=33188, st_ino=7876932, st_dev=234881026, + st_nlink=1, st_uid=501, st_gid=501, st_size=264, st_atime=1297230295, + st_mtime=1297230027, st_ctime=1297230027) >>> statinfo.st_size - 926 + 264 Availability: Unix, Windows. @@ -1256,21 +1402,55 @@ Files and Directories .. function:: statvfs(path) - Perform a :cfunc:`statvfs` system call on the given path. The return value is + Perform a :c:func:`statvfs` system call on the given path. The return value is an object whose attributes describe the filesystem on the given path, and - correspond to the members of the :ctype:`statvfs` structure, namely: + correspond to the members of the :c:type:`statvfs` structure, namely: :attr:`f_bsize`, :attr:`f_frsize`, :attr:`f_blocks`, :attr:`f_bfree`, :attr:`f_bavail`, :attr:`f_files`, :attr:`f_ffree`, :attr:`f_favail`, :attr:`f_flag`, :attr:`f_namemax`. + Two module-level constants are defined for the :attr:`f_flag` attribute's + bit-flags: if :const:`ST_RDONLY` is set, the filesystem is mounted + read-only, and if :const:`ST_NOSUID` is set, the semantics of + setuid/setgid bits are disabled or not supported. + + .. versionchanged:: 3.2 + The :const:`ST_RDONLY` and :const:`ST_NOSUID` constants were added. + Availability: Unix. .. function:: symlink(source, link_name) + symlink(source, link_name, target_is_directory=False) Create a symbolic link pointing to *source* named *link_name*. - Availability: Unix. + On Windows, symlink version takes an additional optional parameter, + *target_is_directory*, which defaults to ``False``. + + On Windows, a symlink represents a file or a directory, and does not morph to + the target dynamically. If *target_is_directory* is set to ``True``, the + symlink will be created as a directory symlink, otherwise as a file symlink + (the default). + + Symbolic link support was introduced in Windows 6.0 (Vista). :func:`symlink` + will raise a :exc:`NotImplementedError` on Windows versions earlier than 6.0. + + .. note:: + + The *SeCreateSymbolicLinkPrivilege* is required in order to successfully + create symlinks. This privilege is not typically granted to regular + users but is available to accounts which can escalate privileges to the + administrator level. Either obtaining the privilege or running your + application as an administrator are ways to successfully create symlinks. + + :exc:`OSError` is raised when the function is called by an unprivileged + user. + + Availability: Unix, Windows. + + .. versionchanged:: 3.2 + Added support for Windows 6.0 (Vista) symbolic links. .. function:: unlink(path) @@ -1298,7 +1478,7 @@ Files and Directories Availability: Unix, Windows. -.. function:: walk(top[, topdown=True [, onerror=None[, followlinks=False]]]) +.. function:: walk(top, topdown=True, onerror=None, followlinks=False) .. index:: single: directory; walking @@ -1331,7 +1511,7 @@ Files and Directories ineffective, because in bottom-up mode the directories in *dirnames* are generated before *dirpath* itself is generated. - By default errors from the :func:`listdir` call are ignored. If optional + By default, errors from the :func:`listdir` call are ignored. If optional argument *onerror* is specified, it should be a function; it will be called with one argument, an :exc:`OSError` instance. It can report the error to continue with the walk, or raise the exception to abort the walk. Note that the filename @@ -1392,7 +1572,7 @@ The various :func:`exec\*` functions take a list of arguments for the new program loaded into the process. In each case, the first of these arguments is passed to the new program as its own name rather than as an argument a user may have typed on a command line. For the C programmer, this is the ``argv[0]`` -passed to a program's :cfunc:`main`. For example, ``os.execv('/bin/echo', +passed to a program's :c:func:`main`. For example, ``os.execv('/bin/echo', ['foo', 'bar'])`` will only print ``bar`` on standard output; ``foo`` will seem to be ignored. @@ -1401,8 +1581,9 @@ to be ignored. Generate a :const:`SIGABRT` signal to the current process. On Unix, the default behavior is to produce a core dump; on Windows, the process immediately returns - an exit code of ``3``. Be aware that programs which use :func:`signal.signal` - to register a handler for :const:`SIGABRT` will behave differently. + an exit code of ``3``. Be aware that calling this function will not call the + Python signal handler registered for :const:`SIGABRT` with + :func:`signal.signal`. Availability: Unix, Windows. @@ -1635,7 +1816,17 @@ written in Python, such as a mail server's external command delivery program. Send signal *sig* to the process *pid*. Constants for the specific signals available on the host platform are defined in the :mod:`signal` module. - Availability: Unix. + + Windows: The :data:`signal.CTRL_C_EVENT` and + :data:`signal.CTRL_BREAK_EVENT` signals are special signals which can + only be sent to console processes which share a common console window, + e.g., some subprocesses. Any other value for *sig* will cause the process + to be unconditionally killed by the TerminateProcess API, and the exit code + will be set to *sig*. The Windows version of :func:`kill` additionally takes + process handles to be killed. + + .. versionadded:: 3.2 + Windows support. .. function:: killpg(pgid, sig) @@ -1731,7 +1922,9 @@ written in Python, such as a mail server's external command delivery program. os.spawnvpe(os.P_WAIT, 'cp', L, os.environ) Availability: Unix, Windows. :func:`spawnlp`, :func:`spawnlpe`, :func:`spawnvp` - and :func:`spawnvpe` are not available on Windows. + and :func:`spawnvpe` are not available on Windows. :func:`spawnle` and + :func:`spawnve` are not thread-safe on Windows; we advise you to use the + :mod:`subprocess` module instead. .. data:: P_NOWAIT @@ -1786,7 +1979,7 @@ written in Python, such as a mail server's external command delivery program. There is no option to wait for the application to close, and no way to retrieve the application's exit status. The *path* parameter is relative to the current directory. If you want to use an absolute path, make sure the first character - is not a slash (``'/'``); the underlying Win32 :cfunc:`ShellExecute` function + is not a slash (``'/'``); the underlying Win32 :c:func:`ShellExecute` function doesn't work if it is. Use the :func:`os.path.normpath` function to ensure that the path is properly encoded for Win32. @@ -1796,14 +1989,14 @@ written in Python, such as a mail server's external command delivery program. .. function:: system(command) Execute the command (a string) in a subshell. This is implemented by calling - the Standard C function :cfunc:`system`, and has the same limitations. + the Standard C function :c:func:`system`, and has the same limitations. Changes to :data:`sys.stdin`, etc. are not reflected in the environment of the executed command. If *command* generates any output, it will be sent to the interpreter standard output stream. On Unix, the return value is the exit status of the process encoded in the format specified for :func:`wait`. Note that POSIX does not specify the - meaning of the return value of the C :cfunc:`system` function, so the return + meaning of the return value of the C :c:func:`system` function, so the return value of the Python function is system-dependent. On Windows, the return value is that returned by the system shell after diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst index 3972f14bcb..0a08428409 100644 --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -1,4 +1,3 @@ - :mod:`ossaudiodev` --- Access to OSS-compatible audio devices ============================================================= @@ -15,7 +14,7 @@ the standard audio interface for Linux and recent versions of FreeBSD. ALSA is in the standard kernel as of 2.5.x. Presumably if you use ALSA, you'll have to make sure its OSS compatibility layer is active to use ossaudiodev, but you're gonna need it for the vast - majority of Linux audio apps anyways. + majority of Linux audio apps anyway. Sounds like things are also complicated for other BSDs. In response to my python-dev query, Thomas Wouters said: @@ -57,7 +56,7 @@ the standard audio interface for Linux and recent versions of FreeBSD. what went wrong. (If :mod:`ossaudiodev` receives an error from a system call such as - :cfunc:`open`, :cfunc:`write`, or :cfunc:`ioctl`, it raises :exc:`IOError`. + :c:func:`open`, :c:func:`write`, or :c:func:`ioctl`, it raises :exc:`IOError`. Errors detected directly by :mod:`ossaudiodev` result in :exc:`OSSAudioError`.) (For backwards compatibility, the exception class is also available as @@ -160,6 +159,11 @@ and (read-only) attributes: is only useful in non-blocking mode. Has no return value, since the amount of data written is always equal to the amount of data supplied. +.. versionchanged:: 3.2 + Audio device objects also support the context manager protocol, i.e. they can + be used in a :keyword:`with` statement. + + The following methods each map to exactly one :func:`ioctl` system call. The correspondence is obvious: for example, :meth:`setfmt` corresponds to the ``SNDCTL_DSP_SETFMT`` ioctl, and :meth:`sync` to ``SNDCTL_DSP_SYNC`` (this can @@ -347,6 +351,10 @@ The mixer object provides two file-like methods: Returns the file handle number of the open mixer device file. +.. versionchanged:: 3.2 + Mixer objects also support the context manager protocol. + + The remaining methods are specific to audio mixing: diff --git a/Doc/library/othergui.rst b/Doc/library/othergui.rst index 1f4aed73d1..da66003b62 100644 --- a/Doc/library/othergui.rst +++ b/Doc/library/othergui.rst @@ -3,42 +3,22 @@ Other Graphical User Interface Packages ======================================= -There are an number of extension widget sets to :mod:`tkinter`. +Major cross-platform (Windows, Mac OS X, Unix-like) GUI toolkits are +available for Python: .. seealso:: - `Python megawidgets <http://pmw.sourceforge.net/>`_ - is a toolkit for building high-level compound widgets in Python using the - :mod:`tkinter` package. It consists of a set of base classes and a library of - flexible and extensible megawidgets built on this foundation. These megawidgets - include notebooks, comboboxes, selection widgets, paned widgets, scrolled - widgets, dialog windows, etc. Also, with the Pmw.Blt interface to BLT, the - busy, graph, stripchart, tabset and vector commands are be available. - - The initial ideas for Pmw were taken from the Tk ``itcl`` extensions ``[incr - Tk]`` by Michael McLennan and ``[incr Widgets]`` by Mark Ulferts. Several of the - megawidgets are direct translations from the itcl to Python. It offers most of - the range of widgets that ``[incr Widgets]`` does, and is almost as complete as - Tix, lacking however Tix's fast :class:`HList` widget for drawing trees. - - `Tkinter3000 Widget Construction Kit (WCK) <http://tkinter.effbot.org/>`_ - is a library that allows you to write new Tkinter widgets in pure Python. The - WCK framework gives you full control over widget creation, configuration, screen - appearance, and event handling. WCK widgets can be very fast and light-weight, - since they can operate directly on Python data structures, without having to - transfer data through the Tk/Tcl layer. - - -The major cross-platform (Windows, Mac OS X, Unix-like) GUI toolkits that are -also available for Python: - -.. seealso:: - - `PyGTK <http://www.pygtk.org/>`_ - is a set of bindings for the `GTK <http://www.gtk.org/>`_ widget set. It - provides an object oriented interface that is slightly higher level than - the C one. It comes with many more widgets than Tkinter provides, and has - good Python-specific reference documentation. There are also bindings to + `PyGObject <https://live.gnome.org/PyGObject>`_ + provides introspection bindings for C libraries using + `GObject <http://developer.gnome.org/gobject/stable/>`_. One of + these libraries is the `GTK+ 3 <http://www.gtk.org/>`_ widget set. + GTK+ comes with many more widgets than Tkinter provides. An online + `Python GTK+ 3 Tutorial <http://python-gtk-3-tutorial.readthedocs.org/en/latest/>`_ + is available. + + `PyGTK <http://www.pygtk.org/>`_ provides bindings for an older version + of the library, GTK+ 2. It provides an object oriented interface that + is slightly higher level than the C one. There are also bindings to `GNOME <http://www.gnome.org>`_. One well known PyGTK application is `PythonCAD <http://www.pythoncad.org/>`_. An online `tutorial <http://www.pygtk.org/pygtk2tutorial/index.html>`_ is available. @@ -55,6 +35,11 @@ also available for Python: with Python and Qt <http://www.qtrac.eu/pyqtbook.html>`_, by Mark Summerfield. + `PySide <http://www.pyside.org/>`_ + is a newer binding to the Qt toolkit, provided by Nokia. + Compared to PyQt, its licensing scheme is friendlier to non-open source + applications. + `wxPython <http://www.wxpython.org>`_ wxPython is a cross-platform GUI toolkit for Python that is built around the popular `wxWidgets <http://www.wxwidgets.org/>`_ (formerly wxWindows) diff --git a/Doc/library/parser.rst b/Doc/library/parser.rst index 50b94c1750..3e1e31bc79 100644 --- a/Doc/library/parser.rst +++ b/Doc/library/parser.rst @@ -1,4 +1,3 @@ - :mod:`parser` --- Access Python parse trees =========================================== @@ -165,7 +164,7 @@ executable code objects. Parse trees may be extracted with or without line numbering information. -.. function:: st2list(st[, line_info]) +.. function:: st2list(st, line_info=False, col_info=False) This function accepts an ST object from the caller in *st* and returns a Python list representing the equivalent parse tree. The resulting list @@ -183,7 +182,7 @@ numbering information. This information is omitted if the flag is false or omitted. -.. function:: st2tuple(st[, line_info]) +.. function:: st2tuple(st, line_info=False, col_info=False) This function accepts an ST object from the caller in *st* and returns a Python tuple representing the equivalent parse tree. Other than returning a @@ -194,7 +193,7 @@ numbering information. information is omitted if the flag is false or omitted. -.. function:: compilest(st[, filename='<syntax-tree>']) +.. function:: compilest(st, filename='<syntax-tree>') .. index:: builtin: exec @@ -293,7 +292,7 @@ ST objects (using the :mod:`pickle` module) is also supported. ST objects have the following methods: -.. method:: ST.compile([filename]) +.. method:: ST.compile(filename='<syntax-tree>') Same as ``compilest(st, filename)``. @@ -308,14 +307,14 @@ ST objects have the following methods: Same as ``issuite(st)``. -.. method:: ST.tolist([line_info]) +.. method:: ST.tolist(line_info=False, col_info=False) - Same as ``st2list(st, line_info)``. + Same as ``st2list(st, line_info, col_info)``. -.. method:: ST.totuple([line_info]) +.. method:: ST.totuple(line_info=False, col_info=False) - Same as ``st2tuple(st, line_info)``. + Same as ``st2tuple(st, line_info, col_info)``. Example: Emulation of :func:`compile` diff --git a/Doc/library/pdb.rst b/Doc/library/pdb.rst index ee63ebb15b..1e9de63b69 100644 --- a/Doc/library/pdb.rst +++ b/Doc/library/pdb.rst @@ -20,9 +20,9 @@ supports post-mortem debugging and can be called under program control. module: bdb module: cmd -The debugger is extensible --- it is actually defined as the class :class:`Pdb`. +The debugger is extensible -- it is actually defined as the class :class:`Pdb`. This is currently undocumented but easily understood by reading the source. The -extension interface uses the modules :mod:`bdb` (undocumented) and :mod:`cmd`. +extension interface uses the modules :mod:`bdb` and :mod:`cmd`. The debugger's prompt is ``(Pdb)``. Typical usage to run a program under control of the debugger is:: @@ -44,19 +44,23 @@ example:: python3 -m pdb myscript.py When invoked as a script, pdb will automatically enter post-mortem debugging if -the program being debugged exits abnormally. After post-mortem debugging (or -after normal exit of the program), pdb will restart the program. Automatic +the program being debugged exits abnormally. After post-mortem debugging (or +after normal exit of the program), pdb will restart the program. Automatic restarting preserves pdb's state (such as breakpoints) and in most cases is more useful than quitting the debugger upon program's exit. +.. versionadded:: 3.2 + :file:`pdb.py` now accepts a ``-c`` option that executes commands as if given + in a :file:`.pdbrc` file, see :ref:`debugger-commands`. + The typical usage to break into the debugger from a running program is to insert :: import pdb; pdb.set_trace() at the location you want to break into the debugger. You can then step through -the code following this statement, and continue running without the debugger using -the ``c`` command. +the code following this statement, and continue running without the debugger +using the :pdbcmd:`continue` command. The typical usage to inspect a crashed program is:: @@ -79,29 +83,31 @@ The typical usage to inspect a crashed program is:: The module defines the following functions; each enters the debugger in a slightly different way: -.. function:: run(statement[, globals[, locals]]) +.. function:: run(statement, globals=None, locals=None) - Execute the *statement* (given as a string) under debugger control. The - debugger prompt appears before any code is executed; you can set breakpoints and - type ``continue``, or you can step through the statement using ``step`` or - ``next`` (all these commands are explained below). The optional *globals* and - *locals* arguments specify the environment in which the code is executed; by - default the dictionary of the module :mod:`__main__` is used. (See the - explanation of the built-in :func:`exec` or :func:`eval` functions.) + Execute the *statement* (given as a string or a code object) under debugger + control. The debugger prompt appears before any code is executed; you can + set breakpoints and type :pdbcmd:`continue`, or you can step through the + statement using :pdbcmd:`step` or :pdbcmd:`next` (all these commands are + explained below). The optional *globals* and *locals* arguments specify the + environment in which the code is executed; by default the dictionary of the + module :mod:`__main__` is used. (See the explanation of the built-in + :func:`exec` or :func:`eval` functions.) -.. function:: runeval(expression[, globals[, locals]]) +.. function:: runeval(expression, globals=None, locals=None) - Evaluate the *expression* (given as a string) under debugger control. When - :func:`runeval` returns, it returns the value of the expression. Otherwise this - function is similar to :func:`run`. + Evaluate the *expression* (given as a string or a code object) under debugger + control. When :func:`runeval` returns, it returns the value of the + expression. Otherwise this function is similar to :func:`run`. -.. function:: runcall(function[, argument, ...]) +.. function:: runcall(function, *args, **kwds) - Call the *function* (a function or method object, not a string) with the given - arguments. When :func:`runcall` returns, it returns whatever the function call - returned. The debugger prompt appears as soon as the function is entered. + Call the *function* (a function or method object, not a string) with the + given arguments. When :func:`runcall` returns, it returns whatever the + function call returned. The debugger prompt appears as soon as the function + is entered. .. function:: set_trace() @@ -111,7 +117,7 @@ slightly different way: being debugged (e.g. when an assertion fails). -.. function:: post_mortem([traceback]) +.. function:: post_mortem(traceback=None) Enter post-mortem debugging of the given *traceback* object. If no *traceback* is given, it uses the one of the exception that is currently @@ -129,7 +135,8 @@ The ``run*`` functions and :func:`set_trace` are aliases for instantiating the :class:`Pdb` class and calling the method of the same name. If you want to access further features, you have to do this yourself: -.. class:: Pdb(completekey='tab', stdin=None, stdout=None, skip=None) +.. class:: Pdb(completekey='tab', stdin=None, stdout=None, skip=None, \ + nosigint=False) :class:`Pdb` is the debugger class. @@ -140,6 +147,11 @@ access further features, you have to do this yourself: patterns. The debugger will not step into frames that originate in a module that matches one of these patterns. [1]_ + By default, Pdb sets a handler for the SIGINT signal (which is sent when the + user presses Ctrl-C on the console) when you give a ``continue`` command. + This allows you to break into the debugger again by pressing Ctrl-C. If you + want Pdb not to touch the SIGINT handler, set *nosigint* tot true. + Example call to enable tracing with *skip*:: import pdb; pdb.Pdb(skip=['django.*']).set_trace() @@ -147,9 +159,13 @@ access further features, you have to do this yourself: .. versionadded:: 3.1 The *skip* argument. - .. method:: run(statement[, globals[, locals]]) - runeval(expression[, globals[, locals]]) - runcall(function[, argument, ...]) + .. versionadded:: 3.2 + The *nosigint* argument. Previously, a SIGINT handler was never set by + Pdb. + + .. method:: run(statement, globals=None, locals=None) + runeval(expression, globals=None, locals=None) + runcall(function, *args, **kwds) set_trace() See the documentation for the functions explained above. @@ -160,16 +176,17 @@ access further features, you have to do this yourself: Debugger Commands ----------------- -The debugger recognizes the following commands. Most commands can be -abbreviated to one or two letters; e.g. ``h(elp)`` means that either ``h`` or -``help`` can be used to enter the help command (but not ``he`` or ``hel``, nor -``H`` or ``Help`` or ``HELP``). Arguments to commands must be separated by -whitespace (spaces or tabs). Optional arguments are enclosed in square brackets -(``[]``) in the command syntax; the square brackets must not be typed. -Alternatives in the command syntax are separated by a vertical bar (``|``). +The commands recognized by the debugger are listed below. Most commands can be +abbreviated to one or two letters as indicated; e.g. ``h(elp)`` means that +either ``h`` or ``help`` can be used to enter the help command (but not ``he`` +or ``hel``, nor ``H`` or ``Help`` or ``HELP``). Arguments to commands must be +separated by whitespace (spaces or tabs). Optional arguments are enclosed in +square brackets (``[]``) in the command syntax; the square brackets must not be +typed. Alternatives in the command syntax are separated by a vertical bar +(``|``). Entering a blank line repeats the last command entered. Exception: if the last -command was a ``list`` command, the next 11 lines are listed. +command was a :pdbcmd:`list` command, the next 11 lines are listed. Commands that the debugger doesn't recognize are assumed to be Python statements and are executed in the context of the program being debugged. Python @@ -179,92 +196,112 @@ change a variable or call a function. When an exception occurs in such a statement, the exception name is printed but the debugger's state is not changed. +The debugger supports :ref:`aliases <debugger-aliases>`. Aliases can have +parameters which allows one a certain level of adaptability to the context under +examination. + Multiple commands may be entered on a single line, separated by ``;;``. (A single ``;`` is not used as it is the separator for multiple commands in a line -that is passed to the Python parser.) No intelligence is applied to separating +that is passed to the Python parser.) No intelligence is applied to separating the commands; the input is split at the first ``;;`` pair, even if it is in the middle of a quoted string. -The debugger supports aliases. Aliases can have parameters which allows one a -certain level of adaptability to the context under examination. - .. index:: pair: .pdbrc; file triple: debugger; configuration; file -If a file :file:`.pdbrc` exists in the user's home directory or in the current +If a file :file:`.pdbrc` exists in the user's home directory or in the current directory, it is read in and executed as if it had been typed at the debugger -prompt. This is particularly useful for aliases. If both files exist, the one +prompt. This is particularly useful for aliases. If both files exist, the one in the home directory is read first and aliases defined there can be overridden by the local file. -h(elp) [*command*] +.. versionchanged:: 3.2 + :file:`.pdbrc` can now contain commands that continue debugging, such as + :pdbcmd:`continue` or :pdbcmd:`next`. Previously, these commands had no + effect. + + +.. pdbcommand:: h(elp) [command] + Without argument, print the list of available commands. With a *command* as argument, print help about that command. ``help pdb`` displays the full - documentation file; if the environment variable :envvar:`PAGER` is defined, the - file is piped through that command instead. Since the *command* argument must - be an identifier, ``help exec`` must be entered to get help on the ``!`` - command. + documentation (the docstring of the :mod:`pdb` module). Since the *command* + argument must be an identifier, ``help exec`` must be entered to get help on + the ``!`` command. + +.. pdbcommand:: w(here) -w(here) Print a stack trace, with the most recent frame at the bottom. An arrow indicates the current frame, which determines the context of most commands. -d(own) - Move the current frame one level down in the stack trace (to a newer frame). +.. pdbcommand:: d(own) [count] + + Move the current frame *count* (default one) levels down in the stack trace + (to a newer frame). + +.. pdbcommand:: u(p) [count] + + Move the current frame *count* (default one) levels up in the stack trace (to + an older frame). -u(p) - Move the current frame one level up in the stack trace (to an older frame). +.. pdbcommand:: b(reak) [([filename:]lineno | function) [, condition]] -b(reak) [[*filename*:]\ *lineno* | *function*\ [, *condition*]] With a *lineno* argument, set a break there in the current file. With a - *function* argument, set a break at the first executable statement within that - function. The line number may be prefixed with a filename and a colon, to - specify a breakpoint in another file (probably one that hasn't been loaded yet). - The file is searched on ``sys.path``. Note that each breakpoint is assigned a - number to which all the other breakpoint commands refer. + *function* argument, set a break at the first executable statement within + that function. The line number may be prefixed with a filename and a colon, + to specify a breakpoint in another file (probably one that hasn't been loaded + yet). The file is searched on :data:`sys.path`. Note that each breakpoint + is assigned a number to which all the other breakpoint commands refer. - If a second argument is present, it is an expression which must evaluate to true - before the breakpoint is honored. + If a second argument is present, it is an expression which must evaluate to + true before the breakpoint is honored. - Without argument, list all breaks, including for each breakpoint, the number of - times that breakpoint has been hit, the current ignore count, and the associated - condition if any. + Without argument, list all breaks, including for each breakpoint, the number + of times that breakpoint has been hit, the current ignore count, and the + associated condition if any. -tbreak [[*filename*:]\ *lineno* | *function*\ [, *condition*]] - Temporary breakpoint, which is removed automatically when it is first hit. The - arguments are the same as break. +.. pdbcommand:: tbreak [([filename:]lineno | function) [, condition]] + + Temporary breakpoint, which is removed automatically when it is first hit. + The arguments are the same as for :pdbcmd:`break`. + +.. pdbcommand:: cl(ear) [filename:lineno | bpnumber [bpnumber ...]] -cl(ear) [*filename:lineno* | *bpnumber* [*bpnumber ...*]] With a *filename:lineno* argument, clear all the breakpoints at this line. With a space separated list of breakpoint numbers, clear those breakpoints. Without argument, clear all breaks (but first ask confirmation). -disable [*bpnumber* [*bpnumber ...*]] - Disables the breakpoints given as a space separated list of breakpoint numbers. - Disabling a breakpoint means it cannot cause the program to stop execution, but - unlike clearing a breakpoint, it remains in the list of breakpoints and can be - (re-)enabled. +.. pdbcommand:: disable [bpnumber [bpnumber ...]] + + Disable the breakpoints given as a space separated list of breakpoint + numbers. Disabling a breakpoint means it cannot cause the program to stop + execution, but unlike clearing a breakpoint, it remains in the list of + breakpoints and can be (re-)enabled. -enable [*bpnumber* [*bpnumber ...*]] - Enables the breakpoints specified. +.. pdbcommand:: enable [bpnumber [bpnumber ...]] -ignore *bpnumber* [*count*] - Sets the ignore count for the given breakpoint number. If count is omitted, the - ignore count is set to 0. A breakpoint becomes active when the ignore count is - zero. When non-zero, the count is decremented each time the breakpoint is - reached and the breakpoint is not disabled and any associated condition - evaluates to true. + Enable the breakpoints specified. -condition *bpnumber* [*condition*] - Condition is an expression which must evaluate to true before the breakpoint is - honored. If condition is absent, any existing condition is removed; i.e., the - breakpoint is made unconditional. +.. pdbcommand:: ignore bpnumber [count] + + Set the ignore count for the given breakpoint number. If count is omitted, + the ignore count is set to 0. A breakpoint becomes active when the ignore + count is zero. When non-zero, the count is decremented each time the + breakpoint is reached and the breakpoint is not disabled and any associated + condition evaluates to true. + +.. pdbcommand:: condition bpnumber [condition] + + Set a new *condition* for the breakpoint, an expression which must evaluate + to true before the breakpoint is honored. If *condition* is absent, any + existing condition is removed; i.e., the breakpoint is made unconditional. + +.. pdbcommand:: commands [bpnumber] -commands [*bpnumber*] Specify a list of commands for breakpoint number *bpnumber*. The commands - themselves appear on the following lines. Type a line containing just 'end' to - terminate the commands. An example:: + themselves appear on the following lines. Type a line containing just + ``end`` to terminate the commands. An example:: (Pdb) commands 1 (com) print some_variable @@ -272,12 +309,12 @@ commands [*bpnumber*] (Pdb) To remove all commands from a breakpoint, type commands and follow it - immediately with end; that is, give no commands. + immediately with ``end``; that is, give no commands. With no *bpnumber* argument, commands refers to the last breakpoint set. - You can use breakpoint commands to start your program up again. Simply use the - continue command, or step, or any other command that resumes execution. + You can use breakpoint commands to start your program up again. Simply use + the continue command, or step, or any other command that resumes execution. Specifying any command resuming execution (currently continue, step, next, return, jump, quit and their abbreviations) terminates the command list (as if @@ -291,91 +328,169 @@ commands [*bpnumber*] that are to print a specific message and then continue. If none of the other commands print anything, you see no sign that the breakpoint was reached. -s(tep) +.. pdbcommand:: s(tep) + Execute the current line, stop at the first possible occasion (either in a function that is called or on the next line in the current function). -n(ext) - Continue execution until the next line in the current function is reached or it - returns. (The difference between ``next`` and ``step`` is that ``step`` stops - inside a called function, while ``next`` executes called functions at (nearly) - full speed, only stopping at the next line in the current function.) +.. pdbcommand:: n(ext) + + Continue execution until the next line in the current function is reached or + it returns. (The difference between :pdbcmd:`next` and :pdbcmd:`step` is + that :pdbcmd:`step` stops inside a called function, while :pdbcmd:`next` + executes called functions at (nearly) full speed, only stopping at the next + line in the current function.) + +.. pdbcommand:: unt(il) [lineno] -unt(il) - Continue execution until the line with the line number greater than the - current one is reached or when returning from current frame. + Without argument, continue execution until the line with a number greater + than the current one is reached. + + With a line number, continue execution until a line with a number greater or + equal to that is reached. In both cases, also stop when the current frame + returns. + + .. versionchanged:: 3.2 + Allow giving an explicit line number. + +.. pdbcommand:: r(eturn) -r(eturn) Continue execution until the current function returns. -c(ont(inue)) +.. pdbcommand:: c(ont(inue)) + Continue execution, only stop when a breakpoint is encountered. -j(ump) *lineno* +.. pdbcommand:: j(ump) lineno + Set the next line that will be executed. Only available in the bottom-most - frame. This lets you jump back and execute code again, or jump forward to skip - code that you don't want to run. + frame. This lets you jump back and execute code again, or jump forward to + skip code that you don't want to run. - It should be noted that not all jumps are allowed --- for instance it is not + It should be noted that not all jumps are allowed -- for instance it is not possible to jump into the middle of a :keyword:`for` loop or out of a :keyword:`finally` clause. -l(ist) [*first*\ [, *last*]] - List source code for the current file. Without arguments, list 11 lines around - the current line or continue the previous listing. With one argument, list 11 - lines around at that line. With two arguments, list the given range; if the - second argument is less than the first, it is interpreted as a count. +.. pdbcommand:: l(ist) [first[, last]] + + List source code for the current file. Without arguments, list 11 lines + around the current line or continue the previous listing. With ``.`` as + argument, list 11 lines around the current line. With one argument, + list 11 lines around at that line. With two arguments, list the given range; + if the second argument is less than the first, it is interpreted as a count. + + The current line in the current frame is indicated by ``->``. If an + exception is being debugged, the line where the exception was originally + raised or propagated is indicated by ``>>``, if it differs from the current + line. + + .. versionadded:: 3.2 + The ``>>`` marker. + +.. pdbcommand:: ll | longlist + + List all source code for the current function or frame. Interesting lines + are marked as for :pdbcmd:`list`. + + .. versionadded:: 3.2 + +.. pdbcommand:: a(rgs) -a(rgs) Print the argument list of the current function. -p(rint) *expression* +.. pdbcommand:: p(rint) expression + Evaluate the *expression* in the current context and print its value. -pp *expression* - Like the ``p`` command, except the value of the expression is pretty-printed - using the :mod:`pprint` module. +.. pdbcommand:: pp expression + + Like the :pdbcmd:`print` command, except the value of the expression is + pretty-printed using the :mod:`pprint` module. + +.. pdbcommand:: whatis expression + + Print the type of the *expression*. + +.. pdbcommand:: source expression + + Try to get source code for the given object and display it. + + .. versionadded:: 3.2 + +.. pdbcommand:: display [expression] + + Display the value of the expression if it changed, each time execution stops + in the current frame. -alias [*name* [command]] - Creates an alias called *name* that executes *command*. The command must *not* - be enclosed in quotes. Replaceable parameters can be indicated by ``%1``, - ``%2``, and so on, while ``%*`` is replaced by all the parameters. If no - command is given, the current alias for *name* is shown. If no arguments are - given, all aliases are listed. + Without expression, list all display expressions for the current frame. - Aliases may be nested and can contain anything that can be legally typed at the - pdb prompt. Note that internal pdb commands *can* be overridden by aliases. - Such a command is then hidden until the alias is removed. Aliasing is - recursively applied to the first word of the command line; all other words in - the line are left alone. + .. versionadded:: 3.2 + +.. pdbcommand:: undisplay [expression] + + Do not display the expression any more in the current frame. Without + expression, clear all display expressions for the current frame. + + .. versionadded:: 3.2 + +.. pdbcommand:: interact + + Start an interative interpreter (using the :mod:`code` module) whose global + namespace contains all the (global and local) names found in the current + scope. + + .. versionadded:: 3.2 + +.. _debugger-aliases: + +.. pdbcommand:: alias [name [command]] + + Create an alias called *name* that executes *command*. The command must + *not* be enclosed in quotes. Replaceable parameters can be indicated by + ``%1``, ``%2``, and so on, while ``%*`` is replaced by all the parameters. + If no command is given, the current alias for *name* is shown. If no + arguments are given, all aliases are listed. + + Aliases may be nested and can contain anything that can be legally typed at + the pdb prompt. Note that internal pdb commands *can* be overridden by + aliases. Such a command is then hidden until the alias is removed. Aliasing + is recursively applied to the first word of the command line; all other words + in the line are left alone. As an example, here are two useful aliases (especially when placed in the :file:`.pdbrc` file):: - #Print instance variables (usage "pi classInst") + # Print instance variables (usage "pi classInst") alias pi for k in %1.__dict__.keys(): print("%1.",k,"=",%1.__dict__[k]) - #Print instance variables in self + # Print instance variables in self alias ps pi self -unalias *name* - Deletes the specified alias. +.. pdbcommand:: unalias name + + Delete the specified alias. + +.. pdbcommand:: ! statement -[!]\ *statement* Execute the (one-line) *statement* in the context of the current stack frame. The exclamation point can be omitted unless the first word of the statement - resembles a debugger command. To set a global variable, you can prefix the - assignment command with a ``global`` command on the same line, e.g.:: + resembles a debugger command. To set a global variable, you can prefix the + assignment command with a :keyword:`global` statement on the same line, + e.g.:: (Pdb) global list_options; list_options = ['-l'] (Pdb) -run [*args* ...] - Restart the debugged Python program. If an argument is supplied, it is split - with "shlex" and the result is used as the new sys.argv. History, breakpoints, - actions and debugger options are preserved. "restart" is an alias for "run". +.. pdbcommand:: run [args ...] + restart [args ...] + + Restart the debugged Python program. If an argument is supplied, it is split + with :mod:`shlex` and the result is used as the new :data:`sys.argv`. + History, breakpoints, actions and debugger options are preserved. + :pdbcmd:`restart` is an alias for :pdbcmd:`run`. + +.. pdbcommand:: q(uit) -q(uit) - Quit from the debugger. The program being executed is aborted. + Quit from the debugger. The program being executed is aborted. .. rubric:: Footnotes diff --git a/Doc/library/persistence.rst b/Doc/library/persistence.rst index b90b2e1a80..d5bb193fa3 100644 --- a/Doc/library/persistence.rst +++ b/Doc/library/persistence.rst @@ -1,4 +1,3 @@ - .. _persistence: **************** diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index 1850b00374..bf0a72e876 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -141,7 +141,7 @@ an unpickler, then you call the unpickler's :meth:`load` method. The The :mod:`pickle` module provides the following functions to make the pickling process more convenient: -.. function:: dump(obj, file[, protocol, \*, fix_imports=True]) +.. function:: dump(obj, file, protocol=None, \*, fix_imports=True) Write a pickled representation of *obj* to the open :term:`file object` *file*. This is equivalent to ``Pickler(file, protocol).dump(obj)``. @@ -163,7 +163,7 @@ process more convenient: map the new Python 3.x names to the old module names used in Python 2.x, so that the pickle data stream is readable with Python 2.x. -.. function:: dumps(obj[, protocol, \*, fix_imports=True]) +.. function:: dumps(obj, protocol=None, \*, fix_imports=True) Return the pickled representation of the object as a :class:`bytes` object, instead of writing it to a file. @@ -180,7 +180,7 @@ process more convenient: map the new Python 3.x names to the old module names used in Python 2.x, so that the pickle data stream is readable with Python 2.x. -.. function:: load(file, [\*, fix_imports=True, encoding="ASCII", errors="strict"]) +.. function:: load(file, \*, fix_imports=True, encoding="ASCII", errors="strict") Read a pickled object representation from the open :term:`file object` *file* and return the reconstituted object hierarchy specified therein. This is @@ -203,7 +203,7 @@ process more convenient: *errors* tell pickle how to decode 8-bit string instances pickled by Python 2.x; these default to 'ASCII' and 'strict', respectively. -.. function:: loads(bytes_object, [\*, fix_imports=True, encoding="ASCII", errors="strict"]) +.. function:: loads(bytes_object, \*, fix_imports=True, encoding="ASCII", errors="strict") Read a pickled object hierarchy from a :class:`bytes` object and return the reconstituted object hierarchy specified therein @@ -237,7 +237,7 @@ The :mod:`pickle` module defines three exceptions: .. exception:: UnpicklingError - Error raised when there a problem unpickling an object, such as a data + Error raised when there is a problem unpickling an object, such as a data corruption or a security violation. It inherits :exc:`PickleError`. Note that other exceptions may also be raised during unpickling, including @@ -248,7 +248,7 @@ The :mod:`pickle` module defines three exceptions: The :mod:`pickle` module exports two classes, :class:`Pickler` and :class:`Unpickler`: -.. class:: Pickler(file[, protocol, \*, fix_imports=True]) +.. class:: Pickler(file, protocol=None, \*, fix_imports=True) This takes a binary file for writing a pickle data stream. @@ -296,7 +296,7 @@ The :mod:`pickle` module exports two classes, :class:`Pickler` and Use :func:`pickletools.optimize` if you need more compact pickles. -.. class:: Unpickler(file, [\*, fix_imports=True, encoding="ASCII", errors="strict"]) +.. class:: Unpickler(file, \*, fix_imports=True, encoding="ASCII", errors="strict") This takes a binary file for reading a pickle data stream. @@ -324,11 +324,11 @@ The :mod:`pickle` module exports two classes, :class:`Pickler` and .. method:: persistent_load(pid) - Raise an :exc:`UnpickingError` by default. + Raise an :exc:`UnpicklingError` by default. If defined, :meth:`persistent_load` should return the object specified by the persistent ID *pid*. If an invalid persistent ID is encountered, an - :exc:`UnpickingError` should be raised. + :exc:`UnpicklingError` should be raised. See :ref:`pickle-persistent` for details and examples of uses. @@ -377,7 +377,7 @@ raised in this case. You can carefully raise this limit with Note that functions (built-in and user-defined) are pickled by "fully qualified" name reference, not by value. This means that only the function name is -pickled, along with the name of module the function is defined in. Neither the +pickled, along with the name of the module the function is defined in. Neither the function's code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised. [#]_ @@ -668,7 +668,7 @@ inoffensive, it is not difficult to imagine one that could damage your system. For this reason, you may want to control what gets unpickled by customizing :meth:`Unpickler.find_class`. Unlike its name suggests, :meth:`find_class` is called whenever a global (i.e., a class or a function) is requested. Thus it is -possible to either forbid completely globals or restrict them to a safe subset. +possible to either completely forbid globals or restrict them to a safe subset. Here is an example of an unpickler allowing only few safe classes from the :mod:`builtins` module to be loaded:: diff --git a/Doc/library/pickletools.rst b/Doc/library/pickletools.rst index 88ecbab919..4c0a148bc3 100644 --- a/Doc/library/pickletools.rst +++ b/Doc/library/pickletools.rst @@ -2,7 +2,13 @@ ================================================== .. module:: pickletools - :synopsis: Contains extensive comments about the pickle protocols and pickle-machine opcodes, as well as some useful functions. + :synopsis: Contains extensive comments about the pickle protocols and + pickle-machine opcodes, as well as some useful functions. + +**Source code:** :source:`Lib/pickletools.py` + +-------------- + This module contains various constants relating to the intimate details of the :mod:`pickle` module, some lengthy comments about the implementation, and a @@ -11,15 +17,81 @@ are useful for Python core developers who are working on the :mod:`pickle`; ordinary users of the :mod:`pickle` module probably won't find the :mod:`pickletools` module relevant. +Command line usage +------------------ + +.. versionadded:: 3.2 + +When invoked from the command line, ``python -m pickletools`` will +disassemble the contents of one or more pickle files. Note that if +you want to see the Python object stored in the pickle rather than the +details of pickle format, you may want to use ``-m pickle`` instead. +However, when the pickle file that you want to examine comes from an +untrusted source, ``-m pickletools`` is a safer option because it does +not execute pickle bytecode. + +For example, with a tuple ``(1, 2)`` pickled in file ``x.pickle``:: + + $ python -m pickle x.pickle + (1, 2) + + $ python -m pickletools x.pickle + 0: \x80 PROTO 3 + 2: K BININT1 1 + 4: K BININT1 2 + 6: \x86 TUPLE2 + 7: q BINPUT 0 + 9: . STOP + highest protocol among opcodes = 2 + +Command line options +^^^^^^^^^^^^^^^^^^^^ + +.. program:: pickletools + +.. cmdoption:: -a, --annotate + + Annotate each line with a short opcode description. + +.. cmdoption:: -o, --output=<file> + + Name of a file where the output should be written. + +.. cmdoption:: -l, --indentlevel=<num> + + The number of blanks by which to indent a new MARK level. + +.. cmdoption:: -m, --memo + + When multiple objects are disassembled, preserve memo between + disassemblies. + +.. cmdoption:: -p, --preamble=<preamble> + + When more than one pickle file are specified, print given preamble + before each disassembly. + + + +Programmatic Interface +---------------------- + + +.. function:: dis(pickle, out=None, memo=None, indentlevel=4, annotate=0) -.. function:: dis(pickle[, out=None, memo=None, indentlevel=4]) + Outputs a symbolic disassembly of the pickle to the file-like + object *out*, defaulting to ``sys.stdout``. *pickle* can be a + string or a file-like object. *memo* can be a Python dictionary + that will be used as the pickle's memo; it can be used to perform + disassemblies across multiple pickles created by the same + pickler. Successive levels, indicated by ``MARK`` opcodes in the + stream, are indented by *indentlevel* spaces. If a nonzero value + is given to *annotate*, each opcode in the output is annotated with + a short description. The value of *annotate* is used as a hint for + the column where annotation should start. - Outputs a symbolic disassembly of the pickle to the file-like object *out*, - defaulting to ``sys.stdout``. *pickle* can be a string or a file-like object. - *memo* can be a Python dictionary that will be used as the pickle's memo; it can - be used to perform disassemblies across multiple pickles created by the same - pickler. Successive levels, indicated by ``MARK`` opcodes in the stream, are - indented by *indentlevel* spaces. + .. versionadded:: 3.2 + The *annotate* argument. .. function:: genops(pickle) diff --git a/Doc/library/pipes.rst b/Doc/library/pipes.rst index 1f2b2ff239..016a720470 100644 --- a/Doc/library/pipes.rst +++ b/Doc/library/pipes.rst @@ -1,4 +1,3 @@ - :mod:`pipes` --- Interface to shell pipelines ============================================= @@ -7,6 +6,9 @@ :synopsis: A Python interface to Unix shell pipelines. .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/pipes.py` + +-------------- The :mod:`pipes` module defines a class to abstract the concept of a *pipeline* --- a sequence of converters from one file to another. diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 5a59f28a05..3118ff2047 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -1,10 +1,13 @@ - :mod:`pkgutil` --- Package extension utility ============================================ .. module:: pkgutil :synopsis: Utilities for the import system. +**Source code:** :source:`Lib/pkgutil.py` + +-------------- + This module provides utilities for the import system, in particular package support. @@ -168,10 +171,10 @@ support. Get a resource from a package. - This is a wrapper for the PEP 302 loader :func:`get_data` API. The package - argument should be the name of a package, in standard module format - (foo.bar). The resource argument should be in the form of a relative - filename, using ``/`` as the path separator. The parent directory name + This is a wrapper for the :pep:`302` loader :func:`get_data` API. The + *package* argument should be the name of a package, in standard module format + (``foo.bar``). The *resource* argument should be in the form of a relative + filename, using ``/`` as the path separator. The parent directory name ``..`` is not allowed, and nor is a rooted name (starting with a ``/``). The function returns a binary string that is the contents of the specified @@ -183,5 +186,5 @@ support. d = os.path.dirname(sys.modules[package].__file__) data = open(os.path.join(d, resource), 'rb').read() - If the package cannot be located or loaded, or it uses a PEP 302 loader + If the package cannot be located or loaded, or it uses a :pep:`302` loader which does not support :func:`get_data`, then ``None`` is returned. diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index 36275ed226..85eca9a178 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Marc-Andre Lemburg <mal@egenix.com> .. sectionauthor:: Bjorn Pettersen <bpettersen@corp.fairisaac.com> +**Source code:** :source:`Lib/platform.py` + +-------------- .. note:: @@ -27,8 +30,8 @@ Cross Platform returned as strings. Values that cannot be determined are returned as given by the parameter presets. - If bits is given as ``''``, the :cfunc:`sizeof(pointer)` (or - :cfunc:`sizeof(long)` on Python version < 1.5.2) is used as indicator for the + If bits is given as ``''``, the :c:func:`sizeof(pointer)` (or + :c:func:`sizeof(long)` on Python version < 1.5.2) is used as indicator for the supported pointer size. The function relies on the system's :file:`file` command to do the actual work. @@ -185,8 +188,8 @@ Windows Platform .. function:: win32_ver(release='', version='', csd='', ptype='') Get additional version information from the Windows Registry and return a tuple - ``(version, csd, ptype)`` referring to version number, CSD level and OS type - (multi/single processor). + ``(version, csd, ptype)`` referring to version number, CSD level + (service pack) and OS type (multi/single processor). As a hint: *ptype* is ``'Uniprocessor Free'`` on single processor NT machines and ``'Multiprocessor Free'`` on multi processor machines. The *'Free'* refers @@ -225,9 +228,6 @@ Mac OS Platform Entries which cannot be determined are set to ``''``. All tuple entries are strings. - Documentation for the underlying :cfunc:`gestalt` API is available online at - http://www.rgaros.nl/gestalt/. - Unix Platforms -------------- diff --git a/Doc/library/plistlib.rst b/Doc/library/plistlib.rst index 36c9eb3087..ae5e94d1a9 100644 --- a/Doc/library/plistlib.rst +++ b/Doc/library/plistlib.rst @@ -11,6 +11,10 @@ pair: plist; file single: property list +**Source code:** :source:`Lib/plistlib.py` + +-------------- + This module provides an interface for reading and writing the "property list" XML files used mainly by Mac OS X. diff --git a/Doc/library/poplib.rst b/Doc/library/poplib.rst index 0b2c033af5..d11d937b1f 100644 --- a/Doc/library/poplib.rst +++ b/Doc/library/poplib.rst @@ -1,4 +1,3 @@ - :mod:`poplib` --- POP3 protocol client ====================================== @@ -9,6 +8,10 @@ .. index:: pair: POP3; protocol +**Source code:** :source:`Lib/poplib.py` + +-------------- + This module defines a class, :class:`POP3`, which encapsulates a connection to a POP3 server and implements the protocol as defined in :rfc:`1725`. The :class:`POP3` class supports both the minimal and optional command sets. @@ -24,7 +27,7 @@ mailserver supports IMAP, you would be better off using the A single class is provided by the :mod:`poplib` module: -.. class:: POP3(host[, port[, timeout]]) +.. class:: POP3(host, port=POP3_PORT[, timeout]) This class implements the actual POP3 protocol. The connection is created when the instance is initialized. If *port* is omitted, the standard POP3 port (110) @@ -33,12 +36,19 @@ A single class is provided by the :mod:`poplib` module: be used). -.. class:: POP3_SSL(host[, port[, keyfile[, certfile]]]) +.. class:: POP3_SSL(host, port=POP3_SSL_PORT, keyfile=None, certfile=None, timeout=None, context=None) This is a subclass of :class:`POP3` that connects to the server over an SSL encrypted socket. If *port* is not specified, 995, the standard POP3-over-SSL port is used. *keyfile* and *certfile* are also optional - they can contain a PEM formatted private key and certificate chain file for the SSL connection. + *timeout* works as in the :class:`POP3` constructor. *context* parameter is a + :class:`ssl.SSLContext` object which allows bundling SSL configuration + options, certificates and private keys into a single (potentially long-lived) + structure. + + .. versionchanged:: 3.2 + *context* parameter added. One exception is defined as an attribute of the :mod:`poplib` module: @@ -160,7 +170,7 @@ An :class:`POP3` instance has the following methods: POP3 servers you will use before trusting it. -.. method:: POP3.uidl([which]) +.. method:: POP3.uidl(which=None) Return message digest (unique id) list. If *which* is specified, result contains the unique id for that message in the form ``'response mesgnum uid``, otherwise diff --git a/Doc/library/posix.rst b/Doc/library/posix.rst index c33d9e59ad..07db2b2af8 100644 --- a/Doc/library/posix.rst +++ b/Doc/library/posix.rst @@ -38,13 +38,13 @@ Large File Support Several operating systems (including AIX, HP-UX, Irix and Solaris) provide support for files that are larger than 2 GB from a C programming model where -:ctype:`int` and :ctype:`long` are 32-bit values. This is typically accomplished +:c:type:`int` and :c:type:`long` are 32-bit values. This is typically accomplished by defining the relevant size and offset types as 64-bit values. Such files are sometimes referred to as :dfn:`large files`. -Large file support is enabled in Python when the size of an :ctype:`off_t` is -larger than a :ctype:`long` and the :ctype:`long long` type is available and is -at least as large as an :ctype:`off_t`. +Large file support is enabled in Python when the size of an :c:type:`off_t` is +larger than a :c:type:`long` and the :c:type:`long long` type is available and is +at least as large as an :c:type:`off_t`. It may be necessary to configure and compile Python with certain compiler flags to enable this mode. For example, it is enabled by default with recent versions of Irix, but with Solaris 2.6 and 2.7 you need to do something like:: @@ -69,17 +69,22 @@ In addition to many functions described in the :mod:`os` module documentation, .. data:: environ A dictionary representing the string environment at the time the interpreter - was started. For example, ``environ['HOME']`` is the pathname of your home - directory, equivalent to ``getenv("HOME")`` in C. + was started. Keys and values are bytes on Unix and str on Windows. For + example, ``environ[b'HOME']`` (``environ['HOME']`` on Windows) is the + pathname of your home directory, equivalent to ``getenv("HOME")`` in C. Modifying this dictionary does not affect the string environment passed on by :func:`execv`, :func:`popen` or :func:`system`; if you need to change the environment, pass ``environ`` to :func:`execve` or add variable assignments and export statements to the command string for :func:`system` or :func:`popen`. + .. versionchanged:: 3.2 + On Unix, keys and values are bytes. + .. note:: - The :mod:`os` module provides an alternate implementation of ``environ`` which - updates the environment on modification. Note also that updating ``os.environ`` - will render this dictionary obsolete. Use of the :mod:`os` module version of - this is recommended over direct access to the :mod:`posix` module. + The :mod:`os` module provides an alternate implementation of ``environ`` + which updates the environment on modification. Note also that updating + :data:`os.environ` will render this dictionary obsolete. Use of the + :mod:`os` module version of this is recommended over direct access to the + :mod:`posix` module. diff --git a/Doc/library/pprint.rst b/Doc/library/pprint.rst index 4169b6475e..3a863318fe 100644 --- a/Doc/library/pprint.rst +++ b/Doc/library/pprint.rst @@ -1,4 +1,3 @@ - :mod:`pprint` --- Data pretty printer ===================================== @@ -7,6 +6,9 @@ .. moduleauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/pprint.py` + +-------------- The :mod:`pprint` module provides a capability to "pretty-print" arbitrary Python data structures in a form which can be used as input to the interpreter. @@ -27,7 +29,7 @@ The :mod:`pprint` module defines one class: .. First the implementation class: -.. class:: PrettyPrinter(...) +.. class:: PrettyPrinter(indent=1, width=80, depth=None, stream=None) Construct a :class:`PrettyPrinter` instance. This constructor understands several keyword parameters. An output stream may be set using the *stream* @@ -62,21 +64,20 @@ The :mod:`pprint` module defines one class: >>> pp.pprint(tup) ('spam', ('eggs', ('lumberjack', ('knights', ('ni', ('dead', (...))))))) -The :class:`PrettyPrinter` class supports several derivative functions: -.. Now the derivative functions: +The :class:`PrettyPrinter` class supports several derivative functions: -.. function:: pformat(object[, indent[, width[, depth]]]) +.. function:: pformat(object, indent=1, width=80, depth=None) Return the formatted representation of *object* as a string. *indent*, *width* and *depth* will be passed to the :class:`PrettyPrinter` constructor as formatting parameters. -.. function:: pprint(object[, stream[, indent[, width[, depth]]]]) +.. function:: pprint(object, stream=None, indent=1, width=80, depth=None) Prints the formatted representation of *object* on *stream*, followed by a - newline. If *stream* is omitted, ``sys.stdout`` is used. This may be used + newline. If *stream* is ``None``, ``sys.stdout`` is used. This may be used in the interactive interpreter instead of the :func:`print` function for inspecting values (you can even reassign ``print = pprint.pprint`` for use within a scope). *indent*, *width* and *depth* will be passed to the diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst index bda977f269..ca54021630 100644 --- a/Doc/library/profile.rst +++ b/Doc/library/profile.rst @@ -1,4 +1,3 @@ - .. _profile: ******************** @@ -10,6 +9,9 @@ The Python Profilers .. module:: profile :synopsis: Python source profiler. +**Source code:** :source:`Lib/profile.py` and :source:`Lib/pstats.py` + +-------------- .. _profiler-introduction: @@ -219,7 +221,7 @@ discussion of how to derive "better" profilers from the classes presented, or reading the source code for these modules. -.. function:: run(command[, filename]) +.. function:: run(command, filename=None, sort=-1) This function takes a single argument that can be passed to the :func:`exec` function, and an optional file name. In all cases this routine attempts to @@ -248,8 +250,8 @@ reading the source code for these modules. for the number of calls, tottime - for the total time spent in the given function (and excluding time made in calls - to sub-functions), + for the total time spent in the given function (and excluding time made in + calls to sub-functions), percall is the quotient of ``tottime`` divided by ``ncalls`` @@ -269,24 +271,25 @@ reading the source code for these modules. calls. Note that when the function does not recurse, these two values are the same, and only the single figure is printed. + If *sort* is given, it can be one of ``'stdname'`` (sort by filename:lineno), + ``'calls'`` (sort by number of calls), ``'time'`` (sort by total time) or + ``'cumulative'`` (sort by cumulative time). The default is ``'stdname'``. + -.. function:: runctx(command, globals, locals[, filename]) +.. function:: runctx(command, globals, locals, filename=None) This function is similar to :func:`run`, with added arguments to supply the globals and locals dictionaries for the *command* string. -Analysis of the profiler data is done using the :class:`Stats` class. - -.. note:: - The :class:`Stats` class is defined in the :mod:`pstats` module. +Analysis of the profiler data is done using the :class:`pstats.Stats` class. .. module:: pstats :synopsis: Statistics object for use with the profiler. -.. class:: Stats(filename[, stream=sys.stdout[, ...]]) +.. class:: Stats(*filenames, stream=sys.stdout) This class constructor creates an instance of a "statistics object" from a *filename* (or set of filenames). :class:`Stats` objects are manipulated by @@ -326,7 +329,7 @@ The :class:`Stats` Class accumulated into a single entry. -.. method:: Stats.add(filename[, ...]) +.. method:: Stats.add(*filenames) This method of the :class:`Stats` class accumulates additional profiling information into the current profiling object. Its arguments should refer to @@ -343,7 +346,7 @@ The :class:`Stats` Class :class:`profile.Profile` and :class:`cProfile.Profile` classes. -.. method:: Stats.sort_stats(key[, ...]) +.. method:: Stats.sort_stats(*keys) This method modifies the :class:`Stats` object by sorting it according to the supplied criteria. The argument is typically a string identifying the basis of @@ -410,7 +413,7 @@ The :class:`Stats` Class .. This method is provided primarily for compatibility with the old profiler. -.. method:: Stats.print_stats([restriction, ...]) +.. method:: Stats.print_stats(*restrictions) This method for the :class:`Stats` class prints out a report as described in the :func:`profile.run` definition. @@ -439,7 +442,7 @@ The :class:`Stats` Class then proceed to only print the first 10% of them. -.. method:: Stats.print_callers([restriction, ...]) +.. method:: Stats.print_callers(*restrictions) This method for the :class:`Stats` class prints a list of all functions that called each function in the profiled database. The ordering is identical to @@ -457,7 +460,7 @@ The :class:`Stats` Class the current function while it was invoked by this specific caller. -.. method:: Stats.print_callees([restriction, ...]) +.. method:: Stats.print_callees(*restrictions) This method for the :class:`Stats` class prints a list of all function that were called by the indicated function. Aside from this reversal of direction of diff --git a/Doc/library/pty.rst b/Doc/library/pty.rst index be879f24ac..2b9385b5f9 100644 --- a/Doc/library/pty.rst +++ b/Doc/library/pty.rst @@ -1,4 +1,3 @@ - :mod:`pty` --- Pseudo-terminal utilities ======================================== @@ -46,3 +45,50 @@ The :mod:`pty` module defines the following functions: a file descriptor. The defaults try to read 1024 bytes each time they are called. + +Example +------- + +.. sectionauthor:: Steen Lumholt + +The following program acts like the Unix command :manpage:`script(1)`, using a +pseudo-terminal to record all input and output of a terminal session in a +"typescript". :: + + import sys, os, time, getopt + import pty + + mode = 'wb' + shell = 'sh' + filename = 'typescript' + if 'SHELL' in os.environ: + shell = os.environ['SHELL'] + + try: + opts, args = getopt.getopt(sys.argv[1:], 'ap') + except getopt.error as msg: + print('%s: %s' % (sys.argv[0], msg)) + sys.exit(2) + + for opt, arg in opts: + # option -a: append to typescript file + if opt == '-a': + mode = 'ab' + # option -p: use a Python shell as the terminal command + elif opt == '-p': + shell = sys.executable + if args: + filename = args[0] + + script = open(filename, mode) + + def read(fd): + data = os.read(fd, 1024) + script.write(data) + return data + + sys.stdout.write('Script started, file is %s\n' % filename) + script.write(('Script started on %s\n' % time.asctime()).encode()) + pty.spawn(shell, read) + script.write(('Script done on %s\n' % time.asctime()).encode()) + sys.stdout.write('Script done, file is %s\n' % filename) diff --git a/Doc/library/pwd.rst b/Doc/library/pwd.rst index 562afd920f..2c17d9e036 100644 --- a/Doc/library/pwd.rst +++ b/Doc/library/pwd.rst @@ -1,4 +1,3 @@ - :mod:`pwd` --- The password database ==================================== diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst index 0891862496..07ddc25422 100644 --- a/Doc/library/py_compile.rst +++ b/Doc/library/py_compile.rst @@ -8,6 +8,10 @@ .. index:: pair: file; byte-code +**Source code:** :source:`Lib/py_compile.py` + +-------------- + The :mod:`py_compile` module provides a function to generate a byte-code file from a source file, and another function used when the module source file is invoked as a script. @@ -22,24 +26,42 @@ byte-code cache files in the directory containing the source code. Exception raised when an error occurs while attempting to compile the file. -.. function:: compile(file[, cfile[, dfile[, doraise]]]) +.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1) + + Compile a source file to byte-code and write out the byte-code cache file. + The source code is loaded from the file name *file*. The byte-code is + written to *cfile*, which defaults to the :PEP:`3147` path, ending in + ``.pyc`` (``.pyo`` if optimization is enabled in the current interpreter). + For example, if *file* is ``/foo/bar/baz.py`` *cfile* will default to + ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. If *dfile* is + specified, it is used as the name of the source file in error messages when + instead of *file*. If *doraise* is true, a :exc:`PyCompileError` is raised + when an error is encountered while compiling *file*. If *doraise* is false + (the default), an error string is written to ``sys.stderr``, but no exception + is raised. This function returns the path to byte-compiled file, i.e. + whatever *cfile* value was used. - Compile a source file to byte-code and write out the byte-code cache file. The - source code is loaded from the file name *file*. The byte-code is written to - *cfile*, which defaults to *file* ``+`` ``'c'`` (``'o'`` if optimization is - enabled in the current interpreter). If *dfile* is specified, it is used as the - name of the source file in error messages instead of *file*. If *doraise* is - true, a :exc:`PyCompileError` is raised when an error is encountered while - compiling *file*. If *doraise* is false (the default), an error string is - written to ``sys.stderr``, but no exception is raised. + *optimize* controls the optimization level and is passed to the built-in + :func:`compile` function. The default of ``-1`` selects the optimization + level of the current interpreter. + .. versionchanged:: 3.2 + Changed default value of *cfile* to be :PEP:`3147`-compliant. Previous + default was *file* + ``'c'`` (``'o'`` if optimization was enabled). + Also added the *optimize* parameter. -.. function:: main([args]) + +.. function:: main(args=None) Compile several source files. The files named in *args* (or on the command - line, if *args* is not specified) are compiled and the resulting bytecode is + line, if *args* is ``None``) are compiled and the resulting bytecode is cached in the normal manner. This function does not search a directory structure to locate source files; it only compiles files named explicitly. + If ``'-'`` is the only parameter in args, the list of files is taken from + standard input. + + .. versionchanged:: 3.2 + Added support for ``'-'``. When this module is run as a script, the :func:`main` is used to compile all the files named on the command line. The exit status is nonzero if one of the files diff --git a/Doc/library/pyclbr.rst b/Doc/library/pyclbr.rst index 36b46f483d..13eaabf594 100644 --- a/Doc/library/pyclbr.rst +++ b/Doc/library/pyclbr.rst @@ -1,4 +1,3 @@ - :mod:`pyclbr` --- Python class browser support ============================================== @@ -6,6 +5,9 @@ :synopsis: Supports information extraction for a Python class browser. .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/pyclbr.py` + +-------------- The :mod:`pyclbr` module can be used to determine some limited information about the classes, methods and top-level functions defined in a module. The @@ -17,7 +19,7 @@ not implemented in Python, including all standard and optional extension modules. -.. function:: readmodule(module[, path=None]) +.. function:: readmodule(module, path=None) Read a module and return a dictionary mapping class names to class descriptor objects. The parameter *module* should be the name of a @@ -26,7 +28,7 @@ modules. of ``sys.path``, which is used to locate module source code. -.. function:: readmodule_ex(module[, path=None]) +.. function:: readmodule_ex(module, path=None) Like :func:`readmodule`, but the returned dictionary, in addition to mapping class names to class descriptor objects, also maps top-level @@ -43,7 +45,7 @@ Class Objects The :class:`Class` objects used as values in the dictionary returned by :func:`readmodule` and :func:`readmodule_ex` provide the following data -members: +attributes: .. attribute:: Class.module @@ -87,7 +89,7 @@ Function Objects ---------------- The :class:`Function` objects used as values in the dictionary returned by -:func:`readmodule_ex` provide the following data members: +:func:`readmodule_ex` provide the following attributes: .. attribute:: Function.module diff --git a/Doc/library/pydoc.rst b/Doc/library/pydoc.rst index 01f48b1e0f..e100865515 100644 --- a/Doc/library/pydoc.rst +++ b/Doc/library/pydoc.rst @@ -1,4 +1,3 @@ - :mod:`pydoc` --- Documentation generator and online help system =============================================================== @@ -13,6 +12,10 @@ single: documentation; online single: help; online +**Source code:** :source:`Lib/pydoc.py` + +-------------- + The :mod:`pydoc` module automatically generates documentation from Python modules. The documentation can be presented as pages of text on the console, served to a Web browser, or saved to HTML files. @@ -51,12 +54,21 @@ manner similar to the Unix :program:`man` command. The synopsis line of a module is the first line of its documentation string. You can also use :program:`pydoc` to start an HTTP server on the local machine -that will serve documentation to visiting Web browsers. :program:`pydoc -p 1234` -will start a HTTP server on port 1234, allowing you to browse -the documentation at ``http://localhost:1234/`` in your preferred Web browser. +that will serve documentation to visiting Web browsers. :program:`pydoc -p 1234` +will start a HTTP server on port 1234, allowing you to browse the +documentation at ``http://localhost:1234/`` in your preferred Web browser. +Specifying ``0`` as the port number will select an arbitrary unused port. + :program:`pydoc -g` will start the server and additionally bring up a small :mod:`tkinter`\ -based graphical interface to help you search for -documentation pages. +documentation pages. The ``-g`` option is deprecated, since the server can +now be controlled directly from HTTP clients. + +:program:`pydoc -b` will start the server and additionally open a web +browser to a module index page. Each served page has a navigation bar at the +top where you can *Get* help on an individual item, *Search* all modules with a +keyword in their synopsis line, and go to the *Module index*, *Topics* and +*Keywords* pages. When :program:`pydoc` generates documentation, it uses the current environment and path to locate modules. Thus, invoking :program:`pydoc spam` @@ -70,3 +82,5 @@ be overridden by setting the :envvar:`PYTHONDOCS` environment variable to a different URL or to a local directory containing the Library Reference Manual pages. +.. versionchanged:: 3.2 + Added the ``-b`` option, deprecated the ``-g`` option. diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst index 1eb9db1122..a648cfa529 100644 --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -1,4 +1,3 @@ - :mod:`xml.parsers.expat` --- Fast XML parsing using Expat ========================================================= @@ -56,7 +55,7 @@ The :mod:`xml.parsers.expat` module contains two functions: Returns an explanatory string for a given error number *errno*. -.. function:: ParserCreate([encoding[, namespace_separator]]) +.. function:: ParserCreate(encoding=None, namespace_separator=None) Creates and returns a new :class:`xmlparser` object. *encoding*, if specified, must be a string naming the encoding used by the XML data. Expat doesn't @@ -177,7 +176,7 @@ XMLParser Objects This method can only be called before the :meth:`Parse` or :meth:`ParseFile` methods are called; calling it after either of those have been called causes :exc:`ExpatError` to be raised with the :attr:`code` attribute set to - :const:`errors.XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING`. + ``errors.codes[errors.XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING]``. :class:`xmlparser` objects have the following attributes: @@ -475,8 +474,21 @@ ExpatError Exceptions .. attribute:: ExpatError.code - Expat's internal error number for the specific error. This will match one of - the constants defined in the ``errors`` object from this module. + Expat's internal error number for the specific error. The + :data:`errors.messages` dictionary maps these error numbers to Expat's error + messages. For example:: + + from xml.parsers.expat import ParserCreate, ExpatError, errors + + p = ParserCreate() + try: + p.Parse(some_xml_document) + except ExpatError as err: + print("Error:", errors.messages[err.code]) + + The :mod:`errors` module also provides error message constants and a + dictionary :data:`~errors.codes` mapping these messages back to the error + codes, see below. .. attribute:: ExpatError.lineno @@ -538,15 +550,16 @@ The output from this program is:: Content Model Descriptions -------------------------- -.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +.. module:: xml.parsers.expat.model +.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> Content modules are described using nested tuples. Each tuple contains four values: the type, the quantifier, the name, and a tuple of children. Children are simply additional content module descriptions. -The values of the first two fields are constants defined in the ``model`` object -of the :mod:`xml.parsers.expat` module. These constants can be collected in two +The values of the first two fields are constants defined in the +:mod:`xml.parsers.expat.model` module. These constants can be collected in two groups: the model type group and the quantifier group. The constants in the model type group are: @@ -618,143 +631,139 @@ The constants in the quantifier group are: Expat error constants --------------------- -The following constants are provided in the ``errors`` object of the -:mod:`xml.parsers.expat` module. These constants are useful in interpreting -some of the attributes of the :exc:`ExpatError` exception objects raised when an -error has occurred. +.. module:: xml.parsers.expat.errors + +The following constants are provided in the :mod:`xml.parsers.expat.errors` +module. These constants are useful in interpreting some of the attributes of +the :exc:`ExpatError` exception objects raised when an error has occurred. +Since for backwards compatibility reasons, the constants' value is the error +*message* and not the numeric error *code*, you do this by comparing its +:attr:`code` attribute with +:samp:`errors.codes[errors.XML_ERROR_{CONSTANT_NAME}]`. + +The ``errors`` module has the following attributes: + +.. data:: codes + + A dictionary mapping numeric error codes to their string descriptions. + + .. versionadded:: 3.2 + + +.. data:: messages -The ``errors`` object has the following attributes: + A dictionary mapping string descriptions to their error codes. + + .. versionadded:: 3.2 .. data:: XML_ERROR_ASYNC_ENTITY - :noindex: .. data:: XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF - :noindex: An entity reference in an attribute value referred to an external entity instead of an internal entity. .. data:: XML_ERROR_BAD_CHAR_REF - :noindex: A character reference referred to a character which is illegal in XML (for example, character ``0``, or '``�``'). .. data:: XML_ERROR_BINARY_ENTITY_REF - :noindex: An entity reference referred to an entity which was declared with a notation, so cannot be parsed. .. data:: XML_ERROR_DUPLICATE_ATTRIBUTE - :noindex: An attribute was used more than once in a start tag. .. data:: XML_ERROR_INCORRECT_ENCODING - :noindex: .. data:: XML_ERROR_INVALID_TOKEN - :noindex: Raised when an input byte could not properly be assigned to a character; for example, a NUL byte (value ``0``) in a UTF-8 input stream. .. data:: XML_ERROR_JUNK_AFTER_DOC_ELEMENT - :noindex: Something other than whitespace occurred after the document element. .. data:: XML_ERROR_MISPLACED_XML_PI - :noindex: An XML declaration was found somewhere other than the start of the input data. .. data:: XML_ERROR_NO_ELEMENTS - :noindex: The document contains no elements (XML requires all documents to contain exactly one top-level element).. .. data:: XML_ERROR_NO_MEMORY - :noindex: Expat was not able to allocate memory internally. .. data:: XML_ERROR_PARAM_ENTITY_REF - :noindex: A parameter entity reference was found where it was not allowed. .. data:: XML_ERROR_PARTIAL_CHAR - :noindex: An incomplete character was found in the input. .. data:: XML_ERROR_RECURSIVE_ENTITY_REF - :noindex: An entity reference contained another reference to the same entity; possibly via a different name, and possibly indirectly. .. data:: XML_ERROR_SYNTAX - :noindex: Some unspecified syntax error was encountered. .. data:: XML_ERROR_TAG_MISMATCH - :noindex: An end tag did not match the innermost open start tag. .. data:: XML_ERROR_UNCLOSED_TOKEN - :noindex: Some token (such as a start tag) was not closed before the end of the stream or the next token was encountered. .. data:: XML_ERROR_UNDEFINED_ENTITY - :noindex: A reference was made to a entity which was not defined. .. data:: XML_ERROR_UNKNOWN_ENCODING - :noindex: The document encoding is not supported by Expat. .. data:: XML_ERROR_UNCLOSED_CDATA_SECTION - :noindex: A CDATA marked section was not closed. .. data:: XML_ERROR_EXTERNAL_ENTITY_HANDLING - :noindex: .. data:: XML_ERROR_NOT_STANDALONE - :noindex: The parser determined that the document was not "standalone" though it declared itself to be in the XML declaration, and the :attr:`NotStandaloneHandler` was @@ -762,15 +771,12 @@ The ``errors`` object has the following attributes: .. data:: XML_ERROR_UNEXPECTED_STATE - :noindex: .. data:: XML_ERROR_ENTITY_DECLARED_IN_PE - :noindex: .. data:: XML_ERROR_FEATURE_REQUIRES_XML_DTD - :noindex: An operation was requested that requires DTD support to be compiled in, but Expat was configured without DTD support. This should never be reported by a @@ -778,7 +784,6 @@ The ``errors`` object has the following attributes: .. data:: XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING - :noindex: A behavioral change was requested after parsing started that can only be changed before parsing has started. This is (currently) only raised by @@ -786,63 +791,53 @@ The ``errors`` object has the following attributes: .. data:: XML_ERROR_UNBOUND_PREFIX - :noindex: An undeclared prefix was found when namespace processing was enabled. .. data:: XML_ERROR_UNDECLARING_PREFIX - :noindex: The document attempted to remove the namespace declaration associated with a prefix. .. data:: XML_ERROR_INCOMPLETE_PE - :noindex: A parameter entity contained incomplete markup. .. data:: XML_ERROR_XML_DECL - :noindex: The document contained no document element at all. .. data:: XML_ERROR_TEXT_DECL - :noindex: There was an error parsing a text declaration in an external entity. .. data:: XML_ERROR_PUBLICID - :noindex: Characters were found in the public id that are not allowed. .. data:: XML_ERROR_SUSPENDED - :noindex: The requested operation was made on a suspended parser, but isn't allowed. This includes attempts to provide additional input or to stop the parser. .. data:: XML_ERROR_NOT_SUSPENDED - :noindex: An attempt to resume the parser was made when the parser had not been suspended. .. data:: XML_ERROR_ABORTED - :noindex: This should not be reported to Python applications. .. data:: XML_ERROR_FINISHED - :noindex: The requested operation was made on a parser which was finished parsing input, but isn't allowed. This includes attempts to provide additional input or to @@ -850,7 +845,6 @@ The ``errors`` object has the following attributes: .. data:: XML_ERROR_SUSPEND_PE - :noindex: .. rubric:: Footnotes diff --git a/Doc/library/python.rst b/Doc/library/python.rst index 7d4d8279ba..b67fbfc281 100644 --- a/Doc/library/python.rst +++ b/Doc/library/python.rst @@ -1,4 +1,3 @@ - .. _python: *********************** @@ -13,6 +12,7 @@ overview: .. toctree:: sys.rst + sysconfig.rst builtins.rst __main__.rst warnings.rst @@ -25,3 +25,4 @@ overview: inspect.rst site.rst fpectl.rst + distutils.rst diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index e16d6e98a1..26336ef515 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -1,10 +1,12 @@ - :mod:`queue` --- A synchronized queue class =========================================== .. module:: queue :synopsis: A synchronized queue class. +**Source code:** :source:`Lib/queue.py` + +-------------- The :mod:`queue` module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be @@ -20,17 +22,17 @@ the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the :mod:`heapq` module) and the lowest valued entry is retrieved first. + The :mod:`queue` module defines the following classes and exceptions: -.. class:: Queue(maxsize) +.. class:: Queue(maxsize=0) Constructor for a FIFO queue. *maxsize* is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If *maxsize* is less than or equal to zero, the queue size is infinite. - -.. class:: LifoQueue(maxsize) +.. class:: LifoQueue(maxsize=0) Constructor for a LIFO queue. *maxsize* is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will @@ -38,7 +40,7 @@ The :mod:`queue` module defines the following classes and exceptions: *maxsize* is less than or equal to zero, the queue size is infinite. -.. class:: PriorityQueue(maxsize) +.. class:: PriorityQueue(maxsize=0) Constructor for a priority queue. *maxsize* is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will @@ -61,12 +63,6 @@ The :mod:`queue` module defines the following classes and exceptions: Exception raised when non-blocking :meth:`put` (or :meth:`put_nowait`) is called on a :class:`Queue` object which is full. -.. seealso:: - - :class:`collections.deque` is an alternative implementation of unbounded - queues with fast atomic :func:`append` and :func:`popleft` operations that - do not require locking. - .. _queueobjects: @@ -100,7 +96,7 @@ provide the public methods described below. guarantee that a subsequent call to put() will not block. -.. method:: Queue.put(item[, block[, timeout]]) +.. method:: Queue.put(item, block=True, timeout=None) Put *item* into the queue. If optional args *block* is true and *timeout* is None (the default), block if necessary until a free slot is available. If @@ -116,7 +112,7 @@ provide the public methods described below. Equivalent to ``put(item, False)``. -.. method:: Queue.get([block[, timeout]]) +.. method:: Queue.get(block=True, timeout=None) Remove and return an item from the queue. If optional args *block* is true and *timeout* is None (the default), block if necessary until an item is available. @@ -177,3 +173,14 @@ Example of how to wait for enqueued tasks to be completed:: q.join() # block until all tasks are done + +.. seealso:: + + Class :class:`multiprocessing.Queue` + A queue class for use in a multi-processing (rather than multi-threading) + context. + + :class:`collections.deque` is an alternative implementation of unbounded + queues with fast atomic :func:`append` and :func:`popleft` operations that + do not require locking. + diff --git a/Doc/library/quopri.rst b/Doc/library/quopri.rst index a64337ef7c..755811adfd 100644 --- a/Doc/library/quopri.rst +++ b/Doc/library/quopri.rst @@ -1,4 +1,3 @@ - :mod:`quopri` --- Encode and decode MIME quoted-printable data ============================================================== @@ -10,6 +9,10 @@ pair: quoted-printable; encoding single: MIME; quoted-printable encoding +**Source code:** :source:`Lib/quopri.py` + +-------------- + This module performs quoted-printable transport encoding and decoding, as defined in :rfc:`1521`: "MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies". @@ -18,8 +21,7 @@ few nonprintable characters; the base64 encoding scheme available via the :mod:`base64` module is more compact if there are many such characters, as when sending a graphics file. - -.. function:: decode(input, output[,header]) +.. function:: decode(input, output, header=False) Decode the contents of the *input* file and write the resulting decoded binary data to the *output* file. *input* and *output* must be :term:`file objects @@ -30,7 +32,7 @@ sending a graphics file. Part Two: Message Header Extensions for Non-ASCII Text". -.. function:: encode(input, output, quotetabs) +.. function:: encode(input, output, quotetabs, header=False) Encode the contents of the *input* file and write the resulting quoted-printable data to the *output* file. *input* and *output* must be :term:`file objects @@ -38,24 +40,24 @@ sending a graphics file. empty string. *quotetabs* is a flag which controls whether to encode embedded spaces and tabs; when true it encodes such embedded whitespace, and when false it leaves them unencoded. Note that spaces and tabs appearing at the - end of lines are always encoded, as per :rfc:`1521`. + end of lines are always encoded, as per :rfc:`1521`. *header* is a flag + which controls if spaces are encoded as underscores as per :rfc:`1522`. -.. function:: decodestring(s[,header]) +.. function:: decodestring(s, header=False) Like :func:`decode`, except that it accepts a source string and returns the corresponding decoded string. -.. function:: encodestring(s[, quotetabs]) +.. function:: encodestring(s, quotetabs=False, header=False) Like :func:`encode`, except that it accepts a source string and returns the - corresponding encoded string. *quotetabs* is optional (defaulting to 0), and is - passed straight through to :func:`encode`. + corresponding encoded string. *quotetabs* and *header* are optional + (defaulting to ``False``), and are passed straight through to :func:`encode`. .. seealso:: Module :mod:`base64` Encode and decode MIME base64 data - diff --git a/Doc/library/random.rst b/Doc/library/random.rst index 2976f5e392..31cb945bd3 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -1,17 +1,20 @@ - :mod:`random` --- Generate pseudo-random numbers ================================================ .. module:: random :synopsis: Generate pseudo-random numbers with various common distributions. +**Source code:** :source:`Lib/random.py` + +-------------- This module implements pseudo-random number generators for various distributions. -For integers, uniform selection from a range. For sequences, uniform selection -of a random element, a function to generate a random permutation of a list -in-place, and a function for random sampling without replacement. +For integers, there is uniform selection from a range. For sequences, there is +uniform selection of a random element, a function to generate a random +permutation of a list in-place, and a function for random sampling without +replacement. On the real line, there are functions to compute uniform, normal (Gaussian), lognormal, negative exponential, gamma, and beta distributions. For generating @@ -36,22 +39,30 @@ basic generator of your own devising: in that case, override the :meth:`random`, Optionally, a new generator can supply a :meth:`getrandbits` method --- this allows :meth:`randrange` to produce selections over an arbitrarily large range. +The :mod:`random` module also provides the :class:`SystemRandom` class which +uses the system function :func:`os.urandom` to generate random numbers +from sources provided by the operating system. + Bookkeeping functions: +.. function:: seed([x], version=2) -.. function:: seed([x]) + Initialize the random number generator. - Initialize the basic random number generator. Optional argument *x* can be any - :term:`hashable` object. If *x* is omitted or ``None``, current system time is used; - current system time is also used to initialize the generator when the module is - first imported. If randomness sources are provided by the operating system, - they are used instead of the system time (see the :func:`os.urandom` function - for details on availability). + If *x* is omitted or ``None``, the current system time is used. If + randomness sources are provided by the operating system, they are used + instead of the system time (see the :func:`os.urandom` function for details + on availability). - If *x* is not ``None`` or an int, ``hash(x)`` is used instead. If *x* is an - int, *x* is used directly. + If *x* is an int, it is used directly. + With version 2 (the default), a :class:`str`, :class:`bytes`, or :class:`bytearray` + object gets converted to an :class:`int` and all of its bits are used. With version 1, + the :func:`hash` of *x* is used instead. + + .. versionchanged:: 3.2 + Moved to the version 2 scheme which uses all of the bits in a string seed. .. function:: getstate() @@ -82,6 +93,13 @@ Functions for integers: equivalent to ``choice(range(start, stop, step))``, but doesn't actually build a range object. + The positional argument pattern matches that of :func:`range`. Keyword arguments + should not be used because the function may use them in unexpected ways. + + .. versionchanged:: 3.2 + :meth:`randrange` is more sophisticated about producing equally distributed + values. Formerly it used a style like ``int(random()*n)`` which could produce + slightly uneven distributions. .. function:: randint(a, b) @@ -145,6 +163,7 @@ be found in any statistics text. The end-point value ``b`` may or may not be included in the range depending on floating-point rounding in the equation ``a + (b-a) * random()``. + .. function:: triangular(low, high, mode) Return a random floating point number *N* such that ``low <= N <= high`` and @@ -173,6 +192,12 @@ be found in any statistics text. Gamma distribution. (*Not* the gamma function!) Conditions on the parameters are ``alpha > 0`` and ``beta > 0``. + The probability distribution function is:: + + x ** (alpha - 1) * math.exp(-x / beta) + pdf(x) = -------------------------------------- + math.gamma(alpha) * beta ** alpha + .. function:: gauss(mu, sigma) @@ -213,29 +238,67 @@ be found in any statistics text. parameter. -Alternative Generators: +Alternative Generator: .. class:: SystemRandom([seed]) Class that uses the :func:`os.urandom` function for generating random numbers from sources provided by the operating system. Not available on all systems. - Does not rely on software state and sequences are not reproducible. Accordingly, + Does not rely on software state, and sequences are not reproducible. Accordingly, the :meth:`seed` method has no effect and is ignored. The :meth:`getstate` and :meth:`setstate` methods raise :exc:`NotImplementedError` if called. -Examples of basic usage:: +.. seealso:: + + M. Matsumoto and T. Nishimura, "Mersenne Twister: A 623-dimensionally + equidistributed uniform pseudorandom number generator", ACM Transactions on + Modeling and Computer Simulation Vol. 8, No. 1, January pp.3-30 1998. + + + `Complementary-Multiply-with-Carry recipe + <http://code.activestate.com/recipes/576707/>`_ for a compatible alternative + random number generator with a long period and comparatively simple update + operations. + + +Notes on Reproducibility +------------------------ - >>> random.random() # Random float x, 0.0 <= x < 1.0 +Sometimes it is useful to be able to reproduce the sequences given by a pseudo +random number generator. By re-using a seed value, the same sequence should be +reproducible from run to run as long as multiple threads are not running. + +Most of the random module's algorithms and seeding functions are subject to +change across Python versions, but two aspects are guaranteed not to change: + +* If a new seeding method is added, then a backward compatible seeder will be + offered. + +* The generator's :meth:`random` method will continue to produce the same + sequence when the compatible seeder is given the same seed. + +.. _random-examples: + +Examples and Recipes +-------------------- + +Basic usage:: + + >>> random.random() # Random float x, 0.0 <= x < 1.0 0.37444887175646646 - >>> random.uniform(1, 10) # Random float x, 1.0 <= x < 10.0 + + >>> random.uniform(1, 10) # Random float x, 1.0 <= x < 10.0 1.1800146073117523 - >>> random.randint(1, 10) # Integer from 1 to 10, endpoints included + + >>> random.randrange(10) # Integer from 0 to 9 7 - >>> random.randrange(0, 101, 2) # Even integer from 0 to 100 + + >>> random.randrange(0, 101, 2) # Even integer from 0 to 100 26 - >>> random.choice('abcdefghij') # Choose a random element + + >>> random.choice('abcdefghij') # Single random element 'c' >>> items = [1, 2, 3, 4, 5, 6, 7] @@ -243,19 +306,25 @@ Examples of basic usage:: >>> items [7, 3, 2, 5, 6, 4, 1] - >>> random.sample([1, 2, 3, 4, 5], 3) # Choose 3 elements + >>> random.sample([1, 2, 3, 4, 5], 3) # Three samples without replacement [4, 1, 5] +A common task is to make a :func:`random.choice` with weighted probababilites. +If the weights are small integer ratios, a simple technique is to build a sample +population with repeats:: -.. seealso:: - - M. Matsumoto and T. Nishimura, "Mersenne Twister: A 623-dimensionally - equidistributed uniform pseudorandom number generator", ACM Transactions on - Modeling and Computer Simulation Vol. 8, No. 1, January pp.3-30 1998. + >>> weighted_choices = [('Red', 3), ('Blue', 2), ('Yellow', 1), ('Green', 4)] + >>> population = [val for val, cnt in weighted_choices for i in range(cnt)] + >>> random.choice(population) + 'Green' +A more general approach is to arrange the weights in a cumulative distribution +with :func:`itertools.accumulate`, and then locate the random value with +:func:`bisect.bisect`:: - `Complementary-Multiply-with-Carry recipe - <http://code.activestate.com/recipes/576707/>`_ for a compatible alternative - random number generator with a long period and comparatively simple update - operations. + >>> choices, weights = zip(*weighted_choices) + >>> cumdist = list(itertools.accumulate(weights)) + >>> x = random.random() * cumdist[-1] + >>> choices[bisect.bisect(cumdist, x)] + 'Blue' diff --git a/Doc/library/re.rst b/Doc/library/re.rst index 1d90f791a7..b196a28f9d 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -1,4 +1,3 @@ - :mod:`re` --- Regular expression operations =========================================== @@ -34,8 +33,9 @@ newline. Usually patterns will be expressed in Python code using this raw string notation. It is important to note that most regular expression operations are available as -module-level functions and :class:`RegexObject` methods. The functions are -shortcuts that don't require you to compile a regex object first, but miss some +module-level functions and methods on +:ref:`compiled regular expressions <re-objects>`. The functions are shortcuts +that don't require you to compile a regex object first, but miss some fine-tuning parameters. .. seealso:: @@ -80,7 +80,7 @@ Some characters, like ``'|'`` or ``'('``, are special. Special characters either stand for classes of ordinary characters, or affect how the regular expressions around them are interpreted. Regular expression pattern strings may not contain null bytes, but can specify -the null byte using the ``\number`` notation, e.g., ``'\x00'``. +the null byte using a ``\number`` notation such as ``'\x00'``. The special characters are: @@ -161,30 +161,36 @@ The special characters are: raw strings for all but the simplest expressions. ``[]`` - Used to indicate a set of characters. Characters can be listed individually, or - a range of characters can be indicated by giving two characters and separating - them by a ``'-'``. Special characters are not active inside sets. For example, - ``[akm$]`` will match any of the characters ``'a'``, ``'k'``, - ``'m'``, or ``'$'``; ``[a-z]`` will match any lowercase letter, and - ``[a-zA-Z0-9]`` matches any letter or digit. Character classes such - as ``\w`` or ``\S`` (defined below) are also acceptable inside a - range, although the characters they match depends on whether - :const:`ASCII` or :const:`LOCALE` mode is in force. If you want to - include a ``']'`` or a ``'-'`` inside a set, precede it with a - backslash, or place it as the first character. The pattern ``[]]`` - will match ``']'``, for example. - - You can match the characters not within a range by :dfn:`complementing` the set. - This is indicated by including a ``'^'`` as the first character of the set; - ``'^'`` elsewhere will simply match the ``'^'`` character. For example, - ``[^5]`` will match any character except ``'5'``, and ``[^^]`` will match any - character except ``'^'``. - - Note that inside ``[]`` the special forms and special characters lose - their meanings and only the syntaxes described here are valid. For - example, ``+``, ``*``, ``(``, ``)``, and so on are treated as - literals inside ``[]``, and backreferences cannot be used inside - ``[]``. + Used to indicate a set of characters. In a set: + + * Characters can be listed individually, e.g. ``[amk]`` will match ``'a'``, + ``'m'``, or ``'k'``. + + * Ranges of characters can be indicated by giving two characters and separating + them by a ``'-'``, for example ``[a-z]`` will match any lowercase ASCII letter, + ``[0-5][0-9]`` will match all the two-digits numbers from ``00`` to ``59``, and + ``[0-9A-Fa-f]`` will match any hexadecimal digit. If ``-`` is escaped (e.g. + ``[a\-z]``) or if it's placed as the first or last character (e.g. ``[a-]``), + it will match a literal ``'-'``. + + * Special characters lose their special meaning inside sets. For example, + ``[(+*)]`` will match any of the literal characters ``'('``, ``'+'``, + ``'*'``, or ``')'``. + + * Character classes such as ``\w`` or ``\S`` (defined below) are also accepted + inside a set, although the characters they match depends on whether + :const:`ASCII` or :const:`LOCALE` mode is in force. + + * Characters that are not within a range can be matched by :dfn:`complementing` + the set. If the first character of the set is ``'^'``, all the characters + that are *not* in the set will be matched. For example, ``[^5]`` will match + any character except ``'5'``, and ``[^^]`` will match any character except + ``'^'``. ``^`` has no special meaning if it's not the first character in + the set. + + * To match a literal ``']'`` inside a set, precede it with a backslash, or + place it at the beginning of the set. For example, both ``[()[\]{}]`` and + ``[]()[{}]`` will both match a parenthesis. ``'|'`` ``A|B``, where A and B can be arbitrary REs, creates a regular expression that @@ -339,11 +345,12 @@ the second character. For example, ``\$`` matches the character ``'$'``. ``\d`` For Unicode (str) patterns: - Matches any Unicode digit (which includes ``[0-9]``, and also many - other digit characters). If the :const:`ASCII` flag is used only - ``[0-9]`` is matched (but the flag affects the entire regular - expression, so in such cases using an explicit ``[0-9]`` may be a - better choice). + Matches any Unicode decimal digit (that is, any character in + Unicode character category [Nd]). This includes ``[0-9]``, and + also many other digit characters. If the :const:`ASCII` flag is + used only ``[0-9]`` is matched (but the flag affects the entire + regular expression, so in such cases using an explicit ``[0-9]`` + may be a better choice). For 8-bit (bytes) patterns: Matches any decimal digit; this is equivalent to ``[0-9]``. @@ -404,7 +411,7 @@ accepted by the regular expression parser:: \r \t \v \x \\ -Octal escapes are included in a limited form: If the first digit is a 0, or if +Octal escapes are included in a limited form. If the first digit is a 0, or if there are three octal digits, it is considered an octal escape. Otherwise, it is a group reference. As for string literals, octal escapes are always at most three digits in length. @@ -412,8 +419,8 @@ three digits in length. .. _matching-searching: -Matching vs Searching ---------------------- +Matching vs. Searching +---------------------- .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> @@ -446,7 +453,7 @@ regular expressions. Most non-trivial applications always use the compiled form. -.. function:: compile(pattern[, flags]) +.. function:: compile(pattern, flags=0) Compile a regular expression pattern into a regular expression object, which can be used for matching using its :func:`match` and :func:`search` methods, @@ -491,6 +498,11 @@ form. isn't allowed for bytes). +.. data:: DEBUG + + Display debug information about compiled expression. + + .. data:: I IGNORECASE @@ -547,21 +559,21 @@ form. -.. function:: search(pattern, string[, flags]) +.. function:: search(pattern, string, flags=0) Scan through *string* looking for a location where the regular expression - *pattern* produces a match, and return a corresponding :class:`MatchObject` - instance. Return ``None`` if no position in the string matches the pattern; note - that this is different from finding a zero-length match at some point in the - string. + *pattern* produces a match, and return a corresponding :ref:`match object + <match-objects>`. Return ``None`` if no position in the string matches the + pattern; note that this is different from finding a zero-length match at some + point in the string. -.. function:: match(pattern, string[, flags]) +.. function:: match(pattern, string, flags=0) If zero or more characters at the beginning of *string* match the regular - expression *pattern*, return a corresponding :class:`MatchObject` instance. - Return ``None`` if the string does not match the pattern; note that this is - different from a zero-length match. + expression *pattern*, return a corresponding :ref:`match object + <match-objects>`. Return ``None`` if the string does not match the pattern; + note that this is different from a zero-length match. .. note:: @@ -569,7 +581,7 @@ form. instead. -.. function:: split(pattern, string[, maxsplit=0, flags=0]) +.. function:: split(pattern, string, maxsplit=0, flags=0) Split *string* by the occurrences of *pattern*. If capturing parentheses are used in *pattern*, then the text of all groups in the pattern are also returned @@ -594,8 +606,7 @@ form. ['', '...', 'words', ', ', 'words', '...', ''] That way, separator components are always found at the same relative - indices within the result list (e.g., if there's one capturing group - in the separator, the 0th, the 2nd and so forth). + indices within the result list. Note that *split* will never split a string on an empty pattern match. For example: @@ -609,7 +620,7 @@ form. Added the optional flags argument. -.. function:: findall(pattern, string[, flags]) +.. function:: findall(pattern, string, flags=0) Return all non-overlapping matches of *pattern* in *string*, as a list of strings. The *string* is scanned left-to-right, and matches are returned in @@ -619,22 +630,22 @@ form. beginning of another match. -.. function:: finditer(pattern, string[, flags]) +.. function:: finditer(pattern, string, flags=0) - Return an :term:`iterator` yielding :class:`MatchObject` instances over all - non-overlapping matches for the RE *pattern* in *string*. The *string* is - scanned left-to-right, and matches are returned in the order found. Empty + Return an :term:`iterator` yielding :ref:`match objects <match-objects>` over + all non-overlapping matches for the RE *pattern* in *string*. The *string* + is scanned left-to-right, and matches are returned in the order found. Empty matches are included in the result unless they touch the beginning of another match. -.. function:: sub(pattern, repl, string[, count, flags]) +.. function:: sub(pattern, repl, string, count=0, flags=0) Return the string obtained by replacing the leftmost non-overlapping occurrences of *pattern* in *string* by the replacement *repl*. If the pattern isn't found, *string* is returned unchanged. *repl* can be a string or a function; if it is a string, any backslash escapes in it are processed. That is, ``\n`` is - converted to a single newline character, ``\r`` is converted to a linefeed, and + converted to a single newline character, ``\r`` is converted to a carriage return, and so forth. Unknown escapes such as ``\j`` are left alone. Backreferences, such as ``\6``, are replaced with the substring matched by group 6 in the pattern. For example: @@ -677,7 +688,7 @@ form. Added the optional flags argument. -.. function:: subn(pattern, repl, string[, count, flags]) +.. function:: subn(pattern, repl, string, count=0, flags=0) Perform the same operation as :func:`sub`, but return a tuple ``(new_string, number_of_subs_made)``. @@ -711,107 +722,107 @@ form. Regular Expression Objects -------------------------- -.. class:: RegexObject - - The :class:`RegexObject` class supports the following methods and attributes: +Compiled regular expression objects support the following methods and +attributes: - .. method:: RegexObject.search(string[, pos[, endpos]]) +.. method:: regex.search(string[, pos[, endpos]]) - Scan through *string* looking for a location where this regular expression - produces a match, and return a corresponding :class:`MatchObject` instance. - Return ``None`` if no position in the string matches the pattern; note that this - is different from finding a zero-length match at some point in the string. + Scan through *string* looking for a location where this regular expression + produces a match, and return a corresponding :ref:`match object + <match-objects>`. Return ``None`` if no position in the string matches the + pattern; note that this is different from finding a zero-length match at some + point in the string. - The optional second parameter *pos* gives an index in the string where the - search is to start; it defaults to ``0``. This is not completely equivalent to - slicing the string; the ``'^'`` pattern character matches at the real beginning - of the string and at positions just after a newline, but not necessarily at the - index where the search is to start. + The optional second parameter *pos* gives an index in the string where the + search is to start; it defaults to ``0``. This is not completely equivalent to + slicing the string; the ``'^'`` pattern character matches at the real beginning + of the string and at positions just after a newline, but not necessarily at the + index where the search is to start. - The optional parameter *endpos* limits how far the string will be searched; it - will be as if the string is *endpos* characters long, so only the characters - from *pos* to ``endpos - 1`` will be searched for a match. If *endpos* is less - than *pos*, no match will be found, otherwise, if *rx* is a compiled regular - expression object, ``rx.search(string, 0, 50)`` is equivalent to - ``rx.search(string[:50], 0)``. + The optional parameter *endpos* limits how far the string will be searched; it + will be as if the string is *endpos* characters long, so only the characters + from *pos* to ``endpos - 1`` will be searched for a match. If *endpos* is less + than *pos*, no match will be found; otherwise, if *rx* is a compiled regular + expression object, ``rx.search(string, 0, 50)`` is equivalent to + ``rx.search(string[:50], 0)``. - >>> pattern = re.compile("d") - >>> pattern.search("dog") # Match at index 0 - <_sre.SRE_Match object at ...> - >>> pattern.search("dog", 1) # No match; search doesn't include the "d" + >>> pattern = re.compile("d") + >>> pattern.search("dog") # Match at index 0 + <_sre.SRE_Match object at ...> + >>> pattern.search("dog", 1) # No match; search doesn't include the "d" - .. method:: RegexObject.match(string[, pos[, endpos]]) +.. method:: regex.match(string[, pos[, endpos]]) - If zero or more characters at the *beginning* of *string* match this regular - expression, return a corresponding :class:`MatchObject` instance. Return - ``None`` if the string does not match the pattern; note that this is different - from a zero-length match. + If zero or more characters at the *beginning* of *string* match this regular + expression, return a corresponding :ref:`match object <match-objects>`. + Return ``None`` if the string does not match the pattern; note that this is + different from a zero-length match. - The optional *pos* and *endpos* parameters have the same meaning as for the - :meth:`~RegexObject.search` method. + The optional *pos* and *endpos* parameters have the same meaning as for the + :meth:`~regex.search` method. - .. note:: + .. note:: - If you want to locate a match anywhere in *string*, use - :meth:`~RegexObject.search` instead. + If you want to locate a match anywhere in *string*, use + :meth:`~regex.search` instead. - >>> pattern = re.compile("o") - >>> pattern.match("dog") # No match as "o" is not at the start of "dog". - >>> pattern.match("dog", 1) # Match as "o" is the 2nd character of "dog". - <_sre.SRE_Match object at ...> + >>> pattern = re.compile("o") + >>> pattern.match("dog") # No match as "o" is not at the start of "dog". + >>> pattern.match("dog", 1) # Match as "o" is the 2nd character of "dog". + <_sre.SRE_Match object at ...> - .. method:: RegexObject.split(string[, maxsplit=0]) +.. method:: regex.split(string, maxsplit=0) - Identical to the :func:`split` function, using the compiled pattern. + Identical to the :func:`split` function, using the compiled pattern. - .. method:: RegexObject.findall(string[, pos[, endpos]]) +.. method:: regex.findall(string[, pos[, endpos]]) - Similar to the :func:`findall` function, using the compiled pattern, but - also accepts optional *pos* and *endpos* parameters that limit the search - region like for :meth:`match`. + Similar to the :func:`findall` function, using the compiled pattern, but + also accepts optional *pos* and *endpos* parameters that limit the search + region like for :meth:`match`. - .. method:: RegexObject.finditer(string[, pos[, endpos]]) +.. method:: regex.finditer(string[, pos[, endpos]]) - Similar to the :func:`finditer` function, using the compiled pattern, but - also accepts optional *pos* and *endpos* parameters that limit the search - region like for :meth:`match`. + Similar to the :func:`finditer` function, using the compiled pattern, but + also accepts optional *pos* and *endpos* parameters that limit the search + region like for :meth:`match`. - .. method:: RegexObject.sub(repl, string[, count=0]) +.. method:: regex.sub(repl, string, count=0) - Identical to the :func:`sub` function, using the compiled pattern. + Identical to the :func:`sub` function, using the compiled pattern. - .. method:: RegexObject.subn(repl, string[, count=0]) +.. method:: regex.subn(repl, string, count=0) - Identical to the :func:`subn` function, using the compiled pattern. + Identical to the :func:`subn` function, using the compiled pattern. - .. attribute:: RegexObject.flags +.. attribute:: regex.flags - The flags argument used when the RE object was compiled, or ``0`` if no flags - were provided. + The flags argument used when the RE object was compiled, or ``0`` if no flags + were provided. - .. attribute:: RegexObject.groups +.. attribute:: regex.groups - The number of capturing groups in the pattern. + The number of capturing groups in the pattern. - .. attribute:: RegexObject.groupindex +.. attribute:: regex.groupindex - A dictionary mapping any symbolic group names defined by ``(?P<id>)`` to group - numbers. The dictionary is empty if no symbolic groups were used in the - pattern. + A dictionary mapping any symbolic group names defined by ``(?P<id>)`` to group + numbers. The dictionary is empty if no symbolic groups were used in the + pattern. - .. attribute:: RegexObject.pattern +.. attribute:: regex.pattern - The pattern string from which the RE object was compiled. + The pattern string from which the RE object was compiled. .. _match-objects: @@ -819,184 +830,185 @@ Regular Expression Objects Match Objects ------------- -.. class:: MatchObject +Match objects always have a boolean value of :const:`True`. This lets you +use a simple if-statement to test whether a match was found. Match objects +support the following methods and attributes: + - Match Objects always have a boolean value of :const:`True`, so that you can test - whether e.g. :func:`match` resulted in a match with a simple if statement. They - support the following methods and attributes: +.. method:: match.expand(template) + Return the string obtained by doing backslash substitution on the template + string *template*, as done by the :meth:`~regex.sub` method. + Escapes such as ``\n`` are converted to the appropriate characters, + and numeric backreferences (``\1``, ``\2``) and named backreferences + (``\g<1>``, ``\g<name>``) are replaced by the contents of the + corresponding group. - .. method:: MatchObject.expand(template) - Return the string obtained by doing backslash substitution on the template - string *template*, as done by the :meth:`~RegexObject.sub` method. Escapes - such as ``\n`` are converted to the appropriate characters, and numeric - backreferences (``\1``, ``\2``) and named backreferences (``\g<1>``, - ``\g<name>``) are replaced by the contents of the corresponding group. +.. method:: match.group([group1, ...]) + Returns one or more subgroups of the match. If there is a single argument, the + result is a single string; if there are multiple arguments, the result is a + tuple with one item per argument. Without arguments, *group1* defaults to zero + (the whole match is returned). If a *groupN* argument is zero, the corresponding + return value is the entire matching string; if it is in the inclusive range + [1..99], it is the string matching the corresponding parenthesized group. If a + group number is negative or larger than the number of groups defined in the + pattern, an :exc:`IndexError` exception is raised. If a group is contained in a + part of the pattern that did not match, the corresponding result is ``None``. + If a group is contained in a part of the pattern that matched multiple times, + the last match is returned. - .. method:: MatchObject.group([group1, ...]) + >>> m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist") + >>> m.group(0) # The entire match + 'Isaac Newton' + >>> m.group(1) # The first parenthesized subgroup. + 'Isaac' + >>> m.group(2) # The second parenthesized subgroup. + 'Newton' + >>> m.group(1, 2) # Multiple arguments give us a tuple. + ('Isaac', 'Newton') - Returns one or more subgroups of the match. If there is a single argument, the - result is a single string; if there are multiple arguments, the result is a - tuple with one item per argument. Without arguments, *group1* defaults to zero - (the whole match is returned). If a *groupN* argument is zero, the corresponding - return value is the entire matching string; if it is in the inclusive range - [1..99], it is the string matching the corresponding parenthesized group. If a - group number is negative or larger than the number of groups defined in the - pattern, an :exc:`IndexError` exception is raised. If a group is contained in a - part of the pattern that did not match, the corresponding result is ``None``. - If a group is contained in a part of the pattern that matched multiple times, - the last match is returned. + If the regular expression uses the ``(?P<name>...)`` syntax, the *groupN* + arguments may also be strings identifying groups by their group name. If a + string argument is not used as a group name in the pattern, an :exc:`IndexError` + exception is raised. - >>> m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist") - >>> m.group(0) # The entire match - 'Isaac Newton' - >>> m.group(1) # The first parenthesized subgroup. - 'Isaac' - >>> m.group(2) # The second parenthesized subgroup. - 'Newton' - >>> m.group(1, 2) # Multiple arguments give us a tuple. - ('Isaac', 'Newton') + A moderately complicated example: - If the regular expression uses the ``(?P<name>...)`` syntax, the *groupN* - arguments may also be strings identifying groups by their group name. If a - string argument is not used as a group name in the pattern, an :exc:`IndexError` - exception is raised. + >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds") + >>> m.group('first_name') + 'Malcolm' + >>> m.group('last_name') + 'Reynolds' - A moderately complicated example: + Named groups can also be referred to by their index: - >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds") - >>> m.group('first_name') - 'Malcolm' - >>> m.group('last_name') - 'Reynolds' + >>> m.group(1) + 'Malcolm' + >>> m.group(2) + 'Reynolds' - Named groups can also be referred to by their index: + If a group matches multiple times, only the last match is accessible: - >>> m.group(1) - 'Malcolm' - >>> m.group(2) - 'Reynolds' + >>> m = re.match(r"(..)+", "a1b2c3") # Matches 3 times. + >>> m.group(1) # Returns only the last match. + 'c3' - If a group matches multiple times, only the last match is accessible: - >>> m = re.match(r"(..)+", "a1b2c3") # Matches 3 times. - >>> m.group(1) # Returns only the last match. - 'c3' +.. method:: match.groups(default=None) - .. method:: MatchObject.groups([default]) + Return a tuple containing all the subgroups of the match, from 1 up to however + many groups are in the pattern. The *default* argument is used for groups that + did not participate in the match; it defaults to ``None``. - Return a tuple containing all the subgroups of the match, from 1 up to however - many groups are in the pattern. The *default* argument is used for groups that - did not participate in the match; it defaults to ``None``. + For example: - For example: + >>> m = re.match(r"(\d+)\.(\d+)", "24.1632") + >>> m.groups() + ('24', '1632') - >>> m = re.match(r"(\d+)\.(\d+)", "24.1632") - >>> m.groups() - ('24', '1632') + If we make the decimal place and everything after it optional, not all groups + might participate in the match. These groups will default to ``None`` unless + the *default* argument is given: - If we make the decimal place and everything after it optional, not all groups - might participate in the match. These groups will default to ``None`` unless - the *default* argument is given: + >>> m = re.match(r"(\d+)\.?(\d+)?", "24") + >>> m.groups() # Second group defaults to None. + ('24', None) + >>> m.groups('0') # Now, the second group defaults to '0'. + ('24', '0') - >>> m = re.match(r"(\d+)\.?(\d+)?", "24") - >>> m.groups() # Second group defaults to None. - ('24', None) - >>> m.groups('0') # Now, the second group defaults to '0'. - ('24', '0') +.. method:: match.groupdict(default=None) - .. method:: MatchObject.groupdict([default]) + Return a dictionary containing all the *named* subgroups of the match, keyed by + the subgroup name. The *default* argument is used for groups that did not + participate in the match; it defaults to ``None``. For example: - Return a dictionary containing all the *named* subgroups of the match, keyed by - the subgroup name. The *default* argument is used for groups that did not - participate in the match; it defaults to ``None``. For example: + >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds") + >>> m.groupdict() + {'first_name': 'Malcolm', 'last_name': 'Reynolds'} - >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds") - >>> m.groupdict() - {'first_name': 'Malcolm', 'last_name': 'Reynolds'} +.. method:: match.start([group]) + match.end([group]) - .. method:: MatchObject.start([group]) - MatchObject.end([group]) + Return the indices of the start and end of the substring matched by *group*; + *group* defaults to zero (meaning the whole matched substring). Return ``-1`` if + *group* exists but did not contribute to the match. For a match object *m*, and + a group *g* that did contribute to the match, the substring matched by group *g* + (equivalent to ``m.group(g)``) is :: - Return the indices of the start and end of the substring matched by *group*; - *group* defaults to zero (meaning the whole matched substring). Return ``-1`` if - *group* exists but did not contribute to the match. For a match object *m*, and - a group *g* that did contribute to the match, the substring matched by group *g* - (equivalent to ``m.group(g)``) is :: + m.string[m.start(g):m.end(g)] - m.string[m.start(g):m.end(g)] + Note that ``m.start(group)`` will equal ``m.end(group)`` if *group* matched a + null string. For example, after ``m = re.search('b(c?)', 'cba')``, + ``m.start(0)`` is 1, ``m.end(0)`` is 2, ``m.start(1)`` and ``m.end(1)`` are both + 2, and ``m.start(2)`` raises an :exc:`IndexError` exception. - Note that ``m.start(group)`` will equal ``m.end(group)`` if *group* matched a - null string. For example, after ``m = re.search('b(c?)', 'cba')``, - ``m.start(0)`` is 1, ``m.end(0)`` is 2, ``m.start(1)`` and ``m.end(1)`` are both - 2, and ``m.start(2)`` raises an :exc:`IndexError` exception. + An example that will remove *remove_this* from email addresses: - An example that will remove *remove_this* from email addresses: + >>> email = "tony@tiremove_thisger.net" + >>> m = re.search("remove_this", email) + >>> email[:m.start()] + email[m.end():] + 'tony@tiger.net' - >>> email = "tony@tiremove_thisger.net" - >>> m = re.search("remove_this", email) - >>> email[:m.start()] + email[m.end():] - 'tony@tiger.net' +.. method:: match.span([group]) - .. method:: MatchObject.span([group]) + For a match *m*, return the 2-tuple ``(m.start(group), m.end(group))``. Note + that if *group* did not contribute to the match, this is ``(-1, -1)``. + *group* defaults to zero, the entire match. - For :class:`MatchObject` *m*, return the 2-tuple ``(m.start(group), - m.end(group))``. Note that if *group* did not contribute to the match, this is - ``(-1, -1)``. *group* defaults to zero, the entire match. +.. attribute:: match.pos - .. attribute:: MatchObject.pos + The value of *pos* which was passed to the :meth:`~regex.search` or + :meth:`~regex.match` method of a :ref:`match object <match-objects>`. This + is the index into the string at which the RE engine started looking for a + match. - The value of *pos* which was passed to the :meth:`~RegexObject.search` or - :meth:`~RegexObject.match` method of the :class:`RegexObject`. This is the - index into the string at which the RE engine started looking for a match. +.. attribute:: match.endpos - .. attribute:: MatchObject.endpos + The value of *endpos* which was passed to the :meth:`~regex.search` or + :meth:`~regex.match` method of a :ref:`match object <match-objects>`. This + is the index into the string beyond which the RE engine will not go. - The value of *endpos* which was passed to the :meth:`~RegexObject.search` or - :meth:`~RegexObject.match` method of the :class:`RegexObject`. This is the - index into the string beyond which the RE engine will not go. +.. attribute:: match.lastindex - .. attribute:: MatchObject.lastindex + The integer index of the last matched capturing group, or ``None`` if no group + was matched at all. For example, the expressions ``(a)b``, ``((a)(b))``, and + ``((ab))`` will have ``lastindex == 1`` if applied to the string ``'ab'``, while + the expression ``(a)(b)`` will have ``lastindex == 2``, if applied to the same + string. - The integer index of the last matched capturing group, or ``None`` if no group - was matched at all. For example, the expressions ``(a)b``, ``((a)(b))``, and - ``((ab))`` will have ``lastindex == 1`` if applied to the string ``'ab'``, while - the expression ``(a)(b)`` will have ``lastindex == 2``, if applied to the same - string. +.. attribute:: match.lastgroup - .. attribute:: MatchObject.lastgroup + The name of the last matched capturing group, or ``None`` if the group didn't + have a name, or if no group was matched at all. - The name of the last matched capturing group, or ``None`` if the group didn't - have a name, or if no group was matched at all. +.. attribute:: match.re - .. attribute:: MatchObject.re + The regular expression object whose :meth:`~regex.match` or + :meth:`~regex.search` method produced this match instance. - The regular expression object whose :meth:`~RegexObject.match` or - :meth:`~RegexObject.search` method produced this :class:`MatchObject` - instance. +.. attribute:: match.string - .. attribute:: MatchObject.string + The string passed to :meth:`~regex.match` or :meth:`~regex.search`. - The string passed to :meth:`~RegexObject.match` or - :meth:`~RegexObject.search`. +.. _re-examples: -Examples --------- +Regular Expression Examples +--------------------------- -Checking For a Pair +Checking for a Pair ^^^^^^^^^^^^^^^^^^^ In this example, we'll use the following helper function to display match @@ -1011,16 +1023,16 @@ objects a little more gracefully: Suppose you are writing a poker program where a player's hand is represented as a 5-character string with each character representing a card, "a" for ace, "k" -for king, "q" for queen, j for jack, "0" for 10, and "1" through "9" +for king, "q" for queen, "j" for jack, "t" for 10, and "2" through "9" representing the card with that value. To see if a given string is a valid hand, one could do the following: - >>> valid = re.compile(r"[0-9akqj]{5}$") - >>> displaymatch(valid.match("ak05q")) # Valid. - "<Match: 'ak05q', groups=()>" - >>> displaymatch(valid.match("ak05e")) # Invalid. - >>> displaymatch(valid.match("ak0")) # Invalid. + >>> valid = re.compile(r"^[a2-9tjqk]{5}$") + >>> displaymatch(valid.match("akt5q")) # Valid. + "<Match: 'akt5q', groups=()>" + >>> displaymatch(valid.match("akt5e")) # Invalid. + >>> displaymatch(valid.match("akt")) # Invalid. >>> displaymatch(valid.match("727ak")) # Valid. "<Match: '727ak', groups=()>" @@ -1035,8 +1047,7 @@ To match this with a regular expression, one could use backreferences as such: "<Match: '354aa', groups=('a',)>" To find out what card the pair consists of, one could use the -:meth:`~MatchObject.group` method of :class:`MatchObject` in the following -manner: +:meth:`~match.group` method of the match object in the following manner: .. doctest:: @@ -1059,14 +1070,14 @@ Simulating scanf() .. index:: single: scanf() -Python does not currently have an equivalent to :cfunc:`scanf`. Regular +Python does not currently have an equivalent to :c:func:`scanf`. Regular expressions are generally more powerful, though also more verbose, than -:cfunc:`scanf` format strings. The table below offers some more-or-less -equivalent mappings between :cfunc:`scanf` format tokens and regular +:c:func:`scanf` format strings. The table below offers some more-or-less +equivalent mappings between :c:func:`scanf` format tokens and regular expressions. +--------------------------------+---------------------------------------------+ -| :cfunc:`scanf` Token | Regular Expression | +| :c:func:`scanf` Token | Regular Expression | +================================+=============================================+ | ``%c`` | ``.`` | +--------------------------------+---------------------------------------------+ @@ -1091,7 +1102,7 @@ To extract the filename and numbers from a string like :: /usr/sbin/sendmail - 0 errors, 4 warnings -you would use a :cfunc:`scanf` format like :: +you would use a :c:func:`scanf` format like :: %s - %d errors, %d warnings @@ -1105,13 +1116,13 @@ Avoiding recursion If you create regular expressions that require the engine to perform a lot of recursion, you may encounter a :exc:`RuntimeError` exception with the message -``maximum recursion limit`` exceeded. For example, :: +``maximum recursion limit exceeded``. For example, :: >>> s = 'Begin ' + 1000*'a very long string ' + 'end' >>> re.match('Begin (\w| )*? end', s).end() Traceback (most recent call last): File "<stdin>", line 1, in ? - File "/usr/local/lib/python3.1/re.py", line 132, in match + File "/usr/local/lib/python3.2/re.py", line 132, in match return _compile(pattern, flags).match(string) RuntimeError: maximum recursion limit exceeded @@ -1250,10 +1261,10 @@ Finding all Adverbs and their Positions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If one wants more information about all matches of a pattern than the matched -text, :func:`finditer` is useful as it provides instances of -:class:`MatchObject` instead of strings. Continuing with the previous example, -if one was a writer who wanted to find all of the adverbs *and their positions* -in some text, he or she would use :func:`finditer` in the following manner: +text, :func:`finditer` is useful as it provides :ref:`match objects +<match-objects>` instead of strings. Continuing with the previous example, if +one was a writer who wanted to find all of the adverbs *and their positions* in +some text, he or she would use :func:`finditer` in the following manner: >>> text = "He was carefully disguised but captured quickly by police." >>> for m in re.finditer(r"\w+ly", text): @@ -1284,3 +1295,83 @@ functionally identical: <_sre.SRE_Match object at ...> >>> re.match("\\\\", r"\\") <_sre.SRE_Match object at ...> + + +Writing a Tokenizer +^^^^^^^^^^^^^^^^^^^ + +A `tokenizer or scanner <http://en.wikipedia.org/wiki/Lexical_analysis>`_ +analyzes a string to categorize groups of characters. This is a useful first +step in writing a compiler or interpreter. + +The text categories are specified with regular expressions. The technique is +to combine those into a single master regular expression and to loop over +successive matches:: + + import collections + import re + + Token = collections.namedtuple('Token', ['typ', 'value', 'line', 'column']) + + def tokenize(s): + keywords = {'IF', 'THEN', 'ENDIF', 'FOR', 'NEXT', 'GOSUB', 'RETURN'} + token_specification = [ + ('NUMBER', r'\d+(\.\d*)?'), # Integer or decimal number + ('ASSIGN', r':='), # Assignment operator + ('END', r';'), # Statement terminator + ('ID', r'[A-Za-z]+'), # Identifiers + ('OP', r'[+*\/\-]'), # Arithmetic operators + ('NEWLINE', r'\n'), # Line endings + ('SKIP', r'[ \t]'), # Skip over spaces and tabs + ] + tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification) + get_token = re.compile(tok_regex).match + line = 1 + pos = line_start = 0 + mo = get_token(s) + while mo is not None: + typ = mo.lastgroup + if typ == 'NEWLINE': + line_start = pos + line += 1 + elif typ != 'SKIP': + val = mo.group(typ) + if typ == 'ID' and val in keywords: + typ = val + yield Token(typ, val, line, mo.start()-line_start) + pos = mo.end() + mo = get_token(s, pos) + if pos != len(s): + raise RuntimeError('Unexpected character %r on line %d' %(s[pos], line)) + + statements = ''' + IF quantity THEN + total := total + price * quantity; + tax := price * 0.05; + ENDIF; + ''' + + for token in tokenize(statements): + print(token) + +The tokenizer produces the following output:: + + Token(typ='IF', value='IF', line=2, column=5) + Token(typ='ID', value='quantity', line=2, column=8) + Token(typ='THEN', value='THEN', line=2, column=17) + Token(typ='ID', value='total', line=3, column=9) + Token(typ='ASSIGN', value=':=', line=3, column=15) + Token(typ='ID', value='total', line=3, column=18) + Token(typ='OP', value='+', line=3, column=24) + Token(typ='ID', value='price', line=3, column=26) + Token(typ='OP', value='*', line=3, column=32) + Token(typ='ID', value='quantity', line=3, column=34) + Token(typ='END', value=';', line=3, column=42) + Token(typ='ID', value='tax', line=4, column=9) + Token(typ='ASSIGN', value=':=', line=4, column=13) + Token(typ='ID', value='price', line=4, column=16) + Token(typ='OP', value='*', line=4, column=22) + Token(typ='NUMBER', value='0.05', line=4, column=24) + Token(typ='END', value=';', line=4, column=28) + Token(typ='ENDIF', value='ENDIF', line=5, column=5) + Token(typ='END', value=';', line=5, column=10) diff --git a/Doc/library/readline.rst b/Doc/library/readline.rst index ab7b4b6ecb..ab5519775b 100644 --- a/Doc/library/readline.rst +++ b/Doc/library/readline.rst @@ -1,4 +1,3 @@ - :mod:`readline` --- GNU readline interface ========================================== @@ -25,6 +24,7 @@ function. you can check for the text "libedit" in :const:`readline.__doc__` to differentiate between GNU readline and libedit. + The :mod:`readline` module defines the following functions: @@ -177,7 +177,6 @@ The :mod:`readline` module defines the following functions: Append a line to the history buffer, as if it was the last line typed. - .. seealso:: Module :mod:`rlcompleter` diff --git a/Doc/library/reprlib.rst b/Doc/library/reprlib.rst index 958ead61a5..24a8e529e7 100644 --- a/Doc/library/reprlib.rst +++ b/Doc/library/reprlib.rst @@ -1,11 +1,13 @@ :mod:`reprlib` --- Alternate :func:`repr` implementation ======================================================== - .. module:: reprlib :synopsis: Alternate repr() implementation with size limits. .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/reprlib.py` + +-------------- The :mod:`reprlib` module provides a means for producing object representations with limits on the size of the resulting strings. This is used in the Python @@ -35,13 +37,36 @@ This module provides a class, an instance, and a function: similar to that returned by the built-in function of the same name, but with limits on most sizes. +In addition to size-limiting tools, the module also provides a decorator for +detecting recursive calls to :meth:`__repr__` and substituting a placeholder +string instead. + +.. decorator:: recursive_repr(fillvalue="...") + + Decorator for :meth:`__repr__` methods to detect recursive calls within the + same thread. If a recursive call is made, the *fillvalue* is returned, + otherwise, the usual :meth:`__repr__` call is made. For example: + + >>> class MyList(list): + ... @recursive_repr() + ... def __repr__(self): + ... return '<' + '|'.join(map(repr, self)) + '>' + ... + >>> m = MyList('abc') + >>> m.append(m) + >>> m.append('x') + >>> print(m) + <'a'|'b'|'c'|...|'x'> + + .. versionadded:: 3.2 + .. _repr-objects: Repr Objects ------------ -:class:`Repr` instances provide several members which can be used to provide +:class:`Repr` instances provide several attributes which can be used to provide size limits for the representations of different object types, and methods which format specific object types. diff --git a/Doc/library/resource.rst b/Doc/library/resource.rst index fbd7204f3a..c16b01301e 100644 --- a/Doc/library/resource.rst +++ b/Doc/library/resource.rst @@ -1,4 +1,3 @@ - :mod:`resource` --- Resource usage information ============================================== @@ -218,14 +217,14 @@ function to specify which processes information should be provided for. .. data:: RUSAGE_SELF - :const:`RUSAGE_SELF` should be used to request information pertaining only to - the process itself. + Pass to :func:`getrusage` to request resources consumed by the calling + process, which is the sum of resources used by all threads in the process. .. data:: RUSAGE_CHILDREN - Pass to :func:`getrusage` to request resource information for child processes of - the calling process. + Pass to :func:`getrusage` to request resources consumed by child processes + of the calling process which have been terminated and waited for. .. data:: RUSAGE_BOTH @@ -233,3 +232,10 @@ function to specify which processes information should be provided for. Pass to :func:`getrusage` to request resources consumed by both the current process and child processes. May not be available on all systems. + +.. data:: RUSAGE_THREAD + + Pass to :func:`getrusage` to request resources consumed by the current + thread. May not be available on all systems. + + .. versionadded:: 3.2 diff --git a/Doc/library/rlcompleter.rst b/Doc/library/rlcompleter.rst index 6b3befcfee..633088d897 100644 --- a/Doc/library/rlcompleter.rst +++ b/Doc/library/rlcompleter.rst @@ -1,4 +1,3 @@ - :mod:`rlcompleter` --- Completion function for GNU readline =========================================================== @@ -6,6 +5,9 @@ :synopsis: Python identifier completion, suitable for the GNU readline library. .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/rlcompleter.py` + +-------------- The :mod:`rlcompleter` module defines a completion function suitable for the :mod:`readline` module by completing valid Python identifiers and keywords. diff --git a/Doc/library/runpy.rst b/Doc/library/runpy.rst index 1b481678e4..4df622cb1f 100644 --- a/Doc/library/runpy.rst +++ b/Doc/library/runpy.rst @@ -5,71 +5,132 @@ :synopsis: Locate and run Python modules without importing them first. .. moduleauthor:: Nick Coghlan <ncoghlan@gmail.com> +**Source code:** :source:`Lib/runpy.py` -The :mod:`runpy` module is used to locate and run Python modules without -importing them first. Its main use is to implement the :option:`-m` command line -switch that allows scripts to be located using the Python module namespace -rather than the filesystem. - -When executed as a script, the module effectively operates as follows:: +-------------- - del sys.argv[0] # Remove the runpy module from the arguments - run_module(sys.argv[0], run_name="__main__", alter_sys=True) +The :mod:`runpy` module is used to locate and run Python modules without +importing them first. Its main use is to implement the :option:`-m` command +line switch that allows scripts to be located using the Python module +namespace rather than the filesystem. -The :mod:`runpy` module provides a single function: +The :mod:`runpy` module provides two functions: -.. function:: run_module(mod_name[, init_globals] [, run_name][, alter_sys]) +.. function:: run_module(mod_name, init_globals=None, run_name=None, alter_sys=False) - Execute the code of the specified module and return the resulting module globals - dictionary. The module's code is first located using the standard import - mechanism (refer to PEP 302 for details) and then executed in a fresh module - namespace. + Execute the code of the specified module and return the resulting module + globals dictionary. The module's code is first located using the standard + import mechanism (refer to :pep:`302` for details) and then executed in a + fresh module namespace. - If the supplied module name refers to a package rather than a normal module, - then that package is imported and the ``__main__`` submodule within that - package is then executed and the resulting module globals dictionary returned. + If the supplied module name refers to a package rather than a normal + module, then that package is imported and the ``__main__`` submodule within + that package is then executed and the resulting module globals dictionary + returned. - The optional dictionary argument *init_globals* may be used to pre-populate the - globals dictionary before the code is executed. The supplied dictionary will not - be modified. If any of the special global variables below are defined in the - supplied dictionary, those definitions are overridden by the ``run_module`` - function. + The optional dictionary argument *init_globals* may be used to pre-populate + the module's globals dictionary before the code is executed. The supplied + dictionary will not be modified. If any of the special global variables + below are defined in the supplied dictionary, those definitions are + overridden by :func:`run_module`. - The special global variables ``__name__``, ``__file__``, ``__loader__``, - ``__builtins__`` and ``__package__`` are set in the globals dictionary before - the module code is executed. + The special global variables ``__name__``, ``__file__``, ``__cached__``, + ``__loader__`` + and ``__package__`` are set in the globals dictionary before the module + code is executed (Note that this is a minimal set of variables - other + variables may be set implicitly as an interpreter implementation detail). - ``__name__`` is set to *run_name* if this optional argument is supplied, to - ``mod_name + '.__main__'`` if the named module is a package and to the - *mod_name* argument otherwise. + ``__name__`` is set to *run_name* if this optional argument is not + :const:`None`, to ``mod_name + '.__main__'`` if the named module is a + package and to the *mod_name* argument otherwise. - ``__loader__`` is set to the PEP 302 module loader used to retrieve the code for - the module (This loader may be a wrapper around the standard import mechanism). + ``__file__`` is set to the name provided by the module loader. If the + loader does not make filename information available, this variable is set + to :const:`None`. - ``__file__`` is set to the name provided by the module loader. If the loader - does not make filename information available, this variable is set to ``None``. + ``__cached__`` will be set to ``None``. - ``__builtins__`` is automatically initialised with a reference to the top level - namespace of the :mod:`builtins` module. + ``__loader__`` is set to the :pep:`302` module loader used to retrieve the + code for the module (This loader may be a wrapper around the standard + import mechanism). - ``__package__`` is set to *mod_name* if the named module is a package and to - ``mod_name.rpartition('.')[0]`` otherwise. + ``__package__`` is set to *mod_name* if the named module is a package and + to ``mod_name.rpartition('.')[0]`` otherwise. - If the argument *alter_sys* is supplied and evaluates to ``True``, then - ``sys.argv[0]`` is updated with the value of ``__file__`` and + If the argument *alter_sys* is supplied and evaluates to :const:`True`, + then ``sys.argv[0]`` is updated with the value of ``__file__`` and ``sys.modules[__name__]`` is updated with a temporary module object for the module being executed. Both ``sys.argv[0]`` and ``sys.modules[__name__]`` are restored to their original values before the function returns. - Note that this manipulation of :mod:`sys` is not thread-safe. Other threads may - see the partially initialised module, as well as the altered list of arguments. - It is recommended that the :mod:`sys` module be left alone when invoking this - function from threaded code. + Note that this manipulation of :mod:`sys` is not thread-safe. Other threads + may see the partially initialised module, as well as the altered list of + arguments. It is recommended that the :mod:`sys` module be left alone when + invoking this function from threaded code. + .. versionchanged:: 3.1 Added ability to execute packages by looking for a ``__main__`` submodule. + .. versionchanged:: 3.2 + Added ``__cached__`` global variable (see :PEP:`3147`). + + +.. function:: run_path(file_path, init_globals=None, run_name=None) + + Execute the code at the named filesystem location and return the resulting + module globals dictionary. As with a script name supplied to the CPython + command line, the supplied path may refer to a Python source file, a + compiled bytecode file or a valid sys.path entry containing a ``__main__`` + module (e.g. a zipfile containing a top-level ``__main__.py`` file). + + For a simple script, the specified code is simply executed in a fresh + module namespace. For a valid sys.path entry (typically a zipfile or + directory), the entry is first added to the beginning of ``sys.path``. The + function then looks for and executes a :mod:`__main__` module using the + updated path. Note that there is no special protection against invoking + an existing :mod:`__main__` entry located elsewhere on ``sys.path`` if + there is no such module at the specified location. + + The optional dictionary argument *init_globals* may be used to pre-populate + the module's globals dictionary before the code is executed. The supplied + dictionary will not be modified. If any of the special global variables + below are defined in the supplied dictionary, those definitions are + overridden by :func:`run_path`. + + The special global variables ``__name__``, ``__file__``, ``__loader__`` + and ``__package__`` are set in the globals dictionary before the module + code is executed (Note that this is a minimal set of variables - other + variables may be set implicitly as an interpreter implementation detail). + + ``__name__`` is set to *run_name* if this optional argument is not + :const:`None` and to ``'<run_path>'`` otherwise. + + ``__file__`` is set to the name provided by the module loader. If the + loader does not make filename information available, this variable is set + to :const:`None`. For a simple script, this will be set to ``file_path``. + + ``__loader__`` is set to the :pep:`302` module loader used to retrieve the + code for the module (This loader may be a wrapper around the standard + import mechanism). For a simple script, this will be set to :const:`None`. + + ``__package__`` is set to ``__name__.rpartition('.')[0]``. + + A number of alterations are also made to the :mod:`sys` module. Firstly, + ``sys.path`` may be altered as described above. ``sys.argv[0]`` is updated + with the value of ``file_path`` and ``sys.modules[__name__]`` is updated + with a temporary module object for the module being executed. All + modifications to items in :mod:`sys` are reverted before the function + returns. + + Note that, unlike :func:`run_module`, the alterations made to :mod:`sys` + are not optional in this function as these adjustments are essential to + allowing the execution of sys.path entries. As the thread-safety + limitations still apply, use of this function in threaded code should be + either serialised with the import lock or delegated to a separate process. + + .. versionadded:: 3.2 .. seealso:: @@ -79,3 +140,4 @@ The :mod:`runpy` module provides a single function: :pep:`366` - Main module explicit relative imports PEP written and implemented by Nick Coghlan. + :ref:`using-on-general` - CPython command line details diff --git a/Doc/library/sched.rst b/Doc/library/sched.rst index 0290ec41cd..000dba00c4 100644 --- a/Doc/library/sched.rst +++ b/Doc/library/sched.rst @@ -7,10 +7,13 @@ .. index:: single: event scheduling +**Source code:** :source:`Lib/sched.py` + +-------------- + The :mod:`sched` module defines a class which implements a general purpose event scheduler: - .. class:: scheduler(timefunc, delayfunc) The :class:`scheduler` class defines a generic interface to scheduling events. @@ -92,7 +95,7 @@ Scheduler Objects .. method:: scheduler.enter(delay, priority, action, argument) - Schedule an event for *delay* more time units. Other then the relative time, the + Schedule an event for *delay* more time units. Other than the relative time, the other arguments, the effect and the return value are the same as those for :meth:`enterabs`. diff --git a/Doc/library/select.rst b/Doc/library/select.rst index 70f73700b4..f1fd126d94 100644 --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -1,4 +1,3 @@ - :mod:`select` --- Waiting for I/O completion ============================================ @@ -6,9 +5,9 @@ :synopsis: Wait for I/O completion on multiple streams. -This module provides access to the :cfunc:`select` and :cfunc:`poll` functions -available in most operating systems, :cfunc:`epoll` available on Linux 2.5+ and -:cfunc:`kqueue` available on most BSD. +This module provides access to the :c:func:`select` and :c:func:`poll` functions +available in most operating systems, :c:func:`epoll` available on Linux 2.5+ and +:c:func:`kqueue` available on most BSD. Note that on Windows, it only works for sockets; on other operating systems, it also works for other file types (in particular, on Unix, it works on pipes). It cannot be used on regular files to determine whether a file has grown since @@ -20,11 +19,11 @@ The module defines the following: .. exception:: error The exception raised when an error occurs. The accompanying value is a pair - containing the numeric error code from :cdata:`errno` and the corresponding - string, as would be printed by the C function :cfunc:`perror`. + containing the numeric error code from :c:data:`errno` and the corresponding + string, as would be printed by the C function :c:func:`perror`. -.. function:: epoll([sizehint=-1]) +.. function:: epoll(sizehint=-1) (Only supported on Linux 2.5.44 and newer.) Returns an edge polling object, which can be used as Edge or Level Triggered interface for I/O events; see @@ -54,7 +53,7 @@ The module defines the following: .. function:: select(rlist, wlist, xlist[, timeout]) - This is a straightforward interface to the Unix :cfunc:`select` system call. + This is a straightforward interface to the Unix :c:func:`select` system call. The first three arguments are sequences of 'waitable objects': either integers representing file descriptors or objects with a parameterless method named :meth:`fileno` returning such an integer: @@ -91,10 +90,21 @@ The module defines the following: .. index:: single: WinSock File objects on Windows are not acceptable, but sockets are. On Windows, - the underlying :cfunc:`select` function is provided by the WinSock + the underlying :c:func:`select` function is provided by the WinSock library, and does not handle file descriptors that don't originate from WinSock. +.. attribute:: PIPE_BUF + + The minimum number of bytes which can be written without blocking to a pipe + when the pipe has been reported as ready for writing by :func:`select`, + :func:`poll` or another interface in this module. This doesn't apply + to other kind of file-like objects such as sockets. + + This value is guaranteed by POSIX to be at least 512. Availability: Unix. + + .. versionadded:: 3.2 + .. _epoll-objects: @@ -124,15 +134,15 @@ Edge and Level Trigger Polling (epoll) Objects | :const:`EPOLLONESHOT` | Set one-shot behavior. After one event is | | | pulled out, the fd is internally disabled | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLRDNORM` | ??? | + | :const:`EPOLLRDNORM` | Equivalent to :const:`EPOLLIN` | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLRDBAND` | ??? | + | :const:`EPOLLRDBAND` | Priority data band can be read. | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLWRNORM` | ??? | + | :const:`EPOLLWRNORM` | Equivalent to :const:`EPOLLOUT` | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLWRBAND` | ??? | + | :const:`EPOLLWRBAND` | Priority data may be written. | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLMSG` | ??? | + | :const:`EPOLLMSG` | Ignored. | +-----------------------+-----------------------------------------------+ @@ -181,13 +191,13 @@ Edge and Level Trigger Polling (epoll) Objects Polling Objects --------------- -The :cfunc:`poll` system call, supported on most Unix systems, provides better +The :c:func:`poll` system call, supported on most Unix systems, provides better scalability for network servers that service many, many clients at the same -time. :cfunc:`poll` scales better because the system call only requires listing -the file descriptors of interest, while :cfunc:`select` builds a bitmap, turns +time. :c:func:`poll` scales better because the system call only requires listing +the file descriptors of interest, while :c:func:`select` builds a bitmap, turns on bits for the fds of interest, and then afterward the whole bitmap has to be -linearly scanned again. :cfunc:`select` is O(highest file descriptor), while -:cfunc:`poll` is O(number of file descriptors). +linearly scanned again. :c:func:`select` is O(highest file descriptor), while +:c:func:`poll` is O(number of file descriptors). .. method:: poll.register(fd[, eventmask]) @@ -226,7 +236,7 @@ linearly scanned again. :cfunc:`select` is O(highest file descriptor), while .. method:: poll.modify(fd, eventmask) Modifies an already registered fd. This has the same effect as - :meth:`register(fd, eventmask)`. Attempting to modify a file descriptor + ``register(fd, eventmask)``. Attempting to modify a file descriptor that was never registered causes an :exc:`IOError` exception with errno :const:`ENOENT` to be raised. diff --git a/Doc/library/shelve.rst b/Doc/library/shelve.rst index b2818141ec..9d7d5045e2 100644 --- a/Doc/library/shelve.rst +++ b/Doc/library/shelve.rst @@ -7,6 +7,10 @@ .. index:: module: pickle +**Source code:** :source:`Lib/shelve.py` + +-------------- + A "shelf" is a persistent, dictionary-like object. The difference with "dbm" databases is that the values (not the keys!) in a shelf can be essentially arbitrary Python objects --- anything that the :mod:`pickle` module can handle. @@ -14,7 +18,7 @@ This includes most class instances, recursive data types, and objects containing lots of shared sub-objects. The keys are ordinary strings. -.. function:: open(filename[, flag='c'[, protocol=None[, writeback=False]]]) +.. function:: open(filename, flag='c', protocol=None, writeback=False) Open a persistent dictionary. The filename specified is the base filename for the underlying database. As a side-effect, an extension may be added to the @@ -97,7 +101,7 @@ Restrictions implementation used. -.. class:: Shelf(dict[, protocol=None[, writeback=False]]) +.. class:: Shelf(dict, protocol=None, writeback=False, keyencoding='utf-8') A subclass of :class:`collections.MutableMapping` which stores pickled values in the *dict* object. @@ -111,8 +115,15 @@ Restrictions This allows natural operations on mutable entries, but can consume much more memory and make sync and close take a long time. + The *keyencoding* parameter is the encoding used to encode keys before they + are used with the underlying dict. + + .. versionadded:: 3.2 + The *keyencoding* parameter; previously, keys were always encoded in + UTF-8. -.. class:: BsdDbShelf(dict[, protocol=None[, writeback=False]]) + +.. class:: BsdDbShelf(dict, protocol=None, writeback=False, keyencoding='utf-8') A subclass of :class:`Shelf` which exposes :meth:`first`, :meth:`!next`, :meth:`previous`, :meth:`last` and :meth:`set_location` which are available @@ -121,11 +132,11 @@ Restrictions modules. The *dict* object passed to the constructor must support those methods. This is generally accomplished by calling one of :func:`bsddb.hashopen`, :func:`bsddb.btopen` or :func:`bsddb.rnopen`. The - optional *protocol* and *writeback* parameters have the same interpretation - as for the :class:`Shelf` class. + optional *protocol*, *writeback*, and *keyencoding* parameters have the same + interpretation as for the :class:`Shelf` class. -.. class:: DbfilenameShelf(filename[, flag='c'[, protocol=None[, writeback=False]]]) +.. class:: DbfilenameShelf(filename, flag='c', protocol=None, writeback=False) A subclass of :class:`Shelf` which accepts a *filename* instead of a dict-like object. The underlying file will be opened using :func:`dbm.open`. By diff --git a/Doc/library/shlex.rst b/Doc/library/shlex.rst index ee241f4a52..0113fb7db3 100644 --- a/Doc/library/shlex.rst +++ b/Doc/library/shlex.rst @@ -1,4 +1,3 @@ - :mod:`shlex` --- Simple lexical analysis ======================================== @@ -9,6 +8,9 @@ .. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com> .. sectionauthor:: Gustavo Niemeyer <niemeyer@conectiva.com> +**Source code:** :source:`Lib/shlex.py` + +-------------- The :class:`shlex` class makes it easy to write lexical analyzers for simple syntaxes resembling that of the Unix shell. This will often be useful for @@ -18,30 +20,31 @@ applications) or for parsing quoted strings. The :mod:`shlex` module defines the following functions: -.. function:: split(s[, comments[, posix]]) +.. function:: split(s, comments=False, posix=True) Split the string *s* using shell-like syntax. If *comments* is :const:`False` (the default), the parsing of comments in the given string will be disabled - (setting the :attr:`commenters` member of the :class:`shlex` instance to the - empty string). This function operates in POSIX mode by default, but uses + (setting the :attr:`commenters` attribute of the :class:`shlex` instance to + the empty string). This function operates in POSIX mode by default, but uses non-POSIX mode if the *posix* argument is false. .. note:: - Since the :func:`split` function instantiates a :class:`shlex` instance, passing - ``None`` for *s* will read the string to split from standard input. + Since the :func:`split` function instantiates a :class:`shlex` instance, + passing ``None`` for *s* will read the string to split from standard + input. The :mod:`shlex` module defines the following class: -.. class:: shlex([instream[, infile[, posix]]]) +.. class:: shlex(instream=None, infile=None, posix=False) A :class:`shlex` instance or subclass instance is a lexical analyzer object. The initialization argument, if present, specifies where to read characters from. It must be a file-/stream-like object with :meth:`read` and :meth:`readline` methods, or a string. If no argument is given, input will be taken from ``sys.stdin``. The second optional argument is a filename - string, which sets the initial value of the :attr:`infile` member. If the + string, which sets the initial value of the :attr:`infile` attribute. If the *instream* argument is omitted or equal to ``sys.stdin``, this second argument defaults to "stdin". The *posix* argument defines the operational mode: when *posix* is not true (default), the :class:`shlex` instance will @@ -111,7 +114,7 @@ A :class:`shlex` instance has the following methods: :meth:`pop_source` methods. -.. method:: shlex.push_source(stream[, filename]) +.. method:: shlex.push_source(newstream, newfile=None) Push an input source stream onto the input stack. If the filename argument is specified it will later be available for use in error messages. This is the @@ -124,7 +127,7 @@ A :class:`shlex` instance has the following methods: used internally when the lexer reaches EOF on a stacked input stream. -.. method:: shlex.error_leader([file[, line]]) +.. method:: shlex.error_leader(infile=None, lineno=None) This method generates an error message leader in the format of a Unix C compiler error label; the format is ``'"%s", line %d: '``, where the ``%s`` is replaced @@ -199,8 +202,8 @@ either control lexical analysis or can be used for debugging: .. attribute:: shlex.source - This member is ``None`` by default. If you assign a string to it, that string - will be recognized as a lexical-level inclusion request similar to the + This attribute is ``None`` by default. If you assign a string to it, that + string will be recognized as a lexical-level inclusion request similar to the ``source`` keyword in various shells. That is, the immediately following token will opened as a filename and input taken from that stream until EOF, at which point the :meth:`close` method of that stream will be called and the input @@ -210,7 +213,7 @@ either control lexical analysis or can be used for debugging: .. attribute:: shlex.debug - If this member is numeric and ``1`` or more, a :class:`shlex` instance will + If this attribute is numeric and ``1`` or more, a :class:`shlex` instance will print verbose progress output on its behavior. If you need to use this, you can read the module source code to learn the details. diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 7cf8550d47..18f6485184 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -1,4 +1,3 @@ - :mod:`shutil` --- High-level file operations ============================================ @@ -11,6 +10,10 @@ single: file; copying single: copying files +**Source code:** :source:`Lib/shutil.py` + +-------------- + The :mod:`shutil` module offers a number of high-level operations on files and collections of files. In particular, functions are provided which support file copying and removal. For operations on individual files, see also the @@ -18,8 +21,8 @@ copying and removal. For operations on individual files, see also the .. warning:: - Even the higher-level file copying functions (:func:`copy`, :func:`copy2`) - can't copy all file metadata. + Even the higher-level file copying functions (:func:`shutil.copy`, + :func:`shutil.copy2`) cannot copy all file metadata. On POSIX platforms, this means that file owner and group are lost as well as ACLs. On Mac OS, the resource fork and other metadata are not used. @@ -28,6 +31,11 @@ copying and removal. For operations on individual files, see also the are not copied. +.. _file-operations: + +Directory and files operations +------------------------------ + .. function:: copyfileobj(fsrc, fdst[, length]) Copy the contents of the file-like object *fsrc* to the file-like object *fdst*. @@ -41,10 +49,10 @@ copying and removal. For operations on individual files, see also the .. function:: copyfile(src, dst) - Copy the contents (no metadata) of the file named *src* to a file named *dst*. - *dst* must be the complete target file name; look at :func:`copy` for a copy that - accepts a target directory path. If *src* and *dst* are the same files, - :exc:`Error` is raised. + Copy the contents (no metadata) of the file named *src* to a file named + *dst*. *dst* must be the complete target file name; look at + :func:`shutil.copy` for a copy that accepts a target directory path. If + *src* and *dst* are the same files, :exc:`Error` is raised. The destination location must be writable; otherwise, an :exc:`IOError` exception will be raised. If *dst* already exists, it will be replaced. Special files such as character or block devices and pipes cannot be copied with this @@ -74,9 +82,9 @@ copying and removal. For operations on individual files, see also the .. function:: copy2(src, dst) - Similar to :func:`copy`, but metadata is copied as well -- in fact, this is just - :func:`copy` followed by :func:`copystat`. This is similar to the - Unix command :program:`cp -p`. + Similar to :func:`shutil.copy`, but metadata is copied as well -- in fact, + this is just :func:`shutil.copy` followed by :func:`copystat`. This is + similar to the Unix command :program:`cp -p`. .. function:: ignore_patterns(\*patterns) @@ -86,17 +94,25 @@ copying and removal. For operations on individual files, see also the match one of the glob-style *patterns* provided. See the example below. -.. function:: copytree(src, dst[, symlinks=False[, ignore=None]]) +.. function:: copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, ignore_dangling_symlinks=False) Recursively copy an entire directory tree rooted at *src*. The destination - directory, named by *dst*, must not already exist; it will be created as well - as missing parent directories. Permissions and times of directories are - copied with :func:`copystat`, individual files are copied using - :func:`copy2`. + directory, named by *dst*, must not already exist; it will be created as + well as missing parent directories. Permissions and times of directories + are copied with :func:`copystat`, individual files are copied using + :func:`shutil.copy2`. If *symlinks* is true, symbolic links in the source tree are represented as - symbolic links in the new tree; if false or omitted, the contents of the - linked files are copied to the new tree. + symbolic links in the new tree, but the metadata of the original links is NOT + copied; if false or omitted, the contents and metadata of the linked files + are copied to the new tree. + + When *symlinks* is false, if the file pointed by the symlink doesn't + exist, a exception will be added in the list of errors raised in + a :exc:`Error` exception at the end of the copy process. + You can set the optional *ignore_dangling_symlinks* flag to true if you + want to silence this exception. Notice that this option has no effect + on platforms that don't support :func:`os.symlink`. If *ignore* is given, it must be a callable that will receive as its arguments the directory being visited by :func:`copytree`, and a list of its @@ -110,11 +126,21 @@ copying and removal. For operations on individual files, see also the If exception(s) occur, an :exc:`Error` is raised with a list of reasons. - The source code for this should be considered an example rather than the - ultimate tool. + If *copy_function* is given, it must be a callable that will be used to copy + each file. It will be called with the source path and the destination path + as arguments. By default, :func:`shutil.copy2` is used, but any function + that supports the same signature (like :func:`copy`) can be used. + + .. versionchanged:: 3.2 + Added the *copy_function* argument to be able to provide a custom copy + function. + + .. versionchanged:: 3.2 + Added the *ignore_dangling_symlinks* argument to silent dangling symlinks + errors when *symlinks* is false. -.. function:: rmtree(path[, ignore_errors[, onerror]]) +.. function:: rmtree(path, ignore_errors=False, onerror=None) .. index:: single: directory; deleting @@ -136,23 +162,31 @@ copying and removal. For operations on individual files, see also the .. function:: move(src, dst) - Recursively move a file or directory to another location. + Recursively move a file or directory (*src*) to another location (*dst*). + + If the destination is a directory or a symlink to a directory, then *src* is + moved inside that directory. + + The destination directory must not already exist. If the destination already + exists but is not a directory, it may be overwritten depending on + :func:`os.rename` semantics. - If the destination is on the current filesystem, then simply use rename. - Otherwise, copy src (with :func:`copy2`) to the dst and then remove src. + If the destination is on the current filesystem, then :func:`os.rename` is + used. Otherwise, *src* is copied (using :func:`shutil.copy2`) to *dst* and + then removed. .. exception:: Error - This exception collects exceptions that raised during a multi-file operation. For - :func:`copytree`, the exception argument is a list of 3-tuples (*srcname*, - *dstname*, *exception*). + This exception collects exceptions that are raised during a multi-file + operation. For :func:`copytree`, the exception argument is a list of 3-tuples + (*srcname*, *dstname*, *exception*). -.. _shutil-example: +.. _shutil-copytree-example: -Example -------- +copytree example +:::::::::::::::: This example is the implementation of the :func:`copytree` function, described above, with the docstring omitted. It demonstrates many of the other functions @@ -190,3 +224,185 @@ provided by this module. :: if errors: raise Error(errors) +Another example that uses the :func:`ignore_patterns` helper:: + + from shutil import copytree, ignore_patterns + + copytree(source, destination, ignore=ignore_patterns('*.pyc', 'tmp*')) + +This will copy everything except ``.pyc`` files and files or directories whose +name starts with ``tmp``. + +Another example that uses the *ignore* argument to add a logging call:: + + from shutil import copytree + import logging + + def _logpath(path, names): + logging.info('Working in %s' % path) + return [] # nothing will be ignored + + copytree(source, destination, ignore=_logpath) + + +.. _archiving-operations: + +Archiving operations +-------------------- + +High-level utilities to create and read compressed and archived files are also +provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. + +.. function:: make_archive(base_name, format, [root_dir, [base_dir, [verbose, [dry_run, [owner, [group, [logger]]]]]]]) + + Create an archive file (such as zip or tar) and return its name. + + *base_name* is the name of the file to create, including the path, minus + any format-specific extension. *format* is the archive format: one of + "zip", "tar", "bztar" (if the :mod:`bz2` module is available) or "gztar". + + *root_dir* is a directory that will be the root directory of the + archive; for example, we typically chdir into *root_dir* before creating the + archive. + + *base_dir* is the directory where we start archiving from; + i.e. *base_dir* will be the common prefix of all files and + directories in the archive. + + *root_dir* and *base_dir* both default to the current directory. + + *owner* and *group* are used when creating a tar archive. By default, + uses the current owner and group. + + *logger* must be an object compatible with :pep:`282`, usually an instance of + :class:`logging.Logger`. + + .. versionadded:: 3.2 + + +.. function:: get_archive_formats() + + Return a list of supported formats for archiving. + Each element of the returned sequence is a tuple ``(name, description)`` + + By default :mod:`shutil` provides these formats: + + - *gztar*: gzip'ed tar-file + - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *tar*: uncompressed tar file + - *zip*: ZIP file + + You can register new formats or provide your own archiver for any existing + formats, by using :func:`register_archive_format`. + + .. versionadded:: 3.2 + + +.. function:: register_archive_format(name, function, [extra_args, [description]]) + + Register an archiver for the format *name*. *function* is a callable that + will be used to invoke the archiver. + + If given, *extra_args* is a sequence of ``(name, value)`` pairs that will be + used as extra keywords arguments when the archiver callable is used. + + *description* is used by :func:`get_archive_formats` which returns the + list of archivers. Defaults to an empty list. + + .. versionadded:: 3.2 + + +.. function:: unregister_archive_format(name) + + Remove the archive format *name* from the list of supported formats. + + .. versionadded:: 3.2 + + +.. function:: unpack_archive(filename[, extract_dir[, format]]) + + Unpack an archive. *filename* is the full path of the archive. + + *extract_dir* is the name of the target directory where the archive is + unpacked. If not provided, the current working directory is used. + + *format* is the archive format: one of "zip", "tar", or "gztar". Or any + other format registered with :func:`register_unpack_format`. If not + provided, :func:`unpack_archive` will use the archive file name extension + and see if an unpacker was registered for that extension. In case none is + found, a :exc:`ValueError` is raised. + + .. versionadded:: 3.2 + + +.. function:: register_unpack_format(name, extensions, function[, extra_args[, description]]) + + Registers an unpack format. *name* is the name of the format and + *extensions* is a list of extensions corresponding to the format, like + ``.zip`` for Zip files. + + *function* is the callable that will be used to unpack archives. The + callable will receive the path of the archive, followed by the directory + the archive must be extracted to. + + When provided, *extra_args* is a sequence of ``(name, value)`` tuples that + will be passed as keywords arguments to the callable. + + *description* can be provided to describe the format, and will be returned + by the :func:`get_unpack_formats` function. + + .. versionadded:: 3.2 + + +.. function:: unregister_unpack_format(name) + + Unregister an unpack format. *name* is the name of the format. + + .. versionadded:: 3.2 + + +.. function:: get_unpack_formats() + + Return a list of all registered formats for unpacking. + Each element of the returned sequence is a tuple + ``(name, extensions, description)``. + + By default :mod:`shutil` provides these formats: + + - *gztar*: gzip'ed tar-file + - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *tar*: uncompressed tar file + - *zip*: ZIP file + + You can register new formats or provide your own unpacker for any existing + formats, by using :func:`register_unpack_format`. + + .. versionadded:: 3.2 + + +.. _shutil-archiving-example: + +Archiving example +::::::::::::::::: + +In this example, we create a gzip'ed tar-file archive containing all files +found in the :file:`.ssh` directory of the user:: + + >>> from shutil import make_archive + >>> import os + >>> archive_name = os.path.expanduser(os.path.join('~', 'myarchive')) + >>> root_dir = os.path.expanduser(os.path.join('~', '.ssh')) + >>> make_archive(archive_name, 'gztar', root_dir) + '/Users/tarek/myarchive.tar.gz' + +The resulting archive contains:: + + $ tar -tzvf /Users/tarek/myarchive.tar.gz + drwx------ tarek/staff 0 2010-02-01 16:23:40 ./ + -rw-r--r-- tarek/staff 609 2008-06-09 13:26:54 ./authorized_keys + -rwxr-xr-x tarek/staff 65 2008-06-09 13:26:54 ./config + -rwx------ tarek/staff 668 2008-06-09 13:26:54 ./id_dsa + -rwxr-xr-x tarek/staff 609 2008-06-09 13:26:54 ./id_dsa.pub + -rw------- tarek/staff 1675 2008-06-09 13:26:54 ./id_rsa + -rw-r--r-- tarek/staff 397 2008-06-09 13:26:54 ./id_rsa.pub + -rw-r--r-- tarek/staff 37192 2010-02-06 18:23:10 ./known_hosts diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 309f71b3ce..698b1e74f4 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -1,4 +1,3 @@ - :mod:`signal` --- Set handlers for asynchronous events ====================================================== @@ -69,12 +68,32 @@ The variables defined in the :mod:`signal` module are: All the signal numbers are defined symbolically. For example, the hangup signal is defined as :const:`signal.SIGHUP`; the variable names are identical to the names used in C programs, as found in ``<signal.h>``. The Unix man page for - ':cfunc:`signal`' lists the existing signals (on some systems this is + ':c:func:`signal`' lists the existing signals (on some systems this is :manpage:`signal(2)`, on others the list is in :manpage:`signal(7)`). Note that not all systems define the same set of signal names; only those names defined by the system are defined by this module. +.. data:: CTRL_C_EVENT + + The signal corresponding to the CTRL+C keystroke event. This signal can + only be used with :func:`os.kill`. + + Availability: Windows. + + .. versionadded:: 3.2 + + +.. data:: CTRL_BREAK_EVENT + + The signal corresponding to the CTRL+BREAK keystroke event. This signal can + only be used with :func:`os.kill`. + + Availability: Windows. + + .. versionadded:: 3.2 + + .. data:: NSIG One more than the number of the highest signal number. @@ -82,7 +101,8 @@ The variables defined in the :mod:`signal` module are: .. data:: ITIMER_REAL - Decrements interval timer in real time, and delivers :const:`SIGALRM` upon expiration. + Decrements interval timer in real time, and delivers :const:`SIGALRM` upon + expiration. .. data:: ITIMER_VIRTUAL @@ -190,7 +210,7 @@ The :mod:`signal` module defines the following functions: Note that installing a signal handler with :func:`signal` will reset the restart behaviour to interruptible by implicitly calling - :cfunc:`siginterrupt` with a true *flag* value for the given signal. + :c:func:`siginterrupt` with a true *flag* value for the given signal. .. function:: signal(signalnum, handler) diff --git a/Doc/library/site.rst b/Doc/library/site.rst index bf65d177ed..db96adde82 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -1,17 +1,22 @@ - :mod:`site` --- Site-specific configuration hook ================================================ .. module:: site - :synopsis: A standard way to reference site-specific modules. + :synopsis: Module responsible for site-specific configuration. + +**Source code:** :source:`Lib/site.py` +-------------- + +.. highlightlang:: none **This module is automatically imported during initialization.** The automatic import can be suppressed using the interpreter's :option:`-S` option. .. index:: triple: module; search; path -Importing this module will append site-specific paths to the module search path. +Importing this module will append site-specific paths to the module search path +and add a few builtins. .. index:: pair: site-python; directory @@ -26,11 +31,11 @@ Unix and Macintosh). For each of the distinct head-tail combinations, it sees if it refers to an existing directory, and if so, adds it to ``sys.path`` and also inspects the newly added path for configuration files. -A path configuration file is a file whose name has the form :file:`package.pth` +A path configuration file is a file whose name has the form :file:`{name}.pth` and exists in one of the four directories mentioned above; its contents are additional items (one per line) to be added to ``sys.path``. Non-existing items -are never added to ``sys.path``, but no check is made that the item refers to a -directory (rather than a file). No item is added to ``sys.path`` more than +are never added to ``sys.path``, and no check is made that the item refers to a +directory rather than a file. No item is added to ``sys.path`` more than once. Blank lines and lines beginning with ``#`` are skipped. Lines starting with ``import`` (followed by space or tab) are executed. @@ -40,8 +45,7 @@ with ``import`` (followed by space or tab) are executed. For example, suppose ``sys.prefix`` and ``sys.exec_prefix`` are set to :file:`/usr/local`. The Python X.Y library is then installed in -:file:`/usr/local/lib/python{X.Y}` (where only the first three characters of -``sys.version`` are used to form the installation path name). Suppose this has +:file:`/usr/local/lib/python{X.Y}`. Suppose this has a subdirectory :file:`/usr/local/lib/python{X.Y}/site-packages` with three subsubdirectories, :file:`foo`, :file:`bar` and :file:`spam`, and two path configuration files, :file:`foo.pth` and :file:`bar.pth`. Assume @@ -74,48 +78,124 @@ not mentioned in either path configuration file. After these path manipulations, an attempt is made to import a module named :mod:`sitecustomize`, which can perform arbitrary site-specific customizations. -If this import fails with an :exc:`ImportError` exception, it is silently -ignored. +It is typically created by a system administrator in the site-packages +directory. If this import fails with an :exc:`ImportError` exception, it is +silently ignored. -.. index:: module: sitecustomize +.. index:: module: usercustomize + +After this, an attempt is made to import a module named :mod:`usercustomize`, +which can perform arbitrary user-specific customizations, if +:data:`ENABLE_USER_SITE` is true. This file is intended to be created in the +user site-packages directory (see below), which is part of ``sys.path`` unless +disabled by :option:`-s`. An :exc:`ImportError` will be silently ignored. Note that for some non-Unix systems, ``sys.prefix`` and ``sys.exec_prefix`` are empty, and the path manipulations are skipped; however the import of -:mod:`sitecustomize` is still attempted. +:mod:`sitecustomize` and :mod:`usercustomize` is still attempted. .. data:: PREFIXES - A list of prefixes for site package directories + A list of prefixes for site-packages directories. .. data:: ENABLE_USER_SITE - Flag showing the status of the user site directory. True means the - user site directory is enabled and added to sys.path. When the flag - is None the user site directory is disabled for security reasons. + Flag showing the status of the user site-packages directory. ``True`` means + that it is enabled and was added to ``sys.path``. ``False`` means that it + was disabled by user request (with :option:`-s` or + :envvar:`PYTHONNOUSERSITE`). ``None`` means it was disabled for security + reasons (mismatch between user or group id and effective id) or by an + administrator. .. data:: USER_SITE - Path to the user site directory for the current Python version or None + Path to the user site-packages for the running Python. Can be ``None`` if + :func:`getusersitepackages` hasn't been called yet. Default value is + :file:`~/.local/lib/python{X.Y}/site-packages` for UNIX and non-framework Mac + OS X builds, :file:`~/Library/Python/{X.Y}/lib/python/site-packages` for Mac + framework builds, and :file:`{%APPDATA%}\\Python\\Python{XY}\\site-packages` + on Windows. This directory is a site directory, which means that + :file:`.pth` files in it will be processed. .. data:: USER_BASE - Path to the base directory for user site directories + Path to the base directory for the user site-packages. Can be ``None`` if + :func:`getuserbase` hasn't been called yet. Default value is + :file:`~/.local` for UNIX and Mac OS X non-framework builds, + :file:`~/Library/Python/{X.Y}` for Mac framework builds, and + :file:`{%APPDATA%}\\Python` for Windows. This value is used by Distutils to + compute the installation directories for scripts, data files, Python modules, + etc. for the :ref:`user installation scheme <inst-alt-install-user>`. See + also :envvar:`PYTHONUSERBASE`. -.. envvar:: PYTHONNOUSERSITE +.. function:: addsitedir(sitedir, known_paths=None) + Add a directory to sys.path and process its :file:`.pth` files. Typically + used in :mod:`sitecustomize` or :mod:`usercustomize` (see above). -.. envvar:: PYTHONUSERBASE +.. function:: getsitepackages() -.. function:: addsitedir(sitedir, known_paths=None) + Return a list containing all global site-packages directories (and possibly + site-python). + + .. versionadded:: 3.2 + + +.. function:: getuserbase() + + Return the path of the user base directory, :data:`USER_BASE`. If it is not + initialized yet, this function will also set it, respecting + :envvar:`PYTHONUSERBASE`. + + .. versionadded:: 3.2 + + +.. function:: getusersitepackages() + + Return the path of the user-specific site-packages directory, + :data:`USER_SITE`. If it is not initialized yet, this function will also set + it, respecting :envvar:`PYTHONNOUSERSITE` and :data:`USER_BASE`. + + .. versionadded:: 3.2 + + +The :mod:`site` module also provides a way to get the user directories from the +command line: + +.. code-block:: sh + + $ python3 -m site --user-site + /home/user/.local/lib/python3.3/site-packages + +.. program:: site + +If it is called without arguments, it will print the contents of +:data:`sys.path` on the standard output, followed by the value of +:data:`USER_BASE` and whether the directory exists, then the same thing for +:data:`USER_SITE`, and finally the value of :data:`ENABLE_USER_SITE`. + +.. cmdoption:: --user-base + + Print the path to the user base directory. + +.. cmdoption:: --user-site + + Print the path to the user site-packages directory. + +If both options are given, user base and user site will be printed (always in +this order), separated by :data:`os.pathsep`. - Adds a directory to sys.path and processes its pth files. +If any option is given, the script will exit with one of these values: ``O`` if +the user site-packages directory is enabled, ``1`` if it was disabled by the +user, ``2`` if it is disabled for security reasons or by an administrator, and a +value greater than 2 if there is an error. +.. seealso:: -.. XXX Update documentation -.. XXX document python -m site --user-base --user-site + :pep:`370` -- Per user site-packages directory diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst index 276751634d..c391f710df 100644 --- a/Doc/library/smtpd.rst +++ b/Doc/library/smtpd.rst @@ -7,13 +7,18 @@ .. moduleauthor:: Barry Warsaw <barry@zope.com> .. sectionauthor:: Moshe Zadka <moshez@moshez.org> +**Source code:** :source:`Lib/smtpd.py` +-------------- +This module offers several classes to implement SMTP (email) servers. -This module offers several classes to implement SMTP servers. One is a generic +Several server implementations are present; one is a generic do-nothing implementation, which can be overridden, while the other two offer specific mail-sending strategies. +Additionally the SMTPChannel may be extended to implement very specific +interaction behaviour with SMTP clients. SMTPServer Objects ------------------ @@ -26,7 +31,6 @@ SMTPServer Objects inherits from :class:`asyncore.dispatcher`, and so will insert itself into :mod:`asyncore`'s event loop on instantiation. - .. method:: process_message(peer, mailfrom, rcpttos, data) Raise :exc:`NotImplementedError` exception. Override this in subclasses to @@ -37,6 +41,11 @@ SMTPServer Objects containing the contents of the e-mail (which should be in :rfc:`2822` format). + .. attribute:: channel_class + + Override this in subclasses to use a custom :class:`SMTPChannel` for + managing SMTP clients. + DebuggingServer Objects ----------------------- @@ -71,3 +80,91 @@ MailmanProxy Objects running this has a good chance to make you into an open relay, so please be careful. +SMTPChannel Objects +------------------- + +.. class:: SMTPChannel(server, conn, addr) + + Create a new :class:`SMTPChannel` object which manages the communication + between the server and a single SMTP client. + + To use a custom SMTPChannel implementation you need to override the + :attr:`SMTPServer.channel_class` of your :class:`SMTPServer`. + + The :class:`SMTPChannel` has the following instance variables: + + .. attribute:: smtp_server + + Holds the :class:`SMTPServer` that spawned this channel. + + .. attribute:: conn + + Holds the socket object connecting to the client. + + .. attribute:: addr + + Holds the address of the client, the second value returned by + socket.accept() + + .. attribute:: received_lines + + Holds a list of the line strings (decoded using UTF-8) received from + the client. The lines have their "\r\n" line ending translated to "\n". + + .. attribute:: smtp_state + + Holds the current state of the channel. This will be either + :attr:`COMMAND` initially and then :attr:`DATA` after the client sends + a "DATA" line. + + .. attribute:: seen_greeting + + Holds a string containing the greeting sent by the client in its "HELO". + + .. attribute:: mailfrom + + Holds a string containing the address identified in the "MAIL FROM:" line + from the client. + + .. attribute:: rcpttos + + Holds a list of strings containing the addresses identified in the + "RCPT TO:" lines from the client. + + .. attribute:: received_data + + Holds a string containing all of the data sent by the client during the + DATA state, up to but not including the terminating "\r\n.\r\n". + + .. attribute:: fqdn + + Holds the fully-qualified domain name of the server as returned by + ``socket.getfqdn()``. + + .. attribute:: peer + + Holds the name of the client peer as returned by ``conn.getpeername()`` + where ``conn`` is :attr:`conn`. + + The :class:`SMTPChannel` operates by invoking methods named ``smtp_<command>`` + upon reception of a command line from the client. Built into the base + :class:`SMTPChannel` class are methods for handling the following commands + (and responding to them appropriately): + + ======== =================================================================== + Command Action taken + ======== =================================================================== + HELO Accepts the greeting from the client and stores it in + :attr:`seen_greeting`. + NOOP Takes no action. + QUIT Closes the connection cleanly. + MAIL Accepts the "MAIL FROM:" syntax and stores the supplied address as + :attr:`mailfrom`. + RCPT Accepts the "RCPT TO:" syntax and stores the supplied addresses in + the :attr:`rcpttos` list. + RSET Resets the :attr:`mailfrom`, :attr:`rcpttos`, and + :attr:`received_data`, but not the greeting. + DATA Sets the internal state to :attr:`DATA` and stores remaining lines + from the client in :attr:`received_data` until the terminator + "\r\n.\r\n" is received. + ======== =================================================================== diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index 9303c3edb5..3101ab7284 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -1,4 +1,3 @@ - :mod:`smtplib` --- SMTP protocol client ======================================= @@ -11,13 +10,17 @@ pair: SMTP; protocol single: Simple Mail Transfer Protocol +**Source code:** :source:`Lib/smtplib.py` + +-------------- + The :mod:`smtplib` module defines an SMTP client session object that can be used to send mail to any Internet machine with an SMTP or ESMTP listener daemon. For details of SMTP and ESMTP operation, consult :rfc:`821` (Simple Mail Transfer Protocol) and :rfc:`1869` (SMTP Service Extensions). -.. class:: SMTP([host[, port[, local_hostname[, timeout]]]]) +.. class:: SMTP(host='', port=0, local_hostname=None[, timeout]) A :class:`SMTP` instance encapsulates an SMTP connection. It has methods that support a full repertoire of SMTP and ESMTP operations. If the optional @@ -32,13 +35,13 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). :meth:`sendmail`, and :meth:`quit` methods. An example is included below. -.. class:: SMTP_SSL([host[, port[, local_hostname[, keyfile[, certfile[, timeout]]]]]]) +.. class:: SMTP_SSL(host='', port=0, local_hostname=None, keyfile=None, certfile=None[, timeout]) A :class:`SMTP_SSL` instance behaves exactly the same as instances of :class:`SMTP`. :class:`SMTP_SSL` should be used for situations where SSL is required from the beginning of the connection and using :meth:`starttls` is not appropriate. If *host* is not specified, the local host is used. If - *port* is omitted, the standard SMTP-over-SSL port (465) is used. *keyfile* + *port* is zero, the standard SMTP-over-SSL port (465) is used. *keyfile* and *certfile* are also optional, and can contain a PEM formatted private key and certificate chain file for the SSL connection. The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the @@ -46,7 +49,7 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). will be used). -.. class:: LMTP([host[, port[, local_hostname]]]) +.. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None) The LMTP protocol, which is very similar to ESMTP, is heavily based on the standard SMTP client. It's common to use Unix sockets for LMTP, so our :meth:`connect` @@ -142,7 +145,7 @@ An :class:`SMTP` instance has the following methods: for connection and for all messages sent to and received from the server. -.. method:: SMTP.connect([host[, port]]) +.. method:: SMTP.connect(host='localhost', port=0) Connect to a host on a given port. The defaults are to connect to the local host at the standard SMTP port (25). If the hostname ends with a colon (``':'``) @@ -151,9 +154,9 @@ An :class:`SMTP` instance has the following methods: the constructor if a host is specified during instantiation. -.. method:: SMTP.docmd(cmd, [, argstring]) +.. method:: SMTP.docmd(cmd, args='') - Send a command *cmd* to the server. The optional argument *argstring* is simply + Send a command *cmd* to the server. The optional argument *args* is simply concatenated to the command, separated by a space. This returns a 2-tuple composed of a numeric response code and the actual @@ -167,7 +170,7 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPServerDisconnected` will be raised. -.. method:: SMTP.helo([hostname]) +.. method:: SMTP.helo(name='') Identify yourself to the SMTP server using ``HELO``. The hostname argument defaults to the fully qualified domain name of the local host. @@ -178,7 +181,7 @@ An :class:`SMTP` instance has the following methods: It will be implicitly called by the :meth:`sendmail` when necessary. -.. method:: SMTP.ehlo([hostname]) +.. method:: SMTP.ehlo(name='') Identify yourself to an ESMTP server using ``EHLO``. The hostname argument defaults to the fully qualified domain name of the local host. Examine the @@ -239,7 +242,7 @@ An :class:`SMTP` instance has the following methods: No suitable authentication method was found. -.. method:: SMTP.starttls([keyfile[, certfile]]) +.. method:: SMTP.starttls(keyfile=None, certfile=None) Put the SMTP connection in TLS (Transport Layer Security) mode. All SMTP commands that follow will be encrypted. You should then call :meth:`ehlo` @@ -261,7 +264,7 @@ An :class:`SMTP` instance has the following methods: SSL/TLS support is not available to your Python interpreter. -.. method:: SMTP.sendmail(from_addr, to_addrs, msg[, mail_options, rcpt_options]) +.. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=[], rcpt_options=[]) Send mail. The required arguments are an :rfc:`822` from-address string, a list of :rfc:`822` to-address strings (a bare string will be treated as a list with 1 @@ -275,9 +278,14 @@ An :class:`SMTP` instance has the following methods: .. note:: The *from_addr* and *to_addrs* parameters are used to construct the message - envelope used by the transport agents. The :class:`SMTP` does not modify the + envelope used by the transport agents. ``sendmail`` does not modify the message headers in any way. + *msg* may be a string containing characters in the ASCII range, or a byte + string. A string is encoded to bytes using the ascii codec, and lone ``\r`` + and ``\n`` characters are converted to ``\r\n`` characters. A byte string is + not modified. + If there has been no previous ``EHLO`` or ``HELO`` command this session, this method tries ESMTP ``EHLO`` first. If the server does ESMTP, message size and each of the specified options will be passed to it (if the option is in the @@ -312,6 +320,38 @@ An :class:`SMTP` instance has the following methods: Unless otherwise noted, the connection will be open even after an exception is raised. + .. versionchanged:: 3.2 *msg* may be a byte string. + + +.. method:: SMTP.send_message(msg, from_addr=None, to_addrs=None, \ + mail_options=[], rcpt_options=[]) + + This is a convenience method for calling :meth:`sendmail` with the message + represented by an :class:`email.message.Message` object. The arguments have + the same meaning as for :meth:`sendmail`, except that *msg* is a ``Message`` + object. + + If *from_addr* is ``None`` or *to_addrs* is ``None``, ``send_message`` fills + those arguments with addresses extracted from the headers of *msg* as + specified in :rfc:`2822`\: *from_addr* is set to the :mailheader:`Sender` + field if it is present, and otherwise to the :mailheader:`From` field. + *to_adresses* combines the values (if any) of the :mailheader:`To`, + :mailheader:`Cc`, and :mailheader:`Bcc` fields from *msg*. If exactly one + set of :mailheader:`Resent-*` headers appear in the message, the regular + headers are ignored and the :mailheader:`Resent-*` headers are used instead. + If the message contains more than one set of :mailheader:`Resent-*` headers, + a :exc:`ValueError` is raised, since there is no way to unambiguously detect + the most recent set of :mailheader:`Resent-` headers. + + ``send_message`` serializes *msg* using + :class:`~email.generator.BytesGenerator` with ``\r\n`` as the *linesep*, and + calls :meth:`sendmail` to transmit the resulting message. Regardless of the + values of *from_addr* and *to_addrs*, ``send_message`` does not transmit any + :mailheader:`Bcc` or :mailheader:`Resent-Bcc` headers that may appear + in *msg*. + + .. versionadded:: 3.2 + .. method:: SMTP.quit() @@ -367,5 +407,5 @@ example doesn't do any processing of the :rfc:`822` headers. In particular, the .. note:: In general, you will want to use the :mod:`email` package's features to - construct an email message, which you can then convert to a string and send - via :meth:`sendmail`; see :ref:`email-examples`. + construct an email message, which you can then send + via :meth:`~smtplib.SMTP.send_message`; see :ref:`email-examples`. diff --git a/Doc/library/sndhdr.rst b/Doc/library/sndhdr.rst index 01a3917a32..f36df68703 100644 --- a/Doc/library/sndhdr.rst +++ b/Doc/library/sndhdr.rst @@ -1,4 +1,3 @@ - :mod:`sndhdr` --- Determine type of sound file ============================================== @@ -11,6 +10,10 @@ single: A-LAW single: u-LAW +**Source code:** :source:`Lib/sndhdr.py` + +-------------- + The :mod:`sndhdr` provides utility functions which attempt to determine the type of sound data which is in a file. When these functions are able to determine what type of sound data is stored in a file, they return a tuple ``(type, diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index d61398c9f7..f236d30f26 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -1,4 +1,3 @@ - :mod:`socket` --- Low-level networking interface ================================================ @@ -65,20 +64,20 @@ Socket addresses are represented as follows: tuple, and the fields depend on the address type. The general tuple form is ``(addr_type, v1, v2, v3 [, scope])``, where: - - *addr_type* is one of TIPC_ADDR_NAMESEQ, TIPC_ADDR_NAME, or - TIPC_ADDR_ID. - - *scope* is one of TIPC_ZONE_SCOPE, TIPC_CLUSTER_SCOPE, and - TIPC_NODE_SCOPE. - - If *addr_type* is TIPC_ADDR_NAME, then *v1* is the server type, *v2* is + - *addr_type* is one of :const:`TIPC_ADDR_NAMESEQ`, :const:`TIPC_ADDR_NAME`, + or :const:`TIPC_ADDR_ID`. + - *scope* is one of :const:`TIPC_ZONE_SCOPE`, :const:`TIPC_CLUSTER_SCOPE`, and + :const:`TIPC_NODE_SCOPE`. + - If *addr_type* is :const:`TIPC_ADDR_NAME`, then *v1* is the server type, *v2* is the port identifier, and *v3* should be 0. - If *addr_type* is TIPC_ADDR_NAMESEQ, then *v1* is the server type, *v2* + If *addr_type* is :const:`TIPC_ADDR_NAMESEQ`, then *v1* is the server type, *v2* is the lower port number, and *v3* is the upper port number. - If *addr_type* is TIPC_ADDR_ID, then *v1* is the node, *v2* is the + If *addr_type* is :const:`TIPC_ADDR_ID`, then *v1* is the node, *v2* is the reference, and *v3* should be set to 0. - If *addr_type* is TIPC_ADDR_ID, then *v1* is the node, *v2* is the + If *addr_type* is :const:`TIPC_ADDR_ID`, then *v1* is the node, *v2* is the reference, and *v3* should be set to 0. - Certain other address families (:const:`AF_BLUETOOTH`, :const:`AF_PACKET`) @@ -118,39 +117,44 @@ The module :mod:`socket` exports the following constants and functions: .. index:: module: errno - This exception is raised for socket-related errors. The accompanying value is - either a string telling what went wrong or a pair ``(errno, string)`` - representing an error returned by a system call, similar to the value - accompanying :exc:`os.error`. See the module :mod:`errno`, which contains names - for the error codes defined by the underlying operating system. + A subclass of :exc:`IOError`, this exception is raised for socket-related + errors. It is recommended that you inspect its ``errno`` attribute to + discriminate between different kinds of errors. + .. seealso:: + The :mod:`errno` module contains symbolic names for the error codes + defined by the underlying operating system. -.. exception:: herror - This exception is raised for address-related errors, i.e. for functions that use - *h_errno* in the C API, including :func:`gethostbyname_ex` and - :func:`gethostbyaddr`. +.. exception:: herror - The accompanying value is a pair ``(h_errno, string)`` representing an error - returned by a library call. *string* represents the description of *h_errno*, as - returned by the :cfunc:`hstrerror` C function. + A subclass of :exc:`socket.error`, this exception is raised for + address-related errors, i.e. for functions that use *h_errno* in the POSIX + C API, including :func:`gethostbyname_ex` and :func:`gethostbyaddr`. + The accompanying value is a pair ``(h_errno, string)`` representing an + error returned by a library call. *h_errno* is a numeric value, while + *string* represents the description of *h_errno*, as returned by the + :c:func:`hstrerror` C function. .. exception:: gaierror - This exception is raised for address-related errors, for :func:`getaddrinfo` and - :func:`getnameinfo`. The accompanying value is a pair ``(error, string)`` - representing an error returned by a library call. *string* represents the - description of *error*, as returned by the :cfunc:`gai_strerror` C function. The - *error* value will match one of the :const:`EAI_\*` constants defined in this - module. + A subclass of :exc:`socket.error`, this exception is raised for + address-related errors by :func:`getaddrinfo` and :func:`getnameinfo`. + The accompanying value is a pair ``(error, string)`` representing an error + returned by a library call. *string* represents the description of + *error*, as returned by the :c:func:`gai_strerror` C function. The + numeric *error* value will match one of the :const:`EAI_\*` constants + defined in this module. .. exception:: timeout - This exception is raised when a timeout occurs on a socket which has had - timeouts enabled via a prior call to :meth:`~socket.settimeout`. The - accompanying value is a string whose value is currently always "timed out". + A subclass of :exc:`socket.error`, this exception is raised when a timeout + occurs on a socket which has had timeouts enabled via a prior call to + :meth:`~socket.settimeout` (or implicitly through + :func:`~socket.setdefaulttimeout`). The accompanying value is a string + whose value is currently always "timed out". .. data:: AF_UNIX @@ -174,6 +178,21 @@ The module :mod:`socket` exports the following constants and functions: (Only :const:`SOCK_STREAM` and :const:`SOCK_DGRAM` appear to be generally useful.) +.. data:: SOCK_CLOEXEC + SOCK_NONBLOCK + + These two constants, if defined, can be combined with the socket types and + allow you to set some flags atomically (thus avoiding possible race + conditions and the need for separate calls). + + .. seealso:: + + `Secure File Descriptor Handling <http://udrepper.livejournal.com/20407.html>`_ + for a more thorough explanation. + + Availability: Linux >= 2.6.27. + + .. versionadded:: 3.2 .. data:: SO_* SOMAXCONN @@ -215,16 +234,33 @@ The module :mod:`socket` exports the following constants and functions: this platform. -.. function:: create_connection(address[, timeout]) +.. function:: create_connection(address[, timeout[, source_address]]) + + Connect to a TCP service listening on the Internet *address* (a 2-tuple + ``(host, port)``), and return the socket object. This is a higher-level + function than :meth:`socket.connect`: if *host* is a non-numeric hostname, + it will try to resolve it for both :data:`AF_INET` and :data:`AF_INET6`, + and then try to connect to all possible addresses in turn until a + connection succeeds. This makes it easy to write clients that are + compatible to both IPv4 and IPv6. - Convenience function. Connect to *address* (a 2-tuple ``(host, port)``), - and return the socket object. Passing the optional *timeout* parameter will - set the timeout on the socket instance before attempting to connect. If no - *timeout* is supplied, the global default timeout setting returned by + Passing the optional *timeout* parameter will set the timeout on the + socket instance before attempting to connect. If no *timeout* is + supplied, the global default timeout setting returned by :func:`getdefaulttimeout` is used. + If supplied, *source_address* must be a 2-tuple ``(host, port)`` for the + socket to bind to as its source address before connecting. If host or port + are '' or 0 respectively the OS default behavior will be used. -.. function:: getaddrinfo(host, port, family=0, socktype=0, proto=0, flags=0) + .. versionchanged:: 3.2 + *source_address* was added. + + .. versionchanged:: 3.2 + support for the :keyword:`with` statement was added. + + +.. function:: getaddrinfo(host, port, family=0, type=0, proto=0, flags=0) Translate the *host*/*port* argument into a sequence of 5-tuples that contain all the necessary arguments for creating a socket connected to that service. @@ -233,7 +269,7 @@ The module :mod:`socket` exports the following constants and functions: port number or ``None``. By passing ``None`` as the value of *host* and *port*, you can pass ``NULL`` to the underlying C API. - The *family*, *socktype* and *proto* arguments can be optionally specified + The *family*, *type* and *proto* arguments can be optionally specified in order to narrow the list of addresses returned. Passing zero as a value for each of these arguments selects the full range of results. The *flags* argument can be one or several of the ``AI_*`` constants, @@ -243,9 +279,9 @@ The module :mod:`socket` exports the following constants and functions: The function returns a list of 5-tuples with the following structure: - ``(family, socktype, proto, canonname, sockaddr)`` + ``(family, type, proto, canonname, sockaddr)`` - In these tuples, *family*, *socktype*, *proto* are all integers and are + In these tuples, *family*, *type*, *proto* are all integers and are meant to be passed to the :func:`socket` function. *canonname* will be a string representing the canonical name of the *host* if :const:`AI_CANONNAME` is part of the *flags* argument; else *canonname* @@ -259,10 +295,13 @@ The module :mod:`socket` exports the following constants and functions: connection to ``www.python.org`` on port 80 (results may differ on your system if IPv6 isn't enabled):: - >>> socket.getaddrinfo("www.python.org", 80, 0, 0, socket.SOL_TCP) + >>> socket.getaddrinfo("www.python.org", 80, proto=socket.SOL_TCP) [(2, 1, 6, '', ('82.94.164.162', 80)), (10, 1, 6, '', ('2001:888:2000:d::a2', 80, 0, 0))] + .. versionchanged:: 3.2 + parameters can now be passed as single keyword arguments. + .. function:: getfqdn([name]) Return a fully qualified domain name for *name*. If *name* is omitted or empty, @@ -368,6 +407,10 @@ The module :mod:`socket` exports the following constants and functions: if defined on the platform; otherwise, the default is :const:`AF_INET`. Availability: Unix. + .. versionchanged:: 3.2 + The returned socket objects now support the whole socket API, rather + than a subset. + .. function:: fromfd(fd, family, type[, proto]) @@ -379,7 +422,6 @@ The module :mod:`socket` exports the following constants and functions: This function is rarely needed, but can be used to get or set socket options on a socket passed to a program as standard input or output (such as a server started by the Unix inet daemon). The socket is assumed to be in blocking mode. - Availability: Unix. .. function:: ntohl(x) @@ -415,7 +457,7 @@ The module :mod:`socket` exports the following constants and functions: Convert an IPv4 address from dotted-quad string format (for example, '123.45.67.89') to 32-bit packed binary format, as a bytes object four characters in length. This is useful when conversing with a program that uses the standard C - library and needs objects of type :ctype:`struct in_addr`, which is the C type + library and needs objects of type :c:type:`struct in_addr`, which is the C type for the 32-bit packed binary this function returns. :func:`inet_aton` also accepts strings with less than three dots; see the @@ -423,7 +465,7 @@ The module :mod:`socket` exports the following constants and functions: If the IPv4 address string passed to this function is invalid, :exc:`socket.error` will be raised. Note that exactly what is valid depends on - the underlying C implementation of :cfunc:`inet_aton`. + the underlying C implementation of :c:func:`inet_aton`. :func:`inet_aton` does not support IPv6, and :func:`inet_pton` should be used instead for IPv4/v6 dual stack support. @@ -434,7 +476,7 @@ The module :mod:`socket` exports the following constants and functions: Convert a 32-bit packed IPv4 address (a bytes object four characters in length) to its standard dotted-quad string representation (for example, '123.45.67.89'). This is useful when conversing with a program that uses the - standard C library and needs objects of type :ctype:`struct in_addr`, which + standard C library and needs objects of type :c:type:`struct in_addr`, which is the C type for the 32-bit packed binary data this function takes as an argument. @@ -448,14 +490,14 @@ The module :mod:`socket` exports the following constants and functions: Convert an IP address from its family-specific string format to a packed, binary format. :func:`inet_pton` is useful when a library or network protocol - calls for an object of type :ctype:`struct in_addr` (similar to - :func:`inet_aton`) or :ctype:`struct in6_addr`. + calls for an object of type :c:type:`struct in_addr` (similar to + :func:`inet_aton`) or :c:type:`struct in6_addr`. Supported values for *address_family* are currently :const:`AF_INET` and :const:`AF_INET6`. If the IP address string *ip_string* is invalid, :exc:`socket.error` will be raised. Note that exactly what is valid depends on both the value of *address_family* and the underlying implementation of - :cfunc:`inet_pton`. + :c:func:`inet_pton`. Availability: Unix (maybe not all platforms). @@ -465,8 +507,8 @@ The module :mod:`socket` exports the following constants and functions: Convert a packed IP address (a bytes object of some number of characters) to its standard, family-specific string representation (for example, ``'7.10.0.5'`` or ``'5aef:2b::8'``). :func:`inet_ntop` is useful when a library or network protocol - returns an object of type :ctype:`struct in_addr` (similar to :func:`inet_ntoa`) - or :ctype:`struct in6_addr`. + returns an object of type :c:type:`struct in_addr` (similar to :func:`inet_ntoa`) + or :c:type:`struct in6_addr`. Supported values for *address_family* are currently :const:`AF_INET` and :const:`AF_INET6`. If the string *packed_ip* is not the correct length for the @@ -478,14 +520,14 @@ The module :mod:`socket` exports the following constants and functions: .. function:: getdefaulttimeout() - Return the default timeout in floating seconds for new socket objects. A value + Return the default timeout in seconds (float) for new socket objects. A value of ``None`` indicates that new socket objects have no timeout. When the socket module is first imported, the default is ``None``. .. function:: setdefaulttimeout(timeout) - Set the default timeout in floating seconds for new socket objects. When + Set the default timeout in seconds (float) for new socket objects. When the socket module is first imported, the default is ``None``. See :meth:`~socket.settimeout` for possible values and their respective meanings. @@ -542,13 +584,22 @@ correspond to Unix system calls applicable to sockets. .. method:: socket.connect_ex(address) Like ``connect(address)``, but return an error indicator instead of raising an - exception for errors returned by the C-level :cfunc:`connect` call (other + exception for errors returned by the C-level :c:func:`connect` call (other problems, such as "host not found," can still raise exceptions). The error indicator is ``0`` if the operation succeeded, otherwise the value of the - :cdata:`errno` variable. This is useful to support, for example, asynchronous + :c:data:`errno` variable. This is useful to support, for example, asynchronous connects. +.. method:: socket.detach() + + Put the socket object into closed state without actually closing the + underlying file descriptor. The file descriptor is returned, and can + be reused for other purposes. + + .. versionadded:: 3.2 + + .. method:: socket.fileno() Return the socket's file descriptor (a small integer). This is useful with @@ -588,7 +639,7 @@ correspond to Unix system calls applicable to sockets. .. method:: socket.gettimeout() - Return the timeout in floating seconds associated with socket operations, + Return the timeout in seconds (float) associated with socket operations, or ``None`` if no timeout is set. This reflects the last call to :meth:`setblocking` or :meth:`settimeout`. @@ -608,11 +659,12 @@ correspond to Unix system calls applicable to sockets. .. method:: socket.listen(backlog) Listen for connections made to the socket. The *backlog* argument specifies the - maximum number of queued connections and should be at least 1; the maximum value - is system-dependent (usually 5). + maximum number of queued connections and should be at least 0; the maximum value + is system-dependent (usually 5), the minimum value is forced to 0. -.. method:: socket.makefile(mode='r', buffering=None, *, encoding=None, errors=None, newline=None) +.. method:: socket.makefile(mode='r', buffering=None, *, encoding=None, \ + errors=None, newline=None) .. index:: single: I/O control; buffering @@ -668,9 +720,9 @@ correspond to Unix system calls applicable to sockets. Receive up to *nbytes* bytes from the socket, storing the data into a buffer rather than creating a new bytestring. If *nbytes* is not specified (or 0), - receive up to the size available in the given buffer. See the Unix manual page - :manpage:`recv(2)` for the meaning of the optional argument *flags*; it defaults - to zero. + receive up to the size available in the given buffer. Returns the number of + bytes received. See the Unix manual page :manpage:`recv(2)` for the meaning + of the optional argument *flags*; it defaults to zero. .. method:: socket.send(bytes[, flags]) @@ -679,7 +731,8 @@ correspond to Unix system calls applicable to sockets. optional *flags* argument has the same meaning as for :meth:`recv` above. Returns the number of bytes sent. Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the - application needs to attempt delivery of the remaining data. + application needs to attempt delivery of the remaining data. For further + information on this topic, consult the :ref:`socket-howto`. .. method:: socket.sendall(bytes[, flags]) @@ -807,6 +860,21 @@ before calling :meth:`~socket.connect` or pass a timeout parameter to return a connection timeout error of its own regardless of any Python socket timeout setting. +Timeouts and the ``accept`` method +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If :func:`getdefaulttimeout` is not :const:`None`, sockets returned by +the :meth:`~socket.accept` method inherit that timeout. Otherwise, the +behaviour depends on settings of the listening socket: + +* if the listening socket is in *blocking mode* or in *timeout mode*, + the socket returned by :meth:`~socket.accept` is in *blocking mode*; + +* if the listening socket is in *non-blocking mode*, whether the socket + returned by :meth:`~socket.accept` is in blocking or non-blocking mode + is operating system-dependent. If you want to ensure cross-platform + behaviour, it is recommended you manually override this setting. + .. _socket-example: @@ -819,8 +887,8 @@ using it. Note that a server must perform the sequence :func:`socket`, :meth:`~socket.bind`, :meth:`~socket.listen`, :meth:`~socket.accept` (possibly repeating the :meth:`~socket.accept` to service more than one client), while a client only needs the sequence :func:`socket`, :meth:`~socket.connect`. Also -note that the server does not :meth:`~socket.send`/:meth:`~socket.recv` on the -socket it is listening on but on the new socket returned by +note that the server does not :meth:`~socket.sendall`/:meth:`~socket.recv` on +the socket it is listening on but on the new socket returned by :meth:`~socket.accept`. The first two examples support IPv4 only. :: @@ -838,7 +906,7 @@ The first two examples support IPv4 only. :: while True: data = conn.recv(1024) if not data: break - conn.send(data) + conn.sendall(data) conn.close() :: @@ -850,7 +918,7 @@ The first two examples support IPv4 only. :: PORT = 50007 # The same port as used by the server s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((HOST, PORT)) - s.send(b'Hello, world') + s.sendall(b'Hello, world') data = s.recv(1024) s.close() print('Received', repr(data)) @@ -922,7 +990,7 @@ sends traffic to the first one connected successfully. :: if s is None: print('could not open socket') sys.exit(1) - s.send(b'Hello, world') + s.sendall(b'Hello, world') data = s.recv(1024) s.close() print('Received', repr(data)) @@ -954,6 +1022,25 @@ the interface:: s.ioctl(socket.SIO_RCVALL, socket.RCVALL_OFF) +Running an example several times with too small delay between executions, could +lead to this error:: + + socket.error: [Errno 98] Address already in use + +This is because the previous execution has left the socket in a ``TIME_WAIT`` +state, and can't be immediately reused. + +There is a :mod:`socket` flag to set, in order to prevent this, +:data:`socket.SO_REUSEADDR`:: + + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + s.bind((HOST, PORT)) + +the :data:`SO_REUSEADDR` flag tells the kernel to reuse a local socket in +``TIME_WAIT`` state, without waiting for its natural timeout to expire. + + .. seealso:: For an introduction to socket programming (in C), see the following papers: diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst index 82d1107b12..5287f17a28 100644 --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -1,10 +1,13 @@ - :mod:`socketserver` --- A framework for network servers ======================================================= .. module:: socketserver :synopsis: A framework for network servers. +**Source code:** :source:`Lib/socketserver.py` + +-------------- + The :mod:`socketserver` module simplifies the task of writing network servers. There are four basic server classes: :class:`TCPServer` uses the Internet TCP @@ -36,11 +39,12 @@ process one or many requests. When inheriting from :class:`ThreadingMixIn` for threaded connection behavior, you should explicitly declare how you want your threads to behave on an abrupt -shutdown. The :class:`ThreadingMixIn` class defines an attribute +shutdown. The :class:`ThreadingMixIn` class defines an attribute *daemon_threads*, which indicates whether or not the server should wait for -thread termination. You should set the flag explicitly if you would like threads -to behave autonomously; the default is :const:`False`, meaning that Python will -not exit until all threads created by :class:`ThreadingMixIn` have exited. +thread termination. You should set the flag explicitly if you would like +threads to behave autonomously; the default is :const:`False`, meaning that +Python will not exit until all threads created by :class:`ThreadingMixIn` have +exited. Server classes have the same external methods and attributes, no matter what network protocol they use. @@ -78,7 +82,7 @@ a threading UDP server class is created as follows:: class ThreadingUDPServer(ThreadingMixIn, UDPServer): pass The mix-in class must come first, since it overrides a method defined in -:class:`UDPServer`. Setting the various member variables also changes the +:class:`UDPServer`. Setting the various attributes also change the behavior of the underlying server mechanism. To implement a service, you must derive a class from :class:`BaseRequestHandler` @@ -112,8 +116,8 @@ or inappropriate for the service) is to maintain an explicit table of partially finished requests and to use :func:`select` to decide which request to work on next (or whether to handle a new incoming request). This is particularly important for stream services where each client can potentially be connected for -a long time (if threads or subprocesses cannot be used). See :mod:`asyncore` for -another way to manage this. +a long time (if threads or subprocesses cannot be used). See :mod:`asyncore` +for another way to manage this. .. XXX should data and methods be intermingled, or separate? how should the distinction between class and instance variables be drawn? @@ -149,13 +153,14 @@ Server Objects .. method:: BaseServer.serve_forever(poll_interval=0.5) - Handle requests until an explicit :meth:`shutdown` request. Polls for - shutdown every *poll_interval* seconds. + Handle requests until an explicit :meth:`shutdown` request. + Poll for shutdown every *poll_interval* seconds. Ignores :attr:`self.timeout`. + If you need to do periodic tasks, do them in another thread. .. method:: BaseServer.shutdown() - Tells the :meth:`serve_forever` loop to stop and waits until it does. + Tell the :meth:`serve_forever` loop to stop and wait until it does. .. attribute:: BaseServer.address_family @@ -189,7 +194,7 @@ The server classes support the following class variables: .. attribute:: BaseServer.allow_reuse_address - Whether the server will allow the reuse of an address. This defaults to + Whether the server will allow the reuse of an address. This defaults to :const:`False`, and can be set in subclasses to change the policy. @@ -266,7 +271,7 @@ users of the server object. .. method:: BaseServer.server_activate() Called by the server's constructor to activate the server. The default behavior - just :meth:`listen`\ s to the server's socket. May be overridden. + just :meth:`listen`\ s to the server's socket. May be overridden. .. method:: BaseServer.server_bind() @@ -277,10 +282,10 @@ users of the server object. .. method:: BaseServer.verify_request(request, client_address) - Must return a Boolean value; if the value is :const:`True`, the request will be - processed, and if it's :const:`False`, the request will be denied. This function - can be overridden to implement access controls for a server. The default - implementation always returns :const:`True`. + Must return a Boolean value; if the value is :const:`True`, the request will + be processed, and if it's :const:`False`, the request will be denied. This + function can be overridden to implement access controls for a server. The + default implementation always returns :const:`True`. RequestHandler Objects @@ -345,10 +350,10 @@ This is the server side:: def handle(self): # self.request is the TCP socket connected to the client self.data = self.request.recv(1024).strip() - print("%s wrote:" % self.client_address[0]) + print("{} wrote:".format(self.client_address[0])) print(self.data) # just send back the same data, but upper-cased - self.request.send(self.data.upper()) + self.request.sendall(self.data.upper()) if __name__ == "__main__": HOST, PORT = "localhost", 9999 @@ -369,7 +374,7 @@ objects that simplify communication by providing the standard file interface):: # self.rfile is a file-like object created by the handler; # we can now use e.g. readline() instead of raw recv() calls self.data = self.rfile.readline().strip() - print("%s wrote:" % self.client_address[0]) + print("{} wrote:".format(self.client_address[0])) print(self.data) # Likewise, self.wfile is a file-like object used to write back # to the client @@ -378,7 +383,7 @@ objects that simplify communication by providing the standard file interface):: The difference is that the ``readline()`` call in the second handler will call ``recv()`` multiple times until it encounters a newline character, while the single ``recv()`` call in the first handler will just return what has been sent -from the client in one ``send()`` call. +from the client in one ``sendall()`` call. This is the client side:: @@ -392,16 +397,18 @@ This is the client side:: # Create a socket (SOCK_STREAM means a TCP socket) sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - # Connect to server and send data - sock.connect((HOST, PORT)) - sock.send(bytes(data + "\n","utf8")) + try: + # Connect to server and send data + sock.connect((HOST, PORT)) + sock.sendall(bytes(data + "\n", "utf-8")) - # Receive data from the server and shut down - received = sock.recv(1024) - sock.close() + # Receive data from the server and shut down + received = str(sock.recv(1024), "utf-8") + finally: + sock.close() - print("Sent: %s" % data) - print("Received: %s" % received) + print("Sent: {}".format(data)) + print("Received: {}".format(received)) The output of the example should look something like this: @@ -418,10 +425,10 @@ Client:: $ python TCPClient.py hello world with TCP Sent: hello world with TCP - Received: b'HELLO WORLD WITH TCP' + Received: HELLO WORLD WITH TCP $ python TCPClient.py python is nice Sent: python is nice - Received: b'PYTHON IS NICE' + Received: PYTHON IS NICE :class:`socketserver.UDPServer` Example @@ -442,7 +449,7 @@ This is the server side:: def handle(self): data = self.request[0].strip() socket = self.request[1] - print("%s wrote:" % self.client_address[0]) + print("{} wrote:".format(self.client_address[0])) print(data) socket.sendto(data.upper(), self.client_address) @@ -464,11 +471,11 @@ This is the client side:: # As you can see, there is no connect() call; UDP has no connections. # Instead, data is directly sent to the recipient via sendto(). - sock.sendto(bytes(data + "\n","utf8"), (HOST, PORT)) - received = sock.recv(1024) + sock.sendto(bytes(data + "\n", "utf-8"), (HOST, PORT)) + received = str(sock.recv(1024), "utf-8") - print("Sent: %s" % data) - print("Received: %s" % received) + print("Sent: {}".format(data)) + print("Received: {}".format(received)) The output of the example should look exactly like for the TCP server example. @@ -488,10 +495,10 @@ An example for the :class:`ThreadingMixIn` class:: class ThreadedTCPRequestHandler(socketserver.BaseRequestHandler): def handle(self): - data = self.request.recv(1024) + data = str(self.request.recv(1024), 'ascii') cur_thread = threading.current_thread() - response = bytes("%s: %s" % (cur_thread.getName(), data),'ascii') - self.request.send(response) + response = bytes("{}: {}".format(cur_thread.name, data), 'ascii') + self.request.sendall(response) class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer): pass @@ -499,10 +506,12 @@ An example for the :class:`ThreadingMixIn` class:: def client(ip, port, message): sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.connect((ip, port)) - sock.send(message) - response = sock.recv(1024) - print("Received: %s" % response) - sock.close() + try: + sock.sendall(bytes(message, 'ascii')) + response = str(sock.recv(1024), 'ascii') + print("Received: {}".format(response)) + finally: + sock.close() if __name__ == "__main__": # Port 0 means to select an arbitrary unused port @@ -515,13 +524,13 @@ An example for the :class:`ThreadingMixIn` class:: # more thread for each request server_thread = threading.Thread(target=server.serve_forever) # Exit the server thread when the main thread terminates - server_thread.setDaemon(True) + server_thread.daemon = True server_thread.start() print("Server loop running in thread:", server_thread.name) - client(ip, port, b"Hello World 1") - client(ip, port, b"Hello World 2") - client(ip, port, b"Hello World 3") + client(ip, port, "Hello World 1") + client(ip, port, "Hello World 2") + client(ip, port, "Hello World 3") server.shutdown() @@ -530,9 +539,9 @@ The output of the example should look something like this:: $ python ThreadedTCPServer.py Server loop running in thread: Thread-1 - Received: b"Thread-2: b'Hello World 1'" - Received: b"Thread-3: b'Hello World 2'" - Received: b"Thread-4: b'Hello World 3'" + Received: Thread-2: Hello World 1 + Received: Thread-3: Hello World 2 + Received: Thread-4: Hello World 3 The :class:`ForkingMixIn` class is used in the same way, except that the server diff --git a/Doc/library/someos.rst b/Doc/library/someos.rst index cf3eb6b6bf..d2009bbc18 100644 --- a/Doc/library/someos.rst +++ b/Doc/library/someos.rst @@ -15,6 +15,7 @@ some other systems as well (e.g. Windows). Here's an overview: select.rst threading.rst multiprocessing.rst + concurrent.futures.rst mmap.rst readline.rst rlcompleter.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 0603738c67..f0fd86cb85 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -90,7 +90,7 @@ This example uses the iterator form:: .. seealso:: - http://www.pysqlite.org + http://code.google.com/p/pysqlite/ The pysqlite web page -- sqlite3 is developed externally under the name "pysqlite". @@ -227,6 +227,12 @@ Connection Objects one of "DEFERRED", "IMMEDIATE" or "EXCLUSIVE". See section :ref:`sqlite3-controlling-transactions` for a more detailed explanation. +.. attribute:: Connection.in_transaction + + :const:`True` if a transaction is active (there are uncommitted changes), + :const:`False` otherwise. Read-only attribute. + + .. versionadded:: 3.2 .. method:: Connection.cursor([cursorClass]) @@ -237,7 +243,7 @@ Connection Objects .. method:: Connection.commit() This method commits the current transaction. If you don't call this method, - anything you did since the last call to ``commit()`` is not visible from from + anything you did since the last call to ``commit()`` is not visible from other database connections. If you wonder why you don't see the data you've written to the database, please check you didn't forget to call this method. @@ -363,6 +369,29 @@ Connection Objects method with :const:`None` for *handler*. +.. method:: Connection.enable_load_extension(enabled) + + This routine allows/disallows the SQLite engine to load SQLite extensions + from shared libraries. SQLite extensions can define new functions, + aggregates or whole new virtual table implementations. One well-known + extension is the fulltext-search extension distributed with SQLite. + + .. versionadded:: 3.2 + + .. literalinclude:: ../includes/sqlite3/load_extension.py + + Loadable extensions are disabled by default. See [#f1]_. + +.. method:: Connection.load_extension(path) + + This routine loads a SQLite extension from a shared library. You have to + enable extension loading with :meth:`enable_load_extension` before you can + use this routine. + + .. versionadded:: 3.2 + + Loadable extensions are disabled by default. See [#f1]_. + .. attribute:: Connection.row_factory You can change this attribute to a callable that accepts the cursor and the @@ -443,14 +472,10 @@ Cursor Objects kinds of placeholders: question marks (qmark style) and named placeholders (named style). - This example shows how to use parameters with qmark style: + Here's an example of both styles: .. literalinclude:: ../includes/sqlite3/execute_1.py - This example shows how to use the named style: - - .. literalinclude:: ../includes/sqlite3/execute_2.py - :meth:`execute` will only execute a single SQL statement. If you try to execute more than one statement with it, it will raise a Warning. Use :meth:`executescript` if you want to execute multiple SQL statements with one @@ -518,18 +543,17 @@ Cursor Objects attribute, the database engine's own support for the determination of "rows affected"/"rows selected" is quirky. - For ``DELETE`` statements, SQLite reports :attr:`rowcount` as 0 if you make a - ``DELETE FROM table`` without any condition. - For :meth:`executemany` statements, the number of modifications are summed up into :attr:`rowcount`. As required by the Python DB API Spec, the :attr:`rowcount` attribute "is -1 in case no ``executeXX()`` has been performed on the cursor or the rowcount of the - last operation is not determinable by the interface". + last operation is not determinable by the interface". This includes ``SELECT`` + statements because we cannot determine the number of rows a query produced + until all rows were fetched. - This includes ``SELECT`` statements because we cannot determine the number of - rows a query produced until all rows were fetched. + With SQLite versions before 3.6.5, :attr:`rowcount` is set to 0 if + you make a ``DELETE FROM table`` without any condition. .. attribute:: Cursor.lastrowid @@ -570,43 +594,43 @@ Row Objects Let's assume we initialize a table as in the example given above:: - conn = sqlite3.connect(":memory:") - c = conn.cursor() - c.execute('''create table stocks - (date text, trans text, symbol text, - qty real, price real)''') - c.execute("""insert into stocks - values ('2006-01-05','BUY','RHAT',100,35.14)""") - conn.commit() - c.close() + conn = sqlite3.connect(":memory:") + c = conn.cursor() + c.execute('''create table stocks + (date text, trans text, symbol text, + qty real, price real)''') + c.execute("""insert into stocks + values ('2006-01-05','BUY','RHAT',100,35.14)""") + conn.commit() + c.close() Now we plug :class:`Row` in:: - >>> conn.row_factory = sqlite3.Row - >>> c = conn.cursor() - >>> c.execute('select * from stocks') - <sqlite3.Cursor object at 0x7f4e7dd8fa80> - >>> r = c.fetchone() - >>> type(r) - <class 'sqlite3.Row'> - >>> tuple(r) - ('2006-01-05', 'BUY', 'RHAT', 100.0, 35.14) - >>> len(r) - 5 - >>> r[2] - 'RHAT' - >>> r.keys() - ['date', 'trans', 'symbol', 'qty', 'price'] - >>> r['qty'] - 100.0 - >>> for member in r: - ... print(member) - ... - 2006-01-05 - BUY - RHAT - 100.0 - 35.14 + >>> conn.row_factory = sqlite3.Row + >>> c = conn.cursor() + >>> c.execute('select * from stocks') + <sqlite3.Cursor object at 0x7f4e7dd8fa80> + >>> r = c.fetchone() + >>> type(r) + <class 'sqlite3.Row'> + >>> tuple(r) + ('2006-01-05', 'BUY', 'RHAT', 100.0, 35.14) + >>> len(r) + 5 + >>> r[2] + 'RHAT' + >>> r.keys() + ['date', 'trans', 'symbol', 'qty', 'price'] + >>> r['qty'] + 100.0 + >>> for member in r: + ... print(member) + ... + 2006-01-05 + BUY + RHAT + 100.0 + 35.14 .. _sqlite3-types: @@ -732,7 +756,7 @@ and constructs a :class:`Point` object from it. :: def convert_point(s): - x, y = map(float, s.split(";")) + x, y = map(float, s.split(b";")) return Point(x, y) Now you need to make the :mod:`sqlite3` module know that what you select from @@ -785,7 +809,8 @@ So if you are within a transaction and issue a command like ``CREATE TABLE before executing that command. There are two reasons for doing that. The first is that some of these commands don't work within transactions. The other reason is that sqlite3 needs to keep track of the transaction state (if a transaction -is active or not). +is active or not). The current transaction state is exposed through the +:attr:`Connection.in_transaction` attribute of the connection object. You can control which kind of ``BEGIN`` statements sqlite3 implicitly executes (or none at all) via the *isolation_level* parameter to the :func:`connect` @@ -852,3 +877,11 @@ threads. If you still try to do so, you will get an exception at runtime. The only exception is calling the :meth:`~Connection.interrupt` method, which only makes sense to call from a different thread. + +.. rubric:: Footnotes + +.. [#f1] The sqlite3 module is not built with loadable extension support by + default, because some platforms (notably Mac OS X) have SQLite + libraries which are compiled without this feature. To get loadable + extension support, you must pass --enable-loadable-sqlite-extensions to + configure. diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index ca3242945b..8cd07d75dc 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -12,6 +12,10 @@ .. index:: TLS, SSL, Transport Layer Security, Secure Sockets Layer +**Source code:** :source:`Lib/ssl.py` + +-------------- + This module provides access to Transport Layer Security (often known as "Secure Sockets Layer") encryption and peer authentication facilities for network sockets, both client-side and server-side. This module uses the OpenSSL @@ -31,23 +35,43 @@ the documents in the "See Also" section at the bottom. This module provides a class, :class:`ssl.SSLSocket`, which is derived from the :class:`socket.socket` type, and provides a socket-like wrapper that also encrypts and decrypts the data going over the socket with SSL. It supports -additional :meth:`read` and :meth:`write` methods, along with a method, -:meth:`getpeercert`, to retrieve the certificate of the other side of the -connection, and a method, :meth:`cipher`, to retrieve the cipher being used for -the secure connection. +additional methods such as :meth:`getpeercert`, which retrieves the +certificate of the other side of the connection, and :meth:`cipher`,which +retrieves the cipher being used for the secure connection. + +For more sophisticated applications, the :class:`ssl.SSLContext` class +helps manage settings and certificates, which can then be inherited +by SSL sockets created through the :meth:`SSLContext.wrap_socket` method. + Functions, Constants, and Exceptions ------------------------------------ .. exception:: SSLError - Raised to signal an error from the underlying SSL implementation. This - signifies some problem in the higher-level encryption and authentication - layer that's superimposed on the underlying network connection. This error + Raised to signal an error from the underlying SSL implementation + (currently provided by the OpenSSL library). This signifies some + problem in the higher-level encryption and authentication layer that's + superimposed on the underlying network connection. This error is a subtype of :exc:`socket.error`, which in turn is a subtype of - :exc:`IOError`. + :exc:`IOError`. The error code and message of :exc:`SSLError` instances + are provided by the OpenSSL library. + +.. exception:: CertificateError + + Raised to signal an error with a certificate (such as mismatching + hostname). Certificate errors detected by OpenSSL, though, raise + an :exc:`SSLError`. + + +Socket creation +^^^^^^^^^^^^^^^ + +The following function allows for standalone socket creation. Starting from +Python 3.2, it can be more flexible to use :meth:`SSLContext.wrap_socket` +instead. -.. function:: wrap_socket(sock, keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE, ssl_version={see docs}, ca_certs=None, do_handshake_on_connect=True, suppress_ragged_eofs=True) +.. function:: wrap_socket(sock, keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE, ssl_version={see docs}, ca_certs=None, do_handshake_on_connect=True, suppress_ragged_eofs=True, ciphers=None) Takes an instance ``sock`` of :class:`socket.socket`, and returns an instance of :class:`ssl.SSLSocket`, a subtype of :class:`socket.socket`, which wraps @@ -64,19 +88,6 @@ Functions, Constants, and Exceptions connection. See the discussion of :ref:`ssl-certificates` for more information on how the certificate is stored in the ``certfile``. - Often the private key is stored in the same file as the certificate; in this - case, only the ``certfile`` parameter need be passed. If the private key is - stored in a separate file, both parameters must be used. If the private key - is stored in the ``certfile``, it should come before the first certificate in - the certificate chain:: - - -----BEGIN RSA PRIVATE KEY----- - ... (private key in base64 encoding) ... - -----END RSA PRIVATE KEY----- - -----BEGIN CERTIFICATE----- - ... (certificate in base64 PEM encoding) ... - -----END CERTIFICATE----- - The parameter ``server_side`` is a boolean which identifies whether server-side or client-side behavior is desired from this socket. @@ -97,9 +108,8 @@ Functions, Constants, and Exceptions The parameter ``ssl_version`` specifies which version of the SSL protocol to use. Typically, the server chooses a particular protocol version, and the client must adapt to the server's choice. Most of the versions are not - interoperable with the other versions. If not specified, for client-side - operation, the default SSL version is SSLv3; for server-side operation, - SSLv23. These version selections provide the most compatibility with other + interoperable with the other versions. If not specified, the default is + :data:`PROTOCOL_SSLv23`; it provides the most compatibility with other versions. Here's a table showing which versions in a client (down the side) can connect @@ -110,14 +120,26 @@ Functions, Constants, and Exceptions ======================== ========= ========= ========== ========= *client* / **server** **SSLv2** **SSLv3** **SSLv23** **TLSv1** ------------------------ --------- --------- ---------- --------- - *SSLv2* yes no yes* no - *SSLv3* yes yes yes no + *SSLv2* yes no yes no + *SSLv3* no yes yes no *SSLv23* yes no yes no *TLSv1* no no yes yes ======================== ========= ========= ========== ========= - In some older versions of OpenSSL (for instance, 0.9.7l on OS X 10.4), an - SSLv2 client could not connect to an SSLv23 server. + .. note:: + + Which connections succeed will vary depending on the version of + OpenSSL. For instance, in some older versions of OpenSSL (such + as 0.9.7l on OS X 10.4), an SSLv2 client could not connect to an + SSLv23 server. Another example: beginning with OpenSSL 1.0.0, + an SSLv23 client will not actually attempt SSLv2 connections + unless you explicitly enable SSLv2 ciphers; for example, you + might specify ``"ALL"`` or ``"SSLv2"`` as the *ciphers* parameter + to enable them. + + The *ciphers* parameter sets the available ciphers for this SSL object. + It should be a string in the `OpenSSL cipher list format + <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`_. The parameter ``do_handshake_on_connect`` specifies whether to do the SSL handshake automatically after doing a :meth:`socket.connect`, or whether the @@ -127,10 +149,17 @@ Functions, Constants, and Exceptions blocking behavior of the socket I/O involved in the handshake. The parameter ``suppress_ragged_eofs`` specifies how the - :meth:`SSLSocket.read` method should signal unexpected EOF from the other end + :meth:`SSLSocket.recv` method should signal unexpected EOF from the other end of the connection. If specified as :const:`True` (the default), it returns a - normal EOF in response to unexpected EOF errors raised from the underlying - socket; if :const:`False`, it will raise the exceptions back to the caller. + normal EOF (an empty bytes object) in response to unexpected EOF errors + raised from the underlying socket; if :const:`False`, it will raise the + exceptions back to the caller. + + .. versionchanged:: 3.2 + New optional argument *ciphers*. + +Random generation +^^^^^^^^^^^^^^^^^ .. function:: RAND_status() @@ -157,6 +186,32 @@ Functions, Constants, and Exceptions string (so you can always use :const:`0.0`). See :rfc:`1750` for more information on sources of entropy. +Certificate handling +^^^^^^^^^^^^^^^^^^^^ + +.. function:: match_hostname(cert, hostname) + + Verify that *cert* (in decoded format as returned by + :meth:`SSLSocket.getpeercert`) matches the given *hostname*. The rules + applied are those for checking the identity of HTTPS servers as outlined + in :rfc:`2818`, except that IP addresses are not currently supported. + In addition to HTTPS, this function should be suitable for checking the + identity of servers in various SSL-based protocols such as FTPS, IMAPS, + POPS and others. + + :exc:`CertificateError` is raised on failure. On success, the function + returns nothing:: + + >>> cert = {'subject': ((('commonName', 'example.com'),),)} + >>> ssl.match_hostname(cert, "example.com") + >>> ssl.match_hostname(cert, "example.org") + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + File "/home/py3k/Lib/ssl.py", line 130, in match_hostname + ssl.CertificateError: hostname 'example.org' doesn't match 'example.com' + + .. versionadded:: 3.2 + .. function:: cert_time_to_seconds(timestring) Returns a floating-point value containing a normal seconds-after-the-epoch @@ -171,7 +226,6 @@ Functions, Constants, and Exceptions >>> import time >>> time.ctime(ssl.cert_time_to_seconds("May 9 00:00:00 2007 GMT")) 'Wed May 9 00:00:00 2007' - >>> .. function:: get_server_certificate(addr, ssl_version=PROTOCOL_SSLv3, ca_certs=None) @@ -194,26 +248,41 @@ Functions, Constants, and Exceptions Given a certificate as an ASCII PEM string, returns a DER-encoded sequence of bytes for that same certificate. +Constants +^^^^^^^^^ + .. data:: CERT_NONE - Value to pass to the ``cert_reqs`` parameter to :func:`sslobject` when no - certificates will be required or validated from the other side of the socket - connection. + Possible value for :attr:`SSLContext.verify_mode`, or the ``cert_reqs`` + parameter to :func:`wrap_socket`. In this mode (the default), no + certificates will be required from the other side of the socket connection. + If a certificate is received from the other end, no attempt to validate it + is made. + + See the discussion of :ref:`ssl-security` below. .. data:: CERT_OPTIONAL - Value to pass to the ``cert_reqs`` parameter to :func:`sslobject` when no - certificates will be required from the other side of the socket connection, - but if they are provided, will be validated. Note that use of this setting - requires a valid certificate validation file also be passed as a value of the - ``ca_certs`` parameter. + Possible value for :attr:`SSLContext.verify_mode`, or the ``cert_reqs`` + parameter to :func:`wrap_socket`. In this mode no certificates will be + required from the other side of the socket connection; but if they + are provided, validation will be attempted and an :class:`SSLError` + will be raised on failure. + + Use of this setting requires a valid set of CA certificates to + be passed, either to :meth:`SSLContext.load_verify_locations` or as a + value of the ``ca_certs`` parameter to :func:`wrap_socket`. .. data:: CERT_REQUIRED - Value to pass to the ``cert_reqs`` parameter to :func:`sslobject` when - certificates will be required from the other side of the socket connection. - Note that use of this setting requires a valid certificate validation file - also be passed as a value of the ``ca_certs`` parameter. + Possible value for :attr:`SSLContext.verify_mode`, or the ``cert_reqs`` + parameter to :func:`wrap_socket`. In this mode, certificates are + required from the other side of the socket connection; an :class:`SSLError` + will be raised if no certificate is provided, or if its validation fails. + + Use of this setting requires a valid set of CA certificates to + be passed, either to :meth:`SSLContext.load_verify_locations` or as a + value of the ``ca_certs`` parameter to :func:`wrap_socket`. .. data:: PROTOCOL_SSLv2 @@ -244,42 +313,111 @@ Functions, Constants, and Exceptions modern version, and probably the best choice for maximum protection, if both sides can speak it. +.. data:: OP_ALL + + Enables workarounds for various bugs present in other SSL implementations. + This option is set by default. It does not necessarily set the same + flags as OpenSSL's ``SSL_OP_ALL`` constant. + + .. versionadded:: 3.2 + +.. data:: OP_NO_SSLv2 + + Prevents an SSLv2 connection. This option is only applicable in + conjunction with :const:`PROTOCOL_SSLv23`. It prevents the peers from + choosing SSLv2 as the protocol version. + + .. versionadded:: 3.2 -SSLSocket Objects ------------------ +.. data:: OP_NO_SSLv3 -.. method:: SSLSocket.read(nbytes=1024, buffer=None) + Prevents an SSLv3 connection. This option is only applicable in + conjunction with :const:`PROTOCOL_SSLv23`. It prevents the peers from + choosing SSLv3 as the protocol version. - Reads up to ``nbytes`` bytes from the SSL-encrypted channel and returns them. - If the ``buffer`` is specified, it will attempt to read into the buffer the - minimum of the size of the buffer and ``nbytes``, if that is specified. If - no buffer is specified, an immutable buffer is allocated and returned with - the data read from the socket. + .. versionadded:: 3.2 -.. method:: SSLSocket.write(data) +.. data:: OP_NO_TLSv1 - Writes the ``data`` to the other side of the connection, using the SSL - channel to encrypt. Returns the number of bytes written. + Prevents a TLSv1 connection. This option is only applicable in + conjunction with :const:`PROTOCOL_SSLv23`. It prevents the peers from + choosing TLSv1 as the protocol version. + + .. versionadded:: 3.2 + +.. data:: HAS_SNI + + Whether the OpenSSL library has built-in support for the *Server Name + Indication* extension to the SSLv3 and TLSv1 protocols (as defined in + :rfc:`4366`). When true, you can use the *server_hostname* argument to + :meth:`SSLContext.wrap_socket`. + + .. versionadded:: 3.2 + +.. data:: OPENSSL_VERSION + + The version string of the OpenSSL library loaded by the interpreter:: + + >>> ssl.OPENSSL_VERSION + 'OpenSSL 0.9.8k 25 Mar 2009' + + .. versionadded:: 3.2 + +.. data:: OPENSSL_VERSION_INFO + + A tuple of five integers representing version information about the + OpenSSL library:: + + >>> ssl.OPENSSL_VERSION_INFO + (0, 9, 8, 11, 15) + + .. versionadded:: 3.2 + +.. data:: OPENSSL_VERSION_NUMBER + + The raw version number of the OpenSSL library, as a single integer:: + + >>> ssl.OPENSSL_VERSION_NUMBER + 9470143 + >>> hex(ssl.OPENSSL_VERSION_NUMBER) + '0x9080bf' + + .. versionadded:: 3.2 + + +SSL Sockets +----------- + +SSL sockets provide the following methods of :ref:`socket-objects`: + +- :meth:`~socket.socket.accept()` +- :meth:`~socket.socket.bind()` +- :meth:`~socket.socket.close()` +- :meth:`~socket.socket.connect()` +- :meth:`~socket.socket.detach()` +- :meth:`~socket.socket.fileno()` +- :meth:`~socket.socket.getpeername()`, :meth:`~socket.socket.getsockname()` +- :meth:`~socket.socket.getsockopt()`, :meth:`~socket.socket.setsockopt()` +- :meth:`~socket.socket.gettimeout()`, :meth:`~socket.socket.settimeout()`, + :meth:`~socket.socket.setblocking()` +- :meth:`~socket.socket.listen()` +- :meth:`~socket.socket.makefile()` +- :meth:`~socket.socket.recv()`, :meth:`~socket.socket.recv_into()` + (but passing a non-zero ``flags`` argument is not allowed) +- :meth:`~socket.socket.send()`, :meth:`~socket.socket.sendall()` (with + the same limitation) +- :meth:`~socket.socket.shutdown()` + +However, since the SSL (and TLS) protocol has its own framing atop +of TCP, the SSL sockets abstraction can, in certain respects, diverge from +the specification of normal, OS-level sockets. See especially the +:ref:`notes on non-blocking sockets <ssl-nonblocking>`. + +SSL sockets also have the following additional methods and attributes: .. method:: SSLSocket.do_handshake() - Performs the SSL setup handshake. If the socket is non-blocking, this method - may raise :exc:`SSLError` with the value of the exception instance's - ``args[0]`` being either :const:`SSL_ERROR_WANT_READ` or - :const:`SSL_ERROR_WANT_WRITE`, and should be called again until it stops - raising those exceptions. Here's an example of how to do that:: - - while True: - try: - sock.do_handshake() - break - except ssl.SSLError as err: - if err.args[0] == ssl.SSL_ERROR_WANT_READ: - select.select([sock], [], []) - elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE: - select.select([], [sock], []) - else: - raise + Perform the SSL setup handshake. .. method:: SSLSocket.getpeercert(binary_form=False) @@ -291,11 +429,9 @@ SSLSocket Objects certificate was not validated, the dict is empty. If the certificate was validated, it returns a dict with the keys ``subject`` (the principal for which the certificate was issued), and ``notAfter`` (the time after which the - certificate should not be trusted). The certificate was already validated, - so the ``notBefore`` and ``issuer`` fields are not returned. If a - certificate contains an instance of the *Subject Alternative Name* extension - (see :rfc:`3280`), there will also be a ``subjectAltName`` key in the - dictionary. + certificate should not be trusted). If a certificate contains an instance + of the *Subject Alternative Name* extension (see :rfc:`3280`), there will + also be a ``subjectAltName`` key in the dictionary. The "subject" field is a tuple containing the sequence of relative distinguished names (RDNs) given in the certificate's data structure for the @@ -317,6 +453,10 @@ SSLSocket Objects been validated, but if :const:`CERT_NONE` was used to establish the connection, the certificate, if present, will not have been validated. + .. versionchanged:: 3.2 + The returned dictionary includes additional items such as ``issuer`` + and ``notBefore``. + .. method:: SSLSocket.cipher() Returns a three-value tuple containing the name of the cipher being used, the @@ -332,6 +472,142 @@ SSLSocket Objects returned socket should always be used for further communication with the other side of the connection, rather than the original socket. + +.. attribute:: SSLSocket.context + + The :class:`SSLContext` object this SSL socket is tied to. If the SSL + socket was created using the top-level :func:`wrap_socket` function + (rather than :meth:`SSLContext.wrap_socket`), this is a custom context + object created for this SSL socket. + + .. versionadded:: 3.2 + + +SSL Contexts +------------ + +.. versionadded:: 3.2 + +An SSL context holds various data longer-lived than single SSL connections, +such as SSL configuration options, certificate(s) and private key(s). +It also manages a cache of SSL sessions for server-side sockets, in order +to speed up repeated connections from the same clients. + +.. class:: SSLContext(protocol) + + Create a new SSL context. You must pass *protocol* which must be one + of the ``PROTOCOL_*`` constants defined in this module. + :data:`PROTOCOL_SSLv23` is recommended for maximum interoperability. + + +:class:`SSLContext` objects have the following methods and attributes: + +.. method:: SSLContext.load_cert_chain(certfile, keyfile=None) + + Load a private key and the corresponding certificate. The *certfile* + string must be the path to a single file in PEM format containing the + certificate as well as any number of CA certificates needed to establish + the certificate's authenticity. The *keyfile* string, if present, must + point to a file containing the private key in. Otherwise the private + key will be taken from *certfile* as well. See the discussion of + :ref:`ssl-certificates` for more information on how the certificate + is stored in the *certfile*. + + An :class:`SSLError` is raised if the private key doesn't + match with the certificate. + +.. method:: SSLContext.load_verify_locations(cafile=None, capath=None) + + Load a set of "certification authority" (CA) certificates used to validate + other peers' certificates when :data:`verify_mode` is other than + :data:`CERT_NONE`. At least one of *cafile* or *capath* must be specified. + + The *cafile* string, if present, is the path to a file of concatenated + CA certificates in PEM format. See the discussion of + :ref:`ssl-certificates` for more information about how to arrange the + certificates in this file. + + The *capath* string, if present, is + the path to a directory containing several CA certificates in PEM format, + following an `OpenSSL specific layout + <http://www.openssl.org/docs/ssl/SSL_CTX_load_verify_locations.html>`_. + +.. method:: SSLContext.set_default_verify_paths() + + Load a set of default "certification authority" (CA) certificates from + a filesystem path defined when building the OpenSSL library. Unfortunately, + there's no easy way to know whether this method succeeds: no error is + returned if no certificates are to be found. When the OpenSSL library is + provided as part of the operating system, though, it is likely to be + configured properly. + +.. method:: SSLContext.set_ciphers(ciphers) + + Set the available ciphers for sockets created with this context. + It should be a string in the `OpenSSL cipher list format + <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`_. + If no cipher can be selected (because compile-time options or other + configuration forbids use of all the specified ciphers), an + :class:`SSLError` will be raised. + + .. note:: + when connected, the :meth:`SSLSocket.cipher` method of SSL sockets will + give the currently selected cipher. + +.. method:: SSLContext.wrap_socket(sock, server_side=False, \ + do_handshake_on_connect=True, suppress_ragged_eofs=True, \ + server_hostname=None) + + Wrap an existing Python socket *sock* and return an :class:`SSLSocket` + object. The SSL socket is tied to the context, its settings and + certificates. The parameters *server_side*, *do_handshake_on_connect* + and *suppress_ragged_eofs* have the same meaning as in the top-level + :func:`wrap_socket` function. + + On client connections, the optional parameter *server_hostname* specifies + the hostname of the service which we are connecting to. This allows a + single server to host multiple SSL-based services with distinct certificates, + quite similarly to HTTP virtual hosts. Specifying *server_hostname* + will raise a :exc:`ValueError` if the OpenSSL library doesn't have support + for it (that is, if :data:`HAS_SNI` is :const:`False`). Specifying + *server_hostname* will also raise a :exc:`ValueError` if *server_side* + is true. + +.. method:: SSLContext.session_stats() + + Get statistics about the SSL sessions created or managed by this context. + A dictionary is returned which maps the names of each `piece of information + <http://www.openssl.org/docs/ssl/SSL_CTX_sess_number.html>`_ to their + numeric values. For example, here is the total number of hits and misses + in the session cache since the context was created:: + + >>> stats = context.session_stats() + >>> stats['hits'], stats['misses'] + (0, 0) + +.. attribute:: SSLContext.options + + An integer representing the set of SSL options enabled on this context. + The default value is :data:`OP_ALL`, but you can specify other options + such as :data:`OP_NO_SSLv2` by ORing them together. + + .. note:: + With versions of OpenSSL older than 0.9.8m, it is only possible + to set options, not to clear them. Attempting to clear an option + (by resetting the corresponding bits) will raise a ``ValueError``. + +.. attribute:: SSLContext.protocol + + The protocol version chosen when constructing the context. This attribute + is read-only. + +.. attribute:: SSLContext.verify_mode + + Whether to try to verify other peers' certificates and how to behave + if verification fails. This attribute must be one of + :data:`CERT_NONE`, :data:`CERT_OPTIONAL` or :data:`CERT_REQUIRED`. + + .. index:: single: certificates .. index:: single: X509 certificate @@ -377,6 +653,9 @@ and a footer line:: ... (certificate in base64 PEM encoding) ... -----END CERTIFICATE----- +Certificate chains +^^^^^^^^^^^^^^^^^^ + The Python files which contain certificates can contain a sequence of certificates, sometimes called a *certificate chain*. This chain should start with the specific certificate for the principal who "is" the client or server, @@ -400,6 +679,9 @@ certification authority's certificate:: ... (the root certificate for the CA's issuer)... -----END CERTIFICATE----- +CA certificates +^^^^^^^^^^^^^^^ + If you are going to require validation of the other side of the connection's certificate, you need to provide a "CA certs" file, filled with the certificate chains for each issuer you are willing to trust. Again, this file just contains @@ -419,6 +701,25 @@ peer is supposed to furnish the other certificates necessary to chain from its certificate to a root certificate. See :rfc:`4158` for more discussion of the way in which certification chains can be built. +Combined key and certificate +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Often the private key is stored in the same file as the certificate; in this +case, only the ``certfile`` parameter to :meth:`SSLContext.load_cert_chain` +and :func:`wrap_socket` needs to be passed. If the private key is stored +with the certificate, it should come before the first certificate in +the certificate chain:: + + -----BEGIN RSA PRIVATE KEY----- + ... (private key in base64 encoding) ... + -----END RSA PRIVATE KEY----- + -----BEGIN CERTIFICATE----- + ... (certificate in base64 PEM encoding) ... + -----END CERTIFICATE----- + +Self-signed certificates +^^^^^^^^^^^^^^^^^^^^^^^^ + If you are going to create a server that provides SSL-encrypted connection services, you will need to acquire a certificate for that service. There are many ways of acquiring appropriate certificates, such as buying one from a @@ -472,101 +773,270 @@ should use the following idiom:: Client-side operation ^^^^^^^^^^^^^^^^^^^^^ -This example connects to an SSL server, prints the server's address and -certificate, sends some bytes, and reads part of the response:: +This example connects to an SSL server and prints the server's certificate:: import socket, ssl, pprint s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - # require a certificate from the server ssl_sock = ssl.wrap_socket(s, ca_certs="/etc/ca_certs_file", cert_reqs=ssl.CERT_REQUIRED) - ssl_sock.connect(('www.verisign.com', 443)) - print(repr(ssl_sock.getpeername())) pprint.pprint(ssl_sock.getpeercert()) - print(pprint.pformat(ssl_sock.getpeercert())) - - # Set a simple HTTP request -- use http.client in actual code. - ssl_sock.write("""GET / HTTP/1.0\r - Host: www.verisign.com\r\n\r\n""") - - # Read a chunk of data. Will not necessarily - # read all the data returned by the server. - data = ssl_sock.read() - # note that closing the SSLSocket will also close the underlying socket ssl_sock.close() -As of September 6, 2007, the certificate printed by this program looked like +As of January 6, 2012, the certificate printed by this program looks like this:: - {'notAfter': 'May 8 23:59:59 2009 GMT', - 'subject': ((('serialNumber', '2497886'),), - (('1.3.6.1.4.1.311.60.2.1.3', 'US'),), - (('1.3.6.1.4.1.311.60.2.1.2', 'Delaware'),), - (('countryName', 'US'),), - (('postalCode', '94043'),), - (('stateOrProvinceName', 'California'),), - (('localityName', 'Mountain View'),), - (('streetAddress', '487 East Middlefield Road'),), - (('organizationName', 'VeriSign, Inc.'),), - (('organizationalUnitName', - 'Production Security Services'),), - (('organizationalUnitName', - 'Terms of use at www.verisign.com/rpa (c)06'),), - (('commonName', 'www.verisign.com'),))} - -which is a fairly poorly-formed ``subject`` field. + {'issuer': ((('countryName', 'US'),), + (('organizationName', 'VeriSign, Inc.'),), + (('organizationalUnitName', 'VeriSign Trust Network'),), + (('organizationalUnitName', + 'Terms of use at https://www.verisign.com/rpa (c)06'),), + (('commonName', + 'VeriSign Class 3 Extended Validation SSL SGC CA'),)), + 'notAfter': 'May 25 23:59:59 2012 GMT', + 'notBefore': 'May 26 00:00:00 2010 GMT', + 'serialNumber': '53D2BEF924A7245E83CA01E46CAA2477', + 'subject': ((('1.3.6.1.4.1.311.60.2.1.3', 'US'),), + (('1.3.6.1.4.1.311.60.2.1.2', 'Delaware'),), + (('businessCategory', 'V1.0, Clause 5.(b)'),), + (('serialNumber', '2497886'),), + (('countryName', 'US'),), + (('postalCode', '94043'),), + (('stateOrProvinceName', 'California'),), + (('localityName', 'Mountain View'),), + (('streetAddress', '487 East Middlefield Road'),), + (('organizationName', 'VeriSign, Inc.'),), + (('organizationalUnitName', ' Production Security Services'),), + (('commonName', 'www.verisign.com'),)), + 'subjectAltName': (('DNS', 'www.verisign.com'), + ('DNS', 'verisign.com'), + ('DNS', 'www.verisign.net'), + ('DNS', 'verisign.net'), + ('DNS', 'www.verisign.mobi'), + ('DNS', 'verisign.mobi'), + ('DNS', 'www.verisign.eu'), + ('DNS', 'verisign.eu')), + 'version': 3} + +This other example first creates an SSL context, instructs it to verify +certificates sent by peers, and feeds it a set of recognized certificate +authorities (CA):: + + >>> context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + >>> context.verify_mode = ssl.CERT_REQUIRED + >>> context.load_verify_locations("/etc/ssl/certs/ca-bundle.crt") + +(it is assumed your operating system places a bundle of all CA certificates +in ``/etc/ssl/certs/ca-bundle.crt``; if not, you'll get an error and have +to adjust the location) + +When you use the context to connect to a server, :const:`CERT_REQUIRED` +validates the server certificate: it ensures that the server certificate +was signed with one of the CA certificates, and checks the signature for +correctness:: + + >>> conn = context.wrap_socket(socket.socket(socket.AF_INET)) + >>> conn.connect(("linuxfr.org", 443)) + +You should then fetch the certificate and check its fields for conformity:: + + >>> cert = conn.getpeercert() + >>> ssl.match_hostname(cert, "linuxfr.org") + +Visual inspection shows that the certificate does identify the desired service +(that is, the HTTPS host ``linuxfr.org``):: + + >>> pprint.pprint(cert) + {'issuer': ((('organizationName', 'CAcert Inc.'),), + (('organizationalUnitName', 'http://www.CAcert.org'),), + (('commonName', 'CAcert Class 3 Root'),)), + 'notAfter': 'Jun 7 21:02:24 2013 GMT', + 'notBefore': 'Jun 8 21:02:24 2011 GMT', + 'serialNumber': 'D3E9', + 'subject': ((('commonName', 'linuxfr.org'),),), + 'subjectAltName': (('DNS', 'linuxfr.org'), + ('othername', '<unsupported>'), + ('DNS', 'linuxfr.org'), + ('othername', '<unsupported>'), + ('DNS', 'dev.linuxfr.org'), + ('othername', '<unsupported>'), + ('DNS', 'prod.linuxfr.org'), + ('othername', '<unsupported>'), + ('DNS', 'alpha.linuxfr.org'), + ('othername', '<unsupported>'), + ('DNS', '*.linuxfr.org'), + ('othername', '<unsupported>')), + 'version': 3} + +Now that you are assured of its authenticity, you can proceed to talk with +the server:: + + >>> conn.sendall(b"HEAD / HTTP/1.0\r\nHost: linuxfr.org\r\n\r\n") + >>> pprint.pprint(conn.recv(1024).split(b"\r\n")) + [b'HTTP/1.1 302 Found', + b'Date: Sun, 16 May 2010 13:43:28 GMT', + b'Server: Apache/2.2', + b'Location: https://linuxfr.org/pub/', + b'Vary: Accept-Encoding', + b'Connection: close', + b'Content-Type: text/html; charset=iso-8859-1', + b'', + b''] + +See the discussion of :ref:`ssl-security` below. + Server-side operation ^^^^^^^^^^^^^^^^^^^^^ -For server operation, typically you'd need to have a server certificate, and -private key, each in a file. You'd open a socket, bind it to a port, call -:meth:`listen` on it, then start waiting for clients to connect:: +For server operation, typically you'll need to have a server certificate, and +private key, each in a file. You'll first create a context holding the key +and the certificate, so that clients can check your authenticity. Then +you'll open a socket, bind it to a port, call :meth:`listen` on it, and start +waiting for clients to connect:: import socket, ssl + context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context.load_cert_chain(certfile="mycertfile", keyfile="mykeyfile") + bindsocket = socket.socket() bindsocket.bind(('myaddr.mydomain.com', 10023)) bindsocket.listen(5) -When one did, you'd call :meth:`accept` on the socket to get the new socket from -the other end, and use :func:`wrap_socket` to create a server-side SSL context -for it:: +When a client connects, you'll call :meth:`accept` on the socket to get the +new socket from the other end, and use the context's :meth:`SSLContext.wrap_socket` +method to create a server-side SSL socket for the connection:: while True: newsocket, fromaddr = bindsocket.accept() - connstream = ssl.wrap_socket(newsocket, - server_side=True, - certfile="mycertfile", - keyfile="mykeyfile", - ssl_version=ssl.PROTOCOL_TLSv1) + connstream = context.wrap_socket(newsocket, server_side=True) try: deal_with_client(connstream) finally: connstream.shutdown(socket.SHUT_RDWR) connstream.close() -Then you'd read data from the ``connstream`` and do something with it till you +Then you'll read data from the ``connstream`` and do something with it till you are finished with the client (or the client is finished with you):: def deal_with_client(connstream): - data = connstream.read() - # null data means the client is finished with us + data = connstream.recv(1024) + # empty data means the client is finished with us while data: if not do_something(connstream, data): # we'll assume do_something returns False # when we're finished with client break - data = connstream.read() + data = connstream.recv(1024) # finished with client -And go back to listening for new client connections. +And go back to listening for new client connections (of course, a real server +would probably handle each client connection in a separate thread, or put +the sockets in non-blocking mode and use an event loop). + + +.. _ssl-nonblocking: + +Notes on non-blocking sockets +----------------------------- + +When working with non-blocking sockets, there are several things you need +to be aware of: + +- Calling :func:`~select.select` tells you that the OS-level socket can be + read from (or written to), but it does not imply that there is sufficient + data at the upper SSL layer. For example, only part of an SSL frame might + have arrived. Therefore, you must be ready to handle :meth:`SSLSocket.recv` + and :meth:`SSLSocket.send` failures, and retry after another call to + :func:`~select.select`. + + (of course, similar provisions apply when using other primitives such as + :func:`~select.poll`) + +- The SSL handshake itself will be non-blocking: the + :meth:`SSLSocket.do_handshake` method has to be retried until it returns + successfully. Here is a synopsis using :func:`~select.select` to wait for + the socket's readiness:: + + while True: + try: + sock.do_handshake() + break + except ssl.SSLError as err: + if err.args[0] == ssl.SSL_ERROR_WANT_READ: + select.select([sock], [], []) + elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE: + select.select([], [sock], []) + else: + raise + + +.. _ssl-security: + +Security considerations +----------------------- + +Verifying certificates +^^^^^^^^^^^^^^^^^^^^^^ + +:const:`CERT_NONE` is the default. Since it does not authenticate the other +peer, it can be insecure, especially in client mode where most of time you +would like to ensure the authenticity of the server you're talking to. +Therefore, when in client mode, it is highly recommended to use +:const:`CERT_REQUIRED`. However, it is in itself not sufficient; you also +have to check that the server certificate, which can be obtained by calling +:meth:`SSLSocket.getpeercert`, matches the desired service. For many +protocols and applications, the service can be identified by the hostname; +in this case, the :func:`match_hostname` function can be used. + +In server mode, if you want to authenticate your clients using the SSL layer +(rather than using a higher-level authentication mechanism), you'll also have +to specify :const:`CERT_REQUIRED` and similarly check the client certificate. + + .. note:: + + In client mode, :const:`CERT_OPTIONAL` and :const:`CERT_REQUIRED` are + equivalent unless anonymous ciphers are enabled (they are disabled + by default). + +Protocol versions +^^^^^^^^^^^^^^^^^ + +SSL version 2 is considered insecure and is therefore dangerous to use. If +you want maximum compatibility between clients and servers, it is recommended +to use :const:`PROTOCOL_SSLv23` as the protocol version and then disable +SSLv2 explicitly using the :data:`SSLContext.options` attribute:: + + context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + context.options |= ssl.OP_NO_SSLv2 + +The SSL context created above will allow SSLv3 and TLSv1 connections, but +not SSLv2. + +Cipher selection +^^^^^^^^^^^^^^^^ + +If you have advanced security requirements, fine-tuning of the ciphers +enabled when negotiating a SSL session is possible through the +:meth:`SSLContext.set_ciphers` method. Starting from Python 3.2.3, the +ssl module disables certain weak ciphers by default, but you may want +to further restrict the cipher choice. For example:: + + context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context.set_ciphers('HIGH:!aNULL:!eNULL') + +The ``!aNULL:!eNULL`` part of the cipher spec is necessary to disable ciphers +which don't provide both encryption and authentication. Be sure to read +OpenSSL's documentation about the `cipher list +format <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`_. +If you want to check which ciphers are enabled by a given cipher list, +use the ``openssl ciphers`` command on your system. .. seealso:: @@ -574,8 +1044,8 @@ And go back to listening for new client connections. Class :class:`socket.socket` Documentation of underlying :mod:`socket` class - `Introducing SSL and Certificates using OpenSSL <http://old.pseudonym.org/ssl/wwwj-index.html>`_ - Frederick J. Hirsch + `TLS (Transport Layer Security) and SSL (Secure Socket Layer) <http://www3.rad.com/networks/applications/secure/tls.htm>`_ + Debby Koren `RFC 1422: Privacy Enhancement for Internet Electronic Mail: Part II: Certificate-Based Key Management <http://www.ietf.org/rfc/rfc1422>`_ Steve Kent @@ -585,3 +1055,6 @@ And go back to listening for new client connections. `RFC 3280: Internet X.509 Public Key Infrastructure Certificate and CRL Profile <http://www.ietf.org/rfc/rfc3280>`_ Housley et. al. + + `RFC 4366: Transport Layer Security (TLS) Extensions <http://www.ietf.org/rfc/rfc4366>`_ + Blake-Wilson et. al. diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index e8ed362cc3..7de98b6581 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -6,11 +6,14 @@ os.lstat() and os.fstat(). .. sectionauthor:: Skip Montanaro <skip@automatrix.com> +**Source code:** :source:`Lib/stat.py` + +-------------- The :mod:`stat` module defines constants and functions for interpreting the results of :func:`os.stat`, :func:`os.fstat` and :func:`os.lstat` (if they -exist). For complete details about the :cfunc:`stat`, :cfunc:`fstat` and -:cfunc:`lstat` calls, consult the documentation for your system. +exist). For complete details about the :c:func:`stat`, :c:func:`fstat` and +:c:func:`lstat` calls, consult the documentation for your system. The :mod:`stat` module defines the following functions to test for specific file types: @@ -68,7 +71,7 @@ mode: Normally, you would use the :func:`os.path.is\*` functions for testing the type of a file; the functions here are useful when you are doing multiple tests of -the same file and wish to avoid the overhead of the :cfunc:`stat` system call +the same file and wish to avoid the overhead of the :c:func:`stat` system call for each test. These are also useful when checking for information about a file that isn't handled by :mod:`os.path`, like the tests for block and character devices. @@ -84,7 +87,7 @@ Example:: for f in os.listdir(top): pathname = os.path.join(top, f) - mode = os.stat(pathname)[ST_MODE] + mode = os.stat(pathname).st_mode if S_ISDIR(mode): # It's a directory, recurse into it walktree(pathname, callback) @@ -304,13 +307,21 @@ The following flags can be used in the *flags* argument of :func:`os.chflags`: The file may only be appended to. +.. data:: UF_OPAQUE + + The directory is opaque when viewed through a union stack. + .. data:: UF_NOUNLINK The file may not be renamed or deleted. -.. data:: UF_OPAQUE +.. data:: UF_COMPRESSED - The directory is opaque when viewed through a union stack. + The file is stored compressed (Mac OS X 10.6+). + +.. data:: UF_HIDDEN + + The file should not be displayed in a GUI (Mac OS X 10.5+). .. data:: SF_ARCHIVED diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 2e3e18feac..5f5d3b6ce7 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -54,7 +54,7 @@ following values are considered false: * instances of user-defined classes, if the class defines a :meth:`__bool__` or :meth:`__len__` method, when that method returns the integer zero or - :class:`bool` value ``False``. [#]_ + :class:`bool` value ``False``. [1]_ .. index:: single: true @@ -168,8 +168,9 @@ Objects of different types, except different numeric types, never compare equal. Furthermore, some types (for example, function objects) support only a degenerate notion of comparison where any two objects of that type are unequal. The ``<``, ``<=``, ``>`` and ``>=`` operators will raise a :exc:`TypeError` exception when -any operand is a complex number, the objects are of different types that cannot -be compared, or other cases where there is no defined ordering. +comparing a complex number with another built-in numeric type, when the objects +are of different types that cannot be compared, or in other cases where there is +no defined ordering. .. index:: single: __eq__() (instance method) @@ -196,8 +197,8 @@ exception. operator: in operator: not in -Two more operations with the same syntactic priority, ``in`` and ``not in``, are -supported only by sequence types (below). +Two more operations with the same syntactic priority, :keyword:`in` and +:keyword:`not in`, are supported only by sequence types (below). .. _typesnumeric: @@ -216,7 +217,7 @@ Numeric Types --- :class:`int`, :class:`float`, :class:`complex` There are three distinct numeric types: :dfn:`integers`, :dfn:`floating point numbers`, and :dfn:`complex numbers`. In addition, Booleans are a subtype of integers. Integers have unlimited precision. Floating point -numbers are usually implemented using :ctype:`double` in C; information +numbers are usually implemented using :c:type:`double` in C; information about the precision and internal representation of floating point numbers for the machine on which your program is running is available in :data:`sys.float_info`. Complex numbers have a real and imaginary @@ -260,7 +261,7 @@ Python fully supports mixed arithmetic: when a binary arithmetic operator has operands of different numeric types, the operand with the "narrower" type is widened to that of the other, where integer is narrower than floating point, which is narrower than complex. Comparisons between numbers of mixed type use -the same rule. [#]_ The constructors :func:`int`, :func:`float`, and +the same rule. [2]_ The constructors :func:`int`, :func:`float`, and :func:`complex` can be used to produce numbers of a specific type. All numeric types (except complex) support the following operations, sorted by @@ -378,12 +379,12 @@ modules. .. _bitstring-ops: -Bit-string Operations on Integer Types +Bitwise Operations on Integer Types -------------------------------------- .. index:: triple: operations on; integer; types - pair: bit-string; operations + pair: bitwise; operations pair: shifting; operations pair: masking; operations operator: ^ @@ -391,15 +392,15 @@ Bit-string Operations on Integer Types operator: << operator: >> -Integers support additional operations that make sense only for bit-strings. -Negative numbers are treated as their 2's complement value (this assumes a -sufficiently large number of bits that no overflow occurs during the operation). +Bitwise operations only make sense for integers. Negative numbers are treated +as their 2's complement value (this assumes a sufficiently large number of bits +that no overflow occurs during the operation). The priorities of the binary bitwise operations are all lower than the numeric operations and higher than the comparisons; the unary operation ``~`` has the same priority as the other unary numeric operations (``+`` and ``-``). -This table lists the bit-string operations sorted in ascending priority +This table lists the bitwise operations sorted in ascending priority (operations in the same box have the same priority): +------------+--------------------------------+----------+ @@ -467,6 +468,69 @@ class`. In addition, it provides one more method: .. versionadded:: 3.1 +.. method:: int.to_bytes(length, byteorder, \*, signed=False) + + Return an array of bytes representing an integer. + + >>> (1024).to_bytes(2, byteorder='big') + b'\x04\x00' + >>> (1024).to_bytes(10, byteorder='big') + b'\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00' + >>> (-1024).to_bytes(10, byteorder='big', signed=True) + b'\xff\xff\xff\xff\xff\xff\xff\xff\xfc\x00' + >>> x = 1000 + >>> x.to_bytes((x.bit_length() // 8) + 1, byteorder='little') + b'\xe8\x03' + + The integer is represented using *length* bytes. An :exc:`OverflowError` + is raised if the integer is not representable with the given number of + bytes. + + The *byteorder* argument determines the byte order used to represent the + integer. If *byteorder* is ``"big"``, the most significant byte is at the + beginning of the byte array. If *byteorder* is ``"little"``, the most + significant byte is at the end of the byte array. To request the native + byte order of the host system, use :data:`sys.byteorder` as the byte order + value. + + The *signed* argument determines whether two's complement is used to + represent the integer. If *signed* is ``False`` and a negative integer is + given, an :exc:`OverflowError` is raised. The default value for *signed* + is ``False``. + + .. versionadded:: 3.2 + +.. classmethod:: int.from_bytes(bytes, byteorder, \*, signed=False) + + Return the integer represented by the given array of bytes. + + >>> int.from_bytes(b'\x00\x10', byteorder='big') + 16 + >>> int.from_bytes(b'\x00\x10', byteorder='little') + 4096 + >>> int.from_bytes(b'\xfc\x00', byteorder='big', signed=True) + -1024 + >>> int.from_bytes(b'\xfc\x00', byteorder='big', signed=False) + 64512 + >>> int.from_bytes([255, 0, 0], byteorder='big') + 16711680 + + The argument *bytes* must either support the buffer protocol or be an + iterable producing bytes. :class:`bytes` and :class:`bytearray` are + examples of built-in objects that support the buffer protocol. + + The *byteorder* argument determines the byte order used to represent the + integer. If *byteorder* is ``"big"``, the most significant byte is at the + beginning of the byte array. If *byteorder* is ``"little"``, the most + significant byte is at the end of the byte array. To request the native + byte order of the host system, use :data:`sys.byteorder` as the byte order + value. + + The *signed* argument indicates whether two's complement is used to + represent the integer. + + .. versionadded:: 3.2 + Additional Methods on Float --------------------------- @@ -552,6 +616,109 @@ hexadecimal string representing the same number:: '0x1.d380000000000p+11' +.. _numeric-hash: + +Hashing of numeric types +------------------------ + +For numbers ``x`` and ``y``, possibly of different types, it's a requirement +that ``hash(x) == hash(y)`` whenever ``x == y`` (see the :meth:`__hash__` +method documentation for more details). For ease of implementation and +efficiency across a variety of numeric types (including :class:`int`, +:class:`float`, :class:`decimal.Decimal` and :class:`fractions.Fraction`) +Python's hash for numeric types is based on a single mathematical function +that's defined for any rational number, and hence applies to all instances of +:class:`int` and :class:`fraction.Fraction`, and all finite instances of +:class:`float` and :class:`decimal.Decimal`. Essentially, this function is +given by reduction modulo ``P`` for a fixed prime ``P``. The value of ``P`` is +made available to Python as the :attr:`modulus` attribute of +:data:`sys.hash_info`. + +.. impl-detail:: + + Currently, the prime used is ``P = 2**31 - 1`` on machines with 32-bit C + longs and ``P = 2**61 - 1`` on machines with 64-bit C longs. + +Here are the rules in detail: + + - If ``x = m / n`` is a nonnegative rational number and ``n`` is not divisible + by ``P``, define ``hash(x)`` as ``m * invmod(n, P) % P``, where ``invmod(n, + P)`` gives the inverse of ``n`` modulo ``P``. + + - If ``x = m / n`` is a nonnegative rational number and ``n`` is + divisible by ``P`` (but ``m`` is not) then ``n`` has no inverse + modulo ``P`` and the rule above doesn't apply; in this case define + ``hash(x)`` to be the constant value ``sys.hash_info.inf``. + + - If ``x = m / n`` is a negative rational number define ``hash(x)`` + as ``-hash(-x)``. If the resulting hash is ``-1``, replace it with + ``-2``. + + - The particular values ``sys.hash_info.inf``, ``-sys.hash_info.inf`` + and ``sys.hash_info.nan`` are used as hash values for positive + infinity, negative infinity, or nans (respectively). (All hashable + nans have the same hash value.) + + - For a :class:`complex` number ``z``, the hash values of the real + and imaginary parts are combined by computing ``hash(z.real) + + sys.hash_info.imag * hash(z.imag)``, reduced modulo + ``2**sys.hash_info.width`` so that it lies in + ``range(-2**(sys.hash_info.width - 1), 2**(sys.hash_info.width - + 1))``. Again, if the result is ``-1``, it's replaced with ``-2``. + + +To clarify the above rules, here's some example Python code, +equivalent to the builtin hash, for computing the hash of a rational +number, :class:`float`, or :class:`complex`:: + + + import sys, math + + def hash_fraction(m, n): + """Compute the hash of a rational number m / n. + + Assumes m and n are integers, with n positive. + Equivalent to hash(fractions.Fraction(m, n)). + + """ + P = sys.hash_info.modulus + # Remove common factors of P. (Unnecessary if m and n already coprime.) + while m % P == n % P == 0: + m, n = m // P, n // P + + if n % P == 0: + hash_ = sys.hash_info.inf + else: + # Fermat's Little Theorem: pow(n, P-1, P) is 1, so + # pow(n, P-2, P) gives the inverse of n modulo P. + hash_ = (abs(m) % P) * pow(n, P - 2, P) % P + if m < 0: + hash_ = -hash_ + if hash_ == -1: + hash_ = -2 + return hash_ + + def hash_float(x): + """Compute the hash of a float x.""" + + if math.isnan(x): + return sys.hash_info.nan + elif math.isinf(x): + return sys.hash_info.inf if x > 0 else -sys.hash_info.inf + else: + return hash_fraction(*x.as_integer_ratio()) + + def hash_complex(z): + """Compute the hash of a complex number z.""" + + hash_ = hash_float(z.real) + sys.hash_info.imag * hash_float(z.imag) + # do a signed reduction modulo 2**sys.hash_info.width + M = 2**(sys.hash_info.width - 1) + hash_ = (hash_ & (M - 1)) - (hash & M) + if hash_ == -1: + hash_ == -2 + return hash_ + .. _typeiter: Iterator Types @@ -657,22 +824,20 @@ constructor, :func:`bytes`, and from literals; use a ``b`` prefix with normal string syntax: ``b'xyzzy'``. To construct byte arrays, use the :func:`bytearray` function. -.. warning:: - - While string objects are sequences of characters (represented by strings of - length 1), bytes and bytearray objects are sequences of *integers* (between 0 - and 255), representing the ASCII value of single bytes. That means that for - a bytes or bytearray object *b*, ``b[0]`` will be an integer, while - ``b[0:1]`` will be a bytes or bytearray object of length 1. The - representation of bytes objects uses the literal format (``b'...'``) since it - is generally more useful than e.g. ``bytes([50, 19, 100])``. You can always - convert a bytes object into a list of integers using ``list(b)``. - - Also, while in previous Python versions, byte strings and Unicode strings - could be exchanged for each other rather freely (barring encoding issues), - strings and bytes are now completely separate concepts. There's no implicit - en-/decoding if you pass an object of the wrong type. A string always - compares unequal to a bytes or bytearray object. +While string objects are sequences of characters (represented by strings of +length 1), bytes and bytearray objects are sequences of *integers* (between 0 +and 255), representing the ASCII value of single bytes. That means that for +a bytes or bytearray object *b*, ``b[0]`` will be an integer, while +``b[0:1]`` will be a bytes or bytearray object of length 1. The +representation of bytes objects uses the literal format (``b'...'``) since it +is generally more useful than e.g. ``bytes([50, 19, 100])``. You can always +convert a bytes object into a list of integers using ``list(b)``. + +Also, while in previous Python versions, byte strings and Unicode strings +could be exchanged for each other rather freely (barring encoding issues), +strings and bytes are now completely separate concepts. There's no implicit +en-/decoding if you pass an object of the wrong type. A string always +compares unequal to a bytes or bytearray object. Lists are constructed with square brackets, separating items with commas: ``[a, b, c]``. Tuples are constructed by the comma operator (not within square @@ -681,17 +846,17 @@ the enclosing parentheses, such as ``a, b, c`` or ``()``. A single item tuple must have a trailing comma, such as ``(d,)``. Objects of type range are created using the :func:`range` function. They don't -support slicing, concatenation or repetition, and using ``in``, ``not in``, -:func:`min` or :func:`max` on them is inefficient. +support concatenation or repetition, and using :func:`min` or :func:`max` on +them is inefficient. Most sequence types support the following operations. The ``in`` and ``not in`` operations have the same priorities as the comparison operations. The ``+`` and ``*`` operations have the same priority as the corresponding numeric operations. -[#]_ Additional methods are provided for :ref:`typesseq-mutable`. +[3]_ Additional methods are provided for :ref:`typesseq-mutable`. This table lists the sequence operations sorted in ascending priority (operations in the same box have the same priority). In the table, *s* and *t* -are sequences of the same type; *n*, *i* and *j* are integers: +are sequences of the same type; *n*, *i*, *j* and *k* are integers. +------------------+--------------------------------+----------+ | Operation | Result | Notes | @@ -708,7 +873,7 @@ are sequences of the same type; *n*, *i* and *j* are integers: | ``s * n, n * s`` | *n* shallow copies of *s* | \(2) | | | concatenated | | +------------------+--------------------------------+----------+ -| ``s[i]`` | *i*'th item of *s*, origin 0 | \(3) | +| ``s[i]`` | *i*\ th item of *s*, origin 0 | \(3) | +------------------+--------------------------------+----------+ | ``s[i:j]`` | slice of *s* from *i* to *j* | (3)(4) | +------------------+--------------------------------+----------+ @@ -799,15 +964,18 @@ Notes: If *k* is ``None``, it is treated like ``1``. (6) - .. impl-detail:: + Concatenating immutable strings always results in a new object. This means + that building up a string by repeated concatenation will have a quadratic + runtime cost in the total string length. To get a linear runtime cost, + you must switch to one of the alternatives below: - If *s* and *t* are both strings, some Python implementations such as - CPython can usually perform an in-place optimization for assignments of - the form ``s = s + t`` or ``s += t``. When applicable, this optimization - makes quadratic run-time much less likely. This optimization is both - version and implementation dependent. For performance sensitive code, it - is preferable to use the :meth:`str.join` method which assures consistent - linear concatenation performance across versions and implementations. + * if concatenating :class:`str` objects, you can build a list and use + :meth:`str.join` at the end; + + * if concatenating :class:`bytes` objects, you can similarly use + :meth:`bytes.join`, or you can do in-place concatenation with a + :class:`bytearray` object. :class:`bytearray` objects are mutable and + have an efficient overallocation mechanism. .. _string-methods: @@ -817,11 +985,10 @@ String Methods .. index:: pair: string; methods -String objects support the methods listed below. Note that none of these -methods take keyword arguments. +String objects support the methods listed below. -In addition, Python's strings support the sequence type methods described in -the :ref:`typesseq` section. To output formatted strings, see the +In addition, Python's strings support the sequence type methods described in the +:ref:`typesseq` section. To output formatted strings, see the :ref:`string-formatting` section. Also, see the :mod:`re` module for string functions based on regular expressions. @@ -844,12 +1011,12 @@ functions based on regular expressions. interpreted as in slice notation. -.. method:: str.encode([encoding[, errors]]) +.. method:: str.encode(encoding="utf-8", errors="strict") - Return an encoded version of the string as a bytes object. Default encoding - is the current default string encoding. *errors* may be given to set a - different error handling scheme. The default for *errors* is ``'strict'``, - meaning that encoding errors raise a :exc:`UnicodeError`. Other possible + Return an encoded version of the string as a bytes object. Default encoding + is ``'utf-8'``. *errors* may be given to set a different error handling scheme. + The default for *errors* is ``'strict'``, meaning that encoding errors raise + a :exc:`UnicodeError`. Other possible values are ``'ignore'``, ``'replace'``, ``'xmlcharrefreplace'``, ``'backslashreplace'`` and any other name registered via :func:`codecs.register_error`, see section :ref:`codec-base-classes`. For a @@ -869,7 +1036,7 @@ functions based on regular expressions. .. method:: str.expandtabs([tabsize]) - Return a copy of the string where all tab characters are replaced by one or + Return a copy of the string where all tab characters are replaced by zero or more spaces, depending on the current column and the given tab size. The column number is reset to zero after each newline occurring in the string. If *tabsize* is not given, a tab size of ``8`` characters is assumed. This @@ -895,12 +1062,12 @@ functions based on regular expressions. .. method:: str.format(*args, **kwargs) - Perform a string formatting operation. The *format_string* argument can - contain literal text or replacement fields delimited by braces ``{}``. Each - replacement field contains either the numeric index of a positional argument, - or the name of a keyword argument. Returns a copy of *format_string* where - each replacement field is replaced with the string value of the corresponding - argument. + Perform a string formatting operation. The string on which this method is + called can contain literal text or replacement fields delimited by braces + ``{}``. Each replacement field contains either the numeric index of a + positional argument, or the name of a keyword argument. Returns a copy of + the string where each replacement field is replaced with the string value of + the corresponding argument. >>> "The sum of 1 + 2 is {0}".format(1+2) 'The sum of 1 + 2 is 3' @@ -909,6 +1076,22 @@ functions based on regular expressions. that can be specified in format strings. +.. method:: str.format_map(mapping) + + Similar to ``str.format(**mapping)``, except that ``mapping`` is + used directly and not copied to a :class:`dict` . This is useful + if for example ``mapping`` is a dict subclass: + + >>> class Default(dict): + ... def __missing__(self, key): + ... return key + ... + >>> '{name} was born in {country}'.format_map(Default(name='Guido')) + 'Guido was born in country' + + .. versionadded:: 3.2 + + .. method:: str.index(sub[, start[, end]]) Like :meth:`find`, but raise :exc:`ValueError` when the substring is not found. @@ -937,7 +1120,7 @@ functions based on regular expressions. characters and there is at least one character, false otherwise. Decimal characters are those from general category "Nd". This category includes digit characters, and all characters - that that can be used to form decimal-radix numbers, e.g. U+0660, + that can be used to form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT ZERO. @@ -957,10 +1140,8 @@ functions based on regular expressions. .. method:: str.islower() - Return true if all cased characters in the string are lowercase and there is at - least one cased character, false otherwise. Cased characters are those with - general category property being one of "Lu", "Ll", or "Lt" and lowercase characters - are those with general category property "Ll". + Return true if all cased characters [4]_ in the string are lowercase and + there is at least one cased character, false otherwise. .. method:: str.isnumeric() @@ -1000,17 +1181,15 @@ functions based on regular expressions. .. method:: str.isupper() - Return true if all cased characters in the string are uppercase and there is at - least one cased character, false otherwise. Cased characters are those with - general category property being one of "Lu", "Ll", or "Lt" and uppercase characters - are those with general category property "Lu". + Return true if all cased characters [4]_ in the string are uppercase and + there is at least one cased character, false otherwise. .. method:: str.join(iterable) Return a string which is the concatenation of the strings in the :term:`iterable` *iterable*. A :exc:`TypeError` will be raised if there are - any non-string values in *seq*, including :class:`bytes` objects. The + any non-string values in *iterable*, including :class:`bytes` objects. The separator between elements is the string providing this method. @@ -1018,12 +1197,13 @@ functions based on regular expressions. Return the string left justified in a string of length *width*. Padding is done using the specified *fillchar* (default is a space). The original string is - returned if *width* is less than ``len(s)``. + returned if *width* is less than or equal to ``len(s)``. .. method:: str.lower() - Return a copy of the string converted to lowercase. + Return a copy of the string with all the cased characters [4]_ converted to + lowercase. .. method:: str.lstrip([chars]) @@ -1086,7 +1266,7 @@ functions based on regular expressions. Return the string right justified in a string of length *width*. Padding is done using the specified *fillchar* (default is a space). The original string is - returned if *width* is less than ``len(s)``. + returned if *width* is less than or equal to ``len(s)``. .. method:: str.rpartition(sep) @@ -1224,14 +1404,17 @@ functions based on regular expressions. .. method:: str.upper() - Return a copy of the string converted to uppercase. + Return a copy of the string with all the cased characters [4]_ converted to + uppercase. Note that ``str.upper().isupper()`` might be ``False`` if ``s`` + contains uncased characters or if the Unicode category of the resulting + character(s) is not "Lu" (Letter, uppercase), but e.g. "Lt" (Letter, titlecase). .. method:: str.zfill(width) Return the numeric string left filled with zeros in a string of length *width*. A sign prefix is handled correctly. The original string is - returned if *width* is less than ``len(s)``. + returned if *width* is less than or equal to ``len(s)``. @@ -1261,10 +1444,10 @@ String objects have one unique built-in operation: the ``%`` operator (modulo). This is also known as the string *formatting* or *interpolation* operator. Given ``format % values`` (where *format* is a string), ``%`` conversion specifications in *format* are replaced with zero or more elements of *values*. -The effect is similar to the using :cfunc:`sprintf` in the C language. +The effect is similar to the using :c:func:`sprintf` in the C language. If *format* requires a single argument, *values* may be a single non-tuple -object. [#]_ Otherwise, *values* must be a tuple with exactly the number of +object. [5]_ Otherwise, *values* must be a tuple with exactly the number of items specified by the format string, or a single mapping object (for example, a dictionary). @@ -1284,7 +1467,7 @@ components, which must occur in this order: object to convert comes after the minimum field width and optional precision. #. Precision (optional), given as a ``'.'`` (dot) followed by the precision. If - specified as ``'*'`` (an asterisk), the actual width is read from the next + specified as ``'*'`` (an asterisk), the actual precision is read from the next element of the tuple in *values*, and the value to convert comes after the precision. @@ -1366,9 +1549,12 @@ The conversion types are: | ``'r'`` | String (converts any Python object using | \(5) | | | :func:`repr`). | | +------------+-----------------------------------------------------+-------+ -| ``'s'`` | String (converts any Python object using | | +| ``'s'`` | String (converts any Python object using | \(5) | | | :func:`str`). | | +------------+-----------------------------------------------------+-------+ +| ``'a'`` | String (converts any Python object using | \(5) | +| | :func:`ascii`). | | ++------------+-----------------------------------------------------+-------+ | ``'%'`` | No argument is converted, results in a ``'%'`` | | | | character in the result. | | +------------+-----------------------------------------------------+-------+ @@ -1401,7 +1587,7 @@ Notes: decimal point and defaults to 6. (5) - The precision determines the maximal number of characters used. + If precision is ``N``, the output is truncated to ``N`` characters. (7) @@ -1434,11 +1620,23 @@ Range Type The :class:`range` type is an immutable sequence which is commonly used for looping. The advantage of the :class:`range` type is that an :class:`range` object will always take the same amount of memory, no matter the size of the -range it represents. There are no consistent performance advantages. +range it represents. + +Range objects have relatively little behavior: they support indexing, contains, +iteration, the :func:`len` function, and the following methods: -Range objects have very little behavior: they only support indexing, iteration, -and the :func:`len` function. +.. method:: range.count(x) + Return the number of *i*'s for which ``s[i] == x``. + + .. versionadded:: 3.2 + +.. method:: range.index(x) + + Return the smallest *i* such that ``s[i] == x``. Raises + :exc:`ValueError` when *x* is not in the range. + + .. versionadded:: 3.2 .. _typesseq-mutable: @@ -1556,6 +1754,9 @@ Notes: *key* specifies a function of one argument that is used to extract a comparison key from each list element: ``key=str.lower``. The default value is ``None``. + Use :func:`functools.cmp_to_key` to convert an + old-style *cmp* function to a *key* function. + *reverse* is a boolean value. If set to ``True``, then the list elements are sorted as if each comparison were reversed. @@ -1607,17 +1808,20 @@ Wherever one of these methods needs to interpret the bytes as characters b = a.replace(b"a", b"f") -.. method:: bytes.decode([encoding[, errors]]) - bytearray.decode([encoding[, errors]]) +.. method:: bytes.decode(encoding="utf-8", errors="strict") + bytearray.decode(encoding="utf-8", errors="strict") - Return a string decoded from the given bytes. Default encoding is the - current default string encoding. *errors* may be given to set a different + Return a string decoded from the given bytes. Default encoding is + ``'utf-8'``. *errors* may be given to set a different error handling scheme. The default for *errors* is ``'strict'``, meaning that encoding errors raise a :exc:`UnicodeError`. Other possible values are ``'ignore'``, ``'replace'`` and any other name registered via :func:`codecs.register_error`, see section :ref:`codec-base-classes`. For a list of possible encodings, see section :ref:`standard-encodings`. + .. versionchanged:: 3.1 + Added support for keyword arguments. + The bytes and bytearray types have an additional class method: @@ -1938,8 +2142,20 @@ pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: returned or raised by the ``__missing__(key)`` call if the key is not present. No other operations or methods invoke :meth:`__missing__`. If :meth:`__missing__` is not defined, :exc:`KeyError` is raised. - :meth:`__missing__` must be a method; it cannot be an instance variable. For - an example, see :class:`collections.defaultdict`. + :meth:`__missing__` must be a method; it cannot be an instance variable:: + + >>> class Counter(dict): + ... def __missing__(self, key): + ... return 0 + >>> c = Counter() + >>> c['red'] + 0 + >>> c['red'] += 1 + >>> c['red'] + 1 + + See :class:`collections.Counter` for a complete implementation including + other methods helpful for accumulating and managing tallies. .. describe:: d[key] = value @@ -2076,7 +2292,6 @@ since the entries are generally not unique.) For set-like views, all of the operations defined for the abstract base class :class:`collections.Set` are available (for example, ``==``, ``<``, or ``^``). - An example of dictionary view usage:: >>> dishes = {'eggs': 2, 'sausage': 1, 'bacon': 1, 'spam': 500} @@ -2106,13 +2321,13 @@ An example of dictionary view usage:: >>> keys & {'eggs', 'bacon', 'salad'} {'bacon'} >>> keys ^ {'sausage', 'juice'} - {'juice', 'eggs', 'bacon', 'spam'} + {'juice', 'sausage', 'bacon', 'spam'} .. _typememoryview: -memoryview Types -================ +memoryview type +=============== :class:`memoryview` objects allow Python code to access the internal data of an object that supports the :ref:`buffer protocol <bufferobjects>` without @@ -2160,15 +2375,15 @@ copying. Memory is generally interpreted as simple bytes. bytearray(b'zbcefg') >>> v[1:4] = b'123' >>> data - bytearray(b'a123fg') + bytearray(b'z123fg') >>> v[2] = b'spam' Traceback (most recent call last): - File "<stdin>", line 1, in <module> + File "<stdin>", line 1, in <module> ValueError: cannot modify size of memoryview object Notice how the size of the memoryview object cannot be changed. - :class:`memoryview` has two methods: + :class:`memoryview` has several methods: .. method:: tobytes() @@ -2188,6 +2403,39 @@ copying. Memory is generally interpreted as simple bytes. >>> memoryview(b'abc').tolist() [97, 98, 99] + .. method:: release() + + Release the underlying buffer exposed by the memoryview object. Many + objects take special actions when a view is held on them (for example, + a :class:`bytearray` would temporarily forbid resizing); therefore, + calling release() is handy to remove these restrictions (and free any + dangling resources) as soon as possible. + + After this method has been called, any further operation on the view + raises a :class:`ValueError` (except :meth:`release()` itself which can + be called multiple times):: + + >>> m = memoryview(b'abc') + >>> m.release() + >>> m[0] + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + ValueError: operation forbidden on released memoryview object + + The context management protocol can be used for a similar effect, + using the ``with`` statement:: + + >>> with memoryview(b'abc') as m: + ... m[0] + ... + b'a' + >>> m[0] + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + ValueError: operation forbidden on released memoryview object + + .. versionadded:: 3.2 + There are also several readonly attributes available: .. attribute:: format @@ -2325,7 +2573,7 @@ statement is not, strictly speaking, an operation on a module object; ``import foo`` does not require a module object named *foo* to exist, rather it requires an (external) *definition* for a module named *foo* somewhere.) -A special member of every module is :attr:`__dict__`. This is the dictionary +A special attribute of every module is :attr:`__dict__`. This is the dictionary containing the module's symbol table. Modifying this dictionary will actually change the module's symbol table, but direct assignment to the :attr:`__dict__` attribute is not possible (you can write ``m.__dict__['a'] = 1``, which defines @@ -2467,6 +2715,20 @@ special operations. There is exactly one ellipsis object, named It is written as ``Ellipsis`` or ``...``. +.. _bltin-notimplemented-object: + +The NotImplemented Object +------------------------- + +This object is returned from comparisons and binary operations when they are +asked to operate on types they don't support. See :ref:`comparisons` for more +information. + +It is written as ``NotImplemented``. + + +.. _bltin-boolean-values: + Boolean Values -------------- @@ -2474,9 +2736,9 @@ Boolean values are the two constant objects ``False`` and ``True``. They are used to represent truth values (although other values can also be considered false or true). In numeric contexts (for example when used as the argument to an arithmetic operator), they behave like the integers 0 and 1, respectively. -The built-in function :func:`bool` can be used to cast any value to a Boolean, -if the value can be interpreted as a truth value (see section Truth Value -Testing above). +The built-in function :func:`bool` can be used to convert any value to a +Boolean, if the value can be interpreted as a truth value (see section +:ref:`truth` above). .. index:: single: False @@ -2526,8 +2788,6 @@ types, where they are relevant. Some of these are not reported by the The name of the class or type. -The following attributes are only supported by :term:`new-style class`\ es. - .. attribute:: class.__mro__ This attribute is a tuple of classes that are considered when looking for @@ -2543,23 +2803,26 @@ The following attributes are only supported by :term:`new-style class`\ es. .. method:: class.__subclasses__ - Each new-style class keeps a list of weak references to its immediate - subclasses. This method returns a list of all those references still alive. + Each class keeps a list of weak references to its immediate subclasses. This + method returns a list of all those references still alive. Example:: >>> int.__subclasses__() - [<type 'bool'>] + [<class 'bool'>] .. rubric:: Footnotes -.. [#] Additional information on these special methods may be found in the Python +.. [1] Additional information on these special methods may be found in the Python Reference Manual (:ref:`customization`). -.. [#] As a consequence, the list ``[1, 2]`` is considered equal to ``[1.0, 2.0]``, and +.. [2] As a consequence, the list ``[1, 2]`` is considered equal to ``[1.0, 2.0]``, and similarly for tuples. -.. [#] They must have since the parser can't tell the type of the operands. +.. [3] They must have since the parser can't tell the type of the operands. + +.. [4] Cased characters are those with general category property being one of + "Lu" (Letter, uppercase), "Ll" (Letter, lowercase), or "Lt" (Letter, titlecase). -.. [#] To format only a tuple you should therefore provide a singleton tuple whose only +.. [5] To format only a tuple you should therefore provide a singleton tuple whose only element is the tuple to be formatted. diff --git a/Doc/library/string.rst b/Doc/library/string.rst index b2d0bbaa4d..1f8e2304f0 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -4,6 +4,9 @@ .. module:: string :synopsis: Common string operations. +**Source code:** :source:`Lib/string.py` + +-------------- .. seealso:: @@ -208,11 +211,13 @@ by a colon ``':'``. These specify a non-default format for the replacement valu See also the :ref:`formatspec` section. -The *field_name* itself begins with an *arg_name* that is either either a number or a +The *field_name* itself begins with an *arg_name* that is either a number or a keyword. If it's a number, it refers to a positional argument, and if it's a keyword, it refers to a named keyword argument. If the numerical arg_names in a format string are 0, 1, 2, ... in sequence, they can all be omitted (not just some) and the numbers 0, 1, 2, ... will be automatically inserted in that order. +Because *arg_name* is not quote-delimited, it is not possible to specify arbitrary +dictionary keys (e.g., the strings ``'10'`` or ``':-]'``) within a format string. The *arg_name* can be followed by any number of index or attribute expressions. An expression of the form ``'.name'`` selects the named attribute using :func:`getattr`, while an expression of the form ``'[index]'`` @@ -340,9 +345,18 @@ following: | | positive numbers, and a minus sign on negative numbers. | +---------+----------------------------------------------------------+ -The ``'#'`` option is only valid for integers, and only for binary, octal, or -hexadecimal output. If present, it specifies that the output will be prefixed -by ``'0b'``, ``'0o'``, or ``'0x'``, respectively. + +The ``'#'`` option causes the "alternate form" to be used for the +conversion. The alternate form is defined differently for different +types. This option is only valid for integer, float, complex and +Decimal types. For integers, when binary, octal, or hexadecimal output +is used, this option adds the prefix respective ``'0b'``, ``'0o'``, or +``'0x'`` to the output value. For floats, complex and Decimal the +alternate form causes the result of the conversion to always contain a +decimal-point character, even if no digits follow it. Normally, a +decimal-point character appears in the result of these conversions +only if a digit follows it. In addition, for ``'g'`` and ``'G'`` +conversions, trailing zeros are not removed from the result. The ``','`` option signals the use of a comma for a thousands separator. For a locale aware separator, use the ``'n'`` integer presentation type @@ -569,7 +583,7 @@ Expressing a percentage:: >>> points = 19 >>> total = 22 - >>> 'Correct answers: {:.2%}.'.format(points/total) + >>> 'Correct answers: {:.2%}'.format(points/total) 'Correct answers: 86.36%' Using type-specific formatting:: @@ -696,15 +710,23 @@ placeholder syntax, delimiter character, or the entire regular expression used to parse template strings. To do this, you can override these class attributes: * *delimiter* -- This is the literal string describing a placeholder introducing - delimiter. The default value ``$``. Note that this should *not* be a regular - expression, as the implementation will call :meth:`re.escape` on this string as - needed. + delimiter. The default value is ``$``. Note that this should *not* be a + regular expression, as the implementation will call :meth:`re.escape` on this + string as needed. * *idpattern* -- This is the regular expression describing the pattern for non-braced placeholders (the braces will be added automatically as appropriate). The default value is the regular expression ``[_a-z][_a-z0-9]*``. +* *flags* -- The regular expression flags that will be applied when compiling + the regular expression used for recognizing substitutions. The default value + is ``re.IGNORECASE``. Note that ``re.VERBOSE`` will always be added to the + flags, so custom *idpattern*\ s must follow conventions for verbose regular + expressions. + + .. versionadded:: 3.2 + Alternatively, you can provide the entire regular expression pattern by overriding the class attribute *pattern*. If you do this, the value must be a regular expression object with four named capturing groups. The capturing @@ -727,7 +749,7 @@ rule: Helper functions ---------------- -.. function:: capwords(s[, sep]) +.. function:: capwords(s, sep=None) Split the argument into words using :meth:`str.split`, capitalize each word using :meth:`str.capitalize`, and join the capitalized words using @@ -736,12 +758,3 @@ Helper functions and leading and trailing whitespace are removed, otherwise *sep* is used to split and join the words. - -.. function:: maketrans(frm, to) - - Return a translation table suitable for passing to :meth:`bytes.translate`, - that will map each character in *from* into the character at the same - position in *to*; *from* and *to* must have the same length. - - .. deprecated:: 3.1 - Use the :meth:`bytes.maketrans` static method instead. diff --git a/Doc/library/struct.rst b/Doc/library/struct.rst index 1834cfd1af..12820e0282 100644 --- a/Doc/library/struct.rst +++ b/Doc/library/struct.rst @@ -21,8 +21,8 @@ structs and the intended conversion to/from Python values. alignment is taken into account when unpacking. This behavior is chosen so that the bytes of a packed struct correspond exactly to the layout in memory of the corresponding C struct. To handle platform-independent data formats - or omit implicit pad bytes, use `standard` size and alignment instead of - `native` size and alignment: see :ref:`struct-alignment` for details. + or omit implicit pad bytes, use ``standard`` size and alignment instead of + ``native`` size and alignment: see :ref:`struct-alignment` for details. Functions and Exceptions ------------------------ @@ -157,63 +157,66 @@ is, when the format string starts with one of ``'<'``, ``'>'``, ``'!'`` or ``'='``. When using native size, the size of the packed value is platform-dependent. -+--------+-------------------------+--------------------+----------------+------------+ -| Format | C Type | Python type | Standard size | Notes | -+========+=========================+====================+================+============+ -| ``x`` | pad byte | no value | | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``c`` | :ctype:`char` | bytes of length 1 | 1 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``b`` | :ctype:`signed char` | integer | 1 | \(1) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``B`` | :ctype:`unsigned char` | integer | 1 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``?`` | :ctype:`_Bool` | bool | 1 | \(2) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``h`` | :ctype:`short` | integer | 2 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``H`` | :ctype:`unsigned short` | integer | 2 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``i`` | :ctype:`int` | integer | 4 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``I`` | :ctype:`unsigned int` | integer | 4 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``l`` | :ctype:`long` | integer | 4 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``L`` | :ctype:`unsigned long` | integer | 4 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``q`` | :ctype:`long long` | integer | 8 | \(3) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``Q`` | :ctype:`unsigned long | integer | 8 | \(3) | -| | long` | | | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``f`` | :ctype:`float` | float | 4 | \(4) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``d`` | :ctype:`double` | float | 8 | \(4) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``s`` | :ctype:`char[]` | bytes | | \(1) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``p`` | :ctype:`char[]` | bytes | | \(1) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``P`` | :ctype:`void \*` | integer | | \(5) | -+--------+-------------------------+--------------------+----------------+------------+ ++--------+--------------------------+--------------------+----------------+------------+ +| Format | C Type | Python type | Standard size | Notes | ++========+==========================+====================+================+============+ +| ``x`` | pad byte | no value | | | ++--------+--------------------------+--------------------+----------------+------------+ +| ``c`` | :c:type:`char` | bytes of length 1 | 1 | | ++--------+--------------------------+--------------------+----------------+------------+ +| ``b`` | :c:type:`signed char` | integer | 1 | \(1),\(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``B`` | :c:type:`unsigned char` | integer | 1 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``?`` | :c:type:`_Bool` | bool | 1 | \(1) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``h`` | :c:type:`short` | integer | 2 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``H`` | :c:type:`unsigned short` | integer | 2 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``i`` | :c:type:`int` | integer | 4 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``I`` | :c:type:`unsigned int` | integer | 4 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``l`` | :c:type:`long` | integer | 4 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``L`` | :c:type:`unsigned long` | integer | 4 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``q`` | :c:type:`long long` | integer | 8 | \(2), \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``Q`` | :c:type:`unsigned long | integer | 8 | \(2), \(3) | +| | long` | | | | ++--------+--------------------------+--------------------+----------------+------------+ +| ``f`` | :c:type:`float` | float | 4 | \(4) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``d`` | :c:type:`double` | float | 8 | \(4) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``s`` | :c:type:`char[]` | bytes | | | ++--------+--------------------------+--------------------+----------------+------------+ +| ``p`` | :c:type:`char[]` | bytes | | | ++--------+--------------------------+--------------------+----------------+------------+ +| ``P`` | :c:type:`void \*` | integer | | \(5) | ++--------+--------------------------+--------------------+----------------+------------+ Notes: (1) - The ``c``, ``s`` and ``p`` conversion codes operate on :class:`bytes` - objects, but packing with such codes also supports :class:`str` objects, - which are encoded using UTF-8. + The ``'?'`` conversion code corresponds to the :c:type:`_Bool` type defined by + C99. If this type is not available, it is simulated using a :c:type:`char`. In + standard mode, it is always represented by one byte. (2) - The ``'?'`` conversion code corresponds to the :ctype:`_Bool` type defined by - C99. If this type is not available, it is simulated using a :ctype:`char`. In - standard mode, it is always represented by one byte. + The ``'q'`` and ``'Q'`` conversion codes are available in native mode only if + the platform C compiler supports C :c:type:`long long`, or, on Windows, + :c:type:`__int64`. They are always available in standard modes. (3) - The ``'q'`` and ``'Q'`` conversion codes are available in native mode only if - the platform C compiler supports C :ctype:`long long`, or, on Windows, - :ctype:`__int64`. They are always available in standard modes. + When attempting to pack a non-integer using any of the integer conversion + codes, if the non-integer has a :meth:`__index__` method then that method is + called to convert the argument to an integer before packing. + + .. versionchanged:: 3.2 + Use of the :meth:`__index__` method for non-integers is new in 3.2. (4) For the ``'f'`` and ``'d'`` conversion codes, the packed representation uses @@ -237,10 +240,11 @@ not contain whitespace though. For the ``'s'`` format character, the count is interpreted as the length of the bytes, not a repeat count like for the other format characters; for example, ``'10s'`` means a single 10-byte string, while ``'10c'`` means 10 characters. -For packing, the string is truncated or padded with null bytes as appropriate to -make it fit. For unpacking, the resulting bytes object always has exactly the -specified number of bytes. As a special case, ``'0s'`` means a single, empty -string (while ``'0c'`` means 0 characters). +If a count is not given, it defaults to 1. For packing, the string is +truncated or padded with null bytes as appropriate to make it fit. For +unpacking, the resulting bytes object always has exactly the specified number +of bytes. As a special case, ``'0s'`` means a single, empty string (while +``'0c'`` means 0 characters). When packing a value ``x`` using one of the integer formats (``'b'``, ``'B'``, ``'h'``, ``'H'``, ``'i'``, ``'I'``, ``'l'``, ``'L'``, @@ -302,9 +306,9 @@ the result in a named tuple:: The ordering of format characters may have an impact on size since the padding needed to satisfy alignment requirements is different:: - >>> pack('ci', '*', 0x12131415) + >>> pack('ci', b'*', 0x12131415) b'*\x00\x00\x00\x12\x13\x14\x15' - >>> pack('ic', 0x12131415, '*') + >>> pack('ic', 0x12131415, b'*') b'\x12\x13\x14\x15*' >>> calcsize('ci') 8 diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 932ca8b084..97c0e03183 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -25,10 +25,230 @@ modules and functions can be found in the following sections. Using the subprocess Module --------------------------- -This module defines one class called :class:`Popen`: +The recommended approach to invoking subprocesses is to use the following +convenience functions for all use cases they can handle. For more advanced +use cases, the underlying :class:`Popen` interface can be used directly. -.. class:: Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0) +.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False) + + Run the command described by *args*. Wait for command to complete, then + return the :attr:`returncode` attribute. + + The arguments shown above are merely the most common ones, described below + in :ref:`frequently-used-arguments` (hence the slightly odd notation in + the abbreviated signature). The full function signature is the same as + that of the :class:`Popen` constructor - this functions passes all + supplied arguments directly through to that interface. + + Examples:: + + >>> subprocess.call(["ls", "-l"]) + 0 + + >>> subprocess.call("exit 1", shell=True) + 1 + + .. warning:: + + Invoking the system shell with ``shell=True`` can be a security hazard + if combined with untrusted input. See the warning under + :ref:`frequently-used-arguments` for details. + + .. note:: + + Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As + the pipes are not being read in the current process, the child + process may block if it generates enough output to a pipe to fill up + the OS pipe buffer. + + +.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False) + + Run command with arguments. Wait for command to complete. If the return + code was zero then return, otherwise raise :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`returncode` attribute. + + The arguments shown above are merely the most common ones, described below + in :ref:`frequently-used-arguments` (hence the slightly odd notation in + the abbreviated signature). The full function signature is the same as + that of the :class:`Popen` constructor - this functions passes all + supplied arguments directly through to that interface. + + Examples:: + + >>> subprocess.check_call(["ls", "-l"]) + 0 + + >>> subprocess.check_call("exit 1", shell=True) + Traceback (most recent call last): + ... + subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 + + .. versionadded:: 2.5 + + .. warning:: + + Invoking the system shell with ``shell=True`` can be a security hazard + if combined with untrusted input. See the warning under + :ref:`frequently-used-arguments` for details. + + .. note:: + + Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As + the pipes are not being read in the current process, the child + process may block if it generates enough output to a pipe to fill up + the OS pipe buffer. + + +.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False) + + Run command with arguments and return its output as a byte string. + + If the return code was non-zero it raises a :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`returncode` attribute and any output in the :attr:`output` + attribute. + + The arguments shown above are merely the most common ones, described below + in :ref:`frequently-used-arguments` (hence the slightly odd notation in + the abbreviated signature). The full function signature is largely the + same as that of the :class:`Popen` constructor, except that *stdout* is + not permitted as it is used internally. All other supplied arguments are + passed directly through to the :class:`Popen` constructor. + + Examples:: + + >>> subprocess.check_output(["echo", "Hello World!"]) + b'Hello World!\n' + + >>> subprocess.check_output(["echo", "Hello World!"], universal_newlines=True) + 'Hello World!\n' + + >>> subprocess.check_output("exit 1", shell=True) + Traceback (most recent call last): + ... + subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 + + By default, this function will return the data as encoded bytes. The actual + encoding of the output data may depend on the command being invoked, so the + decoding to text will often need to be handled at the application level. + + This behaviour may be overridden by setting *universal_newlines* to + :const:`True` as described below in :ref:`frequently-used-arguments`. + + To also capture standard error in the result, use + ``stderr=subprocess.STDOUT``:: + + >>> subprocess.check_output( + ... "ls non_existent_file; exit 0", + ... stderr=subprocess.STDOUT, + ... shell=True) + 'ls: non_existent_file: No such file or directory\n' + + .. versionadded:: 2.7 + + .. warning:: + + Invoking the system shell with ``shell=True`` can be a security hazard + if combined with untrusted input. See the warning under + :ref:`frequently-used-arguments` for details. + + .. note:: + + Do not use ``stderr=PIPE`` with this function. As the pipe is not being + read in the current process, the child process may block if it + generates enough output to the pipe to fill up the OS pipe buffer. + + +.. data:: PIPE + + Special value that can be used as the *stdin*, *stdout* or *stderr* argument + to :class:`Popen` and indicates that a pipe to the standard stream should be + opened. + + +.. data:: STDOUT + + Special value that can be used as the *stderr* argument to :class:`Popen` and + indicates that standard error should go into the same handle as standard + output. + + +.. _frequently-used-arguments: + +Frequently Used Arguments +^^^^^^^^^^^^^^^^^^^^^^^^^ + +To support a wide variety of use cases, the :class:`Popen` constructor (and +the convenience functions) accept a large number of optional arguments. For +most typical use cases, many of these arguments can be safely left at their +default values. The arguments that are most commonly needed are: + + *args* is required for all calls and should be a string, or a sequence of + program arguments. Providing a sequence of arguments is generally + preferred, as it allows the module to take care of any required escaping + and quoting of arguments (e.g. to permit spaces in file names). If passing + a single string, either *shell* must be :const:`True` (see below) or else + the string must simply name the program to be executed without specifying + any arguments. + + *stdin*, *stdout* and *stderr* specify the executed program's standard input, + standard output and standard error file handles, respectively. Valid values + are :data:`PIPE`, an existing file descriptor (a positive integer), an + existing file object, and ``None``. :data:`PIPE` indicates that a new pipe + to the child should be created. With the default settings of ``None``, no + redirection will occur; the child's file handles will be inherited from the + parent. Additionally, *stderr* can be :data:`STDOUT`, which indicates that + the stderr data from the child process should be captured into the same file + handle as for stdout. + + When *stdout* or *stderr* are pipes and *universal_newlines* is + :const:`True` then the output data is assumed to be encoded as UTF-8 and + will automatically be decoded to text. All line endings will be converted + to ``'\n'`` as described for the universal newlines `'U'`` mode argument + to :func:`open`. + + If *shell* is :const:`True`, the specified command will be executed through + the shell. This can be useful if you are using Python primarily for the + enhanced control flow it offers over most system shells and still want + access to other shell features such as filename wildcards, shell pipes and + environment variable expansion. + + .. warning:: + + Executing shell commands that incorporate unsanitized input from an + untrusted source makes a program vulnerable to `shell injection + <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_, + a serious security flaw which can result in arbitrary command execution. + For this reason, the use of *shell=True* is **strongly discouraged** in cases + where the command string is constructed from external input:: + + >>> from subprocess import call + >>> filename = input("What file would you like to display?\n") + What file would you like to display? + non_existent; rm -rf / # + >>> call("cat " + filename, shell=True) # Uh-oh. This will end badly... + + ``shell=False`` disables all shell based features, but does not suffer + from this vulnerability; see the Note in the :class:`Popen` constructor + documentation for helpful hints in getting ``shell=False`` to work. + +These options, along with all of the other options, are described in more +detail in the :class:`Popen` constructor documentation. + + +Popen Constructor +^^^^^^^^^^^^^^^^^ + +The underlying process creation and management in this module is handled by +the :class:`Popen` class. It offers a lot of flexibility so that developers +are able to handle the less common cases not covered by the convenience +functions. + + +.. class:: Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=True, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0, restore_signals=True, start_new_session=False, pass_fds=()) Arguments are: @@ -41,7 +261,8 @@ This module defines one class called :class:`Popen`: name for the executing program in utilities such as :program:`ps`. On Unix, with *shell=False* (default): In this case, the Popen class uses - :meth:`os.execvp` to execute the child program. *args* should normally be a + :meth:`os.execvp` like behavior to execute the child program. + *args* should normally be a sequence. If a string is specified for *args*, it will be used as the name or path of the program to execute; this will only work if the program is being given no arguments. @@ -77,21 +298,9 @@ This module defines one class called :class:`Popen`: .. warning:: - Executing shell commands that incorporate unsanitized input from an - untrusted source makes a program vulnerable to `shell injection - <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_, - a serious security flaw which can result in arbitrary command execution. - For this reason, the use of *shell=True* is **strongly discouraged** in cases - where the command string is constructed from external input:: - - >>> from subprocess import call - >>> filename = input("What file would you like to display?\n") - What file would you like to display? - non_existent; rm -rf / # - >>> call("cat " + filename, shell=True) # Uh-oh. This will end badly... - - *shell=False* does not suffer from this vulnerability; the above Note may be - helpful in getting code using *shell=False* to work. + Enabling this option can be a security hazard if combined with untrusted + input. See the warning under :ref:`frequently-used-arguments` + for details. On Windows: the :class:`Popen` class uses CreateProcess() to execute the child program, which operates on strings. If *args* is a sequence, it will @@ -120,35 +329,76 @@ This module defines one class called :class:`Popen`: You don't need ``shell=True`` to run a batch file, nor to run a console-based executable. - *stdin*, *stdout* and *stderr* specify the executed programs' standard input, + *stdin*, *stdout* and *stderr* specify the executed program's standard input, standard output and standard error file handles, respectively. Valid values are :data:`PIPE`, an existing file descriptor (a positive integer), an existing :term:`file object`, and ``None``. :data:`PIPE` indicates that a - new pipe to the child should be created. With ``None``, no redirection will - occur; the child's file handles will be inherited from the parent. Additionally, - *stderr* can be :data:`STDOUT`, which indicates that the stderr data from the - applications should be captured into the same file handle as for stdout. + new pipe to the child should be created. With the default settings of + ``None``, no redirection will occur; the child's file handles will be + inherited from the parent. Additionally, *stderr* can be :data:`STDOUT`, + which indicates that the stderr data from the applications should be + captured into the same file handle as for stdout. If *preexec_fn* is set to a callable object, this object will be called in the - child process just before the child is executed. (Unix only) + child process just before the child is executed. + (Unix only) + + .. warning:: + + The *preexec_fn* parameter is not safe to use in the presence of threads + in your application. The child process could deadlock before exec is + called. + If you must use it, keep it trivial! Minimize the number of libraries + you call into. + + .. note:: + + If you need to modify the environment for the child use the *env* + parameter rather than doing it in a *preexec_fn*. + The *start_new_session* parameter can take the place of a previously + common use of *preexec_fn* to call os.setsid() in the child. If *close_fds* is true, all file descriptors except :const:`0`, :const:`1` and :const:`2` will be closed before the child process is executed. (Unix only). - Or, on Windows, if *close_fds* is true then no handles will be inherited by the + The default varies by platform: Always true on Unix. On Windows it is + true when *stdin*/*stdout*/*stderr* are :const:`None`, false otherwise. + On Windows, if *close_fds* is true then no handles will be inherited by the child process. Note that on Windows, you cannot set *close_fds* to true and also redirect the standard handles by setting *stdin*, *stdout* or *stderr*. - If *shell* is :const:`True`, the specified command will be executed through the - shell. + .. versionchanged:: 3.2 + The default for *close_fds* was changed from :const:`False` to + what is described above. + + *pass_fds* is an optional sequence of file descriptors to keep open + between the parent and child. Providing any *pass_fds* forces + *close_fds* to be :const:`True`. (Unix only) + + .. versionadded:: 3.2 + The *pass_fds* parameter was added. If *cwd* is not ``None``, the child's current directory will be changed to *cwd* before it is executed. Note that this directory is not considered when searching the executable, so you can't specify the program's path relative to *cwd*. + If *restore_signals* is True (the default) all signals that Python has set to + SIG_IGN are restored to SIG_DFL in the child process before the exec. + Currently this includes the SIGPIPE, SIGXFZ and SIGXFSZ signals. + (Unix only) + + .. versionchanged:: 3.2 + *restore_signals* was added. + + If *start_new_session* is True the setsid() system call will be made in the + child process prior to the execution of the subprocess. (Unix only) + + .. versionchanged:: 3.2 + *start_new_session* was added. + If *env* is not ``None``, it must be a mapping that defines the environment - variables for the new process; these are used instead of inheriting the current - process' environment, which is the default behavior. + variables for the new process; these are used instead of the default + behavior of inheriting the current process' environment. .. note:: @@ -173,118 +423,18 @@ This module defines one class called :class:`Popen`: If given, *startupinfo* will be a :class:`STARTUPINFO` object, which is passed to the underlying ``CreateProcess`` function. - *creationflags*, if given, can be :data:`CREATE_NEW_CONSOLE`. (Windows only) - - -.. data:: PIPE - - Special value that can be used as the *stdin*, *stdout* or *stderr* argument - to :class:`Popen` and indicates that a pipe to the standard stream should be - opened. - - -.. data:: STDOUT - - Special value that can be used as the *stderr* argument to :class:`Popen` and - indicates that standard error should go into the same handle as standard - output. - - -Convenience Functions -^^^^^^^^^^^^^^^^^^^^^ - -This module also defines the following shortcut functions: - + *creationflags*, if given, can be :data:`CREATE_NEW_CONSOLE` or + :data:`CREATE_NEW_PROCESS_GROUP`. (Windows only) -.. function:: call(*popenargs, **kwargs) + Popen objects are supported as context managers via the :keyword:`with` statement: + on exit, standard file descriptors are closed, and the process is waited for. + :: - Run command with arguments. Wait for command to complete, then return the - :attr:`returncode` attribute. - - The arguments are the same as for the Popen constructor. Example:: - - >>> retcode = subprocess.call(["ls", "-l"]) - - .. warning:: - - Like :meth:`Popen.wait`, this will deadlock if the child process - generates enough output to a stdout or stderr pipe such that it blocks - waiting for the OS pipe buffer to accept more data. - - -.. function:: check_call(*popenargs, **kwargs) + with Popen(["ifconfig"], stdout=PIPE) as proc: + log.write(proc.stdout.read()) - Run command with arguments. Wait for command to complete. If the exit code was - zero then return, otherwise raise :exc:`CalledProcessError`. The - :exc:`CalledProcessError` object will have the return code in the - :attr:`returncode` attribute. - - The arguments are the same as for the Popen constructor. Example:: - - >>> subprocess.check_call(["ls", "-l"]) - 0 - - .. warning:: - - See the warning for :func:`call`. - - -.. function:: check_output(*popenargs, **kwargs) - - Run command with arguments and return its output as a byte string. - - If the exit code was non-zero it raises a :exc:`CalledProcessError`. The - :exc:`CalledProcessError` object will have the return code in the - :attr:`returncode` - attribute and output in the :attr:`output` attribute. - - The arguments are the same as for the :class:`Popen` constructor. Example:: - - >>> subprocess.check_output(["ls", "-l", "/dev/null"]) - b'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n' - - The stdout argument is not allowed as it is used internally. - To capture standard error in the result, use ``stderr=subprocess.STDOUT``:: - - >>> subprocess.check_output( - ... ["/bin/sh", "-c", "ls non_existent_file; exit 0"], - ... stderr=subprocess.STDOUT) - b'ls: non_existent_file: No such file or directory\n' - - .. versionadded:: 3.1 - - -.. function:: getstatusoutput(cmd) - - Return ``(status, output)`` of executing *cmd* in a shell. - - Execute the string *cmd* in a shell with :func:`os.popen` and return a 2-tuple - ``(status, output)``. *cmd* is actually run as ``{ cmd ; } 2>&1``, so that the - returned output will contain output or error messages. A trailing newline is - stripped from the output. The exit status for the command can be interpreted - according to the rules for the C function :cfunc:`wait`. Example:: - - >>> subprocess.getstatusoutput('ls /bin/ls') - (0, '/bin/ls') - >>> subprocess.getstatusoutput('cat /bin/junk') - (256, 'cat: /bin/junk: No such file or directory') - >>> subprocess.getstatusoutput('/bin/junk') - (256, 'sh: /bin/junk: not found') - - Availability: UNIX. - - -.. function:: getoutput(cmd) - - Return output (stdout and stderr) of executing *cmd* in a shell. - - Like :func:`getstatusoutput`, except the exit status is ignored and the return - value is a string containing the command's output. Example:: - - >>> subprocess.getoutput('ls /bin/ls') - '/bin/ls' - - Availability: UNIX. + .. versionchanged:: 3.2 + Added context manager support. Exceptions @@ -302,16 +452,19 @@ when trying to execute a non-existent file. Applications should prepare for A :exc:`ValueError` will be raised if :class:`Popen` is called with invalid arguments. -check_call() will raise :exc:`CalledProcessError`, if the called process returns -a non-zero return code. +:func:`check_call` and :func:`check_output` will raise +:exc:`CalledProcessError` if the called process returns a non-zero return +code. Security ^^^^^^^^ -Unlike some other popen functions, this implementation will never call /bin/sh -implicitly. This means that all characters, including shell metacharacters, can -safely be passed to child processes. +Unlike some other popen functions, this implementation will never call a +system shell implicitly. This means that all characters, including shell +metacharacters, can safely be passed to child processes. Obviously, if the +shell is invoked explicitly, then it is the application's responsibility to +ensure that all whitespace and metacharacters are quoted appropriately. Popen Objects @@ -333,9 +486,10 @@ Instances of the :class:`Popen` class have the following methods: .. warning:: - This will deadlock if the child process generates enough output to a - stdout or stderr pipe such that it blocks waiting for the OS pipe buffer - to accept more data. Use :meth:`communicate` to avoid that. + This will deadlock when using ``stdout=PIPE`` and/or + ``stderr=PIPE`` and the child process generates enough output to + a pipe such that it blocks waiting for the OS pipe buffer to + accept more data. Use :meth:`communicate` to avoid that. .. method:: Popen.communicate(input=None) @@ -364,14 +518,15 @@ Instances of the :class:`Popen` class have the following methods: .. note:: - On Windows only SIGTERM is supported so far. It's an alias for - :meth:`terminate`. + On Windows, SIGTERM is an alias for :meth:`terminate`. CTRL_C_EVENT and + CTRL_BREAK_EVENT can be sent to processes started with a *creationflags* + parameter which includes `CREATE_NEW_PROCESS_GROUP`. .. method:: Popen.terminate() Stop the child. On Posix OSs the method sends SIGTERM to the - child. On Windows the Win32 API function :cfunc:`TerminateProcess` is called + child. On Windows the Win32 API function :c:func:`TerminateProcess` is called to stop the child. @@ -442,38 +597,39 @@ on Windows. .. attribute:: dwFlags - A bit field that determines whether certain :class:`STARTUPINFO` members - are used when the process creates a window. :: + A bit field that determines whether certain :class:`STARTUPINFO` + attributes are used when the process creates a window. :: si = subprocess.STARTUPINFO() si.dwFlags = subprocess.STARTF_USESTDHANDLES | subprocess.STARTF_USESHOWWINDOW .. attribute:: hStdInput - If :attr:`dwFlags` specifies :data:`STARTF_USESTDHANDLES`, this member is - the standard input handle for the process. If :data:`STARTF_USESTDHANDLES` - is not specified, the default for standard input is the keyboard buffer. + If :attr:`dwFlags` specifies :data:`STARTF_USESTDHANDLES`, this attribute + is the standard input handle for the process. If + :data:`STARTF_USESTDHANDLES` is not specified, the default for standard + input is the keyboard buffer. .. attribute:: hStdOutput - If :attr:`dwFlags` specifies :data:`STARTF_USESTDHANDLES`, this member is - the standard output handle for the process. Otherwise, this member is - ignored and the default for standard output is the console window's + If :attr:`dwFlags` specifies :data:`STARTF_USESTDHANDLES`, this attribute + is the standard output handle for the process. Otherwise, this attribute + is ignored and the default for standard output is the console window's buffer. .. attribute:: hStdError - If :attr:`dwFlags` specifies :data:`STARTF_USESTDHANDLES`, this member is - the standard error handle for the process. Otherwise, this member is + If :attr:`dwFlags` specifies :data:`STARTF_USESTDHANDLES`, this attribute + is the standard error handle for the process. Otherwise, this attribute is ignored and the default for standard error is the console window's buffer. .. attribute:: wShowWindow - If :attr:`dwFlags` specifies :data:`STARTF_USESHOWWINDOW`, this member + If :attr:`dwFlags` specifies :data:`STARTF_USESHOWWINDOW`, this attribute can be any of the values that can be specified in the ``nCmdShow`` parameter for the `ShowWindow <http://msdn.microsoft.com/en-us/library/ms633548(v=vs.85).aspx>`__ - function, except for ``SW_SHOWDEFAULT``. Otherwise, this member is + function, except for ``SW_SHOWDEFAULT``. Otherwise, this attribute is ignored. :data:`SW_HIDE` is provided for this attribute. It is used when @@ -507,12 +663,12 @@ The :mod:`subprocess` module exposes the following constants. .. data:: STARTF_USESTDHANDLES Specifies that the :attr:`STARTUPINFO.hStdInput`, - :attr:`STARTUPINFO.hStdOutput`, and :attr:`STARTUPINFO.hStdError` members + :attr:`STARTUPINFO.hStdOutput`, and :attr:`STARTUPINFO.hStdError` attributes contain additional information. .. data:: STARTF_USESHOWWINDOW - Specifies that the :attr:`STARTUPINFO.wShowWindow` member contains + Specifies that the :attr:`STARTUPINFO.wShowWindow` attribute contains additional information. .. data:: CREATE_NEW_CONSOLE @@ -522,21 +678,35 @@ The :mod:`subprocess` module exposes the following constants. This flag is always set when :class:`Popen` is created with ``shell=True``. +.. data:: CREATE_NEW_PROCESS_GROUP + + A :class:`Popen` ``creationflags`` parameter to specify that a new process + group will be created. This flag is necessary for using :func:`os.kill` + on the subprocess. + + This flag is ignored if :data:`CREATE_NEW_CONSOLE` is specified. + .. _subprocess-replacements: Replacing Older Functions with the subprocess Module ---------------------------------------------------- -In this section, "a ==> b" means that b can be used as a replacement for a. +In this section, "a becomes b" means that b can be used as a replacement for a. .. note:: - All functions in this section fail (more or less) silently if the executed - program cannot be found; this module raises an :exc:`OSError` exception. + All "a" functions in this section fail (more or less) silently if the + executed program cannot be found; the "b" replacements raise :exc:`OSError` + instead. + + In addition, the replacements using :func:`check_output` will fail with a + :exc:`CalledProcessError` if the requested operation produces a non-zero + return code. The output is still available as the ``output`` attribute of + the raised exception. -In the following examples, we assume that the subprocess module is imported with -"from subprocess import \*". +In the following examples, we assume that the relevant functions have already +been imported from the subprocess module. Replacing /bin/sh shell backquote @@ -545,8 +715,8 @@ Replacing /bin/sh shell backquote :: output=`mycmd myarg` - ==> - output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0] + # becomes + output = check_output(["mycmd", "myarg"]) Replacing shell pipeline @@ -555,7 +725,7 @@ Replacing shell pipeline :: output=`dmesg | grep hda` - ==> + # becomes p1 = Popen(["dmesg"], stdout=PIPE) p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. @@ -564,22 +734,27 @@ Replacing shell pipeline The p1.stdout.close() call after starting the p2 is important in order for p1 to receive a SIGPIPE if p2 exits before p1. +Alternatively, for trusted input, the shell's own pipeline support may still +be used directly: + + output=`dmesg | grep hda` + # becomes + output=check_output("dmesg | grep hda", shell=True) + + Replacing :func:`os.system` ^^^^^^^^^^^^^^^^^^^^^^^^^^^ :: sts = os.system("mycmd" + " myarg") - ==> - p = Popen("mycmd" + " myarg", shell=True) - sts = os.waitpid(p.pid, 0)[1] + # becomes + sts = call("mycmd" + " myarg", shell=True) Notes: * Calling the program through the shell is usually not required. -* It's easier to look at the :attr:`returncode` attribute than the exit status. - A more realistic example would look like this:: try: @@ -701,7 +876,50 @@ Replacing functions from the :mod:`popen2` module * ``stdin=PIPE`` and ``stdout=PIPE`` must be specified. * popen2 closes all file descriptors by default, but you have to specify - ``close_fds=True`` with :class:`Popen`. + ``close_fds=True`` with :class:`Popen` to guarantee this behavior on + all platforms or past Python versions. + + +Legacy Shell Invocation Functions +--------------------------------- + +This module also provides the following legacy functions from the 2.x +``commands`` module. These operations implicitly invoke the system shell and +none of the guarantees described above regarding security and exception +handling consistency are valid for these functions. + +.. function:: getstatusoutput(cmd) + + Return ``(status, output)`` of executing *cmd* in a shell. + + Execute the string *cmd* in a shell with :func:`os.popen` and return a 2-tuple + ``(status, output)``. *cmd* is actually run as ``{ cmd ; } 2>&1``, so that the + returned output will contain output or error messages. A trailing newline is + stripped from the output. The exit status for the command can be interpreted + according to the rules for the C function :c:func:`wait`. Example:: + + >>> subprocess.getstatusoutput('ls /bin/ls') + (0, '/bin/ls') + >>> subprocess.getstatusoutput('cat /bin/junk') + (256, 'cat: /bin/junk: No such file or directory') + >>> subprocess.getstatusoutput('/bin/junk') + (256, 'sh: /bin/junk: not found') + + Availability: UNIX. + + +.. function:: getoutput(cmd) + + Return output (stdout and stderr) of executing *cmd* in a shell. + + Like :func:`getstatusoutput`, except the exit status is ignored and the return + value is a string containing the command's output. Example:: + + >>> subprocess.getoutput('ls /bin/ls') + '/bin/ls' + + Availability: UNIX. + Notes ----- @@ -734,4 +952,3 @@ runtime): backslash. If the number of backslashes is odd, the last backslash escapes the next double quotation mark as described in rule 3. - diff --git a/Doc/library/sunau.rst b/Doc/library/sunau.rst index fc141e9eeb..4bdb99bea6 100644 --- a/Doc/library/sunau.rst +++ b/Doc/library/sunau.rst @@ -5,6 +5,9 @@ :synopsis: Provide an interface to the Sun AU sound format. .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/sunau.py` + +-------------- The :mod:`sunau` module provides a convenient interface to the Sun AU sound format. Note that this module is interface-compatible with the modules diff --git a/Doc/library/symbol.rst b/Doc/library/symbol.rst index 5134d47269..ef9ef1e129 100644 --- a/Doc/library/symbol.rst +++ b/Doc/library/symbol.rst @@ -5,6 +5,9 @@ :synopsis: Constants representing internal nodes of the parse tree. .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/symbol.py` + +-------------- This module provides constants which represent the numeric values of internal nodes of the parse tree. Unlike most Python constants, these use lower-case @@ -21,11 +24,3 @@ This module also provides one additional data object: Dictionary mapping the numeric values of the constants defined in this module back to name strings, allowing more human-readable representation of parse trees to be generated. - - -.. seealso:: - - Module :mod:`parser` - The second example for the :mod:`parser` module shows how to use the - :mod:`symbol` module. - diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 95947560f2..063e0a52c0 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -10,6 +10,13 @@ interpreter and to functions that interact strongly with the interpreter. It is always available. +.. data:: abiflags + + On POSIX systems where Python is build with the standard ``configure`` + script, this contains the ABI flags as specified by :pep:`3149`. + + .. versionadded:: 3.2 + .. data:: argv The list of command line arguments passed to a Python script. ``argv[0]`` is the @@ -29,18 +36,6 @@ always available. little-endian (least-significant byte first) platforms. -.. data:: subversion - - A triple (repo, branch, version) representing the Subversion information of the - Python interpreter. *repo* is the name of the repository, ``'CPython'``. - *branch* is a string of one of the forms ``'trunk'``, ``'branches/name'`` or - ``'tags/name'``. *version* is the output of ``svnversion``, if the interpreter - was built from a Subversion checkout; it contains the revision number (range) - and possibly a trailing 'M' if there were local modifications. If the tree was - exported (or svnversion was not available), it is the revision of - ``Include/patchlevel.h`` if the branch is a tag. Otherwise, it is ``None``. - - .. data:: builtin_module_names A tuple of strings giving the names of all modules that are compiled into this @@ -92,13 +87,48 @@ always available. .. function:: displayhook(value) - If *value* is not ``None``, this function prints it to ``sys.stdout``, and saves - it in ``builtins._``. + If *value* is not ``None``, this function prints ``repr(value)`` to + ``sys.stdout``, and saves *value* in ``builtins._``. If ``repr(value)`` is + not encodable to ``sys.stdout.encoding`` with ``sys.stdout.errors`` error + handler (which is probably ``'strict'``), encode it to + ``sys.stdout.encoding`` with ``'backslashreplace'`` error handler. ``sys.displayhook`` is called on the result of evaluating an :term:`expression` entered in an interactive Python session. The display of these values can be customized by assigning another one-argument function to ``sys.displayhook``. + Pseudo-code:: + + def displayhook(value): + if value is None: + return + # Set '_' to None to avoid recursion + builtins._ = None + text = repr(value) + try: + sys.stdout.write(text) + except UnicodeEncodeError: + bytes = text.encode(sys.stdout.encoding, 'backslashreplace') + if hasattr(sys.stdout, 'buffer'): + sys.stdout.buffer.write(bytes) + else: + text = bytes.decode(sys.stdout.encoding, 'strict') + sys.stdout.write(text) + sys.stdout.write("\n") + builtins._ = value + + .. versionchanged:: 3.2 + Use ``'backslashreplace'`` error handler on :exc:`UnicodeEncodeError`. + + +.. data:: dont_write_bytecode + + If this is true, Python won't try to write ``.pyc`` or ``.pyo`` files on the + import of source modules. This value is initially set to ``True`` or + ``False`` depending on the :option:`-B` command line option and the + :envvar:`PYTHONDONTWRITEBYTECODE` environment variable, but you can set it + yourself to control bytecode file generation. + .. function:: excepthook(type, value, traceback) @@ -164,16 +194,18 @@ always available. Python files are installed; by default, this is also ``'/usr/local'``. This can be set at build time with the ``--exec-prefix`` argument to the :program:`configure` script. Specifically, all configuration files (e.g. the - :file:`pyconfig.h` header file) are installed in the directory ``exec_prefix + - '/lib/pythonversion/config'``, and shared library modules are installed in - ``exec_prefix + '/lib/pythonversion/lib-dynload'``, where *version* is equal to - ``version[:3]``. + :file:`pyconfig.h` header file) are installed in the directory + :file:`{exec_prefix}/lib/python{X.Y}/config', and shared library modules are + installed in :file:`{exec_prefix}/lib/python{X.Y}/lib-dynload`, where *X.Y* + is the version number of Python, for example ``3.2``. .. data:: executable - A string giving the name of the executable binary for the Python interpreter, on - systems where this makes sense. + A string giving the absolute path of the executable binary for the Python + interpreter, on systems where this makes sense. If Python is unable to retrieve + the real path to its executable, :data:`sys.executable` will be an empty string + or ``None``. .. function:: exit([arg]) @@ -220,10 +252,14 @@ always available. :const:`ignore_environment` :option:`-E` :const:`verbose` :option:`-v` :const:`bytes_warning` :option:`-b` + :const:`quiet` :option:`-q` :const:`hash_randomization` :option:`-R` ============================= ============================= - .. versionadded:: 3.1.5 + .. versionchanged:: 3.2 + Added ``quiet`` attribute for the new :option:`-q` flag. + + .. versionadded:: 3.2.3 The ``hash_randomization`` attribute. @@ -266,8 +302,12 @@ always available. +---------------------+----------------+--------------------------------------------------+ | :const:`radix` | FLT_RADIX | radix of exponent representation | +---------------------+----------------+--------------------------------------------------+ - | :const:`rounds` | FLT_ROUNDS | constant representing rounding mode | - | | | used for arithmetic operations | + | :const:`rounds` | FLT_ROUNDS | integer constant representing the rounding mode | + | | | used for arithmetic operations. This reflects | + | | | the value of the system FLT_ROUNDS macro at | + | | | interpreter startup time. See section 5.2.4.2.2 | + | | | of the C99 standard for an explanation of the | + | | | possible values and their meanings. | +---------------------+----------------+--------------------------------------------------+ The attribute :attr:`sys.float_info.dig` needs further explanation. If @@ -307,6 +347,9 @@ always available. Return the interpreter's "check interval"; see :func:`setcheckinterval`. + .. deprecated:: 3.2 + Use :func:`getswitchinterval` instead. + .. function:: getdefaultencoding() @@ -316,22 +359,20 @@ always available. .. function:: getdlopenflags() - Return the current value of the flags that are used for :cfunc:`dlopen` calls. + Return the current value of the flags that are used for :c:func:`dlopen` calls. The flag constants are defined in the :mod:`ctypes` and :mod:`DLFCN` modules. Availability: Unix. .. function:: getfilesystemencoding() - Return the name of the encoding used to convert Unicode filenames into system - file names, or ``None`` if the system default encoding is used. The result value - depends on the operating system: + Return the name of the encoding used to convert Unicode filenames into + system file names. The result value depends on the operating system: * On Mac OS X, the encoding is ``'utf-8'``. * On Unix, the encoding is the user's preference according to the result of - nl_langinfo(CODESET), or ``None`` if the ``nl_langinfo(CODESET)`` - failed. + nl_langinfo(CODESET), or ``'utf-8'`` if ``nl_langinfo(CODESET)`` failed. * On Windows NT+, file names are Unicode natively, so no conversion is performed. :func:`getfilesystemencoding` still returns ``'mbcs'``, as @@ -341,6 +382,10 @@ always available. * On Windows 9x, the encoding is ``'mbcs'``. + .. versionchanged:: 3.2 + On Unix, use ``'utf-8'`` instead of ``None`` if ``nl_langinfo(CODESET)`` + failed. :func:`getfilesystemencoding` result cannot be ``None``. + .. function:: getrefcount(object) @@ -371,6 +416,17 @@ always available. additional garbage collector overhead if the object is managed by the garbage collector. + See `recursive sizeof recipe <http://code.activestate.com/recipes/577504>`_ + for an example of using :func:`getsizeof` recursively to find the size of + containers and all their contents. + +.. function:: getswitchinterval() + + Return the interpreter's "thread switch interval"; see + :func:`setswitchinterval`. + + .. versionadded:: 3.2 + .. function:: _getframe([depth]) @@ -412,9 +468,15 @@ always available. .. function:: getwindowsversion() - Return a tuple containing five components, describing the Windows version - currently running. The elements are *major*, *minor*, *build*, *platform*, and - *text*. *text* contains a string while all other values are integers. + Return a named tuple describing the Windows version + currently running. The named elements are *major*, *minor*, + *build*, *platform*, *service_pack*, *service_pack_minor*, + *service_pack_major*, *suite_mask*, and *product_type*. + *service_pack* contains a string while all other values are + integers. The components can also be accessed by name, so + ``sys.getwindowsversion()[0]`` is equivalent to + ``sys.getwindowsversion().major``. For compatibility with prior + versions, only the first 5 elements are retrievable by indexing. *platform* may be one of the following values: @@ -430,11 +492,54 @@ always available. | :const:`3 (VER_PLATFORM_WIN32_CE)` | Windows CE | +-----------------------------------------+-------------------------+ - This function wraps the Win32 :cfunc:`GetVersionEx` function; see the Microsoft - documentation for more information about these fields. + *product_type* may be one of the following values: + + +---------------------------------------+---------------------------------+ + | Constant | Meaning | + +=======================================+=================================+ + | :const:`1 (VER_NT_WORKSTATION)` | The system is a workstation. | + +---------------------------------------+---------------------------------+ + | :const:`2 (VER_NT_DOMAIN_CONTROLLER)` | The system is a domain | + | | controller. | + +---------------------------------------+---------------------------------+ + | :const:`3 (VER_NT_SERVER)` | The system is a server, but not | + | | a domain controller. | + +---------------------------------------+---------------------------------+ + + + This function wraps the Win32 :c:func:`GetVersionEx` function; see the + Microsoft documentation on :c:func:`OSVERSIONINFOEX` for more information + about these fields. Availability: Windows. + .. versionchanged:: 3.2 + Changed to a named tuple and added *service_pack_minor*, + *service_pack_major*, *suite_mask*, and *product_type*. + + +.. data:: hash_info + + A structseq giving parameters of the numeric hash implementation. For + more details about hashing of numeric types, see :ref:`numeric-hash`. + + +---------------------+--------------------------------------------------+ + | attribute | explanation | + +=====================+==================================================+ + | :const:`width` | width in bits used for hash values | + +---------------------+--------------------------------------------------+ + | :const:`modulus` | prime modulus P used for numeric hash scheme | + +---------------------+--------------------------------------------------+ + | :const:`inf` | hash value returned for a positive infinity | + +---------------------+--------------------------------------------------+ + | :const:`nan` | hash value returned for a nan | + +---------------------+--------------------------------------------------+ + | :const:`imag` | multiplier used for the imaginary part of a | + | | complex number | + +---------------------+--------------------------------------------------+ + + .. versionadded:: 3.2 + .. data:: hexversion @@ -529,7 +634,7 @@ always available. .. data:: maxsize - An integer giving the maximum value a variable of type :ctype:`Py_ssize_t` can + An integer giving the maximum value a variable of type :c:type:`Py_ssize_t` can take. It's usually ``2**31 - 1`` on a 32-bit platform and ``2**63 - 1`` on a 64-bit platform. @@ -548,7 +653,7 @@ always available. imported. The :meth:`find_module` method is called at least with the absolute name of the module being imported. If the module to be imported is contained in package then the parent package's :attr:`__path__` attribute - is passed in as a second argument. The method returns :keyword:`None` if + is passed in as a second argument. The method returns ``None`` if the module cannot be found, else returns a :term:`loader`. :data:`sys.meta_path` is searched before any implicit default finders or @@ -601,7 +706,7 @@ always available. A dictionary acting as a cache for :term:`finder` objects. The keys are paths that have been passed to :data:`sys.path_hooks` and the values are the finders that are found. If a path is a valid file system path but no - explicit finder is found on :data:`sys.path_hooks` then :keyword:`None` is + explicit finder is found on :data:`sys.path_hooks` then ``None`` is stored to represent the implicit default finder should be used. If the path is not an existing path then :class:`imp.NullImporter` is set. @@ -613,22 +718,43 @@ always available. This string contains a platform identifier that can be used to append platform-specific components to :data:`sys.path`, for instance. - For Unix systems, this is the lowercased OS name as returned by ``uname -s`` - with the first part of the version as returned by ``uname -r`` appended, - e.g. ``'sunos5'`` or ``'linux2'``, *at the time when Python was built*. + For most Unix systems, this is the lowercased OS name as returned by ``uname + -s`` with the first part of the version as returned by ``uname -r`` appended, + e.g. ``'sunos5'``, *at the time when Python was built*. Unless you want to + test for a specific system version, it is therefore recommended to use the + following idiom:: + + if sys.platform.startswith('freebsd'): + # FreeBSD-specific code here... + elif sys.platform.startswith('linux'): + # Linux-specific code here... + + .. versionchanged:: 3.2.2 + Since lots of code check for ``sys.platform == 'linux2'``, and there is + no essential change between Linux 2.x and 3.x, ``sys.platform`` is always + set to ``'linux2'``, even on Linux 3.x. In Python 3.3 and later, the + value will always be set to ``'linux'``, so it is recommended to always + use the ``startswith`` idiom presented above. + For other systems, the values are: - ================ =========================== - System :data:`platform` value - ================ =========================== - Windows ``'win32'`` - Windows/Cygwin ``'cygwin'`` - Mac OS X ``'darwin'`` - OS/2 ``'os2'`` - OS/2 EMX ``'os2emx'`` - AtheOS ``'atheos'`` - ================ =========================== + ====================== =========================== + System :data:`platform` value + ====================== =========================== + Linux (2.x *and* 3.x) ``'linux2'`` + Windows ``'win32'`` + Windows/Cygwin ``'cygwin'`` + Mac OS X ``'darwin'`` + OS/2 ``'os2'`` + OS/2 EMX ``'os2emx'`` + ====================== =========================== + + .. seealso:: + :attr:`os.name` has a coarser granularity. :func:`os.uname` gives + system-dependent version information. + The :mod:`platform` module provides detailed checks for the + system's identity. .. data:: prefix @@ -636,10 +762,10 @@ always available. independent Python files are installed; by default, this is the string ``'/usr/local'``. This can be set at build time with the ``--prefix`` argument to the :program:`configure` script. The main collection of Python - library modules is installed in the directory ``prefix + '/lib/pythonversion'`` + library modules is installed in the directory :file:`{prefix}/lib/python{X.Y}`` while the platform independent header files (all except :file:`pyconfig.h`) are - stored in ``prefix + '/include/pythonversion'``, where *version* is equal to - ``version[:3]``. + stored in :file:`{prefix}/include/python{X.Y}``, where *X.Y* is the version + number of Python, for example ``3.2``. .. data:: ps1 @@ -657,15 +783,6 @@ always available. implement a dynamic prompt. -.. data:: dont_write_bytecode - - If this is true, Python won't try to write ``.pyc`` or ``.pyo`` files on the - import of source modules. This value is initially set to ``True`` or ``False`` - depending on the ``-B`` command line option and the ``PYTHONDONTWRITEBYTECODE`` - environment variable, but you can set it yourself to control bytecode file - generation. - - .. function:: setcheckinterval(interval) Set the interpreter's "check interval". This integer value determines how often @@ -675,22 +792,15 @@ always available. performance for programs using threads. Setting it to a value ``<=`` 0 checks every virtual instruction, maximizing responsiveness as well as overhead. - -.. function:: setdefaultencoding(name) - - Set the current default string encoding used by the Unicode implementation. If - *name* does not match any available encoding, :exc:`LookupError` is raised. - This function is only intended to be used by the :mod:`site` module - implementation and, where needed, by :mod:`sitecustomize`. Once used by the - :mod:`site` module, it is removed from the :mod:`sys` module's namespace. - - .. Note that :mod:`site` is not imported if the :option:`-S` option is passed - to the interpreter, in which case this function will remain available. + .. deprecated:: 3.2 + This function doesn't have an effect anymore, as the internal logic for + thread switching and asynchronous tasks has been rewritten. Use + :func:`setswitchinterval` instead. .. function:: setdlopenflags(n) - Set the flags used by the interpreter for :cfunc:`dlopen` calls, such as when + Set the flags used by the interpreter for :c:func:`dlopen` calls, such as when the interpreter loads extension modules. Among other things, this will enable a lazy resolving of symbols when importing a module, if called as ``sys.setdlopenflags(0)``. To share symbols across extension modules, call as @@ -700,15 +810,6 @@ always available. :file:`/usr/include/dlfcn.h` using the :program:`h2py` script. Availability: Unix. -.. function:: setfilesystemencoding(enc) - - Set the encoding used when converting Python strings to file names to *enc*. - By default, Python tries to determine the encoding it should use automatically - on Unix; on Windows, it avoids such conversion completely. This function can - be used when Python's determination of the encoding needs to be overwritten, - e.g. when not all file names on disk can be decoded using the encoding that - Python had chosen. - .. function:: setprofile(profilefunc) .. index:: @@ -733,11 +834,24 @@ always available. Python. The highest possible limit is platform-dependent. A user may need to set the - limit higher when she has a program that requires deep recursion and a platform + limit higher when they have a program that requires deep recursion and a platform that supports a higher limit. This should be done with care, because a too-high limit can lead to a crash. +.. function:: setswitchinterval(interval) + + Set the interpreter's thread switch interval (in seconds). This floating-point + value determines the ideal duration of the "timeslices" allocated to + concurrently running Python threads. Please note that the actual value + can be higher, especially if long-running internal functions or methods + are used. Also, which thread becomes scheduled at the end of the interval + is the operating system's decision. The interpreter doesn't have its + own scheduler. + + .. versionadded:: 3.2 + + .. function:: settrace(tracefunc) .. index:: @@ -770,9 +884,11 @@ always available. specifies the local trace function. ``'line'`` - The interpreter is about to execute a new line of code (sometimes multiple - line events on one line exist). The local trace function is called; *arg* - is ``None``; the return value specifies the new local trace function. + The interpreter is about to execute a new line of code or re-execute the + condition of a loop. The local trace function is called; *arg* is + ``None``; the return value specifies the new local trace function. See + :file:`Objects/lnotab_notes.txt` for a detailed explanation of how this + works. ``'return'`` A function (or other code block) is about to return. The local trace @@ -815,36 +931,51 @@ always available. available only if Python was compiled with ``--with-tsc``. To understand the output of this dump, read :file:`Python/ceval.c` in the Python sources. + .. impl-detail:: + This function is intimately bound to CPython implementation details and + thus not likely to be implemented elsewhere. + .. data:: stdin stdout stderr - :term:`File objects <file object>` corresponding to the interpreter's standard - input, output and error streams. ``stdin`` is used for all interpreter input - except for scripts but including calls to :func:`input`. ``stdout`` is used - for the output of :func:`print` and :term:`expression` statements and for the - prompts of :func:`input`. The interpreter's own prompts - and (almost all of) its error messages go to ``stderr``. ``stdout`` and - ``stderr`` needn't be built-in file objects: any object is acceptable as long - as it has a :meth:`write` method that takes a string argument. (Changing these - objects doesn't affect the standard I/O streams of processes executed by - :func:`os.popen`, :func:`os.system` or the :func:`exec\*` family of functions in - the :mod:`os` module.) - - The standard streams are in text mode by default. To write or read binary - data to these, use the underlying binary buffer. For example, to write bytes - to :data:`stdout`, use ``sys.stdout.buffer.write(b'abc')``. Using - :meth:`io.TextIOBase.detach` streams can be made binary by default. This + :term:`File objects <file object>` used by the interpreter for standard + input, output and errors: + + * ``stdin`` is used for all interactive input (including calls to + :func:`input`); + * ``stdout`` is used for the output of :func:`print` and :term:`expression` + statements and for the prompts of :func:`input`; + * The interpreter's own prompts and its error messages go to ``stderr``. + + By default, these streams are regular text streams as returned by the + :func:`open` function. Their parameters are chosen as follows: + + * The character encoding is platform-dependent. Under Windows, if the stream + is interactive (that is, if its :meth:`isatty` method returns True), the + console codepage is used, otherwise the ANSI code page. Under other + platforms, the locale encoding is used (see :meth:`locale.getpreferredencoding`). + + Under all platforms though, you can override this value by setting the + :envvar:`PYTHONIOENCODING` environment variable. + + * When interactive, standard streams are line-buffered. Otherwise, they + are block-buffered like regular text files. You can override this + value with the :option:`-u` command-line option. + + To write or read binary data from/to the standard streams, use the + underlying binary :data:`~io.TextIOBase.buffer`. For example, to write + bytes to :data:`stdout`, use ``sys.stdout.buffer.write(b'abc')``. Using + :meth:`io.TextIOBase.detach`, streams can be made binary by default. This function sets :data:`stdin` and :data:`stdout` to binary:: def make_streams_binary(): sys.stdin = sys.stdin.detach() sys.stdout = sys.stdout.detach() - Note that the streams can be replaced with objects (like - :class:`io.StringIO`) that do not support the - :attr:`~io.BufferedIOBase.buffer` attribute or the + Note that the streams may be replaced with objects (like :class:`io.StringIO`) + that do not support the :attr:`~io.BufferedIOBase.buffer` attribute or the :meth:`~io.BufferedIOBase.detach` method and can raise :exc:`AttributeError` or :exc:`io.UnsupportedOperation`. @@ -870,6 +1001,24 @@ always available. to a console and Python apps started with :program:`pythonw`. +.. data:: subversion + + A triple (repo, branch, version) representing the Subversion information of the + Python interpreter. *repo* is the name of the repository, ``'CPython'``. + *branch* is a string of one of the forms ``'trunk'``, ``'branches/name'`` or + ``'tags/name'``. *version* is the output of ``svnversion``, if the interpreter + was built from a Subversion checkout; it contains the revision number (range) + and possibly a trailing 'M' if there were local modifications. If the tree was + exported (or svnversion was not available), it is the revision of + ``Include/patchlevel.h`` if the branch is a tag. Otherwise, it is ``None``. + + .. deprecated:: 3.2.1 + Python is now `developed <http://docs.python.org/devguide/>`_ using + Mercurial. In recent Python 3.2 bugfix releases, :data:`subversion` + therefore contains placeholder information. It is removed in Python + 3.3. + + .. data:: tracebacklimit When this variable is set to an integer value, it determines the maximum number @@ -921,6 +1070,30 @@ always available. module for informational purposes; modifying this value has no effect on the registry keys used by Python. Availability: Windows. + +.. data:: _xoptions + + A dictionary of the various implementation-specific flags passed through + the :option:`-X` command-line option. Option names are either mapped to + their values, if given explicitly, or to :const:`True`. Example:: + + $ ./python -Xa=b -Xc + Python 3.2a3+ (py3k, Oct 16 2010, 20:14:50) + [GCC 4.4.3] on linux2 + Type "help", "copyright", "credits" or "license" for more information. + >>> import sys + >>> sys._xoptions + {'a': 'b', 'c': True} + + .. impl-detail:: + + This is a CPython-specific way of accessing options passed through + :option:`-X`. Other implementations may export them through other + means, or not at all. + + .. versionadded:: 3.2 + + .. rubric:: Citations .. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf . diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst new file mode 100644 index 0000000000..c47dcce83c --- /dev/null +++ b/Doc/library/sysconfig.rst @@ -0,0 +1,260 @@ +:mod:`sysconfig` --- Provide access to Python's configuration information +========================================================================= + +.. module:: sysconfig + :synopsis: Python's configuration information +.. moduleauthor:: Tarek Ziadé <tarek@ziade.org> +.. sectionauthor:: Tarek Ziadé <tarek@ziade.org> + +.. index:: + single: configuration information + +.. versionadded:: 3.2 + +**Source code:** :source:`Lib/sysconfig.py` + +-------------- + +The :mod:`sysconfig` module provides access to Python's configuration +information like the list of installation paths and the configuration variables +relevant for the current platform. + +Configuration variables +----------------------- + +A Python distribution contains a :file:`Makefile` and a :file:`pyconfig.h` +header file that are necessary to build both the Python binary itself and +third-party C extensions compiled using :mod:`distutils`. + +:mod:`sysconfig` puts all variables found in these files in a dictionary that +can be accessed using :func:`get_config_vars` or :func:`get_config_var`. + +Notice that on Windows, it's a much smaller set. + +.. function:: get_config_vars(\*args) + + With no arguments, return a dictionary of all configuration variables + relevant for the current platform. + + With arguments, return a list of values that result from looking up each + argument in the configuration variable dictionary. + + For each argument, if the value is not found, return ``None``. + + +.. function:: get_config_var(name) + + Return the value of a single variable *name*. Equivalent to + ``get_config_vars().get(name)``. + + If *name* is not found, return ``None``. + +Example of usage:: + + >>> import sysconfig + >>> sysconfig.get_config_var('Py_ENABLE_SHARED') + 0 + >>> sysconfig.get_config_var('LIBDIR') + '/usr/local/lib' + >>> sysconfig.get_config_vars('AR', 'CXX') + ['ar', 'g++'] + + +Installation paths +------------------ + +Python uses an installation scheme that differs depending on the platform and on +the installation options. These schemes are stored in :mod:`sysconfig` under +unique identifiers based on the value returned by :const:`os.name`. + +Every new component that is installed using :mod:`distutils` or a +Distutils-based system will follow the same scheme to copy its file in the right +places. + +Python currently supports seven schemes: + +- *posix_prefix*: scheme for Posix platforms like Linux or Mac OS X. This is + the default scheme used when Python or a component is installed. +- *posix_home*: scheme for Posix platforms used when a *home* option is used + upon installation. This scheme is used when a component is installed through + Distutils with a specific home prefix. +- *posix_user*: scheme for Posix platforms used when a component is installed + through Distutils and the *user* option is used. This scheme defines paths + located under the user home directory. +- *nt*: scheme for NT platforms like Windows. +- *nt_user*: scheme for NT platforms, when the *user* option is used. +- *os2*: scheme for OS/2 platforms. +- *os2_home*: scheme for OS/2 patforms, when the *user* option is used. + +Each scheme is itself composed of a series of paths and each path has a unique +identifier. Python currently uses eight paths: + +- *stdlib*: directory containing the standard Python library files that are not + platform-specific. +- *platstdlib*: directory containing the standard Python library files that are + platform-specific. +- *platlib*: directory for site-specific, platform-specific files. +- *purelib*: directory for site-specific, non-platform-specific files. +- *include*: directory for non-platform-specific header files. +- *platinclude*: directory for platform-specific header files. +- *scripts*: directory for script files. +- *data*: directory for data files. + +:mod:`sysconfig` provides some functions to determine these paths. + +.. function:: get_scheme_names() + + Return a tuple containing all schemes currently supported in + :mod:`sysconfig`. + + +.. function:: get_path_names() + + Return a tuple containing all path names currently supported in + :mod:`sysconfig`. + + +.. function:: get_path(name, [scheme, [vars, [expand]]]) + + Return an installation path corresponding to the path *name*, from the + install scheme named *scheme*. + + *name* has to be a value from the list returned by :func:`get_path_names`. + + :mod:`sysconfig` stores installation paths corresponding to each path name, + for each platform, with variables to be expanded. For instance the *stdlib* + path for the *nt* scheme is: ``{base}/Lib``. + + :func:`get_path` will use the variables returned by :func:`get_config_vars` + to expand the path. All variables have default values for each platform so + one may call this function and get the default value. + + If *scheme* is provided, it must be a value from the list returned by + :func:`get_scheme_names`. Otherwise, the default scheme for the current + platform is used. + + If *vars* is provided, it must be a dictionary of variables that will update + the dictionary return by :func:`get_config_vars`. + + If *expand* is set to ``False``, the path will not be expanded using the + variables. + + If *name* is not found, return ``None``. + + +.. function:: get_paths([scheme, [vars, [expand]]]) + + Return a dictionary containing all installation paths corresponding to an + installation scheme. See :func:`get_path` for more information. + + If *scheme* is not provided, will use the default scheme for the current + platform. + + If *vars* is provided, it must be a dictionary of variables that will + update the dictionary used to expand the paths. + + If *expand* is set to False, the paths will not be expanded. + + If *scheme* is not an existing scheme, :func:`get_paths` will raise a + :exc:`KeyError`. + + +Other functions +--------------- + +.. function:: get_python_version() + + Return the ``MAJOR.MINOR`` Python version number as a string. Similar to + ``sys.version[:3]``. + + +.. function:: get_platform() + + Return a string that identifies the current platform. + + This is used mainly to distinguish platform-specific build directories and + platform-specific built distributions. Typically includes the OS name and + version and the architecture (as supplied by :func:`os.uname`), although the + exact information included depends on the OS; e.g. for IRIX the architecture + isn't particularly important (IRIX only runs on SGI hardware), but for Linux + the kernel version isn't particularly important. + + Examples of returned values: + + - linux-i586 + - linux-alpha (?) + - solaris-2.6-sun4u + - irix-5.3 + - irix64-6.2 + + Windows will return one of: + + - win-amd64 (64bit Windows on AMD64 (aka x86_64, Intel64, EM64T, etc) + - win-ia64 (64bit Windows on Itanium) + - win32 (all others - specifically, sys.platform is returned) + + Mac OS X can return: + + - macosx-10.6-ppc + - macosx-10.4-ppc64 + - macosx-10.3-i386 + - macosx-10.4-fat + + For other non-POSIX platforms, currently just returns :data:`sys.platform`. + + +.. function:: is_python_build() + + Return ``True`` if the current Python installation was built from source. + + +.. function:: parse_config_h(fp[, vars]) + + Parse a :file:`config.h`\-style file. + + *fp* is a file-like object pointing to the :file:`config.h`\-like file. + + A dictionary containing name/value pairs is returned. If an optional + dictionary is passed in as the second argument, it is used instead of a new + dictionary, and updated with the values read in the file. + + +.. function:: get_config_h_filename() + + Return the path of :file:`pyconfig.h`. + +.. function:: get_makefile_filename() + + Return the path of :file:`Makefile`. + +Using :mod:`sysconfig` as a script +---------------------------------- + +You can use :mod:`sysconfig` as a script with Python's *-m* option:: + + $ python -m sysconfig + Platform: "macosx-10.4-i386" + Python version: "3.2" + Current installation scheme: "posix_prefix" + + Paths: + data = "/usr/local" + include = "/Users/tarek/Dev/svn.python.org/py3k/Include" + platinclude = "." + platlib = "/usr/local/lib/python3.2/site-packages" + platstdlib = "/usr/local/lib/python3.2" + purelib = "/usr/local/lib/python3.2/site-packages" + scripts = "/usr/local/bin" + stdlib = "/usr/local/lib/python3.2" + + Variables: + AC_APPLE_UNIVERSAL_BUILD = "0" + AIX_GENUINE_CPLUSPLUS = "0" + AR = "ar" + ARFLAGS = "rc" + ASDLGEN = "./Parser/asdl_c.py" + ... + +This call will print in the standard output the information returned by +:func:`get_platform`, :func:`get_python_version`, :func:`get_path` and +:func:`get_config_vars`. diff --git a/Doc/library/syslog.rst b/Doc/library/syslog.rst index 89dd38f4fc..795d66d99f 100644 --- a/Doc/library/syslog.rst +++ b/Doc/library/syslog.rst @@ -10,42 +10,63 @@ This module provides an interface to the Unix ``syslog`` library routines. Refer to the Unix manual pages for a detailed description of the ``syslog`` facility. +This module wraps the system ``syslog`` family of routines. A pure Python +library that can speak to a syslog server is available in the +:mod:`logging.handlers` module as :class:`SysLogHandler`. + The module defines the following functions: .. function:: syslog([priority,] message) - Send the string *message* to the system logger. A trailing newline is added if - necessary. Each message is tagged with a priority composed of a *facility* and - a *level*. The optional *priority* argument, which defaults to - :const:`LOG_INFO`, determines the message priority. If the facility is not - encoded in *priority* using logical-or (``LOG_INFO | LOG_USER``), the value - given in the :func:`openlog` call is used. + Send the string *message* to the system logger. A trailing newline is added + if necessary. Each message is tagged with a priority composed of a + *facility* and a *level*. The optional *priority* argument, which defaults + to :const:`LOG_INFO`, determines the message priority. If the facility is + not encoded in *priority* using logical-or (``LOG_INFO | LOG_USER``), the + value given in the :func:`openlog` call is used. + + If :func:`openlog` has not been called prior to the call to :func:`syslog`, + ``openlog()`` will be called with no arguments. + +.. function:: openlog([ident[, logoption[, facility]]]) -.. function:: openlog(ident[, logopt[, facility]]) + Logging options of subsequent :func:`syslog` calls can be set by calling + :func:`openlog`. :func:`syslog` will call :func:`openlog` with no arguments + if the log is not currently open. - Logging options other than the defaults can be set by explicitly opening the log - file with :func:`openlog` prior to calling :func:`syslog`. The defaults are - (usually) *ident* = ``'syslog'``, *logopt* = ``0``, *facility* = - :const:`LOG_USER`. The *ident* argument is a string which is prepended to every - message. The optional *logopt* argument is a bit field - see below for possible - values to combine. The optional *facility* argument sets the default facility - for messages which do not have a facility explicitly encoded. + The optional *ident* keyword argument is a string which is prepended to every + message, and defaults to ``sys.argv[0]`` with leading path components + stripped. The optional *logoption* keyword argument (default is 0) is a bit + field -- see below for possible values to combine. The optional *facility* + keyword argument (default is :const:`LOG_USER`) sets the default facility for + messages which do not have a facility explicitly encoded. + + .. versionchanged:: 3.2 + In previous versions, keyword arguments were not allowed, and *ident* was + required. The default for *ident* was dependent on the system libraries, + and often was ``python`` instead of the name of the python program file. .. function:: closelog() - Close the log file. + Reset the syslog module values and call the system library ``closelog()``. + + This causes the module to behave as it does when initially imported. For + example, :func:`openlog` will be called on the first :func:`syslog` call (if + :func:`openlog` hasn't already been called), and *ident* and other + :func:`openlog` parameters are reset to defaults. .. function:: setlogmask(maskpri) - Set the priority mask to *maskpri* and return the previous mask value. Calls to - :func:`syslog` with a priority level not set in *maskpri* are ignored. The - default is to log all priorities. The function ``LOG_MASK(pri)`` calculates the - mask for the individual priority *pri*. The function ``LOG_UPTO(pri)`` - calculates the mask for all priorities up to and including *pri*. + Set the priority mask to *maskpri* and return the previous mask value. Calls + to :func:`syslog` with a priority level not set in *maskpri* are ignored. + The default is to log all priorities. The function ``LOG_MASK(pri)`` + calculates the mask for the individual priority *pri*. The function + ``LOG_UPTO(pri)`` calculates the mask for all priorities up to and including + *pri*. The module defines the following constants: @@ -63,3 +84,24 @@ Log options: :const:`LOG_PID`, :const:`LOG_CONS`, :const:`LOG_NDELAY`, :const:`LOG_NOWAIT` and :const:`LOG_PERROR` if defined in ``<syslog.h>``. + +Examples +-------- + +Simple example +~~~~~~~~~~~~~~ + +A simple set of examples:: + + import syslog + + syslog.syslog('Processing started') + if error: + syslog.syslog(syslog.LOG_ERR, 'Processing started') + +An example of setting some log options, these would include the process ID in +logged messages, and write the messages to the destination facility used for +mail logging:: + + syslog.openlog(logoption=syslog.LOG_PID, facility=syslog.LOG_MAIL) + syslog.syslog('E-mail processing initiated...') diff --git a/Doc/library/tabnanny.rst b/Doc/library/tabnanny.rst index 549ce368b7..4f3e705cab 100644 --- a/Doc/library/tabnanny.rst +++ b/Doc/library/tabnanny.rst @@ -9,6 +9,10 @@ .. rudimentary documentation based on module comments +**Source code:** :source:`Lib/tabnanny.py` + +-------------- + For the time being this module is intended to be called as a script. However it is possible to import it into an IDE and use the function :func:`check` described below. diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index d5a511eaaf..46e4900b40 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -8,10 +8,14 @@ .. moduleauthor:: Lars Gustäbel <lars@gustaebel.de> .. sectionauthor:: Lars Gustäbel <lars@gustaebel.de> +**Source code:** :source:`Lib/tarfile.py` + +-------------- The :mod:`tarfile` module makes it possible to read and write tar archives, including those using gzip or bz2 compression. -(:file:`.zip` files can be read and written using the :mod:`zipfile` module.) +Use the :mod:`zipfile` module to read or write :file:`.zip` files, or the +higher-level functions in :ref:`shutil <archiving-operations>`. Some facts and figures: @@ -20,7 +24,8 @@ Some facts and figures: * read/write support for the POSIX.1-1988 (ustar) format. * read/write support for the GNU tar format including *longname* and *longlink* - extensions, read-only support for the *sparse* extension. + extensions, read-only support for all variants of the *sparse* extension + including restoration of sparse files. * read/write support for the POSIX.1-2001 (pax) format. @@ -97,10 +102,10 @@ Some facts and figures: +-------------+--------------------------------------------+ | ``'w|'`` | Open an uncompressed *stream* for writing. | +-------------+--------------------------------------------+ - | ``'w|gz'`` | Open an gzip compressed *stream* for | + | ``'w|gz'`` | Open a gzip compressed *stream* for | | | writing. | +-------------+--------------------------------------------+ - | ``'w|bz2'`` | Open an bzip2 compressed *stream* for | + | ``'w|bz2'`` | Open a bzip2 compressed *stream* for | | | writing. | +-------------+--------------------------------------------+ @@ -185,8 +190,8 @@ The following variables are available on module level: .. data:: ENCODING - The default character encoding i.e. the value from either - :func:`sys.getfilesystemencoding` or :func:`sys.getdefaultencoding`. + The default character encoding: ``'utf-8'`` on Windows, + :func:`sys.getfilesystemencoding` otherwise. .. seealso:: @@ -209,8 +214,16 @@ a header block followed by data blocks. It is possible to store a file in a tar archive several times. Each archive member is represented by a :class:`TarInfo` object, see :ref:`tarinfo-objects` for details. +A :class:`TarFile` object can be used as a context manager in a :keyword:`with` +statement. It will automatically be closed when the block is completed. Please +note that in the event of an exception an archive opened for writing will not +be finalized; only the internally used file object will be closed. See the +:ref:`tar-examples` section for a use case. -.. class:: TarFile(name=None, mode='r', fileobj=None, format=DEFAULT_FORMAT, tarinfo=TarInfo, dereference=False, ignore_zeros=False, encoding=ENCODING, errors=None, pax_headers=None, debug=0, errorlevel=0) +.. versionadded:: 3.2 + Added support for the context manager protocol. + +.. class:: TarFile(name=None, mode='r', fileobj=None, format=DEFAULT_FORMAT, tarinfo=TarInfo, dereference=False, ignore_zeros=False, encoding=ENCODING, errors='surrogateescape', pax_headers=None, debug=0, errorlevel=0) All following arguments are optional and can be accessed as instance attributes as well. @@ -259,6 +272,9 @@ object, see :ref:`tarinfo-objects` for details. to be handled. The default settings will work for most users. See section :ref:`tar-unicode` for in-depth information. + .. versionchanged:: 3.2 + Use ``'surrogateescape'`` as the default for the *errors* argument. + The *pax_headers* argument is an optional dictionary of strings which will be added as a pax global header if *format* is :const:`PAX_FORMAT`. @@ -324,12 +340,13 @@ object, see :ref:`tarinfo-objects` for details. dots ``".."``. -.. method:: TarFile.extract(member, path="") +.. method:: TarFile.extract(member, path="", set_attrs=True) Extract a member from the archive to the current working directory, using its full name. Its file information is extracted as accurately as possible. *member* may be a filename or a :class:`TarInfo` object. You can specify a different - directory using *path*. + directory using *path*. File attributes (owner, mtime, mode) are set unless + *set_attrs* is False. .. note:: @@ -340,6 +357,8 @@ object, see :ref:`tarinfo-objects` for details. See the warning for :meth:`extractall`. + .. versionchanged:: 3.2 + Added the *set_attrs* parameter. .. method:: TarFile.extractfile(member) @@ -355,15 +374,27 @@ object, see :ref:`tarinfo-objects` for details. and :meth:`close`, and also supports iteration over its lines. -.. method:: TarFile.add(name, arcname=None, recursive=True, exclude=None) +.. method:: TarFile.add(name, arcname=None, recursive=True, exclude=None, *, filter=None) - Add the file *name* to the archive. *name* may be any type of file (directory, - fifo, symbolic link, etc.). If given, *arcname* specifies an alternative name - for the file in the archive. Directories are added recursively by default. This - can be avoided by setting *recursive* to :const:`False`. If *exclude* is given, - it must be a function that takes one filename argument and returns a boolean - value. Depending on this value the respective file is either excluded - (:const:`True`) or added (:const:`False`). + Add the file *name* to the archive. *name* may be any type of file + (directory, fifo, symbolic link, etc.). If given, *arcname* specifies an + alternative name for the file in the archive. Directories are added + recursively by default. This can be avoided by setting *recursive* to + :const:`False`. If *exclude* is given, it must be a function that takes one + filename argument and returns a boolean value. Depending on this value the + respective file is either excluded (:const:`True`) or added + (:const:`False`). If *filter* is specified it must be a keyword argument. It + should be a function that takes a :class:`TarInfo` object argument and + returns the changed :class:`TarInfo` object. If it instead returns + :const:`None` the :class:`TarInfo` object will be excluded from the + archive. See :ref:`tar-examples` for an example. + + .. versionchanged:: 3.2 + Added the *filter* parameter. + + .. deprecated:: 3.2 + The *exclude* parameter is deprecated, please use the *filter* parameter + instead. .. method:: TarFile.addfile(tarinfo, fileobj=None) @@ -430,11 +461,14 @@ It does *not* contain the file's data itself. a :class:`TarInfo` object. -.. method:: TarInfo.tobuf(format=DEFAULT_FORMAT, encoding=ENCODING, errors='strict') +.. method:: TarInfo.tobuf(format=DEFAULT_FORMAT, encoding=ENCODING, errors='surrogateescape') Create a string buffer from a :class:`TarInfo` object. For information on the arguments see the constructor of the :class:`TarFile` class. + .. versionchanged:: 3.2 + Use ``'surrogateescape'`` as the default for the *errors* argument. + A ``TarInfo`` object has the following public data attributes: @@ -582,6 +616,13 @@ How to create an uncompressed tar archive from a list of filenames:: tar.add(name) tar.close() +The same example using the :keyword:`with` statement:: + + import tarfile + with tarfile.open("sample.tar", "w") as tar: + for name in ["foo", "bar", "quux"]: + tar.add(name) + How to read a gzip compressed tar archive and display some member information:: import tarfile @@ -596,6 +637,18 @@ How to read a gzip compressed tar archive and display some member information:: print("something else.") tar.close() +How to create an archive and reset the user information using the *filter* +parameter in :meth:`TarFile.add`:: + + import tarfile + def reset(tarinfo): + tarinfo.uid = tarinfo.gid = 0 + tarinfo.uname = tarinfo.gname = "root" + return tarinfo + tar = tarfile.open("sample.tar.gz", "w:gz") + tar.add("foo", filter=reset) + tar.close() + .. _tar-formats: @@ -663,11 +716,12 @@ metadata must be either decoded or encoded. If *encoding* is not set appropriately, this conversion may fail. The *errors* argument defines how characters are treated that cannot be -converted. Possible values are listed in section :ref:`codec-base-classes`. In -read mode the default scheme is ``'replace'``. This avoids unexpected -:exc:`UnicodeError` exceptions and guarantees that an archive can always be -read. In write mode the default value for *errors* is ``'strict'``. This -ensures that name information is not altered unnoticed. - -In case of writing :const:`PAX_FORMAT` archives, *encoding* is ignored because -non-ASCII metadata is stored using *UTF-8*. +converted. Possible values are listed in section :ref:`codec-base-classes`. +The default scheme is ``'surrogateescape'`` which Python also uses for its +file system calls, see :ref:`os-filenames`. + +In case of :const:`PAX_FORMAT` archives, *encoding* is generally not needed +because all the metadata is stored using *UTF-8*. *encoding* is only used in +the rare cases when binary pax headers are decoded or when strings with +surrogate characters are stored. + diff --git a/Doc/library/telnetlib.rst b/Doc/library/telnetlib.rst index 6e3abdea4d..646634db7c 100644 --- a/Doc/library/telnetlib.rst +++ b/Doc/library/telnetlib.rst @@ -8,6 +8,10 @@ .. index:: single: protocol; Telnet +**Source code:** :source:`Lib/telnetlib.py` + +-------------- + The :mod:`telnetlib` module provides a :class:`Telnet` class that implements the Telnet protocol. See :rfc:`854` for details about the protocol. In addition, it provides symbolic constants for the protocol characters (see below), and for the diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index a13df0dd22..fff6c4eb57 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -12,6 +12,10 @@ pair: temporary; file name pair: temporary; file +**Source code:** :source:`Lib/tempfile.py` + +-------------- + This module generates temporary files and directories. It works on all supported platforms. It provides three new functions, :func:`NamedTemporaryFile`, :func:`mkstemp`, and :func:`mkdtemp`, which should @@ -25,7 +29,7 @@ no longer necessary to use the global *tempdir* and *template* variables. To maintain backward compatibility, the argument order is somewhat odd; it is recommended to use keyword arguments for clarity. -The module defines the following user-callable functions: +The module defines the following user-callable items: .. function:: TemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None) @@ -56,7 +60,7 @@ The module defines the following user-callable functions: This function operates exactly as :func:`TemporaryFile` does, except that the file is guaranteed to have a visible name in the file system (on Unix, the directory entry is not unlinked). That name can be retrieved - from the :attr:`name` member of the file object. Whether the name can be + from the :attr:`name` attribute of the file object. Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later). If *delete* is true (the default), the file is @@ -83,6 +87,24 @@ The module defines the following user-callable functions: used in a :keyword:`with` statement, just like a normal file. +.. function:: TemporaryDirectory(suffix='', prefix='tmp', dir=None) + + This function creates a temporary directory using :func:`mkdtemp` + (the supplied arguments are passed directly to the underlying function). + The resulting object can be used as a context manager (see + :ref:`context-managers`). On completion of the context (or destruction + of the temporary directory object), the newly created temporary directory + and all its contents are removed from the filesystem. + + The directory name can be retrieved from the :attr:`name` attribute + of the returned object. + + The directory can be explicitly cleaned up by calling the + :func:`cleanup` method. + + .. versionadded:: 3.2 + + .. function:: mkstemp(suffix='', prefix='tmp', dir=None, text=False) Creates a temporary file in the most secure manner possible. There are @@ -210,3 +232,36 @@ the appropriate function arguments, instead. Return the filename prefix used to create temporary files. This does not contain the directory component. + +Examples +-------- + +Here are some examples of typical usage of the :mod:`tempfile` module:: + + >>> import tempfile + + # create a temporary file and write some data to it + >>> fp = tempfile.TemporaryFile() + >>> fp.write(b'Hello world!') + # read data from file + >>> fp.seek(0) + >>> fp.read() + b'Hello world!' + # close the file, it will be removed + >>> fp.close() + + # create a temporary file using a context manager + >>> with tempfile.TemporaryFile() as fp: + ... fp.write(b'Hello world!') + ... fp.seek(0) + ... fp.read() + b'Hello world!' + >>> + # file is now closed and removed + + # create a temporary directory using the context manager + >>> with tempfile.TemporaryDirectory() as tmpdirname: + ... print('created temporary directory', tmpdirname) + >>> + # directory and contents have been removed + diff --git a/Doc/library/test.rst b/Doc/library/test.rst index 12f4c13e1b..c27ee08b58 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -5,7 +5,7 @@ :synopsis: Regression tests package containing the testing suite for Python. .. sectionauthor:: Brett Cannon <brett@python.org> -.. warning:: +.. note:: The :mod:`test` package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python's standard library is discouraged as code @@ -96,17 +96,17 @@ The goal for regression testing is to try to break code. This leads to a few guidelines to be followed: * The testing suite should exercise all classes, functions, and constants. This - includes not just the external API that is to be presented to the outside world - but also "private" code. + includes not just the external API that is to be presented to the outside + world but also "private" code. * Whitebox testing (examining the code being tested when the tests are being written) is preferred. Blackbox testing (testing only the published user - interface) is not complete enough to make sure all boundary and edge cases are - tested. + interface) is not complete enough to make sure all boundary and edge cases + are tested. * Make sure all possible values are tested including invalid ones. This makes - sure that not only all valid values are acceptable but also that improper values - are handled correctly. + sure that not only all valid values are acceptable but also that improper + values are handled correctly. * Exhaust as many code paths as possible. Test where branching occurs and thus tailor input to make sure as many different paths through the code are taken. @@ -126,8 +126,8 @@ guidelines to be followed: behavior from side-effects of importing a module. * Try to maximize code reuse. On occasion, tests will vary by something as small - as what type of input is used. Minimize code duplication by subclassing a basic - test class with a class that specifies the input:: + as what type of input is used. Minimize code duplication by subclassing a + basic test class with a class that specifies the input:: class TestFuncAcceptsSequences(unittest.TestCase): @@ -137,13 +137,13 @@ guidelines to be followed: self.func(self.arg) class AcceptLists(TestFuncAcceptsSequences): - arg = [1,2,3] + arg = [1, 2, 3] class AcceptStrings(TestFuncAcceptsSequences): arg = 'abc' class AcceptTuples(TestFuncAcceptsSequences): - arg = (1,2,3) + arg = (1, 2, 3) .. seealso:: @@ -157,46 +157,53 @@ guidelines to be followed: Running tests using the command-line interface ---------------------------------------------- -The :mod:`test.regrtest` module can be run as a script to drive Python's regression -test suite, thanks to the :option:`-m` option: :program:`python -m test.regrtest`. +The :mod:`test` package can be run as a script to drive Python's regression +test suite, thanks to the :option:`-m` option: :program:`python -m test`. Under +the hood, it uses :mod:`test.regrtest`; the call :program:`python -m +test.regrtest` used in previous Python versions still works). Running the script by itself automatically starts running all regression tests in the :mod:`test` package. It does this by finding all modules in the package whose name starts with ``test_``, importing them, and executing the -function :func:`test_main` if present. The names of tests to execute may also be -passed to the script. Specifying a single regression test (:program:`python --m test.regrtest test_spam`) will minimize output and only print whether -the test passed or failed and thus minimize output. - -Running :mod:`test.regrtest` directly allows what resources are available for -tests to use to be set. You do this by using the :option:`-u` command-line -option. Run :program:`python -m test.regrtest -uall` to turn on all +function :func:`test_main` if present. The names of tests to execute may also +be passed to the script. Specifying a single regression test (:program:`python +-m test test_spam`) will minimize output and only print +whether the test passed or failed and thus minimize output. + +Running :mod:`test` directly allows what resources are available for +tests to use to be set. You do this by using the ``-u`` command-line +option. Run :program:`python -m test -uall` to turn on all resources; specifying ``all`` as an option for ``-u`` enables all possible resources. If all but one resource is desired (a more common case), a comma-separated list of resources that are not desired may be listed after -``all``. The command :program:`python -m test.regrtest -uall,-audio,-largefile` -will run :mod:`test.regrtest` with all resources except the ``audio`` and +``all``. The command :program:`python -m test -uall,-audio,-largefile` +will run :mod:`test` with all resources except the ``audio`` and ``largefile`` resources. For a list of all resources and more command-line -options, run :program:`python -m test.regrtest -h`. +options, run :program:`python -m test -h`. Some other ways to execute the regression tests depend on what platform the tests are being executed on. On Unix, you can run :program:`make test` at the -top-level directory where Python was built. On Windows, executing -:program:`rt.bat` from your :file:`PCBuild` directory will run all regression -tests. +top-level directory where Python was built. On Windows, +executing :program:`rt.bat` from your :file:`PCBuild` directory will run all +regression tests. -:mod:`test.support` --- Utility functions for tests -=================================================== +:mod:`test.support` --- Utilities for the Python test suite +=========================================================== .. module:: test.support - :synopsis: Support for Python regression tests. + :synopsis: Support for Python's regression test suite. The :mod:`test.support` module provides support for Python's regression -tests. +test suite. + +.. note:: + :mod:`test.support` is not a public module. It is documented here to help + Python developers write tests. The API of this module is subject to change + without backwards compatibility concerns between releases. -This module defines the following exceptions: +This module defines the following exceptions: .. exception:: TestFailed @@ -205,20 +212,14 @@ This module defines the following exceptions: methods. -.. exception:: TestSkipped - - Subclass of :exc:`TestFailed`. Raised when a test is skipped. This occurs when a - needed resource (such as a network connection) is not available at the time of - testing. - - .. exception:: ResourceDenied - Subclass of :exc:`TestSkipped`. Raised when a resource (such as a network - connection) is not available. Raised by the :func:`requires` function. + Subclass of :exc:`unittest.SkipTest`. Raised when a resource (such as a + network connection) is not available. Raised by the :func:`requires` + function. -The :mod:`test.support` module defines the following constants: +The :mod:`test.support` module defines the following constants: .. data:: verbose @@ -234,44 +235,45 @@ The :mod:`test.support` module defines the following constants: .. data:: TESTFN - Set to the path that a temporary file may be created at. Any temporary that is - created should be closed and unlinked (removed). + Set to a name that is safe to use as the name of a temporary file. Any + temporary file that is created should be closed and unlinked (removed). -The :mod:`test.support` module defines the following functions: +The :mod:`test.support` module defines the following functions: .. function:: forget(module_name) - Removes the module named *module_name* from ``sys.modules`` and deletes any + Remove the module named *module_name* from ``sys.modules`` and delete any byte-compiled files of the module. .. function:: is_resource_enabled(resource) - Returns :const:`True` if *resource* is enabled and available. The list of + Return :const:`True` if *resource* is enabled and available. The list of available resources is only set when :mod:`test.regrtest` is executing the tests. .. function:: requires(resource, msg=None) - Raises :exc:`ResourceDenied` if *resource* is not available. *msg* is the - argument to :exc:`ResourceDenied` if it is raised. Always returns true if called - by a function whose ``__name__`` is ``'__main__'``. Used when tests are executed - by :mod:`test.regrtest`. + Raise :exc:`ResourceDenied` if *resource* is not available. *msg* is the + argument to :exc:`ResourceDenied` if it is raised. Always returns + :const:`True` if called by a function whose ``__name__`` is ``'__main__'``. + Used when tests are executed by :mod:`test.regrtest`. .. function:: findfile(filename) - Return the path to the file named *filename*. If no match is found *filename* is - returned. This does not equal a failure since it could be the path to the file. + Return the path to the file named *filename*. If no match is found + *filename* is returned. This does not equal a failure since it could be the + path to the file. -.. function:: run_unittest(*classes) +.. function:: run_unittest(\*classes) Execute :class:`unittest.TestCase` subclasses passed to the function. The - function scans the classes for methods starting with the prefix ``test_`` and - executes the tests individually. + function scans the classes for methods starting with the prefix ``test_`` + and executes the tests individually. It is also legal to pass strings as parameters; these should be keys in ``sys.modules``. Each associated module will be scanned by @@ -284,37 +286,72 @@ The :mod:`test.support` module defines the following functions: This will run all tests defined in the named module. -.. function:: check_warnings() +.. function:: check_warnings(\*filters, quiet=True) - A convenience wrapper for ``warnings.catch_warnings()`` that makes - it easier to test that a warning was correctly raised with a single - assertion. It is approximately equivalent to calling - ``warnings.catch_warnings(record=True)``. + A convenience wrapper for :func:`warnings.catch_warnings()` that makes it + easier to test that a warning was correctly raised. It is approximately + equivalent to calling ``warnings.catch_warnings(record=True)`` with + :meth:`warnings.simplefilter` set to ``always`` and with the option to + automatically validate the results that are recorded. - The main difference is that on entry to the context manager, a - :class:`WarningRecorder` instance is returned instead of a simple list. - The underlying warnings list is available via the recorder object's - :attr:`warnings` attribute, while the attributes of the last raised - warning are also accessible directly on the object. If no warning has - been raised, then the latter attributes will all be :const:`None`. + ``check_warnings`` accepts 2-tuples of the form ``("message regexp", + WarningCategory)`` as positional arguments. If one or more *filters* are + provided, or if the optional keyword argument *quiet* is :const:`False`, + it checks to make sure the warnings are as expected: each specified filter + must match at least one of the warnings raised by the enclosed code or the + test fails, and if any warnings are raised that do not match any of the + specified filters the test fails. To disable the first of these checks, + set *quiet* to :const:`True`. - A :meth:`reset` method is also provided on the recorder object. This - method simply clears the warning list. + If no arguments are specified, it defaults to:: - The context manager is used like this:: + check_warnings(("", Warning), quiet=True) - with check_warnings() as w: - warnings.simplefilter("always") + In this case all warnings are caught and no errors are raised. + + On entry to the context manager, a :class:`WarningRecorder` instance is + returned. The underlying warnings list from + :func:`~warnings.catch_warnings` is available via the recorder object's + :attr:`warnings` attribute. As a convenience, the attributes of the object + representing the most recent warning can also be accessed directly through + the recorder object (see example below). If no warning has been raised, + then any of the attributes that would otherwise be expected on an object + representing a warning will return :const:`None`. + + The recorder object also has a :meth:`reset` method, which clears the + warnings list. + + The context manager is designed to be used like this:: + + with check_warnings(("assertion is always true", SyntaxWarning), + ("", UserWarning)): + exec('assert(False, "Hey!")') + warnings.warn(UserWarning("Hide me!")) + + In this case if either warning was not raised, or some other warning was + raised, :func:`check_warnings` would raise an error. + + When a test needs to look more deeply into the warnings, rather than + just checking whether or not they occurred, code like this can be used:: + + with check_warnings(quiet=True) as w: warnings.warn("foo") - assert str(w.message) == "foo" + assert str(w.args[0]) == "foo" warnings.warn("bar") - assert str(w.message) == "bar" - assert str(w.warnings[0].message) == "foo" - assert str(w.warnings[1].message) == "bar" + assert str(w.args[0]) == "bar" + assert str(w.warnings[0].args[0]) == "foo" + assert str(w.warnings[1].args[0]) == "bar" w.reset() assert len(w.warnings) == 0 + Here all warnings will be caught, and the test code tests the captured + warnings directly. + + .. versionchanged:: 3.2 + New optional arguments *filters* and *quiet*. + + .. function:: captured_stdout() This is a context manager that runs the :keyword:`with` statement body using @@ -389,18 +426,19 @@ The :mod:`test.support` module defines the following classes: .. class:: EnvironmentVarGuard() - Class used to temporarily set or unset environment variables. Instances can be - used as a context manager and have a complete dictionary interface for - querying/modifying the underlying ``os.environ``. After exit from the context - manager all changes to environment variables done through this instance will - be rolled back. + Class used to temporarily set or unset environment variables. Instances can + be used as a context manager and have a complete dictionary interface for + querying/modifying the underlying ``os.environ``. After exit from the + context manager all changes to environment variables done through this + instance will be rolled back. .. versionchanged:: 3.1 Added dictionary interface. .. method:: EnvironmentVarGuard.set(envvar, value) - Temporarily set the environment variable ``envvar`` to the value of ``value``. + Temporarily set the environment variable ``envvar`` to the value of + ``value``. .. method:: EnvironmentVarGuard.unset(envvar) @@ -412,4 +450,3 @@ The :mod:`test.support` module defines the following classes: Class used to record warnings for unit tests. See documentation of :func:`check_warnings` above for more details. - diff --git a/Doc/library/textwrap.rst b/Doc/library/textwrap.rst index 835701340a..a81496206d 100644 --- a/Doc/library/textwrap.rst +++ b/Doc/library/textwrap.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Greg Ward <gward@python.net> .. sectionauthor:: Greg Ward <gward@python.net> +**Source code:** :source:`Lib/textwrap.py` + +-------------- The :mod:`textwrap` module provides two convenience functions, :func:`wrap` and :func:`fill`, as well as :class:`TextWrapper`, the class that does all the work, @@ -13,7 +16,6 @@ and a utility function :func:`dedent`. If you're just wrapping or filling one or two text strings, the convenience functions should be good enough; otherwise, you should use an instance of :class:`TextWrapper` for efficiency. - .. function:: wrap(text, width=70, **kwargs) Wraps the single paragraph in *text* (a string) so every line is at most diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index fb1880915f..9b3affd979 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -4,6 +4,9 @@ .. module:: threading :synopsis: Thread-based parallelism. +**Source code:** :source:`Lib/threading.py` + +-------------- This module constructs higher-level threading interfaces on top of the lower level :mod:`_thread` module. See also the :mod:`queue` module. @@ -24,8 +27,9 @@ The :mod:`dummy_threading` module is provided for situations where libraries might overcome this limitation). If you want your application to make better of use of the computational resources of multi-core machines, you are advised to use - :mod:`multiprocessing`. However, threading is still an appropriate model - if you want to run multiple I/O-bound tasks simultaneously. + :mod:`multiprocessing` or :class:`concurrent.futures.ProcessPoolExecutor`. + However, threading is still an appropriate model if you want to run + multiple I/O-bound tasks simultaneously. This module defines the following functions and objects: @@ -182,6 +186,18 @@ This module defines the following functions and objects: Availability: Windows, systems with POSIX threads. +This module also defines the following constant: + +.. data:: TIMEOUT_MAX + + The maximum value allowed for the *timeout* parameter of blocking functions + (:meth:`Lock.acquire`, :meth:`RLock.acquire`, :meth:`Condition.wait`, etc.). + Specifying a timeout greater than this value will raise an + :exc:`OverflowError`. + + .. versionadded:: 3.2 + + Detailed interfaces for the objects are documented below. The design of this module is loosely based on Java's threading model. However, @@ -376,7 +392,7 @@ and may vary across implementations. All methods are executed atomically. -.. method:: Lock.acquire([blocking]) +.. method:: Lock.acquire(blocking=True, timeout=-1) Acquire a lock, blocking or non-blocking. @@ -390,6 +406,21 @@ All methods are executed atomically. without an argument would block, return false immediately; otherwise, do the same thing as when called without arguments, and return true. + When invoked with the floating-point *timeout* argument set to a positive + value, block for at most the number of seconds specified by *timeout* + and as long as the lock cannot be acquired. A negative *timeout* argument + specifies an unbounded wait. It is forbidden to specify a *timeout* + when *blocking* is false. + + The return value is ``True`` if the lock is acquired successfully, + ``False`` if not (for example if the *timeout* expired). + + .. versionchanged:: 3.2 + The *timeout* parameter is new. + + .. versionchanged:: 3.2 + Lock acquires can now be interrupted by signals on POSIX. + .. method:: Lock.release() @@ -423,7 +454,7 @@ pair) resets the lock to unlocked and allows another thread blocked in :meth:`acquire` to proceed. -.. method:: RLock.acquire(blocking=True) +.. method:: RLock.acquire(blocking=True, timeout=-1) Acquire a lock, blocking or non-blocking. @@ -442,6 +473,14 @@ pair) resets the lock to unlocked and allows another thread blocked in without an argument would block, return false immediately; otherwise, do the same thing as when called without arguments, and return true. + When invoked with the floating-point *timeout* argument set to a positive + value, block for at most the number of seconds specified by *timeout* + and as long as the lock cannot be acquired. Return true if the lock has + been acquired, false if the timeout has elapsed. + + .. versionchanged:: 3.2 + The *timeout* parameter is new. + .. method:: RLock.release() @@ -513,6 +552,13 @@ state change can be interesting for only one or several waiting threads. E.g. in a typical producer-consumer situation, adding one item to the buffer only needs to wake up one consumer thread. +Note: Condition variables can be, depending on the implementation, subject +to both spurious wakeups (when :meth:`wait` returns without a :meth:`notify` +call) and stolen wakeups (when another thread acquires the lock before the +awoken thread.) For this reason, it is always necessary to verify the state +the thread is waiting for when :meth:`wait` returns and optionally repeat +the call as often as necessary. + .. class:: Condition(lock=None) @@ -553,20 +599,56 @@ needs to wake up one consumer thread. interface is then used to restore the recursion level when the lock is reacquired. - .. method:: notify() + The return value is ``True`` unless a given *timeout* expired, in which + case it is ``False``. + + .. versionchanged:: 3.2 + Previously, the method always returned ``None``. + + .. method:: wait_for(predicate, timeout=None) - Wake up a thread waiting on this condition, if any. If the calling thread - has not acquired the lock when this method is called, a + Wait until a condition evaluates to True. *predicate* should be a + callable which result will be interpreted as a boolean value. + A *timeout* may be provided giving the maximum time to wait. + + This utility method may call :meth:`wait` repeatedly until the predicate + is satisfied, or until a timeout occurs. The return value is + the last return value of the predicate and will evaluate to + ``False`` if the method timed out. + + Ignoring the timeout feature, calling this method is roughly equivalent to + writing:: + + while not predicate(): + cv.wait() + + Therefore, the same rules apply as with :meth:`wait`: The lock must be + held when called and is re-aquired on return. The predicate is evaluated + with the lock held. + + Using this method, the consumer example above can be written thus:: + + with cv: + cv.wait_for(an_item_is_available) + get_an_available_item() + + .. versionadded:: 3.2 + + .. method:: notify(n=1) + + By default, wake up one thread waiting on this condition, if any. If the + calling thread has not acquired the lock when this method is called, a :exc:`RuntimeError` is raised. - This method wakes up one of the threads waiting for the condition - variable, if any are waiting; it is a no-op if no threads are waiting. + This method wakes up at most *n* of the threads waiting for the condition + variable; it is a no-op if no threads are waiting. - The current implementation wakes up exactly one thread, if any are - waiting. However, it's not safe to rely on this behavior. A future, - optimized implementation may occasionally wake up more than one thread. + The current implementation wakes up exactly *n* threads, if at least *n* + threads are waiting. However, it's not safe to rely on this behavior. + A future, optimized implementation may occasionally wake up more than + *n* threads. - Note: the awakened thread does not actually return from its :meth:`wait` + Note: an awakened thread does not actually return from its :meth:`wait` call until it can reacquire the lock. Since :meth:`notify` does not release the lock, its caller should. @@ -599,7 +681,7 @@ waiting until some other thread calls :meth:`release`. defaults to ``1``. If the *value* given is less than 0, :exc:`ValueError` is raised. - .. method:: acquire(blocking=True) + .. method:: acquire(blocking=True, timeout=None) Acquire a semaphore. @@ -610,14 +692,18 @@ waiting until some other thread calls :meth:`release`. interlocking so that if multiple :meth:`acquire` calls are blocked, :meth:`release` will wake exactly one of them up. The implementation may pick one at random, so the order in which blocked threads are awakened - should not be relied on. There is no return value in this case. - - When invoked with *blocking* set to true, do the same thing as when called - without arguments, and return true. + should not be relied on. Returns true (or blocks indefinitely). When invoked with *blocking* set to false, do not block. If a call - without an argument would block, return false immediately; otherwise, do - the same thing as when called without arguments, and return true. + without an argument would block, return false immediately; otherwise, + do the same thing as when called without arguments, and return true. + + When invoked with a *timeout* other than None, it will block for at + most *timeout* seconds. If acquire does not complete successfully in + that interval, return false. Return true otherwise. + + .. versionchanged:: 3.2 + The *timeout* parameter is new. .. method:: release() @@ -696,8 +782,10 @@ An event object manages an internal flag that can be set to true with the floating point number specifying a timeout for the operation in seconds (or fractions thereof). - This method returns the internal flag on exit, so it will always return - ``True`` except if a timeout is given and the operation times out. + This method returns true if and only if the internal flag has been set to + true, either before the wait call or after the wait starts, so it will + always return ``True`` except if a timeout is given and the operation + times out. .. versionchanged:: 3.1 Previously, the method always returned ``None``. @@ -737,6 +825,108 @@ For example:: only work if the timer is still in its waiting stage. +Barrier Objects +--------------- + +.. versionadded:: 3.2 + +This class provides a simple synchronization primitive for use by a fixed number +of threads that need to wait for each other. Each of the threads tries to pass +the barrier by calling the :meth:`wait` method and will block until all of the +threads have made the call. At this points, the threads are released +simultanously. + +The barrier can be reused any number of times for the same number of threads. + +As an example, here is a simple way to synchronize a client and server thread:: + + b = Barrier(2, timeout=5) + + def server(): + start_server() + b.wait() + while True: + connection = accept_connection() + process_server_connection(connection) + + def client(): + b.wait() + while True: + connection = make_connection() + process_client_connection(connection) + + +.. class:: Barrier(parties, action=None, timeout=None) + + Create a barrier object for *parties* number of threads. An *action*, when + provided, is a callable to be called by one of the threads when they are + released. *timeout* is the default timeout value if none is specified for + the :meth:`wait` method. + + .. method:: wait(timeout=None) + + Pass the barrier. When all the threads party to the barrier have called + this function, they are all released simultaneously. If a *timeout* is + provided, it is used in preference to any that was supplied to the class + constructor. + + The return value is an integer in the range 0 to *parties* -- 1, different + for each thread. This can be used to select a thread to do some special + housekeeping, e.g.:: + + i = barrier.wait() + if i == 0: + # Only one thread needs to print this + print("passed the barrier") + + If an *action* was provided to the constructor, one of the threads will + have called it prior to being released. Should this call raise an error, + the barrier is put into the broken state. + + If the call times out, the barrier is put into the broken state. + + This method may raise a :class:`BrokenBarrierError` exception if the + barrier is broken or reset while a thread is waiting. + + .. method:: reset() + + Return the barrier to the default, empty state. Any threads waiting on it + will receive the :class:`BrokenBarrierError` exception. + + Note that using this function may can require some external + synchronization if there are other threads whose state is unknown. If a + barrier is broken it may be better to just leave it and create a new one. + + .. method:: abort() + + Put the barrier into a broken state. This causes any active or future + calls to :meth:`wait` to fail with the :class:`BrokenBarrierError`. Use + this for example if one of the needs to abort, to avoid deadlocking the + application. + + It may be preferable to simply create the barrier with a sensible + *timeout* value to automatically guard against one of the threads going + awry. + + .. attribute:: parties + + The number of threads required to pass the barrier. + + .. attribute:: n_waiting + + The number of threads currently waiting in the barrier. + + .. attribute:: broken + + A boolean that is ``True`` if the barrier is in the broken state. + + +.. exception:: BrokenBarrierError + + This exception, a subclass of :exc:`RuntimeError`, is raised when the + :class:`Barrier` object is reset or broken. + + .. _with-locks: Using locks, conditions, and semaphores in the :keyword:`with` statement diff --git a/Doc/library/time.rst b/Doc/library/time.rst index b91aa539df..7c464ac245 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -24,9 +24,9 @@ An explanation of some terminology and conventions is in order. .. index:: single: Year 2038 -* The functions in this module do not handle dates and times before the epoch or +* The functions in this module may not handle dates and times before the epoch or far in the future. The cut-off point in the future is determined by the C - library; for Unix, it is typically in 2038. + library; for 32-bit systems, it is typically in 2038. .. index:: single: Year 2000 @@ -34,20 +34,31 @@ An explanation of some terminology and conventions is in order. .. _time-y2kissues: -* **Year 2000 (Y2K) issues**: Python depends on the platform's C library, which +* **Year 2000 (Y2K) issues**: Python depends on the platform's C library, which generally doesn't have year 2000 issues, since all dates and times are - represented internally as seconds since the epoch. Functions accepting a - :class:`struct_time` (see below) generally require a 4-digit year. For backward - compatibility, 2-digit years are supported if the module variable - ``accept2dyear`` is a non-zero integer; this variable is initialized to ``1`` - unless the environment variable :envvar:`PYTHONY2K` is set to a non-empty - string, in which case it is initialized to ``0``. Thus, you can set - :envvar:`PYTHONY2K` to a non-empty string in the environment to require 4-digit - years for all year input. When 2-digit years are accepted, they are converted - according to the POSIX or X/Open standard: values 69-99 are mapped to 1969-1999, - and values 0--68 are mapped to 2000--2068. Values 100--1899 are always illegal. - Note that this is new as of Python 1.5.2(a2); earlier versions, up to Python - 1.5.1 and 1.5.2a1, would add 1900 to year values below 1900. + represented internally as seconds since the epoch. Function :func:`strptime` + can parse 2-digit years when given ``%y`` format code. When 2-digit years are + parsed, they are converted according to the POSIX and ISO C standards: values + 69--99 are mapped to 1969--1999, and values 0--68 are mapped to 2000--2068. + + For backward compatibility, years with less than 4 digits are treated + specially by :func:`asctime`, :func:`mktime`, and :func:`strftime` functions + that operate on a 9-tuple or :class:`struct_time` values. If year (the first + value in the 9-tuple) is specified with less than 4 digits, its interpretation + depends on the value of ``accept2dyear`` variable. + + If ``accept2dyear`` is true (default), a backward compatibility behavior is + invoked as follows: + + - for 2-digit year, century is guessed according to POSIX rules for + ``%y`` strptime format. A deprecation warning is issued when century + information is guessed in this way. + + - for 3-digit or negative year, a :exc:`ValueError` exception is raised. + + If ``accept2dyear`` is false (set by the program or as a result of a + non-empty value assigned to ``PYTHONY2K`` environment variable) all year + values are interpreted as given. .. index:: single: UTC @@ -73,8 +84,8 @@ An explanation of some terminology and conventions is in order. * On the other hand, the precision of :func:`time` and :func:`sleep` is better than their Unix equivalents: times are expressed as floating point numbers, :func:`time` returns the most accurate time available (using Unix - :cfunc:`gettimeofday` where available), and :func:`sleep` will accept a time - with a nonzero fraction (Unix :cfunc:`select` is used to implement this, where + :c:func:`gettimeofday` where available), and :func:`sleep` will accept a time + with a nonzero fraction (Unix :c:func:`select` is used to implement this, where available). * The time value as returned by :func:`gmtime`, :func:`localtime`, and @@ -109,10 +120,19 @@ The module defines the following functions and data items: .. data:: accept2dyear - Boolean value indicating whether two-digit year values will be accepted. This - is true by default, but will be set to false if the environment variable - :envvar:`PYTHONY2K` has been set to a non-empty string. It may also be modified - at run time. + Boolean value indicating whether two-digit year values will be + mapped to 1969--2068 range by :func:`asctime`, :func:`mktime`, and + :func:`strftime` functions. This is true by default, but will be + set to false if the environment variable :envvar:`PYTHONY2K` has + been set to a non-empty string. It may also be modified at run + time. + + .. deprecated:: 3.2 + Mapping of 2-digit year values by :func:`asctime`, + :func:`mktime`, and :func:`strftime` functions to 1969--2068 + range is deprecated. Programs that need to process 2-digit + years should use ``%y`` code available in :func:`strptime` + function or convert 2-digit year values to 4-digit themselves. .. data:: altzone @@ -125,7 +145,7 @@ The module defines the following functions and data items: .. function:: asctime([t]) Convert a tuple or :class:`struct_time` representing a time as returned by - :func:`gmtime` or :func:`localtime` to a 24-character string of the following + :func:`gmtime` or :func:`localtime` to a string of the following form: ``'Sun Jun 20 23:21:05 1993'``. If *t* is not provided, the current time as returned by :func:`localtime` is used. Locale information is not used by :func:`asctime`. @@ -149,7 +169,7 @@ The module defines the following functions and data items: On Windows, this function returns wall-clock seconds elapsed since the first call to this function, as a floating point number, based on the Win32 function - :cfunc:`QueryPerformanceCounter`. The resolution is typically better than one + :c:func:`QueryPerformanceCounter`. The resolution is typically better than one microsecond. @@ -288,7 +308,7 @@ The module defines the following functions and data items: | ``%y`` | Year without century as a decimal number | | | | [00,99]. | | +-----------+------------------------------------------------+-------+ - | ``%Y`` | Year with century as a decimal number. | | + | ``%Y`` | Year with century as a decimal number. | \(4) | | | | | +-----------+------------------------------------------------+-------+ | ``%Z`` | Time zone name (no characters if no time zone | | @@ -304,13 +324,20 @@ The module defines the following functions and data items: the output hour field if the ``%I`` directive is used to parse the hour. (2) - The range really is ``0`` to ``61``; this accounts for leap seconds and the - (very rare) double leap seconds. + The range really is ``0`` to ``61``; value ``60`` is valid in + timestamps representing leap seconds and value ``61`` is supported + for historical reasons. (3) When used with the :func:`strptime` function, ``%U`` and ``%W`` are only used in calculations when the day of the week and the year are specified. + (4) + Produces different results depending on the value of + ``time.accept2dyear`` variable. See :ref:`Year 2000 (Y2K) + issues <time-y2kissues>` for details. + + Here is an example, a format for dates compatible with that specified in the :rfc:`2822` Internet email standard. [#]_ :: @@ -380,7 +407,7 @@ The module defines the following functions and data items: +-------+-------------------+---------------------------------+ | 4 | :attr:`tm_min` | range [0, 59] | +-------+-------------------+---------------------------------+ - | 5 | :attr:`tm_sec` | range [0, 61]; see **(1)** in | + | 5 | :attr:`tm_sec` | range [0, 61]; see **(2)** in | | | | :func:`strftime` description | +-------+-------------------+---------------------------------+ | 6 | :attr:`tm_wday` | range [0, 6], Monday is 0 | @@ -526,6 +553,6 @@ The module defines the following functions and data items: preferred hour/minute offset is not supported by all ANSI C libraries. Also, a strict reading of the original 1982 :rfc:`822` standard calls for a two-digit year (%y rather than %Y), but practice moved to 4-digit years long before the - year 2000. The 4-digit year has been mandated by :rfc:`2822`, which obsoletes - :rfc:`822`. + year 2000. After that, :rfc:`822` became obsolete and the 4-digit year has + been first recommended by :rfc:`1123` and then mandated by :rfc:`2822`. diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst index 4a0b9c257b..0112994040 100644 --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -9,6 +9,10 @@ single: Benchmarking single: Performance +**Source code:** :source:`Lib/timeit.py` + +-------------- + This module provides a simple way to time small bits of Python code. It has both command line as well as callable interfaces. It avoids a number of common traps for measuring execution times. See also Tim Peters' introduction to the @@ -187,13 +191,13 @@ interface) that compare the cost of using :func:`hasattr` vs. :keyword:`try`/:keyword:`except` to test for missing and present object attributes. :: - % timeit.py 'try:' ' str.__bool__' 'except AttributeError:' ' pass' + $ python -m timeit 'try:' ' str.__bool__' 'except AttributeError:' ' pass' 100000 loops, best of 3: 15.7 usec per loop - % timeit.py 'if hasattr(str, "__bool__"): pass' + $ python -m timeit 'if hasattr(str, "__bool__"): pass' 100000 loops, best of 3: 4.26 usec per loop - % timeit.py 'try:' ' int.__bool__' 'except AttributeError:' ' pass' + $ python -m timeit 'try:' ' int.__bool__' 'except AttributeError:' ' pass' 1000000 loops, best of 3: 1.43 usec per loop - % timeit.py 'if hasattr(int, "__bool__"): pass' + $ python -m timeit 'if hasattr(int, "__bool__"): pass' 100000 loops, best of 3: 2.23 usec per loop :: @@ -234,10 +238,10 @@ To give the :mod:`timeit` module access to functions you define, you can pass a ``setup`` parameter which contains an import statement:: def test(): - "Stupid test function" + """Stupid test function""" L = [i for i in range(100)] - if __name__=='__main__': + if __name__ == '__main__': from timeit import Timer t = Timer("test()", "from __main__ import test") print(t.timeit()) diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst index 7bb54fd4fa..ae5635f3c8 100644 --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -9,7 +9,9 @@ The :mod:`tkinter` package ("Tk interface") is the standard Python interface to the Tk GUI toolkit. Both Tk and :mod:`tkinter` are available on most Unix platforms, as well as on Windows systems. (Tk itself is not part of Python; it -is maintained at ActiveState.) +is maintained at ActiveState.) You can check that :mod:`tkinter` is properly +installed on your system by running ``python -m tkinter`` from the command line; +this should open a window demonstrating a simple Tk interface. .. seealso:: @@ -657,9 +659,7 @@ relief scrollcommand This is almost always the :meth:`!set` method of some scrollbar widget, but can - be any widget method that takes a single argument. Refer to the file - :file:`Demo/tkinter/matt/canvas-with-scrollbars.py` in the Python source - distribution for an example. + be any widget method that takes a single argument. wrap: Must be one of: ``"none"``, ``"char"``, or ``"word"``. diff --git a/Doc/library/tkinter.tix.rst b/Doc/library/tkinter.tix.rst index beb91e6ee1..289bffd1ad 100644 --- a/Doc/library/tkinter.tix.rst +++ b/Doc/library/tkinter.tix.rst @@ -84,11 +84,7 @@ Tix Widgets ----------- `Tix <http://tix.sourceforge.net/dist/current/man/html/TixCmd/TixIntro.htm>`_ -introduces over 40 widget classes to the :mod:`tkinter` repertoire. There is a -demo of all the :mod:`tkinter.tix` widgets in the :file:`Demo/tix` directory of -the standard distribution. - -.. The Python sample code is still being added to Python, hence commented out +introduces over 40 widget classes to the :mod:`tkinter` repertoire. Basic Widgets diff --git a/Doc/library/tkinter.ttk.rst b/Doc/library/tkinter.ttk.rst index 7bf39b3109..ed351f5a3c 100644 --- a/Doc/library/tkinter.ttk.rst +++ b/Doc/library/tkinter.ttk.rst @@ -1240,7 +1240,7 @@ option. If you don't know the class name of a widget, use the method *layoutspec*, if specified, is expected to be a list or some other sequence type (excluding strings), where each item should be a tuple and the first item is the layout name and the second item should have the - format described described in `Layouts`_. + format described in `Layouts`_. To understand the format, see the following example (it is not intended to do anything useful):: @@ -1291,7 +1291,7 @@ option. If you don't know the class name of a widget, use the method * sticky=spec Specifies how the image is placed within the final parcel. spec - contains zero or more characters “n”, “s”, “w”, or “e”. + contains zero or more characters "n", "s", "w", or "e". * width=width Specifies a minimum width for the element. If less than zero, the diff --git a/Doc/library/token.rst b/Doc/library/token.rst index 991762f4e6..4cd709814c 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -5,6 +5,9 @@ :synopsis: Constants representing terminal nodes of the parse tree. .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/token.py` + +-------------- This module provides constants which represent the numeric values of leaf nodes of the parse tree (terminal tokens). Refer to the file :file:`Grammar/Grammar` @@ -65,7 +68,6 @@ The token constants are: EQUAL DOT PERCENT - BACKQUOTE LBRACE RBRACE EQEQUAL @@ -91,6 +93,8 @@ The token constants are: DOUBLESLASH DOUBLESLASHEQUAL AT + RARROW + ELLIPSIS OP ERRORTOKEN N_TOKENS diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 7017045f61..70919ca998 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -6,12 +6,21 @@ .. moduleauthor:: Ka Ping Yee .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/tokenize.py` + +-------------- The :mod:`tokenize` module provides a lexical scanner for Python source code, implemented in Python. The scanner in this module returns comments as tokens as well, making it useful for implementing "pretty-printers," including colorizers for on-screen displays. +To simplify token stream handling, all :ref:`operators` and :ref:`delimiters` +tokens are returned using the generic :data:`token.OP` token type. The exact +type can be determined by checking the token ``string`` field on the +:term:`named tuple` returned from :func:`tokenize.tokenize` for the character +sequence that identifies a specific operator token. + The primary entry point is a :term:`generator`: .. function:: tokenize(readline) @@ -95,12 +104,25 @@ function it uses to do this is available: It detects the encoding from the presence of a UTF-8 BOM or an encoding cookie as specified in :pep:`263`. If both a BOM and a cookie are present, - but disagree, a SyntaxError will be raised. + but disagree, a SyntaxError will be raised. Note that if the BOM is found, + ``'utf-8-sig'`` will be returned as an encoding. + + If no encoding is specified, then the default of ``'utf-8'`` will be + returned. + + Use :func:`open` to open Python source files: it uses + :func:`detect_encoding` to detect the file encoding. - If no encoding is specified, then the default of ``'utf-8'`` will be returned. +.. function:: open(filename) -Example of a script re-writer that transforms float literals into Decimal + Open a file in read only mode using the encoding detected by + :func:`detect_encoding`. + + .. versionadded:: 3.2 + + +Example of a script rewriter that transforms float literals into Decimal objects:: from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP @@ -142,4 +164,3 @@ objects:: result.append((toknum, tokval)) return untokenize(result).decode('utf-8') - diff --git a/Doc/library/trace.rst b/Doc/library/trace.rst index 7b49c8fd49..c4ddc56cf2 100644 --- a/Doc/library/trace.rst +++ b/Doc/library/trace.rst @@ -4,13 +4,15 @@ .. module:: trace :synopsis: Trace or track Python statement execution. +**Source code:** :source:`Lib/trace.py` + +-------------- The :mod:`trace` module allows you to trace program execution, generate annotated statement coverage listings, print caller/callee relationships and list functions executed during a program run. It can be used in another program or from the command line. - .. _trace-cli: Command-Line Usage diff --git a/Doc/library/turtle-star.pdf b/Doc/library/turtle-star.pdf Binary files differnew file mode 100644 index 0000000000..e354073dd4 --- /dev/null +++ b/Doc/library/turtle-star.pdf diff --git a/Doc/library/turtle-star.png b/Doc/library/turtle-star.png Binary files differnew file mode 100644 index 0000000000..caf36a3ab3 --- /dev/null +++ b/Doc/library/turtle-star.png diff --git a/Doc/library/turtle-star.ps b/Doc/library/turtle-star.ps new file mode 100644 index 0000000000..46362cb9f7 --- /dev/null +++ b/Doc/library/turtle-star.ps @@ -0,0 +1,447 @@ +%!PS-Adobe-3.0 EPSF-3.0 +%%Creator: Tk Canvas Widget +%%For: Alexander Belopolsky +%%Title: Window .4315905424 +%%CreationDate: Tue Nov 9 12:54:06 2010 +%%XBoundingBox: -172 -52 785 845 +%%BoundingBox: 290 290 520 520 +%%Pages: 1 +%%DocumentData: Clean7Bit +%%Orientation: Portrait +%%EndComments + +%%BeginProlog +/CurrentEncoding [ +/space/space/space/space/space/space/space/space +/space/space/space/space/space/space/space/space +/space/space/space/space/space/space/space/space +/space/space/space/space/space/space/space/space +/space/exclam/quotedbl/numbersign/dollar/percent/ampersand/quotesingle +/parenleft/parenright/asterisk/plus/comma/hyphen/period/slash +/zero/one/two/three/four/five/six/seven +/eight/nine/colon/semicolon/less/equal/greater/question +/at/A/B/C/D/E/F/G +/H/I/J/K/L/M/N/O +/P/Q/R/S/T/U/V/W +/X/Y/Z/bracketleft/backslash/bracketright/asciicircum/underscore +/grave/a/b/c/d/e/f/g +/h/i/j/k/l/m/n/o +/p/q/r/s/t/u/v/w +/x/y/z/braceleft/bar/braceright/asciitilde/space +/space/space/space/space/space/space/space/space +/space/space/space/space/space/space/space/space +/space/space/space/space/space/space/space/space +/space/space/space/space/space/space/space/space +/space/exclamdown/cent/sterling/currency/yen/brokenbar/section +/dieresis/copyright/ordfeminine/guillemotleft/logicalnot/hyphen/registered/macron +/degree/plusminus/twosuperior/threesuperior/acute/mu/paragraph/periodcentered +/cedilla/onesuperior/ordmasculine/guillemotright/onequarter/onehalf/threequarters/questiondown +/Agrave/Aacute/Acircumflex/Atilde/Adieresis/Aring/AE/Ccedilla +/Egrave/Eacute/Ecircumflex/Edieresis/Igrave/Iacute/Icircumflex/Idieresis +/Eth/Ntilde/Ograve/Oacute/Ocircumflex/Otilde/Odieresis/multiply +/Oslash/Ugrave/Uacute/Ucircumflex/Udieresis/Yacute/Thorn/germandbls +/agrave/aacute/acircumflex/atilde/adieresis/aring/ae/ccedilla +/egrave/eacute/ecircumflex/edieresis/igrave/iacute/icircumflex/idieresis +/eth/ntilde/ograve/oacute/ocircumflex/otilde/odieresis/divide +/oslash/ugrave/uacute/ucircumflex/udieresis/yacute/thorn/ydieresis +] def + +50 dict begin +% This is a standard prolog for Postscript generated by Tk's canvas +% widget. +% RCS: @(#) $Id$ + +% The definitions below just define all of the variables used in +% any of the procedures here. This is needed for obscure reasons +% explained on p. 716 of the Postscript manual (Section H.2.7, +% "Initializing Variables," in the section on Encapsulated Postscript). + +/baseline 0 def +/stipimage 0 def +/height 0 def +/justify 0 def +/lineLength 0 def +/spacing 0 def +/stipple 0 def +/strings 0 def +/xoffset 0 def +/yoffset 0 def +/tmpstip null def + + +/cstringshow { + { + dup type /stringtype eq + { show } { glyphshow } + ifelse + } + forall +} bind def + + + +/cstringwidth { + 0 exch 0 exch + { + dup type /stringtype eq + { stringwidth } { + currentfont /Encoding get exch 1 exch put (\001) stringwidth + } + ifelse + exch 3 1 roll add 3 1 roll add exch + } + forall +} bind def + +% font ISOEncode font +% This procedure changes the encoding of a font from the default +% Postscript encoding to current system encoding. It's typically invoked just +% before invoking "setfont". The body of this procedure comes from +% Section 5.6.1 of the Postscript book. + +/ISOEncode { + dup length dict begin + {1 index /FID ne {def} {pop pop} ifelse} forall + /Encoding CurrentEncoding def + currentdict + end + + % I'm not sure why it's necessary to use "definefont" on this new + % font, but it seems to be important; just use the name "Temporary" + % for the font. + + /Temporary exch definefont +} bind def + +% StrokeClip +% +% This procedure converts the current path into a clip area under +% the assumption of stroking. It's a bit tricky because some Postscript +% interpreters get errors during strokepath for dashed lines. If +% this happens then turn off dashes and try again. + +/StrokeClip { + {strokepath} stopped { + (This Postscript printer gets limitcheck overflows when) = + (stippling dashed lines; lines will be printed solid instead.) = + [] 0 setdash strokepath} if + clip +} bind def + +% desiredSize EvenPixels closestSize +% +% The procedure below is used for stippling. Given the optimal size +% of a dot in a stipple pattern in the current user coordinate system, +% compute the closest size that is an exact multiple of the device's +% pixel size. This allows stipple patterns to be displayed without +% aliasing effects. + +/EvenPixels { + % Compute exact number of device pixels per stipple dot. + dup 0 matrix currentmatrix dtransform + dup mul exch dup mul add sqrt + + % Round to an integer, make sure the number is at least 1, and compute + % user coord distance corresponding to this. + dup round dup 1 lt {pop 1} if + exch div mul +} bind def + +% width height string StippleFill -- +% +% Given a path already set up and a clipping region generated from +% it, this procedure will fill the clipping region with a stipple +% pattern. "String" contains a proper image description of the +% stipple pattern and "width" and "height" give its dimensions. Each +% stipple dot is assumed to be about one unit across in the current +% user coordinate system. This procedure trashes the graphics state. + +/StippleFill { + % The following code is needed to work around a NeWSprint bug. + + /tmpstip 1 index def + + % Change the scaling so that one user unit in user coordinates + % corresponds to the size of one stipple dot. + 1 EvenPixels dup scale + + % Compute the bounding box occupied by the path (which is now + % the clipping region), and round the lower coordinates down + % to the nearest starting point for the stipple pattern. Be + % careful about negative numbers, since the rounding works + % differently on them. + + pathbbox + 4 2 roll + 5 index div dup 0 lt {1 sub} if cvi 5 index mul 4 1 roll + 6 index div dup 0 lt {1 sub} if cvi 6 index mul 3 2 roll + + % Stack now: width height string y1 y2 x1 x2 + % Below is a doubly-nested for loop to iterate across this area + % in units of the stipple pattern size, going up columns then + % across rows, blasting out a stipple-pattern-sized rectangle at + % each position + + 6 index exch { + 2 index 5 index 3 index { + % Stack now: width height string y1 y2 x y + + gsave + 1 index exch translate + 5 index 5 index true matrix tmpstip imagemask + grestore + } for + pop + } for + pop pop pop pop pop +} bind def + +% -- AdjustColor -- +% Given a color value already set for output by the caller, adjusts +% that value to a grayscale or mono value if requested by the CL +% variable. + +/AdjustColor { + CL 2 lt { + currentgray + CL 0 eq { + .5 lt {0} {1} ifelse + } if + setgray + } if +} bind def + +% x y strings spacing xoffset yoffset justify stipple DrawText -- +% This procedure does all of the real work of drawing text. The +% color and font must already have been set by the caller, and the +% following arguments must be on the stack: +% +% x, y - Coordinates at which to draw text. +% strings - An array of strings, one for each line of the text item, +% in order from top to bottom. +% spacing - Spacing between lines. +% xoffset - Horizontal offset for text bbox relative to x and y: 0 for +% nw/w/sw anchor, -0.5 for n/center/s, and -1.0 for ne/e/se. +% yoffset - Vertical offset for text bbox relative to x and y: 0 for +% nw/n/ne anchor, +0.5 for w/center/e, and +1.0 for sw/s/se. +% justify - 0 for left justification, 0.5 for center, 1 for right justify. +% stipple - Boolean value indicating whether or not text is to be +% drawn in stippled fashion. If text is stippled, +% procedure StippleText must have been defined to call +% StippleFill in the right way. +% +% Also, when this procedure is invoked, the color and font must already +% have been set for the text. + +/DrawText { + /stipple exch def + /justify exch def + /yoffset exch def + /xoffset exch def + /spacing exch def + /strings exch def + + % First scan through all of the text to find the widest line. + + /lineLength 0 def + strings { + cstringwidth pop + dup lineLength gt {/lineLength exch def} {pop} ifelse + newpath + } forall + + % Compute the baseline offset and the actual font height. + + 0 0 moveto (TXygqPZ) false charpath + pathbbox dup /baseline exch def + exch pop exch sub /height exch def pop + newpath + + % Translate coordinates first so that the origin is at the upper-left + % corner of the text's bounding box. Remember that x and y for + % positioning are still on the stack. + + translate + lineLength xoffset mul + strings length 1 sub spacing mul height add yoffset mul translate + + % Now use the baseline and justification information to translate so + % that the origin is at the baseline and positioning point for the + % first line of text. + + justify lineLength mul baseline neg translate + + % Iterate over each of the lines to output it. For each line, + % compute its width again so it can be properly justified, then + % display it. + + strings { + dup cstringwidth pop + justify neg mul 0 moveto + stipple { + + + % The text is stippled, so turn it into a path and print + % by calling StippledText, which in turn calls StippleFill. + % Unfortunately, many Postscript interpreters will get + % overflow errors if we try to do the whole string at + % once, so do it a character at a time. + + gsave + /char (X) def + { + dup type /stringtype eq { + % This segment is a string. + { + char 0 3 -1 roll put + currentpoint + gsave + char true charpath clip StippleText + grestore + char stringwidth translate + moveto + } forall + } { + % This segment is glyph name + % Temporary override + currentfont /Encoding get exch 1 exch put + currentpoint + gsave (\001) true charpath clip StippleText + grestore + (\001) stringwidth translate + moveto + } ifelse + } forall + grestore + } {cstringshow} ifelse + 0 spacing neg translate + } forall +} bind def + +%%EndProlog +%%BeginSetup +/CL 2 def +%%EndSetup + +%%Page: 1 1 +save +306.0 396.0 translate +0.9995 0.9995 scale +4 -449 translate +-483 898 moveto 475 898 lineto 475 0 lineto -483 0 lineto closepath clip newpath +gsave +grestore +gsave +0 445 moveto +200 445 lineto +3.03844939755837 479.729635533386 lineto +190.97697355474 411.325606868252 lineto +17.7718927978523 511.325606868252 lineto +170.980781421648 382.768084930944 lineto +42.42325948434 535.97697355474 lineto +142.42325948434 362.771892797852 lineto +74.0192308192062 550.710416955034 lineto +108.748866352592 353.748866352592 lineto +108.748866352592 553.748866352592 lineto +74.0192308192064 356.787315750151 lineto +142.42325948434 544.725839907333 lineto +42.4232594843401 371.520759150445 lineto +170.980781421648 524.72964777424 lineto +17.7718927978524 396.172125836932 lineto +190.97697355474 496.172125836933 lineto +3.03844939755834 427.768097171799 lineto +200 462.497732705185 lineto +-1.13686837721616e-13 462.497732705185 lineto +196.961550602442 427.768097171799 lineto +9.02302644525972 496.172125836932 lineto +182.228107202148 396.172125836933 lineto +29.0192185783518 524.72964777424 lineto +157.57674051566 371.520759150445 lineto +57.5767405156596 544.725839907332 lineto +125.980769180794 356.787315750151 lineto +91.2511336474073 553.748866352592 lineto +91.2511336474079 353.748866352592 lineto +125.980769180793 550.710416955034 lineto +57.5767405156601 362.771892797852 lineto +157.57674051566 535.97697355474 lineto +29.0192185783522 382.768084930944 lineto +182.228107202148 511.325606868253 lineto +9.02302644525994 411.325606868252 lineto +196.961550602442 479.729635533386 lineto +-1.70530256582424e-13 445 lineto +0 445 lineto +1.000 1.000 0.000 setrgbcolor AdjustColor +eofill +grestore +gsave +0 445 moveto +200 445 lineto +3.03844939755837 479.729635533386 lineto +190.97697355474 411.325606868252 lineto +17.7718927978523 511.325606868252 lineto +170.980781421648 382.768084930944 lineto +42.42325948434 535.97697355474 lineto +142.42325948434 362.771892797852 lineto +74.0192308192062 550.710416955034 lineto +108.748866352592 353.748866352592 lineto +108.748866352592 553.748866352592 lineto +74.0192308192064 356.787315750151 lineto +142.42325948434 544.725839907333 lineto +42.4232594843401 371.520759150445 lineto +170.980781421648 524.72964777424 lineto +17.7718927978524 396.172125836932 lineto +190.97697355474 496.172125836933 lineto +3.03844939755834 427.768097171799 lineto +200 462.497732705185 lineto +-1.13686837721616e-13 462.497732705185 lineto +196.961550602442 427.768097171799 lineto +9.02302644525972 496.172125836932 lineto +182.228107202148 396.172125836933 lineto +29.0192185783518 524.72964777424 lineto +157.57674051566 371.520759150445 lineto +57.5767405156596 544.725839907332 lineto +125.980769180794 356.787315750151 lineto +91.2511336474073 553.748866352592 lineto +91.2511336474079 353.748866352592 lineto +125.980769180793 550.710416955034 lineto +57.5767405156601 362.771892797852 lineto +157.57674051566 535.97697355474 lineto +29.0192185783522 382.768084930944 lineto +182.228107202148 511.325606868253 lineto +9.02302644525994 411.325606868252 lineto +196.961550602442 479.729635533386 lineto +-1.70530256582424e-13 445 lineto +1 setlinecap +1 setlinejoin +1 setlinewidth +[] 0 setdash +1.000 0.000 0.000 setrgbcolor AdjustColor +stroke +grestore +gsave +grestore +gsave +-1.70530256582424e-13 445 moveto +-9.00000000000019 450 lineto +-7.00000000000017 445 lineto +-9.00000000000015 440 lineto +-1.70530256582424e-13 445 lineto +1.000 1.000 0.000 setrgbcolor AdjustColor +eofill +-1.70530256582424e-13 445 moveto +-9.00000000000019 450 lineto +-7.00000000000017 445 lineto +-9.00000000000015 440 lineto +-1.70530256582424e-13 445 lineto +1 setlinejoin 1 setlinecap +1 setlinewidth +[] 0 setdash +1.000 0.000 0.000 setrgbcolor AdjustColor +stroke +grestore +restore showpage + +%%Trailer +end +%%EOF + diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index 1fe9699f96..4373f78d70 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -18,10 +18,20 @@ Turtle graphics is a popular way for introducing programming to kids. It was part of the original Logo programming language developed by Wally Feurzig and Seymour Papert in 1966. -Imagine a robotic turtle starting at (0, 0) in the x-y plane. Give it the +Imagine a robotic turtle starting at (0, 0) in the x-y plane. After an ``import turtle``, give it the command ``turtle.forward(15)``, and it moves (on-screen!) 15 pixels in the direction it is facing, drawing a line as it moves. Give it the command -``turtle.left(25)``, and it rotates in-place 25 degrees clockwise. +``turtle.right(25)``, and it rotates in-place 25 degrees clockwise. + +.. sidebar:: Turtle star + + Turtle can draw intricate shapes using programs that repeat simple + moves. + + .. image:: turtle-star.* + :align: center + + .. literalinclude:: ../includes/turtle-star.py By combining together these and similar commands, intricate shapes and pictures can easily be drawn. @@ -194,7 +204,7 @@ Using screen events | :func:`onkeypress` | :func:`onclick` | :func:`onscreenclick` | :func:`ontimer` - | :func:`mainloop` + | :func:`mainloop` | :func:`done` Settings and special methods | :func:`mode` @@ -1763,6 +1773,7 @@ Using screen events .. function:: mainloop() + done() Starts event loop - calling Tkinter's mainloop function. Must be the last statement in a turtle graphics program. @@ -1868,7 +1879,7 @@ Settings and special methods >>> cv = screen.getcanvas() >>> cv - <turtle.ScrolledCanvas instance at 0x...> + <turtle.ScrolledCanvas object at ...> .. function:: getshapes() @@ -2258,7 +2269,7 @@ There can be a :file:`turtle.cfg` file in the directory where :mod:`turtle` is stored and an additional one in the current working directory. The latter will override the settings of the first one. -The :file:`Demo/turtle` directory contains a :file:`turtle.cfg` file. You can +The :file:`Lib/turtledemo` directory contains a :file:`turtle.cfg` file. You can study it as an example and see its effects when running the demos (preferably not from within the demo-viewer). @@ -2266,29 +2277,35 @@ not from within the demo-viewer). Demo scripts ============ -There is a set of demo scripts in the turtledemo directory located in the -:file:`Demo/turtle` directory in the source distribution. +There is a set of demo scripts in the :mod:`turtledemo` package. These +scripts can be run and viewed using the supplied demo viewer as follows:: + + python -m turtledemo + +Alternatively, you can run the demo scripts individually. For example, :: + + python -m turtledemo.bytedesign -It contains: +The :mod:`turtledemo` package directory contains: - a set of 15 demo scripts demonstrating different features of the new module - :mod:`turtle` -- a demo viewer :file:`turtleDemo.py` which can be used to view the sourcecode + :mod:`turtle`; +- a demo viewer :file:`__main__.py` which can be used to view the sourcecode of the scripts and run them at the same time. 14 of the examples can be accessed via the Examples menu; all of them can also be run standalone. -- The example :file:`turtledemo_two_canvases.py` demonstrates the simultaneous +- The example :mod:`turtledemo.two_canvases` demonstrates the simultaneous use of two canvases with the turtle module. Therefore it only can be run standalone. -- There is a :file:`turtle.cfg` file in this directory, which also serves as an +- There is a :file:`turtle.cfg` file in this directory, which serves as an example for how to write and use such files. -The demoscripts are: +The demo scripts are: +----------------+------------------------------+-----------------------+ | Name | Description | Features | +----------------+------------------------------+-----------------------+ | bytedesign | complex classical | :func:`tracer`, delay,| -| | turtlegraphics pattern | :func:`update` | +| | turtle graphics pattern | :func:`update` | +----------------+------------------------------+-----------------------+ | chaos | graphs Verhulst dynamics, | world coordinates | | | shows that computer's | | @@ -2386,8 +2403,7 @@ Changes since Python 3.0 strings and numbers respectively. - Two example scripts :file:`tdemo_nim.py` and :file:`tdemo_round_dance.py` - have been added to the Demo directory (source distribution only). As usual - they can be viewed and executed within the demo viewer :file:`turtleDemo.py`. + have been added to the :file:`Lib/turtledemo` directory. .. doctest:: diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 7caecaf4f6..d4a76b611b 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -4,6 +4,9 @@ .. module:: types :synopsis: Names for built-in types. +**Source code:** :source:`Lib/types.py` + +-------------- This module defines names for some object types that are used by the standard Python interpreter, but not exposed as builtins like :class:`int` or diff --git a/Doc/library/unicodedata.rst b/Doc/library/unicodedata.rst index e1e6dc130e..42400dfd78 100644 --- a/Doc/library/unicodedata.rst +++ b/Doc/library/unicodedata.rst @@ -13,14 +13,15 @@ single: character pair: Unicode; database -This module provides access to the Unicode Character Database which defines -character properties for all Unicode characters. The data in this database is -based on the :file:`UnicodeData.txt` file version 5.1.0 which is publicly -available from ftp://ftp.unicode.org/. +This module provides access to the Unicode Character Database (UCD) which +defines character properties for all Unicode characters. The data contained in +this database is compiled from the `UCD version 6.0.0 +<http://www.unicode.org/Public/6.0.0/ucd>`_. -The module uses the same names and symbols as defined by the UnicodeData File -Format 5.1.0 (see http://www.unicode.org/Public/5.1.0/ucd/UCD.html). It defines -the following functions: +The module uses the same names and symbols as defined by Unicode +Standard Annex #44, `"Unicode Character Database" +<http://www.unicode.org/reports/tr44/tr44-6.html>`_. It defines the +following functions: .. function:: lookup(name) @@ -104,7 +105,7 @@ the following functions: based on the definition of canonical equivalence and compatibility equivalence. In Unicode, several characters can be expressed in various way. For example, the character U+00C7 (LATIN CAPITAL LETTER C WITH CEDILLA) can also be expressed as - the sequence U+0327 (COMBINING CEDILLA) U+0043 (LATIN CAPITAL LETTER C). + the sequence U+0043 (LATIN CAPITAL LETTER C) U+0327 (COMBINING CEDILLA). For each character, there are two normal forms: normal form C and normal form D. Normal form D (NFD) is also known as canonical decomposition, and translates diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index 01a036d28f..bdf07a40cb 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -97,6 +97,13 @@ need to derive from a specific class. A special-interest-group for discussion of testing, and testing tools, in Python. + The script :file:`Tools/unittestgui/unittestgui.py` in the Python source distribution is + a GUI tool for test discovery and execution. This is intended largely for ease of use + for those new to unit testing. For production environments it is recommended that + tests be driven by a continuous integration system such as `Hudson <http://hudson-ci.org/>`_ + or `Buildbot <http://buildbot.net/trac>`_. + + .. _unittest-minimal-example: Basic example @@ -204,14 +211,141 @@ modules, classes or even individual test methods:: You can pass in a list with any combination of module names, and fully qualified class or method names. +Test modules can be specified by file path as well:: + + python -m unittest tests/test_something.py + +This allows you to use the shell filename completion to specify the test module. +The file specified must still be importable as a module. The path is converted +to a module name by removing the '.py' and converting path separators into '.'. +If you want to execute a test file that isn't importable as a module you should +execute the file directly instead. + You can run tests with more detail (higher verbosity) by passing in the -v flag:: python -m unittest -v test_module +When executed without arguments :ref:`unittest-test-discovery` is started:: + + python -m unittest + For a list of all the command-line options:: python -m unittest -h +.. versionchanged:: 3.2 + In earlier versions it was only possible to run individual test methods and + not modules or classes. + + +Command-line options +~~~~~~~~~~~~~~~~~~~~ + +:program:`unittest` supports these command-line options: + +.. program:: unittest + +.. cmdoption:: -b, --buffer + + The standard output and standard error streams are buffered during the test + run. Output during a passing test is discarded. Output is echoed normally + on test fail or error and is added to the failure messages. + +.. cmdoption:: -c, --catch + + Control-C during the test run waits for the current test to end and then + reports all the results so far. A second control-C raises the normal + :exc:`KeyboardInterrupt` exception. + + See `Signal Handling`_ for the functions that provide this functionality. + +.. cmdoption:: -f, --failfast + + Stop the test run on the first error or failure. + +.. versionadded:: 3.2 + The command-line options ``-b``, ``-c`` and ``-f`` were added. + +The command line can also be used for test discovery, for running all of the +tests in a project or just a subset. + + +.. _unittest-test-discovery: + +Test Discovery +-------------- + +.. versionadded:: 3.2 + +Unittest supports simple test discovery. In order to be compatible with test +discovery, all of the test files must be :ref:`modules <tut-modules>` or +:ref:`packages <tut-packages>` importable from the top-level directory of +the project (this means that their filenames must be valid +:ref:`identifiers <identifiers>`). + +Test discovery is implemented in :meth:`TestLoader.discover`, but can also be +used from the command line. The basic command-line usage is:: + + cd project_directory + python -m unittest discover + +.. note:: + + As a shortcut, ``python -m unittest`` is the equivalent of + ``python -m unittest discover``. If you want to pass arguments to test + discovery the ``discover`` sub-command must be used explicitly. + +The ``discover`` sub-command has the following options: + +.. program:: unittest discover + +.. cmdoption:: -v, --verbose + + Verbose output + +.. cmdoption:: -s directory + + Directory to start discovery (``.`` default) + +.. cmdoption:: -p pattern + + Pattern to match test files (``test*.py`` default) + +.. cmdoption:: -t directory + + Top level directory of project (defaults to start directory) + +The :option:`-s`, :option:`-p`, and :option:`-t` options can be passed in +as positional arguments in that order. The following two command lines +are equivalent:: + + python -m unittest discover -s project_directory -p '*_test.py' + python -m unittest discover project_directory '*_test.py' + +As well as being a path it is possible to pass a package name, for example +``myproject.subpackage.test``, as the start directory. The package name you +supply will then be imported and its location on the filesystem will be used +as the start directory. + +.. caution:: + + Test discovery loads tests by importing them. Once test discovery has found + all the test files from the start directory you specify it turns the paths + into package names to import. For example :file:`foo/bar/baz.py` will be + imported as ``foo.bar.baz``. + + If you have a package installed globally and attempt test discovery on + a different copy of the package then the import *could* happen from the + wrong place. If this happens test discovery will warn you and exit. + + If you supply the start directory as a package name rather than a + path to a directory then discover assumes that whichever location it + imports from is the location you intended, so you will not get the + warning. + +Test modules and packages can customize test loading and discovery by through +the `load_tests protocol`_. + .. _organizing-tests: @@ -220,9 +354,9 @@ Organizing test code The basic building blocks of unit testing are :dfn:`test cases` --- single scenarios that must be set up and checked for correctness. In :mod:`unittest`, -test cases are represented by instances of :mod:`unittest`'s :class:`TestCase` -class. To make your own test cases you must write subclasses of -:class:`TestCase`, or use :class:`FunctionTestCase`. +test cases are represented by :class:`unittest.TestCase` instances. +To make your own test cases you must write subclasses of +:class:`TestCase` or use :class:`FunctionTestCase`. An instance of a :class:`TestCase`\ -derived class is an object that can completely run a single test method, together with optional set-up and tidy-up @@ -242,7 +376,7 @@ The simplest :class:`TestCase` subclass will simply override the widget = Widget('The widget') self.assertEqual(widget.size(), (50, 50), 'incorrect default size') -Note that in order to test something, we use the one of the :meth:`assert\*` +Note that in order to test something, we use one of the :meth:`assert\*` methods provided by the :class:`TestCase` base class. If the test fails, an exception will be raised, and :mod:`unittest` will identify the test case as a :dfn:`failure`. Any other exceptions will be treated as :dfn:`errors`. This @@ -532,24 +666,27 @@ the test unless the passed object has a certain attribute: :: The following decorators implement test skipping and expected failures: -.. function:: skip(reason) +.. decorator:: skip(reason) Unconditionally skip the decorated test. *reason* should describe why the test is being skipped. -.. function:: skipIf(condition, reason) +.. decorator:: skipIf(condition, reason) Skip the decorated test if *condition* is true. -.. function:: skipUnless(condition, reason) +.. decorator:: skipUnless(condition, reason) Skip the decorated test unless *condition* is true. -.. function:: expectedFailure +.. decorator:: expectedFailure Mark the test as an expected failure. If the test fails when run, the test is not counted as a failure. +Skipped tests will not have :meth:`setUp` or :meth:`tearDown` run around them. +Skipped classes will not have :meth:`setUpClass` or :meth:`tearDownClass` run. + .. _unittest-contents: @@ -586,6 +723,11 @@ Test cases Here, we create two instances of :class:`WidgetTestCase`, each of which runs a single test. + .. versionchanged:: 3.2 + :class:`TestCase` can be instantiated successfully without providing a method + name. This makes it easier to experiment with :class:`TestCase` from the + interactive interpreter. + *methodName* defaults to :meth:`runTest`. :class:`TestCase` instances provide three groups of methods: one group used @@ -615,6 +757,36 @@ Test cases the outcome of the test method. The default implementation does nothing. + .. method:: setUpClass() + + A class method called before tests in an individual class run. + ``setUpClass`` is called with the class as the only argument + and must be decorated as a :func:`classmethod`:: + + @classmethod + def setUpClass(cls): + ... + + See `Class and Module Fixtures`_ for more details. + + .. versionadded:: 3.2 + + + .. method:: tearDownClass() + + A class method called after tests in an individual class have run. + ``tearDownClass`` is called with the class as the only argument + and must be decorated as a :meth:`classmethod`:: + + @classmethod + def tearDownClass(cls): + ... + + See `Class and Module Fixtures`_ for more details. + + .. versionadded:: 3.2 + + .. method:: run(result=None) Run the test, collecting the result into the test result object passed as @@ -678,32 +850,42 @@ Test cases | :meth:`assertNotIn(a, b) | ``a not in b`` | 3.1 | | <TestCase.assertNotIn>` | | | +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertIsInstance(a, b) | ``isinstance(a, b)`` | 3.2 | + | <TestCase.assertIsInstance>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertNotIsInstance(a, b) | ``not isinstance(a, b)`` | 3.2 | + | <TestCase.assertNotIsInstance>` | | | + +-----------------------------------------+-----------------------------+---------------+ All the assert methods (except :meth:`assertRaises`, - :meth:`assertRaisesRegexp`, :meth:`assertWarns`, :meth:`assertWarnsRegexp`) + :meth:`assertRaisesRegex`, :meth:`assertWarns`, :meth:`assertWarnsRegex`) accept a *msg* argument that, if specified, is used as the error message on failure (see also :data:`longMessage`). .. method:: assertEqual(first, second, msg=None) - Test that *first* and *second* are equal. If the values do not compare - equal, the test will fail. + Test that *first* and *second* are equal. If the values do not + compare equal, the test will fail. In addition, if *first* and *second* are the exact same type and one of list, tuple, dict, set, frozenset or str or any type that a subclass - registers with :meth:`addTypeEqualityFunc` the type specific equality + registers with :meth:`addTypeEqualityFunc` the type-specific equality function will be called in order to generate a more useful default error message (see also the :ref:`list of type-specific methods <type-specific-methods>`). .. versionchanged:: 3.1 - Added the automatic calling of type specific equality function. + Added the automatic calling of type-specific equality function. + + .. versionchanged:: 3.2 + :meth:`assertMultiLineEqual` added as the default type equality + function for comparing strings. .. method:: assertNotEqual(first, second, msg=None) - Test that *first* and *second* are not equal. If the values do compare - equal, the test will fail. + Test that *first* and *second* are not equal. If the values do + compare equal, the test will fail. .. method:: assertTrue(expr, msg=None) assertFalse(expr, msg=None) @@ -720,7 +902,8 @@ Test cases .. method:: assertIs(first, second, msg=None) assertIsNot(first, second, msg=None) - Test that *first* and *second* evaluate (or don't evaluate) to the same object. + Test that *first* and *second* evaluate (or don't evaluate) to the + same object. .. versionadded:: 3.1 @@ -741,6 +924,16 @@ Test cases .. versionadded:: 3.1 + .. method:: assertIsInstance(obj, cls, msg=None) + assertNotIsInstance(obj, cls, msg=None) + + Test that *obj* is (or is not) an instance of *cls* (which can be a + class or a tuple of classes, as supported by :func:`isinstance`). + To check for the exact type, use :func:`assertIs(type(obj), cls) <assertIs>`. + + .. versionadded:: 3.2 + + It is also possible to check that exceptions and warnings are raised using the following methods: @@ -748,11 +941,17 @@ Test cases +---------------------------------------------------------+--------------------------------------+------------+ | Method | Checks that | New in | +=========================================================+======================================+============+ - | :meth:`assertRaises(exc, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises `exc` | | + | :meth:`assertRaises(exc, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises *exc* | | | <TestCase.assertRaises>` | | | +---------------------------------------------------------+--------------------------------------+------------+ - | :meth:`assertRaisesRegexp(exc, re, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises `exc` | 3.1 | - | <TestCase.assertRaisesRegexp>` | and the message matches `re` | | + | :meth:`assertRaisesRegex(exc, re, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises *exc* | 3.1 | + | <TestCase.assertRaisesRegex>` | and the message matches *re* | | + +---------------------------------------------------------+--------------------------------------+------------+ + | :meth:`assertWarns(warn, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises *warn* | 3.2 | + | <TestCase.assertWarns>` | | | + +---------------------------------------------------------+--------------------------------------+------------+ + | :meth:`assertWarnsRegex(warn, re, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises *warn* | 3.2 | + | <TestCase.assertWarnsRegex>` | and the message matches *re* | | +---------------------------------------------------------+--------------------------------------+------------+ .. method:: assertRaises(exception, callable, *args, **kwds) @@ -771,27 +970,97 @@ Test cases with self.assertRaises(SomeException): do_something() + The context manager will store the caught exception object in its + :attr:`exception` attribute. This can be useful if the intention + is to perform additional checks on the exception raised:: + + with self.assertRaises(SomeException) as cm: + do_something() + + the_exception = cm.exception + self.assertEqual(the_exception.error_code, 3) + .. versionchanged:: 3.1 Added the ability to use :meth:`assertRaises` as a context manager. + .. versionchanged:: 3.2 + Added the :attr:`exception` attribute. - .. method:: assertRaisesRegexp(exception, regexp, callable, *args, **kwds) - assertRaisesRegexp(exception, regexp) - Like :meth:`assertRaises` but also tests that *regexp* matches - on the string representation of the raised exception. *regexp* may be + .. method:: assertRaisesRegex(exception, regex, callable, *args, **kwds) + assertRaisesRegex(exception, regex) + + Like :meth:`assertRaises` but also tests that *regex* matches + on the string representation of the raised exception. *regex* may be a regular expression object or a string containing a regular expression suitable for use by :func:`re.search`. Examples:: - self.assertRaisesRegexp(ValueError, 'invalid literal for.*XYZ$', - int, 'XYZ') + self.assertRaisesRegex(ValueError, 'invalid literal for.*XYZ$', + int, 'XYZ') or:: - with self.assertRaisesRegexp(ValueError, 'literal'): + with self.assertRaisesRegex(ValueError, 'literal'): int('XYZ') .. versionadded:: 3.1 + under the name ``assertRaisesRegexp``. + .. versionchanged:: 3.2 + Renamed to :meth:`assertRaisesRegex`. + + + .. method:: assertWarns(warning, callable, *args, **kwds) + assertWarns(warning) + + Test that a warning is triggered when *callable* is called with any + positional or keyword arguments that are also passed to + :meth:`assertWarns`. The test passes if *warning* is triggered and + fails if it isn't. Also, any unexpected exception is an error. + To catch any of a group of warnings, a tuple containing the warning + classes may be passed as *warnings*. + + If only the *warning* argument is given, returns a context manager so + that the code under test can be written inline rather than as a function:: + + with self.assertWarns(SomeWarning): + do_something() + + The context manager will store the caught warning object in its + :attr:`warning` attribute, and the source line which triggered the + warnings in the :attr:`filename` and :attr:`lineno` attributes. + This can be useful if the intention is to perform additional checks + on the exception raised:: + + with self.assertWarns(SomeWarning) as cm: + do_something() + + self.assertIn('myfile.py', cm.filename) + self.assertEqual(320, cm.lineno) + + This method works regardless of the warning filters in place when it + is called. + + .. versionadded:: 3.2 + + + .. method:: assertWarnsRegex(warning, regex, callable, *args, **kwds) + assertWarnsRegex(warning, regex) + + Like :meth:`assertWarns` but also tests that *regex* matches on the + message of the triggered warning. *regex* may be a regular expression + object or a string containing a regular expression suitable for use + by :func:`re.search`. Example:: + + self.assertWarnsRegex(DeprecationWarning, + r'legacy_function\(\) is deprecated', + legacy_function, 'XYZ') + + or:: + + with self.assertWarnsRegex(RuntimeWarning, 'unsafe frobnicating'): + frobnicate('/etc/passwd') + + .. versionadded:: 3.2 @@ -818,11 +1087,15 @@ Test cases | :meth:`assertLessEqual(a, b) | ``a <= b`` | 3.1 | | <TestCase.assertLessEqual>` | | | +---------------------------------------+--------------------------------+--------------+ - | :meth:`assertRegexpMatches(s, re) | ``regex.search(s)`` | 3.1 | - | <TestCase.assertRegexpMatches>` | | | + | :meth:`assertRegex(s, re) | ``regex.search(s)`` | 3.1 | + | <TestCase.assertRegex>` | | | + +---------------------------------------+--------------------------------+--------------+ + | :meth:`assertNotRegex(s, re) | ``not regex.search(s)`` | 3.2 | + | <TestCase.assertNotRegex>` | | | +---------------------------------------+--------------------------------+--------------+ - | :meth:`assertDictContainsSubset(a, b) | all the key/value pairs | 3.1 | - | <TestCase.assertDictContainsSubset>` | in `a` exist in `b` | | + | :meth:`assertCountEqual(a, b) | *a* and *b* have the same | 3.2 | + | <TestCase.assertCountEqual>` | elements in the same number, | | + | | regardless of their order | | +---------------------------------------+--------------------------------+--------------+ @@ -840,6 +1113,11 @@ Test cases Supplying both *delta* and *places* raises a ``TypeError``. + .. versionchanged:: 3.2 + :meth:`assertAlmostEqual` automatically considers almost equal objects + that compare equal. :meth:`assertNotAlmostEqual` automatically fails + if the objects compare equal. Added the *delta* keyword argument. + .. method:: assertGreater(first, second, msg=None) assertGreaterEqual(first, second, msg=None) @@ -855,38 +1133,64 @@ Test cases .. versionadded:: 3.1 - .. method:: assertRegexpMatches(text, regexp, msg=None) + .. method:: assertRegex(text, regex, msg=None) + assertNotRegex(text, regex, msg=None) - Test that a *regexp* search matches *text*. In case + Test that a *regex* search matches (or does not match) *text*. In case of failure, the error message will include the pattern and the *text* (or - the pattern and the part of *text* that unexpectedly matched). *regexp* + the pattern and the part of *text* that unexpectedly matched). *regex* may be a regular expression object or a string containing a regular expression suitable for use by :func:`re.search`. - .. versionadded:: 3.1 :meth:`~TestCase.assertRegexpMatches` + .. versionadded:: 3.1 + under the name ``assertRegexpMatches``. + .. versionchanged:: 3.2 + The method ``assertRegexpMatches()`` has been renamed to + :meth:`.assertRegex`. + .. versionadded:: 3.2 + :meth:`.assertNotRegex`. + + .. method:: assertDictContainsSubset(subset, dictionary, msg=None) - .. method:: assertDictContainsSubset(expected, actual, msg=None) + Tests whether the key/value pairs in *dictionary* are a superset of + those in *subset*. If not, an error message listing the missing keys + and mismatched values is generated. - Tests whether the key/value pairs in dictionary *actual* are a - superset of those in *expected*. If not, an error message listing - the missing keys and mismatched values is generated. + Note, the arguments are in the opposite order of what the method name + dictates. Instead, consider using the set-methods on :ref:`dictionary + views <dict-views>`, for example: ``d.keys() <= e.keys()`` or + ``d.items() <= d.items()``. .. versionadded:: 3.1 .. deprecated:: 3.2 - .. method:: assertSameElements(actual, expected, msg=None) + .. method:: assertCountEqual(first, second, msg=None) - Test that sequence *expected* contains the same elements as *actual*, + Test that sequence *first* contains the same elements as *second*, + regardless of their order. When they don't, an error message listing the + differences between the sequences will be generated. + + Duplicate elements are *not* ignored when comparing *first* and + *second*. It verifies whether each element has the same count in both + sequences. Equivalent to: + ``assertEqual(Counter(list(first)), Counter(list(second)))`` + but works with sequences of unhashable objects as well. + + .. versionadded:: 3.2 + + .. method:: assertSameElements(first, second, msg=None) + + Test that sequence *first* contains the same elements as *second*, regardless of their order. When they don't, an error message listing the differences between the sequences will be generated. - Duplicate elements are ignored when comparing *actual* and *expected*. - It is the equivalent of ``assertEqual(set(expected), set(actual))`` + Duplicate elements are ignored when comparing *first* and *second*. + It is the equivalent of ``assertEqual(set(first), set(second))`` but it works with sequences of unhashable objects as well. Because duplicates are ignored, this method has been deprecated in favour of - :meth:`assertItemsEqual`. + :meth:`assertCountEqual`. .. versionadded:: 3.1 .. deprecated:: 3.2 @@ -950,10 +1254,10 @@ Test cases .. versionadded:: 3.1 - .. method:: assertSequenceEqual(seq1, seq2, msg=None, seq_type=None) + .. method:: assertSequenceEqual(first, second, msg=None, seq_type=None) Tests that two sequences are equal. If a *seq_type* is supplied, both - *seq1* and *seq2* must be instances of *seq_type* or a failure will + *first* and *second* must be instances of *seq_type* or a failure will be raised. If the sequences are different an error message is constructed that shows the difference between the two. @@ -964,8 +1268,8 @@ Test cases .. versionadded:: 3.1 - .. method:: assertListEqual(list1, list2, msg=None) - assertTupleEqual(tuple1, tuple2, msg=None) + .. method:: assertListEqual(first, second, msg=None) + assertTupleEqual(first, second, msg=None) Tests that two lists or tuples are equal. If not an error message is constructed that shows only the differences between the two. An error @@ -976,19 +1280,19 @@ Test cases .. versionadded:: 3.1 - .. method:: assertSetEqual(set1, set2, msg=None) + .. method:: assertSetEqual(first, second, msg=None) Tests that two sets are equal. If not, an error message is constructed that lists the differences between the sets. This method is used by default when comparing sets or frozensets with :meth:`assertEqual`. - Fails if either of *set1* or *set2* does not have a :meth:`set.difference` + Fails if either of *first* or *second* does not have a :meth:`set.difference` method. .. versionadded:: 3.1 - .. method:: assertDictEqual(expected, actual, msg=None) + .. method:: assertDictEqual(first, second, msg=None) Test that two dictionaries are equal. If not, an error message is constructed that shows the differences in the dictionaries. This @@ -1029,8 +1333,8 @@ Test cases to ``True`` allows you to have a custom error message in addition to the normal one. - This attribute defaults to ``False``, meaning that a custom message passed - to an assert method will silence the normal message. + This attribute defaults to ``True``. If set to False then a custom message + passed to an assert method will silence the normal message. The class setting can be overridden in individual tests by assigning an instance attribute to ``True`` or ``False`` before calling the assert methods. @@ -1038,6 +1342,21 @@ Test cases .. versionadded:: 3.1 + .. attribute:: maxDiff + + This attribute controls the maximum length of diffs output by assert + methods that report diffs on failure. It defaults to 80*8 characters. + Assert methods affected by this attribute are + :meth:`assertSequenceEqual` (including all the sequence comparison + methods that delegate to it), :meth:`assertDictEqual` and + :meth:`assertMultiLineEqual`. + + Setting ``maxDiff`` to None means that there is no maximum length of + diffs. + + .. versionadded:: 3.2 + + Testing frameworks can use the following methods to collect information on the test: @@ -1070,13 +1389,13 @@ Test cases Returns a description of the test, or ``None`` if no description has been provided. The default implementation of this method returns the first line of the test method's docstring, if available, - along with the method name. + or ``None``. .. versionchanged:: 3.1 - In earlier versions this only returned the first line of the test - method's docstring, if available or the :const:`None`. That led to - undesirable behavior of not printing the test name when someone was - thoughtful enough to write a docstring. + In 3.1 this was changed to add the test name to the short description + even in the presence of a docstring. This caused compatibility issues + with unittest extensions and adding the test name was moved to the + :class:`TextTestResult` in Python 3.2. .. method:: addCleanup(function, *args, **kwargs) @@ -1118,6 +1437,8 @@ Test cases :mod:`unittest`-based test framework. +.. _deprecated-aliases: + Deprecated aliases ################## @@ -1125,21 +1446,27 @@ For historical reasons, some of the :class:`TestCase` methods had one or more aliases that are now deprecated. The following table lists the correct names along with their deprecated aliases: - ============================== =============================== - Method Name Deprecated alias(es) - ============================== =============================== - :meth:`.assertEqual` failUnlessEqual, assertEquals - :meth:`.assertNotEqual` failIfEqual - :meth:`.assertTrue` failUnless, assert\_ + ============================== ====================== ====================== + Method Name Deprecated alias Deprecated alias + ============================== ====================== ====================== + :meth:`.assertEqual` failUnlessEqual assertEquals + :meth:`.assertNotEqual` failIfEqual assertNotEquals + :meth:`.assertTrue` failUnless assert\_ :meth:`.assertFalse` failIf :meth:`.assertRaises` failUnlessRaises - :meth:`.assertAlmostEqual` failUnlessAlmostEqual - :meth:`.assertNotAlmostEqual` failIfAlmostEqual - ============================== =============================== + :meth:`.assertAlmostEqual` failUnlessAlmostEqual assertAlmostEquals + :meth:`.assertNotAlmostEqual` failIfAlmostEqual assertNotAlmostEquals + :meth:`.assertRegex` assertRegexpMatches + :meth:`.assertRaisesRegex` assertRaisesRegexp + ============================== ====================== ====================== .. deprecated:: 3.1 - the aliases listed in the second column - + the fail* aliases listed in the second column. + .. deprecated:: 3.2 + the assert* aliases listed in the third column. + .. deprecated:: 3.2 + ``assertRegexpMatches`` and ``assertRaisesRegexp`` have been renamed to + :meth:`.assertRegex` and :meth:`.assertRaisesRegex` .. _testsuite-objects: @@ -1209,6 +1536,11 @@ Grouping tests (for example when counting tests or comparing for equality) so the tests returned must be the same for repeated iterations. + .. versionchanged:: 3.2 + In earlier versions the :class:`TestSuite` accessed tests directly rather + than through iteration, so overriding :meth:`__iter__` wasn't sufficient + for providing tests. + In the typical usage of a :class:`TestSuite` object, the :meth:`run` method is invoked by a :class:`TestRunner` rather than by the end-user test harness. @@ -1248,6 +1580,13 @@ Loading and running tests directly does not play well with this method. Doing so, however, can be useful when the fixtures are different and defined in subclasses. + If a module provides a ``load_tests`` function it will be called to + load the tests. This allows modules to customize test loading. + This is the `load_tests protocol`_. + + .. versionchanged:: 3.2 + Support for ``load_tests`` added. + .. method:: loadTestsFromName(name, module=None) @@ -1287,6 +1626,39 @@ Loading and running tests this should be a subclass of :class:`TestCase`. + .. method:: discover(start_dir, pattern='test*.py', top_level_dir=None) + + Find and return all test modules from the specified start directory, + recursing into subdirectories to find them. Only test files that match + *pattern* will be loaded. (Using shell style pattern matching.) Only + module names that are importable (i.e. are valid Python identifiers) will + be loaded. + + All test modules must be importable from the top level of the project. If + the start directory is not the top level directory then the top level + directory must be specified separately. + + If importing a module fails, for example due to a syntax error, then this + will be recorded as a single error and discovery will continue. + + If a test package name (directory with :file:`__init__.py`) matches the + pattern then the package will be checked for a ``load_tests`` + function. If this exists then it will be called with *loader*, *tests*, + *pattern*. + + If load_tests exists then discovery does *not* recurse into the package, + ``load_tests`` is responsible for loading all tests in the package. + + The pattern is deliberately not stored as a loader attribute so that + packages can continue discovery themselves. *top_level_dir* is stored so + ``load_tests`` does not need to pass this argument in to + ``loader.discover()``. + + *start_dir* can be a dotted module name as well as a directory. + + .. versionadded:: 3.2 + + The following attributes of a :class:`TestLoader` can be configured either by subclassing or assignment on an instance: @@ -1375,6 +1747,24 @@ Loading and running tests The total number of tests run so far. + .. attribute:: buffer + + If set to true, ``sys.stdout`` and ``sys.stderr`` will be buffered in between + :meth:`startTest` and :meth:`stopTest` being called. Collected output will + only be echoed onto the real ``sys.stdout`` and ``sys.stderr`` if the test + fails or errors. Any output is also attached to the failure / error message. + + .. versionadded:: 3.2 + + + .. attribute:: failfast + + If set to true :meth:`stop` will be called on the first failure or error, + halting the test run. + + .. versionadded:: 3.2 + + .. method:: wasSuccessful() Return ``True`` if all tests run so far have passed, otherwise returns @@ -1403,18 +1793,11 @@ Loading and running tests Called when the test case *test* is about to be run. - The default implementation simply increments the instance's :attr:`testsRun` - counter. - - .. method:: stopTest(test) Called after the test case *test* has been executed, regardless of the outcome. - The default implementation does nothing. - - .. method:: startTestRun(test) Called once before any tests are executed. @@ -1485,6 +1868,16 @@ Loading and running tests :attr:`unexpectedSuccesses` attribute. +.. class:: TextTestResult(stream, descriptions, verbosity) + + A concrete implementation of :class:`TestResult` used by the + :class:`TextTestRunner`. + + .. versionadded:: 3.2 + This class was previously named ``_TextTestResult``. The old name still + exists as an alias but is deprecated. + + .. data:: defaultTestLoader Instance of the :class:`TestLoader` class intended to be shared. If no @@ -1492,20 +1885,46 @@ Loading and running tests instead of repeatedly creating new instances. -.. class:: TextTestRunner(stream=sys.stderr, descriptions=True, verbosity=1) +.. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, runnerclass=None, warnings=None) - A basic test runner implementation which prints results on standard error. It + A basic test runner implementation that outputs results to a stream. If *stream* + is ``None``, the default, :data:`sys.stderr` is used as the output stream. This class has a few configurable parameters, but is essentially very simple. Graphical applications which run test suites should provide alternate implementations. + By default this runner shows :exc:`DeprecationWarning`, + :exc:`PendingDeprecationWarning`, and :exc:`ImportWarning` even if they are + :ref:`ignored by default <warning-ignored>`. Deprecation warnings caused by + :ref:`deprecated unittest methods <deprecated-aliases>` are also + special-cased and, when the warning filters are ``'default'`` or ``'always'``, + they will appear only once per-module, in order to avoid too many warning + messages. This behavior can be overridden using the :option:`-Wd` or + :option:`-Wa` options and leaving *warnings* to ``None``. + + .. versionchanged:: 3.2 + Added the ``warnings`` argument. + + .. versionchanged:: 3.2 + The default stream is set to :data:`sys.stderr` at instantiation time rather + than import time. + .. method:: _makeResult() This method returns the instance of ``TestResult`` used by :meth:`run`. It is not intended to be called directly, but can be overridden in subclasses to provide a custom ``TestResult``. + ``_makeResult()`` instantiates the class or callable passed in the + ``TextTestRunner`` constructor as the ``resultclass`` argument. It + defaults to :class:`TextTestResult` if no ``resultclass`` is provided. + The result class is instantiated with the following arguments:: -.. function:: main(module='__main__', defaultTest=None, argv=None, testRunner=TextTestRunner, testLoader=unittest.defaultTestLoader, exit=True) + stream, descriptions, verbosity + + +.. function:: main(module='__main__', defaultTest=None, argv=None, testRunner=None, \ + testLoader=unittest.defaultTestLoader, exit=True, verbosity=1, \ + failfast=None, catchbreak=None, buffer=None, warnings=None) A command-line program that runs a set of tests; this is primarily for making test modules conveniently executable. The simplest use for this function is to @@ -1514,6 +1933,11 @@ Loading and running tests if __name__ == '__main__': unittest.main() + You can run tests with more detailed information by passing in the verbosity + argument:: + + if __name__ == '__main__': + unittest.main(verbosity=2) The *testRunner* argument can either be a test runner class or an already created instance of it. By default ``main`` calls :func:`sys.exit` with @@ -1526,8 +1950,228 @@ Loading and running tests >>> from unittest import main >>> main(module='test_module', exit=False) + The ``failfast``, ``catchbreak`` and ``buffer`` parameters have the same + effect as the same-name `command-line options`_. + + The *warning* argument specifies the :ref:`warning filter <warning-filter>` + that should be used while running the tests. If it's not specified, it will + remain ``None`` if a :option:`-W` option is passed to :program:`python`, + otherwise it will be set to ``'default'``. + Calling ``main`` actually returns an instance of the ``TestProgram`` class. This stores the result of the tests run as the ``result`` attribute. .. versionchanged:: 3.1 The ``exit`` parameter was added. + + .. versionchanged:: 3.2 + The ``verbosity``, ``failfast``, ``catchbreak``, ``buffer`` + and ``warnings`` parameters were added. + + +load_tests Protocol +################### + +.. versionadded:: 3.2 + +Modules or packages can customize how tests are loaded from them during normal +test runs or test discovery by implementing a function called ``load_tests``. + +If a test module defines ``load_tests`` it will be called by +:meth:`TestLoader.loadTestsFromModule` with the following arguments:: + + load_tests(loader, standard_tests, None) + +It should return a :class:`TestSuite`. + +*loader* is the instance of :class:`TestLoader` doing the loading. +*standard_tests* are the tests that would be loaded by default from the +module. It is common for test modules to only want to add or remove tests +from the standard set of tests. +The third argument is used when loading packages as part of test discovery. + +A typical ``load_tests`` function that loads tests from a specific set of +:class:`TestCase` classes may look like:: + + test_cases = (TestCase1, TestCase2, TestCase3) + + def load_tests(loader, tests, pattern): + suite = TestSuite() + for test_class in test_cases: + tests = loader.loadTestsFromTestCase(test_class) + suite.addTests(tests) + return suite + +If discovery is started, either from the command line or by calling +:meth:`TestLoader.discover`, with a pattern that matches a package +name then the package :file:`__init__.py` will be checked for ``load_tests``. + +.. note:: + + The default pattern is 'test*.py'. This matches all Python files + that start with 'test' but *won't* match any test directories. + + A pattern like 'test*' will match test packages as well as + modules. + +If the package :file:`__init__.py` defines ``load_tests`` then it will be +called and discovery not continued into the package. ``load_tests`` +is called with the following arguments:: + + load_tests(loader, standard_tests, pattern) + +This should return a :class:`TestSuite` representing all the tests +from the package. (``standard_tests`` will only contain tests +collected from :file:`__init__.py`.) + +Because the pattern is passed into ``load_tests`` the package is free to +continue (and potentially modify) test discovery. A 'do nothing' +``load_tests`` function for a test package would look like:: + + def load_tests(loader, standard_tests, pattern): + # top level directory cached on loader instance + this_dir = os.path.dirname(__file__) + package_tests = loader.discover(start_dir=this_dir, pattern=pattern) + standard_tests.addTests(package_tests) + return standard_tests + + +Class and Module Fixtures +------------------------- + +Class and module level fixtures are implemented in :class:`TestSuite`. When +the test suite encounters a test from a new class then :meth:`tearDownClass` +from the previous class (if there is one) is called, followed by +:meth:`setUpClass` from the new class. + +Similarly if a test is from a different module from the previous test then +``tearDownModule`` from the previous module is run, followed by +``setUpModule`` from the new module. + +After all the tests have run the final ``tearDownClass`` and +``tearDownModule`` are run. + +Note that shared fixtures do not play well with [potential] features like test +parallelization and they break test isolation. They should be used with care. + +The default ordering of tests created by the unittest test loaders is to group +all tests from the same modules and classes together. This will lead to +``setUpClass`` / ``setUpModule`` (etc) being called exactly once per class and +module. If you randomize the order, so that tests from different modules and +classes are adjacent to each other, then these shared fixture functions may be +called multiple times in a single test run. + +Shared fixtures are not intended to work with suites with non-standard +ordering. A ``BaseTestSuite`` still exists for frameworks that don't want to +support shared fixtures. + +If there are any exceptions raised during one of the shared fixture functions +the test is reported as an error. Because there is no corresponding test +instance an ``_ErrorHolder`` object (that has the same interface as a +:class:`TestCase`) is created to represent the error. If you are just using +the standard unittest test runner then this detail doesn't matter, but if you +are a framework author it may be relevant. + + +setUpClass and tearDownClass +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These must be implemented as class methods:: + + import unittest + + class Test(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls._connection = createExpensiveConnectionObject() + + @classmethod + def tearDownClass(cls): + cls._connection.destroy() + +If you want the ``setUpClass`` and ``tearDownClass`` on base classes called +then you must call up to them yourself. The implementations in +:class:`TestCase` are empty. + +If an exception is raised during a ``setUpClass`` then the tests in the class +are not run and the ``tearDownClass`` is not run. Skipped classes will not +have ``setUpClass`` or ``tearDownClass`` run. If the exception is a +``SkipTest`` exception then the class will be reported as having been skipped +instead of as an error. + + +setUpModule and tearDownModule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These should be implemented as functions:: + + def setUpModule(): + createConnection() + + def tearDownModule(): + closeConnection() + +If an exception is raised in a ``setUpModule`` then none of the tests in the +module will be run and the ``tearDownModule`` will not be run. If the exception is a +``SkipTest`` exception then the module will be reported as having been skipped +instead of as an error. + + +Signal Handling +--------------- + +.. versionadded:: 3.2 + +The :option:`-c/--catch <unittest -c>` command-line option to unittest, +along with the ``catchbreak`` parameter to :func:`unittest.main()`, provide +more friendly handling of control-C during a test run. With catch break +behavior enabled control-C will allow the currently running test to complete, +and the test run will then end and report all the results so far. A second +control-c will raise a :exc:`KeyboardInterrupt` in the usual way. + +The control-c handling signal handler attempts to remain compatible with code or +tests that install their own :const:`signal.SIGINT` handler. If the ``unittest`` +handler is called but *isn't* the installed :const:`signal.SIGINT` handler, +i.e. it has been replaced by the system under test and delegated to, then it +calls the default handler. This will normally be the expected behavior by code +that replaces an installed handler and delegates to it. For individual tests +that need ``unittest`` control-c handling disabled the :func:`removeHandler` +decorator can be used. + +There are a few utility functions for framework authors to enable control-c +handling functionality within test frameworks. + +.. function:: installHandler() + + Install the control-c handler. When a :const:`signal.SIGINT` is received + (usually in response to the user pressing control-c) all registered results + have :meth:`~TestResult.stop` called. + + +.. function:: registerResult(result) + + Register a :class:`TestResult` object for control-c handling. Registering a + result stores a weak reference to it, so it doesn't prevent the result from + being garbage collected. + + Registering a :class:`TestResult` object has no side-effects if control-c + handling is not enabled, so test frameworks can unconditionally register + all results they create independently of whether or not handling is enabled. + + +.. function:: removeResult(result) + + Remove a registered result. Once a result has been removed then + :meth:`~TestResult.stop` will no longer be called on that result object in + response to a control-c. + + +.. function:: removeHandler(function=None) + + When called without arguments this function removes the control-c handler + if it has been installed. This function can also be used as a test decorator + to temporarily remove the handler whilst the test is being executed:: + + @unittest.removeHandler + def test_signal_handling(self): + ... diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index 36fe1b31b5..aece7149ef 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -12,6 +12,10 @@ pair: URL; parsing pair: relative; URL +**Source code:** :source:`Lib/urllib/parse.py` + +-------------- + This module defines a standard interface to break Uniform Resource Locator (URL) strings up in components (addressing scheme, network location, path etc.), to combine the components back into a URL string, and to convert a "relative URL" @@ -24,7 +28,15 @@ following URL schemes: ``file``, ``ftp``, ``gopher``, ``hdl``, ``http``, ``rsync``, ``rtsp``, ``rtspu``, ``sftp``, ``shttp``, ``sip``, ``sips``, ``snews``, ``svn``, ``svn+ssh``, ``telnet``, ``wais``. -The :mod:`urllib.parse` module defines the following functions: +The :mod:`urllib.parse` module defines functions that fall into two broad +categories: URL parsing and URL quoting. These are covered in detail in +the following sections. + +URL Parsing +----------- + +The URL parsing functions focus on splitting a URL string into its components, +or on combining URL components into a URL string. .. function:: urlparse(urlstring, scheme='', allow_fragments=True) @@ -104,8 +116,11 @@ The :mod:`urllib.parse` module defines the following functions: See section :ref:`urlparse-result-object` for more information on the result object. + .. versionchanged:: 3.2 + Added IPv6 URL parsing capabilities. + -.. function:: parse_qs(qs, keep_blank_values=False, strict_parsing=False) +.. function:: parse_qs(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace') Parse a query string given as a string argument (data of type :mimetype:`application/x-www-form-urlencoded`). Data are returned as a @@ -122,11 +137,19 @@ The :mod:`urllib.parse` module defines the following functions: parsing errors. If false (the default), errors are silently ignored. If true, errors raise a :exc:`ValueError` exception. + The optional *encoding* and *errors* parameters specify how to decode + percent-encoded sequences into Unicode characters, as accepted by the + :meth:`bytes.decode` method. + Use the :func:`urllib.parse.urlencode` function to convert such dictionaries into query strings. -.. function:: parse_qsl(qs, keep_blank_values=False, strict_parsing=False) + .. versionchanged:: 3.2 + Add *encoding* and *errors* parameters. + + +.. function:: parse_qsl(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace') Parse a query string given as a string argument (data of type :mimetype:`application/x-www-form-urlencoded`). Data are returned as a list of @@ -142,9 +165,16 @@ The :mod:`urllib.parse` module defines the following functions: parsing errors. If false (the default), errors are silently ignored. If true, errors raise a :exc:`ValueError` exception. + The optional *encoding* and *errors* parameters specify how to decode + percent-encoded sequences into Unicode characters, as accepted by the + :meth:`bytes.decode` method. + Use the :func:`urllib.parse.urlencode` function to convert such lists of pairs into query strings. + .. versionchanged:: 3.2 + Add *encoding* and *errors* parameters. + .. function:: urlunparse(parts) @@ -239,6 +269,162 @@ The :mod:`urllib.parse` module defines the following functions: string. If there is no fragment identifier in *url*, return *url* unmodified and an empty string. + The return value is actually an instance of a subclass of :class:`tuple`. This + class has the following additional read-only convenience attributes: + + +------------------+-------+-------------------------+----------------------+ + | Attribute | Index | Value | Value if not present | + +==================+=======+=========================+======================+ + | :attr:`url` | 0 | URL with no fragment | empty string | + +------------------+-------+-------------------------+----------------------+ + | :attr:`fragment` | 1 | Fragment identifier | empty string | + +------------------+-------+-------------------------+----------------------+ + + See section :ref:`urlparse-result-object` for more information on the result + object. + + .. versionchanged:: 3.2 + Result is a structured object rather than a simple 2-tuple. + +.. _parsing-ascii-encoded-bytes: + +Parsing ASCII Encoded Bytes +--------------------------- + +The URL parsing functions were originally designed to operate on character +strings only. In practice, it is useful to be able to manipulate properly +quoted and encoded URLs as sequences of ASCII bytes. Accordingly, the +URL parsing functions in this module all operate on :class:`bytes` and +:class:`bytearray` objects in addition to :class:`str` objects. + +If :class:`str` data is passed in, the result will also contain only +:class:`str` data. If :class:`bytes` or :class:`bytearray` data is +passed in, the result will contain only :class:`bytes` data. + +Attempting to mix :class:`str` data with :class:`bytes` or +:class:`bytearray` in a single function call will result in a +:exc:`TypeError` being raised, while attempting to pass in non-ASCII +byte values will trigger :exc:`UnicodeDecodeError`. + +To support easier conversion of result objects between :class:`str` and +:class:`bytes`, all return values from URL parsing functions provide +either an :meth:`encode` method (when the result contains :class:`str` +data) or a :meth:`decode` method (when the result contains :class:`bytes` +data). The signatures of these methods match those of the corresponding +:class:`str` and :class:`bytes` methods (except that the default encoding +is ``'ascii'`` rather than ``'utf-8'``). Each produces a value of a +corresponding type that contains either :class:`bytes` data (for +:meth:`encode` methods) or :class:`str` data (for +:meth:`decode` methods). + +Applications that need to operate on potentially improperly quoted URLs +that may contain non-ASCII data will need to do their own decoding from +bytes to characters before invoking the URL parsing methods. + +The behaviour described in this section applies only to the URL parsing +functions. The URL quoting functions use their own rules when producing +or consuming byte sequences as detailed in the documentation of the +individual URL quoting functions. + +.. versionchanged:: 3.2 + URL parsing functions now accept ASCII encoded byte sequences + + +.. _urlparse-result-object: + +Structured Parse Results +------------------------ + +The result objects from the :func:`urlparse`, :func:`urlsplit` and +:func:`urldefrag` functions are subclasses of the :class:`tuple` type. +These subclasses add the attributes listed in the documentation for +those functions, the encoding and decoding support described in the +previous section, as well as an additional method: + +.. method:: urllib.parse.SplitResult.geturl() + + Return the re-combined version of the original URL as a string. This may + differ from the original URL in that the scheme may be normalized to lower + case and empty components may be dropped. Specifically, empty parameters, + queries, and fragment identifiers will be removed. + + For :func:`urldefrag` results, only empty fragment identifiers will be removed. + For :func:`urlsplit` and :func:`urlparse` results, all noted changes will be + made to the URL returned by this method. + + The result of this method remains unchanged if passed back through the original + parsing function: + + >>> from urllib.parse import urlsplit + >>> url = 'HTTP://www.Python.org/doc/#' + >>> r1 = urlsplit(url) + >>> r1.geturl() + 'http://www.Python.org/doc/' + >>> r2 = urlsplit(r1.geturl()) + >>> r2.geturl() + 'http://www.Python.org/doc/' + + +The following classes provide the implementations of the structured parse +results when operating on :class:`str` objects: + +.. class:: DefragResult(url, fragment) + + Concrete class for :func:`urldefrag` results containing :class:`str` + data. The :meth:`encode` method returns a :class:`DefragResultBytes` + instance. + + .. versionadded:: 3.2 + +.. class:: ParseResult(scheme, netloc, path, params, query, fragment) + + Concrete class for :func:`urlparse` results containing :class:`str` + data. The :meth:`encode` method returns a :class:`ParseResultBytes` + instance. + +.. class:: SplitResult(scheme, netloc, path, query, fragment) + + Concrete class for :func:`urlsplit` results containing :class:`str` + data. The :meth:`encode` method returns a :class:`SplitResultBytes` + instance. + + +The following classes provide the implementations of the parse results when +operating on :class:`bytes` or :class:`bytearray` objects: + +.. class:: DefragResultBytes(url, fragment) + + Concrete class for :func:`urldefrag` results containing :class:`bytes` + data. The :meth:`decode` method returns a :class:`DefragResult` + instance. + + .. versionadded:: 3.2 + +.. class:: ParseResultBytes(scheme, netloc, path, params, query, fragment) + + Concrete class for :func:`urlparse` results containing :class:`bytes` + data. The :meth:`decode` method returns a :class:`ParseResult` + instance. + + .. versionadded:: 3.2 + +.. class:: SplitResultBytes(scheme, netloc, path, query, fragment) + + Concrete class for :func:`urlsplit` results containing :class:`bytes` + data. The :meth:`decode` method returns a :class:`SplitResult` + instance. + + .. versionadded:: 3.2 + + +URL Quoting +----------- + +The URL quoting functions focus on taking program data and making it safe +for use as URL components by quoting special characters and appropriately +encoding non-ASCII text. They also support reversing these operations to +recreate the original data from the contents of a URL component if that +task isn't already covered by the URL parsing functions above. .. function:: quote(string, safe='/', encoding=None, errors=None) @@ -319,16 +505,16 @@ The :mod:`urllib.parse` module defines the following functions: If it is a :class:`str`, unescaped non-ASCII characters in *string* are encoded into UTF-8 bytes. - Example: ``unquote_to_bytes('a%26%EF')`` yields - ``b'a&\xef'``. + Example: ``unquote_to_bytes('a%26%EF')`` yields ``b'a&\xef'``. .. function:: urlencode(query, doseq=False, safe='', encoding=None, errors=None) Convert a mapping object or a sequence of two-element tuples, which may - either be a :class:`str` or a :class:`bytes`, to a "percent-encoded" string, - suitable to pass to :func:`urlopen` above as the optional *data* argument. - This is useful to pass a dictionary of form fields to a ``POST`` request. + either be a :class:`str` or a :class:`bytes`, to a "percent-encoded" + string. The resultant string must be converted to bytes using the + user-specified encoding before it is sent to :func:`urlopen` as the optional + *data* argument. The resulting string is a series of ``key=value`` pairs separated by ``'&'`` characters, where both *key* and *value* are quoted using :func:`quote_plus` above. When a sequence of two-element tuples is used as the *query* @@ -337,12 +523,16 @@ The :mod:`urllib.parse` module defines the following functions: the optional parameter *doseq* is evaluates to *True*, individual ``key=value`` pairs separated by ``'&'`` are generated for each element of the value sequence for the key. The order of parameters in the encoded - string will match the order of parameter tuples in the sequence. This module - provides the functions :func:`parse_qs` and :func:`parse_qsl` which are used - to parse query strings into Python data structures. + string will match the order of parameter tuples in the sequence. When *query* parameter is a :class:`str`, the *safe*, *encoding* and *error* - parameters are sent the :func:`quote_plus` for encoding. + parameters are passed down to :func:`quote_plus` for encoding. + + To reverse this encoding process, :func:`parse_qs` and :func:`parse_qsl` are + provided in this module to parse query strings into Python data structures. + + Refer to :ref:`urllib examples <urllib-examples>` to find out how urlencode + method can be used for generating query string for a URL or data for POST. .. versionchanged:: 3.2 Query parameter supports bytes and string objects. @@ -356,6 +546,9 @@ The :mod:`urllib.parse` module defines the following functions: mostly for backward compatibility purposes and for certain de-facto parsing requirements as commonly observed in major browsers. + :rfc:`2732` - Format for Literal IPv6 Addresses in URL's. + This specifies the parsing requirements of IPv6 URLs. + :rfc:`2396` - Uniform Resource Identifiers (URI): Generic Syntax Document describing the generic syntactic requirements for both Uniform Resource Names (URNs) and Uniform Resource Locators (URLs). @@ -370,57 +563,3 @@ The :mod:`urllib.parse` module defines the following functions: :rfc:`1738` - Uniform Resource Locators (URL) This specifies the formal syntax and semantics of absolute URLs. - - -.. _urlparse-result-object: - -Results of :func:`urlparse` and :func:`urlsplit` ------------------------------------------------- - -The result objects from the :func:`urlparse` and :func:`urlsplit` functions are -subclasses of the :class:`tuple` type. These subclasses add the attributes -described in those functions, as well as provide an additional method: - -.. method:: ParseResult.geturl() - - Return the re-combined version of the original URL as a string. This may differ - from the original URL in that the scheme will always be normalized to lower case - and empty components may be dropped. Specifically, empty parameters, queries, - and fragment identifiers will be removed. - - The result of this method is a fixpoint if passed back through the original - parsing function: - - >>> import urllib.parse - >>> url = 'HTTP://www.Python.org/doc/#' - - >>> r1 = urllib.parse.urlsplit(url) - >>> r1.geturl() - 'http://www.Python.org/doc/' - - >>> r2 = urllib.parse.urlsplit(r1.geturl()) - >>> r2.geturl() - 'http://www.Python.org/doc/' - - -The following classes provide the implementations of the parse results: - -.. class:: BaseResult - - Base class for the concrete result classes. This provides most of the - attribute definitions. It does not provide a :meth:`geturl` method. It is - derived from :class:`tuple`, but does not override the :meth:`__init__` or - :meth:`__new__` methods. - - -.. class:: ParseResult(scheme, netloc, path, params, query, fragment) - - Concrete class for :func:`urlparse` results. The :meth:`__new__` method is - overridden to support checking that the right number of arguments are passed. - - -.. class:: SplitResult(scheme, netloc, path, query, fragment) - - Concrete class for :func:`urlsplit` results. The :meth:`__new__` method is - overridden to support checking that the right number of arguments are passed. - diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 3ba2c15023..cc759a4465 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -15,28 +15,37 @@ authentication, redirections, cookies and more. The :mod:`urllib.request` module defines the following functions: -.. function:: urlopen(url, data=None[, timeout]) +.. function:: urlopen(url, data=None[, timeout], *, cafile=None, capath=None) Open the URL *url*, which can be either a string or a :class:`Request` object. - .. warning:: - HTTPS requests do not do any verification of the server's certificate. - - *data* may be a string specifying additional data to send to the - server, or ``None`` if no such data is needed. Currently HTTP - requests are the only ones that use *data*; the HTTP request will - be a POST instead of a GET when the *data* parameter is provided. - *data* should be a buffer in the standard + *data* may be a bytes object specifying additional data to send to the + server, or ``None`` if no such data is needed. *data* may also be an + iterable object and in that case Content-Length value must be specified in + the headers. Currently HTTP requests are the only ones that use *data*; the + HTTP request will be a POST instead of a GET when the *data* parameter is + provided. *data* should be a buffer in the standard :mimetype:`application/x-www-form-urlencoded` format. The - :func:`urllib.parse.urlencode` function takes a mapping or sequence - of 2-tuples and returns a string in this format. + :func:`urllib.parse.urlencode` function takes a mapping or sequence of + 2-tuples and returns a string in this format. urllib.request module uses + HTTP/1.1 and includes ``Connection:close`` header in its HTTP requests. The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). This actually only works for HTTP, HTTPS and FTP connections. + The optional *cafile* and *capath* parameters specify a set of trusted + CA certificates for HTTPS requests. *cafile* should point to a single + file containing a bundle of CA certificates, whereas *capath* should + point to a directory of hashed certificate files. More information can + be found in :meth:`ssl.SSLContext.load_verify_locations`. + + .. warning:: + If neither *cafile* nor *capath* is specified, an HTTPS request + will not do any verification of the server's certificate. + This function returns a file-like object with two additional methods from the :mod:`urllib.response` module @@ -61,6 +70,16 @@ The :mod:`urllib.request` module defines the following functions: Proxy handling, which was done by passing a dictionary parameter to ``urllib.urlopen``, can be obtained by using :class:`ProxyHandler` objects. + .. versionchanged:: 3.2 + *cafile* and *capath* were added. + + .. versionchanged:: 3.2 + HTTPS virtual hosts are now supported if possible (that is, if + :data:`ssl.HAS_SNI` is true). + + .. versionadded:: 3.2 + *data* can be an iterable object. + .. function:: install_opener(opener) Install an :class:`OpenerDirector` instance as the default global opener. @@ -86,54 +105,8 @@ The :mod:`urllib.request` module defines the following functions: can be imported), :class:`HTTPSHandler` will also be added. A :class:`BaseHandler` subclass may also change its :attr:`handler_order` - member variable to modify its position in the handlers list. - - -.. function:: urlretrieve(url, filename=None, reporthook=None, data=None) - - Copy a network object denoted by a URL to a local file, if necessary. If the URL - points to a local file, or a valid cached copy of the object exists, the object - is not copied. Return a tuple ``(filename, headers)`` where *filename* is the - local file name under which the object can be found, and *headers* is whatever - the :meth:`info` method of the object returned by :func:`urlopen` returned (for - a remote object, possibly cached). Exceptions are the same as for - :func:`urlopen`. - - The second argument, if present, specifies the file location to copy to (if - absent, the location will be a tempfile with a generated name). The third - argument, if present, is a hook function that will be called once on - establishment of the network connection and once after each block read - thereafter. The hook will be passed three arguments; a count of blocks - transferred so far, a block size in bytes, and the total size of the file. The - third argument may be ``-1`` on older FTP servers which do not return a file - size in response to a retrieval request. - - If the *url* uses the :file:`http:` scheme identifier, the optional *data* - argument may be given to specify a ``POST`` request (normally the request type - is ``GET``). The *data* argument must in standard - :mimetype:`application/x-www-form-urlencoded` format; see the :func:`urlencode` - function below. - - :func:`urlretrieve` will raise :exc:`ContentTooShortError` when it detects that - the amount of data available was less than the expected amount (which is the - size reported by a *Content-Length* header). This can occur, for example, when - the download is interrupted. - - The *Content-Length* is treated as a lower bound: if there's more data to read, - :func:`urlretrieve` reads more data, but if less data is available, it raises - the exception. - - You can still retrieve the downloaded data in this case, it is stored in the - :attr:`content` attribute of the exception instance. + attribute to modify its position in the handlers list. - If no *Content-Length* header was supplied, :func:`urlretrieve` can not - check the size of the data it has downloaded, and just returns it. In - this case you just have to assume that the download was successful. - -.. function:: urlcleanup() - - Clear the cache that may have been built up by previous calls to - :func:`urlretrieve`. .. function:: pathname2url(path) @@ -151,10 +124,10 @@ The :mod:`urllib.request` module defines the following functions: .. function:: getproxies() This helper function returns a dictionary of scheme to proxy server URL - mappings. It scans the environment for variables named ``<scheme>_proxy`` - for all operating systems first, and when it cannot find it, looks for proxy - information from Mac OSX System Configuration for Mac OS X and Windows - Systems Registry for Windows. + mappings. It scans the environment for variables named ``<scheme>_proxy``, + in a case insensitive approach, for all operating systems first, and when it + cannot find it, looks for proxy information from Mac OSX System + Configuration for Mac OS X and Windows Systems Registry for Windows. The following classes are provided: @@ -203,116 +176,6 @@ The following classes are provided: fetching of the image, this should be true. -.. class:: URLopener(proxies=None, **x509) - - Base class for opening and reading URLs. Unless you need to support opening - objects using schemes other than :file:`http:`, :file:`ftp:`, or :file:`file:`, - you probably want to use :class:`FancyURLopener`. - - By default, the :class:`URLopener` class sends a :mailheader:`User-Agent` header - of ``urllib/VVV``, where *VVV* is the :mod:`urllib` version number. - Applications can define their own :mailheader:`User-Agent` header by subclassing - :class:`URLopener` or :class:`FancyURLopener` and setting the class attribute - :attr:`version` to an appropriate string value in the subclass definition. - - The optional *proxies* parameter should be a dictionary mapping scheme names to - proxy URLs, where an empty dictionary turns proxies off completely. Its default - value is ``None``, in which case environmental proxy settings will be used if - present, as discussed in the definition of :func:`urlopen`, above. - - Additional keyword parameters, collected in *x509*, may be used for - authentication of the client when using the :file:`https:` scheme. The keywords - *key_file* and *cert_file* are supported to provide an SSL key and certificate; - both are needed to support client authentication. - - :class:`URLopener` objects will raise an :exc:`IOError` exception if the server - returns an error code. - - .. method:: open(fullurl, data=None) - - Open *fullurl* using the appropriate protocol. This method sets up cache and - proxy information, then calls the appropriate open method with its input - arguments. If the scheme is not recognized, :meth:`open_unknown` is called. - The *data* argument has the same meaning as the *data* argument of - :func:`urlopen`. - - - .. method:: open_unknown(fullurl, data=None) - - Overridable interface to open unknown URL types. - - - .. method:: retrieve(url, filename=None, reporthook=None, data=None) - - Retrieves the contents of *url* and places it in *filename*. The return value - is a tuple consisting of a local filename and either a - :class:`email.message.Message` object containing the response headers (for remote - URLs) or ``None`` (for local URLs). The caller must then open and read the - contents of *filename*. If *filename* is not given and the URL refers to a - local file, the input filename is returned. If the URL is non-local and - *filename* is not given, the filename is the output of :func:`tempfile.mktemp` - with a suffix that matches the suffix of the last path component of the input - URL. If *reporthook* is given, it must be a function accepting three numeric - parameters. It will be called after each chunk of data is read from the - network. *reporthook* is ignored for local URLs. - - If the *url* uses the :file:`http:` scheme identifier, the optional *data* - argument may be given to specify a ``POST`` request (normally the request type - is ``GET``). The *data* argument must in standard - :mimetype:`application/x-www-form-urlencoded` format; see the :func:`urlencode` - function below. - - - .. attribute:: version - - Variable that specifies the user agent of the opener object. To get - :mod:`urllib` to tell servers that it is a particular user agent, set this in a - subclass as a class variable or in the constructor before calling the base - constructor. - - -.. class:: FancyURLopener(...) - - :class:`FancyURLopener` subclasses :class:`URLopener` providing default handling - for the following HTTP response codes: 301, 302, 303, 307 and 401. For the 30x - response codes listed above, the :mailheader:`Location` header is used to fetch - the actual URL. For 401 response codes (authentication required), basic HTTP - authentication is performed. For the 30x response codes, recursion is bounded - by the value of the *maxtries* attribute, which defaults to 10. - - For all other response codes, the method :meth:`http_error_default` is called - which you can override in subclasses to handle the error appropriately. - - .. note:: - - According to the letter of :rfc:`2616`, 301 and 302 responses to POST requests - must not be automatically redirected without confirmation by the user. In - reality, browsers do allow automatic redirection of these responses, changing - the POST to a GET, and :mod:`urllib` reproduces this behaviour. - - The parameters to the constructor are the same as those for :class:`URLopener`. - - .. note:: - - When performing basic authentication, a :class:`FancyURLopener` instance calls - its :meth:`prompt_user_passwd` method. The default implementation asks the - users for the required information on the controlling terminal. A subclass may - override this method to support more appropriate behavior if needed. - - The :class:`FancyURLopener` class offers one additional method that should be - overloaded to provide the appropriate behavior: - - .. method:: prompt_user_passwd(host, realm) - - Return information needed to authenticate the user at the given host in the - specified security realm. The return value should be a tuple, ``(user, - password)``, which can be used for basic authentication. - - The implementation prompts for this information on the terminal; an application - should override this method to use an appropriate interaction model in the local - environment. - - .. class:: OpenerDirector() The :class:`OpenerDirector` class opens URLs via :class:`BaseHandler`\ s chained @@ -421,9 +284,13 @@ The following classes are provided: A class to handle opening of HTTP URLs. -.. class:: HTTPSHandler() +.. class:: HTTPSHandler(debuglevel=0, context=None, check_hostname=None) - A class to handle opening of HTTPS URLs. + A class to handle opening of HTTPS URLs. *context* and *check_hostname* + have the same meaning as in :class:`http.client.HTTPSConnection`. + + .. versionchanged:: 3.2 + *context* and *check_hostname* were added. .. class:: FileHandler() @@ -446,6 +313,11 @@ The following classes are provided: A catch-all class to handle unknown URLs. +.. class:: HTTPErrorProcessor() + + Process HTTP error responses. + + .. _request-objects: Request Objects @@ -669,7 +541,7 @@ intended for direct use: Remove any parents. -The following members and methods should only be used by classes derived from +The following attribute and methods should only be used by classes derived from :class:`BaseHandler`. .. note:: @@ -994,8 +866,12 @@ FileHandler Objects .. method:: FileHandler.file_open(req) Open the file locally, if there is no host name, or the host name is - ``'localhost'``. Change the protocol to ``ftp`` otherwise, and retry opening it - using :attr:`parent`. + ``'localhost'``. + + This method is applicable only for local hostnames. When a remote hostname + is given, an :exc:`URLError` is raised. + +.. versionchanged:: 3.2 .. _ftp-handler-objects: @@ -1045,7 +921,7 @@ UnknownHandler Objects HTTPErrorProcessor Objects -------------------------- -.. method:: HTTPErrorProcessor.unknown_open() +.. method:: HTTPErrorProcessor.http_response() Process HTTP error responses. @@ -1057,6 +933,13 @@ HTTPErrorProcessor Objects :exc:`HTTPError` if no other handler handles the error. +.. method:: HTTPErrorProcessor.https_response() + + Process HTTPS error responses. + + The behavior is same as :meth:`http_response`. + + .. _urllib-request-examples: Examples @@ -1100,7 +983,7 @@ when the Python installation supports SSL. :: >>> import urllib.request >>> req = urllib.request.Request(url='https://localhost/cgi-bin/test.cgi', - ... data='This data is passed to stdin of the CGI') + ... data=b'This data is passed to stdin of the CGI') >>> f = urllib.request.urlopen(req) >>> print(f.read().decode('utf-8')) Got Data: "This data is passed to stdin of the CGI" @@ -1176,11 +1059,13 @@ containing parameters:: >>> f = urllib.request.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params) >>> print(f.read().decode('utf-8')) -The following example uses the ``POST`` method instead:: +The following example uses the ``POST`` method instead. Note that params output +from urlencode is encoded to bytes before it is sent to urlopen as data:: >>> import urllib.request >>> import urllib.parse >>> params = urllib.parse.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) + >>> params = params.encode('utf-8') >>> f = urllib.request.urlopen("http://www.musi-cal.com/cgi-bin/query", params) >>> print(f.read().decode('utf-8')) @@ -1201,6 +1086,170 @@ The following example uses no proxies at all, overriding environment settings:: >>> f.read().decode('utf-8') +Legacy interface +---------------- + +The following functions and classes are ported from the Python 2 module +``urllib`` (as opposed to ``urllib2``). They might become deprecated at +some point in the future. + + +.. function:: urlretrieve(url, filename=None, reporthook=None, data=None) + + Copy a network object denoted by a URL to a local file, if necessary. If the URL + points to a local file, or a valid cached copy of the object exists, the object + is not copied. Return a tuple ``(filename, headers)`` where *filename* is the + local file name under which the object can be found, and *headers* is whatever + the :meth:`info` method of the object returned by :func:`urlopen` returned (for + a remote object, possibly cached). Exceptions are the same as for + :func:`urlopen`. + + The second argument, if present, specifies the file location to copy to (if + absent, the location will be a tempfile with a generated name). The third + argument, if present, is a hook function that will be called once on + establishment of the network connection and once after each block read + thereafter. The hook will be passed three arguments; a count of blocks + transferred so far, a block size in bytes, and the total size of the file. The + third argument may be ``-1`` on older FTP servers which do not return a file + size in response to a retrieval request. + + If the *url* uses the :file:`http:` scheme identifier, the optional *data* + argument may be given to specify a ``POST`` request (normally the request type + is ``GET``). The *data* argument must in standard + :mimetype:`application/x-www-form-urlencoded` format; see the :func:`urlencode` + function below. + + :func:`urlretrieve` will raise :exc:`ContentTooShortError` when it detects that + the amount of data available was less than the expected amount (which is the + size reported by a *Content-Length* header). This can occur, for example, when + the download is interrupted. + + The *Content-Length* is treated as a lower bound: if there's more data to read, + :func:`urlretrieve` reads more data, but if less data is available, it raises + the exception. + + You can still retrieve the downloaded data in this case, it is stored in the + :attr:`content` attribute of the exception instance. + + If no *Content-Length* header was supplied, :func:`urlretrieve` can not check + the size of the data it has downloaded, and just returns it. In this case + you just have to assume that the download was successful. + +.. function:: urlcleanup() + + Clear the cache that may have been built up by previous calls to + :func:`urlretrieve`. + +.. class:: URLopener(proxies=None, **x509) + + Base class for opening and reading URLs. Unless you need to support opening + objects using schemes other than :file:`http:`, :file:`ftp:`, or :file:`file:`, + you probably want to use :class:`FancyURLopener`. + + By default, the :class:`URLopener` class sends a :mailheader:`User-Agent` header + of ``urllib/VVV``, where *VVV* is the :mod:`urllib` version number. + Applications can define their own :mailheader:`User-Agent` header by subclassing + :class:`URLopener` or :class:`FancyURLopener` and setting the class attribute + :attr:`version` to an appropriate string value in the subclass definition. + + The optional *proxies* parameter should be a dictionary mapping scheme names to + proxy URLs, where an empty dictionary turns proxies off completely. Its default + value is ``None``, in which case environmental proxy settings will be used if + present, as discussed in the definition of :func:`urlopen`, above. + + Additional keyword parameters, collected in *x509*, may be used for + authentication of the client when using the :file:`https:` scheme. The keywords + *key_file* and *cert_file* are supported to provide an SSL key and certificate; + both are needed to support client authentication. + + :class:`URLopener` objects will raise an :exc:`IOError` exception if the server + returns an error code. + + .. method:: open(fullurl, data=None) + + Open *fullurl* using the appropriate protocol. This method sets up cache and + proxy information, then calls the appropriate open method with its input + arguments. If the scheme is not recognized, :meth:`open_unknown` is called. + The *data* argument has the same meaning as the *data* argument of + :func:`urlopen`. + + + .. method:: open_unknown(fullurl, data=None) + + Overridable interface to open unknown URL types. + + + .. method:: retrieve(url, filename=None, reporthook=None, data=None) + + Retrieves the contents of *url* and places it in *filename*. The return value + is a tuple consisting of a local filename and either a + :class:`email.message.Message` object containing the response headers (for remote + URLs) or ``None`` (for local URLs). The caller must then open and read the + contents of *filename*. If *filename* is not given and the URL refers to a + local file, the input filename is returned. If the URL is non-local and + *filename* is not given, the filename is the output of :func:`tempfile.mktemp` + with a suffix that matches the suffix of the last path component of the input + URL. If *reporthook* is given, it must be a function accepting three numeric + parameters. It will be called after each chunk of data is read from the + network. *reporthook* is ignored for local URLs. + + If the *url* uses the :file:`http:` scheme identifier, the optional *data* + argument may be given to specify a ``POST`` request (normally the request type + is ``GET``). The *data* argument must in standard + :mimetype:`application/x-www-form-urlencoded` format; see the :func:`urlencode` + function below. + + + .. attribute:: version + + Variable that specifies the user agent of the opener object. To get + :mod:`urllib` to tell servers that it is a particular user agent, set this in a + subclass as a class variable or in the constructor before calling the base + constructor. + + +.. class:: FancyURLopener(...) + + :class:`FancyURLopener` subclasses :class:`URLopener` providing default handling + for the following HTTP response codes: 301, 302, 303, 307 and 401. For the 30x + response codes listed above, the :mailheader:`Location` header is used to fetch + the actual URL. For 401 response codes (authentication required), basic HTTP + authentication is performed. For the 30x response codes, recursion is bounded + by the value of the *maxtries* attribute, which defaults to 10. + + For all other response codes, the method :meth:`http_error_default` is called + which you can override in subclasses to handle the error appropriately. + + .. note:: + + According to the letter of :rfc:`2616`, 301 and 302 responses to POST requests + must not be automatically redirected without confirmation by the user. In + reality, browsers do allow automatic redirection of these responses, changing + the POST to a GET, and :mod:`urllib` reproduces this behaviour. + + The parameters to the constructor are the same as those for :class:`URLopener`. + + .. note:: + + When performing basic authentication, a :class:`FancyURLopener` instance calls + its :meth:`prompt_user_passwd` method. The default implementation asks the + users for the required information on the controlling terminal. A subclass may + override this method to support more appropriate behavior if needed. + + The :class:`FancyURLopener` class offers one additional method that should be + overloaded to provide the appropriate behavior: + + .. method:: prompt_user_passwd(host, realm) + + Return information needed to authenticate the user at the given host in the + specified security realm. The return value should be a tuple, ``(user, + password)``, which can be used for basic authentication. + + The implementation prompts for this information on the terminal; an application + should override this method to use an appropriate interaction model in the local + environment. + + :mod:`urllib.request` Restrictions ---------------------------------- @@ -1208,11 +1257,11 @@ The following example uses no proxies at all, overriding environment settings:: pair: HTTP; protocol pair: FTP; protocol -* Currently, only the following protocols are supported: HTTP, (versions 0.9 and - 1.0), FTP, and local files. +* Currently, only the following protocols are supported: HTTP (versions 0.9 and + 1.0), FTP, and local files. -* The caching feature of :func:`urlretrieve` has been disabled until I find the - time to hack proper processing of Expiration time headers. +* The caching feature of :func:`urlretrieve` has been disabled until someone + finds the time to hack proper processing of Expiration time headers. * There should be a function to query whether a particular URL is in the cache. diff --git a/Doc/library/uu.rst b/Doc/library/uu.rst index 7813e448e8..d61c178831 100644 --- a/Doc/library/uu.rst +++ b/Doc/library/uu.rst @@ -5,6 +5,9 @@ :synopsis: Encode and decode files in uuencode format. .. moduleauthor:: Lance Ellinghouse +**Source code:** :source:`Lib/uu.py` + +-------------- This module encodes and decodes files in uuencode format, allowing arbitrary binary data to be transferred over ASCII-only connections. Wherever a file @@ -56,4 +59,3 @@ The :mod:`uu` module defines the following functions: Module :mod:`binascii` Support module containing ASCII-to-binary and binary-to-ASCII conversions. - diff --git a/Doc/library/uuid.rst b/Doc/library/uuid.rst index 835194675f..7dc46acd05 100644 --- a/Doc/library/uuid.rst +++ b/Doc/library/uuid.rst @@ -222,34 +222,34 @@ Here are some examples of typical usage of the :mod:`uuid` module:: >>> import uuid - # make a UUID based on the host ID and current time + >>> # make a UUID based on the host ID and current time >>> uuid.uuid1() UUID('a8098c1a-f86e-11da-bd1a-00112444be1e') - # make a UUID using an MD5 hash of a namespace UUID and a name + >>> # make a UUID using an MD5 hash of a namespace UUID and a name >>> uuid.uuid3(uuid.NAMESPACE_DNS, 'python.org') UUID('6fa459ea-ee8a-3ca4-894e-db77e160355e') - # make a random UUID + >>> # make a random UUID >>> uuid.uuid4() UUID('16fd2706-8baf-433b-82eb-8c7fada847da') - # make a UUID using a SHA-1 hash of a namespace UUID and a name + >>> # make a UUID using a SHA-1 hash of a namespace UUID and a name >>> uuid.uuid5(uuid.NAMESPACE_DNS, 'python.org') UUID('886313e1-3b8a-5372-9b90-0c9aee199e5d') - # make a UUID from a string of hex digits (braces and hyphens ignored) + >>> # make a UUID from a string of hex digits (braces and hyphens ignored) >>> x = uuid.UUID('{00010203-0405-0607-0809-0a0b0c0d0e0f}') - # convert a UUID to a string of hex digits in standard form + >>> # convert a UUID to a string of hex digits in standard form >>> str(x) '00010203-0405-0607-0809-0a0b0c0d0e0f' - # get the raw 16 bytes of the UUID + >>> # get the raw 16 bytes of the UUID >>> x.bytes b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f' - # make a UUID from a 16-byte string + >>> # make a UUID from a 16-byte string >>> uuid.UUID(bytes=x.bytes) UUID('00010203-0405-0607-0809-0a0b0c0d0e0f') diff --git a/Doc/library/warnings.rst b/Doc/library/warnings.rst index e62be551bb..8af19a282e 100644 --- a/Doc/library/warnings.rst +++ b/Doc/library/warnings.rst @@ -6,6 +6,9 @@ .. module:: warnings :synopsis: Issue warning messages and control their disposition. +**Source code:** :source:`Lib/warnings.py` + +-------------- Warning messages are typically issued in situations where it is useful to alert the user of some condition in a program, where that condition (normally) doesn't @@ -13,7 +16,7 @@ warrant raising an exception and terminating the program. For example, one might want to issue a warning when a program uses an obsolete module. Python programmers issue warnings by calling the :func:`warn` function defined -in this module. (C programmers use :cfunc:`PyErr_WarnEx`; see +in this module. (C programmers use :c:func:`PyErr_WarnEx`; see :ref:`exceptionhandling` for details). Warning messages are normally written to ``sys.stderr``, but their disposition @@ -37,6 +40,10 @@ may be overridden; the default implementation of this function formats the message by calling :func:`formatwarning`, which is also available for use by custom implementations. +.. seealso:: + :func:`logging.captureWarnings` allows you to handle all warnings with + the standard logging infrastructure. + .. _warning-categories: @@ -57,7 +64,7 @@ following warnings category classes are currently defined: | :exc:`UserWarning` | The default category for :func:`warn`. | +----------------------------------+-----------------------------------------------+ | :exc:`DeprecationWarning` | Base category for warnings about deprecated | -| | features. | +| | features (ignored by default). | +----------------------------------+-----------------------------------------------+ | :exc:`SyntaxWarning` | Base category for warnings about dubious | | | syntactic features. | @@ -82,6 +89,9 @@ following warnings category classes are currently defined: | :exc:`BytesWarning` | Base category for warnings related to | | | :class:`bytes` and :class:`buffer`. | +----------------------------------+-----------------------------------------------+ +| :exc:`ResourceWarning` | Base category for warnings related to | +| | resource usage. | ++----------------------------------+-----------------------------------------------+ While these are technically built-in exceptions, they are documented here, @@ -150,14 +160,6 @@ interpreter command line. The interpreter saves the arguments for all :mod:`warnings` module parses these when it is first imported (invalid options are ignored, after printing a message to ``sys.stderr``). -The warnings that are ignored by default may be enabled by passing :option:`-Wd` -to the interpreter. This enables default handling for all warnings, including -those that are normally ignored by default. This is particular useful for -enabling ImportWarning when debugging problems importing a developed package. -ImportWarning can also be enabled explicitly in Python code using:: - - warnings.simplefilter('default', ImportWarning) - Default Warning Filters ~~~~~~~~~~~~~~~~~~~~~~~ @@ -166,12 +168,19 @@ By default, Python installs several warning filters, which can be overridden by the command-line options passed to :option:`-W` and calls to :func:`filterwarnings`. -* :exc:`PendingDeprecationWarning`, and :exc:`ImportWarning` are ignored. +* :exc:`DeprecationWarning` and :exc:`PendingDeprecationWarning`, and + :exc:`ImportWarning` are ignored. * :exc:`BytesWarning` is ignored unless the :option:`-b` option is given once or twice; in this case this warning is either printed (``-b``) or turned into an exception (``-bb``). +* :exc:`ResourceWarning` is ignored unless Python was built in debug mode. + +.. versionchanged:: 3.2 + :exc:`DeprecationWarning` is now ignored by default in addition to + :exc:`PendingDeprecationWarning`. + .. _warning-suppress: @@ -194,7 +203,10 @@ the warning using the :class:`catch_warnings` context manager:: While within the context manager all warnings will simply be ignored. This allows you to use known-deprecated code without having to see the warning while not suppressing the warning for other code that might not be aware of its use -of deprecated code. +of deprecated code. Note: this can only be guaranteed in a single-threaded +application. If two or more threads use the :class:`catch_warnings` context +manager at the same time, the behavior is undefined. + .. _warning-testing: @@ -232,7 +244,9 @@ Once the context manager exits, the warnings filter is restored to its state when the context was entered. This prevents tests from changing the warnings filter in unexpected ways between tests and leading to indeterminate test results. The :func:`showwarning` function in the module is also restored to -its original value. +its original value. Note: this can only be guaranteed in a single-threaded +application. If two or more threads use the :class:`catch_warnings` context +manager at the same time, the behavior is undefined. When testing multiple operations that raise the same kind of warning, it is important to test them in a manner that confirms each operation is raising @@ -242,6 +256,42 @@ continues to increase after each operation, or else delete the previous entries from the warnings list before each new operation). +.. _warning-ignored: + +Updating Code For New Versions of Python +---------------------------------------- + +Warnings that are only of interest to the developer are ignored by default. As +such you should make sure to test your code with typically ignored warnings +made visible. You can do this from the command-line by passing :option:`-Wd` +to the interpreter (this is shorthand for :option:`-W default`). This enables +default handling for all warnings, including those that are ignored by default. +To change what action is taken for encountered warnings you simply change what +argument is passed to :option:`-W`, e.g. :option:`-W error`. See the +:option:`-W` flag for more details on what is possible. + +To programmatically do the same as :option:`-Wd`, use:: + + warnings.simplefilter('default') + +Make sure to execute this code as soon as possible. This prevents the +registering of what warnings have been raised from unexpectedly influencing how +future warnings are treated. + +Having certain warnings ignored by default is done to prevent a user from +seeing warnings that are only of interest to the developer. As you do not +necessarily have control over what interpreter a user uses to run their code, +it is possible that a new version of Python will be released between your +release cycles. The new interpreter release could trigger new warnings in your +code that were not there in an older interpreter, e.g. +:exc:`DeprecationWarning` for a module that you are using. While you as a +developer want to be notified that your code is using a deprecated module, to a +user this information is essentially noise and provides no benefit to them. + +The :mod:`unittest` module has been also updated to use the ``'default'`` +filter while running tests. + + .. _warning-functions: Available Functions @@ -351,3 +401,11 @@ Available Context Managers module returned when you import :mod:`warnings` whose filter will be protected. This argument exists primarily for testing the :mod:`warnings` module itself. + + .. note:: + + The :class:`catch_warnings` manager works by replacing and + then later restoring the module's + :func:`showwarning` function and internal list of filter + specifications. This means the context manager is modifying + global state and therefore is not thread-safe. diff --git a/Doc/library/wave.rst b/Doc/library/wave.rst index 6cfa1a4733..afafb45c55 100644 --- a/Doc/library/wave.rst +++ b/Doc/library/wave.rst @@ -6,6 +6,10 @@ .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> .. Documentations stolen from comments in file. +**Source code:** :source:`Lib/wave.py` + +-------------- + The :mod:`wave` module provides a convenient interface to the WAV sound format. It does not support compression/decompression, but it does support mono/stereo. @@ -162,6 +166,10 @@ Wave_write objects, as returned by :func:`.open`, have the following methods: Set the frame rate to *n*. + .. versionchanged:: 3.2 + A non-integral input to this method is rounded to the nearest + integer. + .. method:: Wave_write.setnframes(n) diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index 10b69c98c0..63545abf3a 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -8,6 +8,9 @@ .. moduleauthor:: Martin von Löwis <martin@loewis.home.cs.tu-berlin.de> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/weakref.py` + +-------------- The :mod:`weakref` module allows the Python programmer to create :dfn:`weak references` to objects. @@ -59,8 +62,11 @@ is exposed by the :mod:`weakref` module for the benefit of advanced uses. Not all objects can be weakly referenced; those objects which can include class instances, functions written in Python (but not in C), instance methods, sets, frozensets, some :term:`file objects <file object>`, :term:`generator`\s, type -objects, sockets, arrays, deques and regular expression pattern objects. +objects, sockets, arrays, deques, regular expression pattern objects, and code +objects. +.. versionchanged:: 3.2 + Added support for thread.lock, threading.Lock, and code objects. Several built-in types such as :class:`list` and :class:`dict` do not directly support weak references but can add support through subclassing:: diff --git a/Doc/library/webbrowser.rst b/Doc/library/webbrowser.rst index 20c091399f..23ba6c5471 100644 --- a/Doc/library/webbrowser.rst +++ b/Doc/library/webbrowser.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/webbrowser.py` + +-------------- The :mod:`webbrowser` module provides a high-level interface to allow displaying Web-based documents to users. Under most circumstances, simply calling the diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst index 68f7e91263..5cf30ee03c 100644 --- a/Doc/library/winreg.rst +++ b/Doc/library/winreg.rst @@ -60,6 +60,33 @@ This module offers the following functions: :exc:`WindowsError` exception is raised. +.. function:: CreateKeyEx(key, sub_key, reserved=0, access=KEY_ALL_ACCESS) + + Creates or opens the specified key, returning a + :ref:`handle object <handle-object>`. + + *key* is an already open key, or one of the predefined + :ref:`HKEY_* constants <hkey-constants>`. + + *sub_key* is a string that names the key this method opens or creates. + + *res* is a reserved integer, and must be zero. The default is zero. + + *sam* is an integer that specifies an access mask that describes the desired + security access for the key. Default is :const:`KEY_ALL_ACCESS`. See + :ref:`Access Rights <access-rights>` for other allowed values. + + If *key* is one of the predefined keys, *sub_key* may be ``None``. In that + case, the handle returned is the same key handle passed in to the function. + + If the key already exists, this function opens the existing key. + + The return value is the handle of the opened key. If the function fails, a + :exc:`WindowsError` exception is raised. + + .. versionadded:: 3.2 + + .. function:: DeleteKey(key, sub_key) Deletes the specified key. @@ -67,8 +94,8 @@ This module offers the following functions: *key* is an already open key, or one of the predefined :ref:`HKEY_* constants <hkey-constants>`. - *sub_key* is a string that must be a subkey of the key identified by the *key* - parameter. This value must not be ``None``, and the key may not have subkeys. + *sub_key* is a string that must be a subkey of the key identified by the *key* + parameter. This value must not be ``None``, and the key may not have subkeys. *This method can not delete keys with subkeys.* @@ -76,6 +103,39 @@ This module offers the following functions: If the method fails, a :exc:`WindowsError` exception is raised. +.. function:: DeleteKeyEx(key, sub_key, access=KEY_ALL_ACCESS, reserved=0) + + Deletes the specified key. + + .. note:: + The :func:`DeleteKeyEx` function is implemented with the RegDeleteKeyEx + Windows API function, which is specific to 64-bit versions of Windows. + See the `RegDeleteKeyEx documentation + <http://msdn.microsoft.com/en-us/library/ms724847%28VS.85%29.aspx>`__. + + *key* is an already open key, or one of the predefined + :ref:`HKEY_* constants <hkey-constants>`. + + *sub_key* is a string that must be a subkey of the key identified by the + *key* parameter. This value must not be ``None``, and the key may not have + subkeys. + + *res* is a reserved integer, and must be zero. The default is zero. + + *sam* is an integer that specifies an access mask that describes the desired + security access for the key. Default is :const:`KEY_ALL_ACCESS`. See + :ref:`Access Rights <access-rights>` for other allowed values. + + *This method can not delete keys with subkeys.* + + If the method succeeds, the entire key, including all of its values, is + removed. If the method fails, a :exc:`WindowsError` exception is raised. + + On unsupported Windows versions, :exc:`NotImplementedError` is raised. + + .. versionadded:: 3.2 + + .. function:: DeleteValue(key, value) Removes a named value from a registry key. @@ -183,7 +243,7 @@ This module offers the following functions: specified in *file_name* is relative to the remote computer. -.. function:: OpenKey(key, sub_key[, res[, sam]]) +.. function:: OpenKey(key, sub_key, reserved=0, access=KEY_ALL_ACCESS) Opens the specified key, returning a :ref:`handle object <handle-object>`. @@ -202,6 +262,8 @@ This module offers the following functions: If the function fails, :exc:`WindowsError` is raised. + .. versionchanged:: 3.2 Allow the use of named arguments. + .. function:: OpenKeyEx() diff --git a/Doc/library/winsound.rst b/Doc/library/winsound.rst index d54c999cd9..8356062078 100644 --- a/Doc/library/winsound.rst +++ b/Doc/library/winsound.rst @@ -22,7 +22,7 @@ provided by Windows platforms. It includes functions and several constants. .. function:: PlaySound(sound, flags) - Call the underlying :cfunc:`PlaySound` function from the Platform API. The + Call the underlying :c:func:`PlaySound` function from the Platform API. The *sound* parameter may be a filename, audio data as a string, or ``None``. Its interpretation depends on the value of *flags*, which can be a bitwise ORed combination of the constants described below. If the *sound* parameter is @@ -32,7 +32,7 @@ provided by Windows platforms. It includes functions and several constants. .. function:: MessageBeep(type=MB_OK) - Call the underlying :cfunc:`MessageBeep` function from the Platform API. This + Call the underlying :c:func:`MessageBeep` function from the Platform API. This plays a sound as specified in the registry. The *type* argument specifies which sound to play; possible values are ``-1``, ``MB_ICONASTERISK``, ``MB_ICONEXCLAMATION``, ``MB_ICONHAND``, ``MB_ICONQUESTION``, and ``MB_OK``, all diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst index 231449ce9f..1fd345145c 100644 --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -22,7 +22,7 @@ be used to add WSGI support to a web server or framework. It provides utilities for manipulating WSGI environment variables and response headers, base classes for implementing WSGI servers, a demo HTTP server that serves WSGI applications, and a validation tool that checks WSGI servers and applications for conformance -to the WSGI specification (:pep:`333`). +to the WSGI specification (:pep:`3333`). See http://www.wsgi.org for more information about WSGI, and links to tutorials and other resources. @@ -39,9 +39,9 @@ and other resources. This module provides a variety of utility functions for working with WSGI environments. A WSGI environment is a dictionary containing HTTP request -variables as described in :pep:`333`. All of the functions taking an *environ* +variables as described in :pep:`3333`. All of the functions taking an *environ* parameter expect a WSGI-compliant dictionary to be supplied; please see -:pep:`333` for a detailed specification. +:pep:`3333` for a detailed specification. .. function:: guess_scheme(environ) @@ -60,7 +60,7 @@ parameter expect a WSGI-compliant dictionary to be supplied; please see .. function:: request_uri(environ, include_query=True) Return the full request URI, optionally including the query string, using the - algorithm found in the "URL Reconstruction" section of :pep:`333`. If + algorithm found in the "URL Reconstruction" section of :pep:`3333`. If *include_query* is false, the query string is not included in the resulting URI. @@ -104,7 +104,7 @@ parameter expect a WSGI-compliant dictionary to be supplied; please see This routine adds various parameters required for WSGI, including ``HTTP_HOST``, ``SERVER_NAME``, ``SERVER_PORT``, ``REQUEST_METHOD``, ``SCRIPT_NAME``, - ``PATH_INFO``, and all of the :pep:`333`\ -defined ``wsgi.*`` variables. It + ``PATH_INFO``, and all of the :pep:`3333`\ -defined ``wsgi.*`` variables. It only supplies default values, and does not replace any existing settings for these variables. @@ -122,8 +122,8 @@ parameter expect a WSGI-compliant dictionary to be supplied; please see def simple_app(environ, start_response): setup_testing_defaults(environ) - status = b'200 OK' - headers = [(b'Content-type', b'text/plain; charset=utf-8')] + status = '200 OK' + headers = [('Content-type', 'text/plain; charset=utf-8')] start_response(status, headers) @@ -152,8 +152,8 @@ also provides these miscellaneous utilities: support both :meth:`__getitem__` and :meth:`__iter__` iteration styles, for compatibility with Python 2.1 and Jython. As the object is iterated over, the optional *blksize* parameter will be repeatedly passed to the *filelike* - object's :meth:`read` method to obtain strings to yield. When :meth:`read` - returns an empty string, iteration is ended and is not resumable. + object's :meth:`read` method to obtain bytestrings to yield. When :meth:`read` + returns an empty bytestring, iteration is ended and is not resumable. If *filelike* has a :meth:`close` method, the returned object will also have a :meth:`close` method, and it will invoke the *filelike* object's :meth:`close` @@ -187,7 +187,7 @@ manipulation of WSGI response headers using a mapping-like interface. .. class:: Headers(headers) Create a mapping-like object wrapping *headers*, which must be a list of header - name/value tuples as described in :pep:`333`. + name/value tuples as described in :pep:`3333`. :class:`Headers` objects support typical mapping operations including :meth:`__getitem__`, :meth:`get`, :meth:`__setitem__`, :meth:`setdefault`, @@ -210,11 +210,11 @@ manipulation of WSGI response headers using a mapping-like interface. :meth:`items`, which is the same as the length of the wrapped header list. In fact, the :meth:`items` method just returns a copy of the wrapped header list. - Calling ``str()`` on a :class:`Headers` object returns a formatted string + Calling ``bytes()`` on a :class:`Headers` object returns a formatted bytestring suitable for transmission as HTTP response headers. Each header is placed on a line with its value, separated by a colon and a space. Each line is terminated - by a carriage return and line feed, and the string is terminated with a blank - line. + by a carriage return and line feed, and the bytestring is terminated with a + blank line. In addition to their mapping interface and formatting features, :class:`Headers` objects also have the following methods for querying and adding multi-valued @@ -272,7 +272,7 @@ request. (E.g., using the :func:`shift_path_info` function from Create a new WSGI server listening on *host* and *port*, accepting connections for *app*. The return value is an instance of the supplied *server_class*, and will process requests using the specified *handler_class*. *app* must be a WSGI - application object, as defined by :pep:`333`. + application object, as defined by :pep:`3333`. Example usage:: @@ -346,7 +346,7 @@ request. (E.g., using the :func:`shift_path_info` function from :attr:`base_environ` dictionary attribute and then adds various headers derived from the HTTP request. Each call to this method should return a new dictionary containing all of the relevant CGI environment variables as specified in - :pep:`333`. + :pep:`3333`. .. method:: WSGIRequestHandler.get_stderr() @@ -376,7 +376,7 @@ application objects that validate communications between a WSGI server or gateway and a WSGI application object, to check both sides for protocol conformance. -Note that this utility does not guarantee complete :pep:`333` compliance; an +Note that this utility does not guarantee complete :pep:`3333` compliance; an absence of errors from this module does not necessarily mean that errors do not exist. However, if this module does produce an error, then it is virtually certain that either the server or application is not 100% compliant. @@ -401,7 +401,7 @@ Paste" library. This wrapper may also generate output using the :mod:`warnings` module to indicate behaviors that are questionable but which may not actually be - prohibited by :pep:`333`. Unless they are suppressed using Python command-line + prohibited by :pep:`3333`. Unless they are suppressed using Python command-line options or the :mod:`warnings` API, any such warnings will be written to ``sys.stderr`` (*not* ``wsgi.errors``, unless they happen to be the same object). @@ -414,8 +414,8 @@ Paste" library. # Our callable object which is intentionally not compliant to the # standard, so the validator is going to break def simple_app(environ, start_response): - status = b'200 OK' # HTTP Status - headers = [(b'Content-type', b'text/plain')] # HTTP Headers + status = '200 OK' # HTTP Status + headers = [('Content-type', 'text/plain')] # HTTP Headers start_response(status, headers) # This is going to break because we need to return a list, and @@ -456,6 +456,32 @@ input, output, and error streams. environment. +.. class:: IISCGIHandler() + + A specialized alternative to :class:`CGIHandler`, for use when deploying on + Microsoft's IIS web server, without having set the config allowPathInfo + option (IIS>=7) or metabase allowPathInfoForScriptMappings (IIS<7). + + By default, IIS gives a ``PATH_INFO`` that duplicates the ``SCRIPT_NAME`` at + the front, causing problems for WSGI applications that wish to implement + routing. This handler strips any such duplicated path. + + IIS can be configured to pass the correct ``PATH_INFO``, but this causes + another bug where ``PATH_TRANSLATED`` is wrong. Luckily this variable is + rarely used and is not guaranteed by WSGI. On IIS<7, though, the + setting can only be made on a vhost level, affecting all other script + mappings, many of which break when exposed to the ``PATH_TRANSLATED`` bug. + For this reason IIS<7 is almost never deployed with the fix. (Even IIS7 + rarely uses it because there is still no UI for it.) + + There is no way for CGI code to tell whether the option was set, so a + separate handler class is provided. It is used in the same way as + :class:`CGIHandler`, i.e., by calling ``IISCGIHandler().run(app)``, where + ``app`` is the WSGI application object you wish to invoke. + + .. versionadded:: 3.2 + + .. class:: BaseCGIHandler(stdin, stdout, stderr, environ, multithread=True, multiprocess=False) Similar to :class:`CGIHandler`, but instead of using the :mod:`sys` and @@ -626,7 +652,7 @@ input, output, and error streams. This method can access the current error information using ``sys.exc_info()``, and should pass that information to *start_response* when calling it (as - described in the "Error Handling" section of :pep:`333`). + described in the "Error Handling" section of :pep:`3333`). The default implementation just uses the :attr:`error_status`, :attr:`error_headers`, and :attr:`error_body` attributes to generate an output @@ -641,23 +667,23 @@ input, output, and error streams. .. attribute:: BaseHandler.error_status The HTTP status used for error responses. This should be a status string as - defined in :pep:`333`; it defaults to a 500 code and message. + defined in :pep:`3333`; it defaults to a 500 code and message. .. attribute:: BaseHandler.error_headers The HTTP headers used for error responses. This should be a list of WSGI - response headers (``(name, value)`` tuples), as described in :pep:`333`. The + response headers (``(name, value)`` tuples), as described in :pep:`3333`. The default list just sets the content type to ``text/plain``. .. attribute:: BaseHandler.error_body - The error response body. This should be an HTTP response body string. It + The error response body. This should be an HTTP response body bytestring. It defaults to the plain text, "A server error occurred. Please contact the administrator." - Methods and attributes for :pep:`333`'s "Optional Platform-Specific File + Methods and attributes for :pep:`3333`'s "Optional Platform-Specific File Handling" feature: @@ -696,6 +722,24 @@ input, output, and error streams. version of the response set to the client. It defaults to ``"1.0"``. +.. function:: read_environ() + + Transcode CGI variables from ``os.environ`` to PEP 3333 "bytes in unicode" + strings, returning a new dictionary. This function is used by + :class:`CGIHandler` and :class:`IISCGIHandler` in place of directly using + ``os.environ``, which is not necessarily WSGI-compliant on all platforms + and web servers using Python 3 -- specifically, ones where the OS's + actual environment is Unicode (i.e. Windows), or ones where the environment + is bytes, but the system encoding used by Python to decode it is anything + other than ISO-8859-1 (e.g. Unix systems using UTF-8). + + If you are implementing a CGI-based handler of your own, you probably want + to use this routine instead of just copying values out of ``os.environ`` + directly. + + .. versionadded:: 3.2 + + Examples -------- @@ -710,8 +754,8 @@ This is a working "Hello World" WSGI application:: # is a dictionary containing CGI-style envrironment variables and the # second variable is the callable object (see PEP 333). def hello_world_app(environ, start_response): - status = b'200 OK' # HTTP Status - headers = [(b'Content-type', b'text/plain; charset=utf-8')] # HTTP Headers + status = '200 OK' # HTTP Status + headers = [('Content-type', 'text/plain; charset=utf-8')] # HTTP Headers start_response(status, headers) # The returned object is going to be printed diff --git a/Doc/library/xdrlib.rst b/Doc/library/xdrlib.rst index 4fcb9cdfb6..5c7dfa454b 100644 --- a/Doc/library/xdrlib.rst +++ b/Doc/library/xdrlib.rst @@ -9,6 +9,10 @@ single: XDR single: External Data Representation +**Source code:** :source:`Lib/xdrlib.py` + +-------------- + The :mod:`xdrlib` module supports the External Data Representation Standard as described in :rfc:`1014`, written by Sun Microsystems, Inc. June 1987. It supports most of the data types described in the RFC. @@ -256,7 +260,7 @@ Exceptions in this module are coded as class instances: .. exception:: Error - The base exception class. :exc:`Error` has a single public data member + The base exception class. :exc:`Error` has a single public attribute :attr:`msg` containing the description of the error. diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst index 12360c0db4..ab5476d91f 100644 --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -7,6 +7,9 @@ .. sectionauthor:: Paul Prescod <paul@prescod.net> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/xml/dom/minidom.py` + +-------------- :mod:`xml.dom.minidom` is a light-weight implementation of the Document Object Model interface. It is intended to be simpler than the full DOM and also @@ -78,22 +81,12 @@ document: the one that holds all others. Here is an example program:: dom3 = parseString("<myxml>Some data</myxml>") assert dom3.documentElement.tagName == "myxml" -When you are finished with a DOM, you should clean it up. This is necessary -because some versions of Python do not support garbage collection of objects -that refer to each other in a cycle. Until this restriction is removed from all -versions of Python, it is safest to write your code as if cycles would not be -cleaned up. - -The way to clean up a DOM is to call its :meth:`unlink` method:: - - dom1.unlink() - dom2.unlink() - dom3.unlink() - -:meth:`unlink` is a :mod:`xml.dom.minidom`\ -specific extension to the DOM API. -After calling :meth:`unlink` on a node, the node and its descendants are -essentially useless. - +When you are finished with a DOM tree, you may optionally call the +:meth:`unlink` method to encourage early cleanup of the now-unneeded +objects. :meth:`unlink` is a :mod:`xml.dom.minidom`\ -specific +extension to the DOM API that renders the node and its descendants are +essentially useless. Otherwise, Python's garbage collector will +eventually take care of the objects in the tree. .. seealso:: @@ -120,6 +113,13 @@ module documentation. This section lists the differences between the API and to be called on the :class:`Document` object, but may be called on child nodes to discard children of that node. + You can avoid calling this method explicitly by using the :keyword:`with` + statement. The following code will automatically unlink *dom* when the + :keyword:`with` block is exited:: + + with xml.dom.minidom.parse(datasource) as dom: + ... # Work with dom. + .. method:: Node.writexml(writer, indent="", addindent="", newl="") @@ -135,18 +135,20 @@ module documentation. This section lists the differences between the API and .. method:: Node.toxml(encoding=None) - Return the XML that the DOM represents as a string. - - With no argument, the XML header does not specify an encoding, and the result is - Unicode string if the default encoding cannot represent all characters in the - document. Encoding this string in an encoding other than UTF-8 is likely - incorrect, since UTF-8 is the default encoding of XML. + Return a string or byte string containing the XML represented by + the DOM node. - With an explicit *encoding* [1]_ argument, the result is a byte string in the - specified encoding. It is recommended that this argument is always specified. To - avoid :exc:`UnicodeError` exceptions in case of unrepresentable text data, the - encoding argument should be specified as "utf-8". + With an explicit *encoding* [1]_ argument, the result is a byte + string in the specified encoding. It is recommended that you + always specify an encoding; you may use any encoding you like, but + an argument of "utf-8" is the most common choice, avoiding + :exc:`UnicodeError` exceptions in case of unrepresentable text + data. + With no *encoding* argument, the result is a Unicode string, and the + XML declaration in the resulting string does not specify an + encoding. Encoding this string in an encoding other than UTF-8 is + likely incorrect, since UTF-8 is the default encoding of XML. .. method:: Node.toprettyxml(indent="", newl="", encoding="") @@ -154,7 +156,8 @@ module documentation. This section lists the differences between the API and indentation string and defaults to a tabulator; *newl* specifies the string emitted at the end of each line and defaults to ``\n``. - There's also an *encoding* argument; see :meth:`toxml`. + The *encoding* argument behaves like the corresponding argument of + :meth:`toxml`. .. _dom-example: @@ -239,7 +242,9 @@ utility to most DOM users. .. rubric:: Footnotes -.. [#] The encoding string included in XML output should conform to the - appropriate standards. For example, "UTF-8" is valid, but "UTF8" is - not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl +.. [#] The encoding name included in the XML output should conform to + the appropriate standards. For example, "UTF-8" is valid, but + "UTF8" is not valid in an XML document's declaration, even though + Python accepts it as an encoding name. + See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl and http://www.iana.org/assignments/character-sets . diff --git a/Doc/library/xml.dom.pulldom.rst b/Doc/library/xml.dom.pulldom.rst index 1893fba894..4a5ef4c135 100644 --- a/Doc/library/xml.dom.pulldom.rst +++ b/Doc/library/xml.dom.pulldom.rst @@ -5,6 +5,9 @@ :synopsis: Support for building partial DOM trees from SAX events. .. moduleauthor:: Paul Prescod <paul@prescod.net> +**Source code:** :source:`Lib/xml/dom/pulldom.py` + +-------------- :mod:`xml.dom.pulldom` allows building only selected portions of a Document Object Model representation of a document from SAX events. diff --git a/Doc/library/xml.dom.rst b/Doc/library/xml.dom.rst index a3414d749f..297fc0c5c0 100644 --- a/Doc/library/xml.dom.rst +++ b/Doc/library/xml.dom.rst @@ -686,18 +686,27 @@ Attr Objects .. attribute:: Attr.name - The attribute name. In a namespace-using document it may have colons in it. + The attribute name. + In a namespace-using document it may include a colon. .. attribute:: Attr.localName - The part of the name following the colon if there is one, else the entire name. + The part of the name following the colon if there is one, else the + entire name. This is a read-only attribute. .. attribute:: Attr.prefix - The part of the name preceding the colon if there is one, else the empty string. + The part of the name preceding the colon if there is one, else the + empty string. + + +.. attribute:: Attr.value + + The text value of the attribute. This is a synonym for the + :attr:`nodeValue` attribute. .. _dom-attributelist-objects: @@ -960,29 +969,24 @@ Python. Type Mapping ^^^^^^^^^^^^ -The primitive IDL types used in the DOM specification are mapped to Python types +The IDL types used in the DOM specification are mapped to Python types according to the following table. +------------------+-------------------------------------------+ | IDL Type | Python Type | +==================+===========================================+ -| ``boolean`` | ``IntegerType`` (with a value of ``0`` or | -| | ``1``) | +| ``boolean`` | ``bool`` or ``int`` | +------------------+-------------------------------------------+ -| ``int`` | ``IntegerType`` | +| ``int`` | ``int`` | +------------------+-------------------------------------------+ -| ``long int`` | ``IntegerType`` | +| ``long int`` | ``int`` | +------------------+-------------------------------------------+ -| ``unsigned int`` | ``IntegerType`` | +| ``unsigned int`` | ``int`` | ++------------------+-------------------------------------------+ +| ``DOMString`` | ``str`` or ``bytes`` | ++------------------+-------------------------------------------+ +| ``null`` | ``None`` | +------------------+-------------------------------------------+ - -Additionally, the :class:`DOMString` defined in the recommendation is mapped to -a bytes or string object. Applications should be able to handle -Unicode whenever a string is returned from the DOM. - -The IDL ``null`` value is mapped to ``None``, which may be accepted or -provided by the implementation whenever ``null`` is allowed by the API. - .. _dom-accessor-methods: diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst index bc04a5355e..a46d99d969 100644 --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -5,10 +5,13 @@ :synopsis: Implementation of the ElementTree API. .. moduleauthor:: Fredrik Lundh <fredrik@pythonware.com> +**Source code:** :source:`Lib/xml/etree/ElementTree.py` -The Element type is a flexible container object, designed to store hierarchical -data structures in memory. The type can be described as a cross between a list -and a dictionary. +-------------- + +The :class:`Element` type is a flexible container object, designed to store +hierarchical data structures in memory. The type can be described as a cross +between a list and a dictionary. Each element has a number of properties associated with it: @@ -23,7 +26,8 @@ Each element has a number of properties associated with it: * a number of child elements, stored in a Python sequence -To create an element instance, use the Element or SubElement factory functions. +To create an element instance, use the :class:`Element` constructor or the +:func:`SubElement` factory function. The :class:`ElementTree` class can be used to wrap an element structure, and convert it from and to XML. @@ -31,8 +35,14 @@ convert it from and to XML. A C implementation of this API is available as :mod:`xml.etree.cElementTree`. See http://effbot.org/zone/element-index.htm for tutorials and links to other -docs. Fredrik Lundh's page is also the location of the development version of the -xml.etree.ElementTree. +docs. Fredrik Lundh's page is also the location of the development version of +the xml.etree.ElementTree. + +.. versionchanged:: 3.2 + The ElementTree API is updated to 1.3. For more information, see + `Introducing ElementTree 1.3 + <http://effbot.org/zone/elementtree-13-intro.htm>`_. + .. _elementtree-functions: @@ -43,16 +53,16 @@ Functions .. function:: Comment(text=None) Comment element factory. This factory function creates a special element - that will be serialized as an XML comment. The comment string can be either - an ASCII-only :class:`bytes` object or a :class:`str` object. *text* is a - string containing the comment string. Returns an element instance + that will be serialized as an XML comment by the standard serializer. The + comment string can be either a bytestring or a Unicode string. *text* is a + string containing the comment string. Returns an element instance representing a comment. .. function:: dump(elem) - Writes an element tree or element structure to sys.stdout. This function should - be used for debugging only. + Writes an element tree or element structure to sys.stdout. This function + should be used for debugging only. The exact output format is implementation dependent. In this version, it's written as an ordinary XML file. @@ -60,39 +70,36 @@ Functions *elem* is an element tree or an individual element. -.. function:: Element(tag, attrib={}, **extra) +.. function:: fromstring(text) - Element factory. This function returns an object implementing the standard - Element interface. The exact class or type of that object is implementation - dependent, but it will always be compatible with the _ElementInterface class in - this module. + Parses an XML section from a string constant. Same as :func:`XML`. *text* + is a string containing XML data. Returns an :class:`Element` instance. - The element name, attribute names, and attribute values can be either an - ASCII-only :class:`bytes` object or a :class:`str` object. *tag* is the - element name. *attrib* is an optional dictionary, containing element - attributes. *extra* contains additional attributes, given as keyword - arguments. Returns an element instance. +.. function:: fromstringlist(sequence, parser=None) -.. function:: fromstring(text) + Parses an XML document from a sequence of string fragments. *sequence* is a + list or other sequence containing XML data fragments. *parser* is an + optional parser instance. If not given, the standard :class:`XMLParser` + parser is used. Returns an :class:`Element` instance. - Parses an XML section from a string constant. Same as XML. *text* is a string - containing XML data. Returns an Element instance. + .. versionadded:: 3.2 .. function:: iselement(element) - Checks if an object appears to be a valid element object. *element* is an - element instance. Returns a true value if this is an element object. + Checks if an object appears to be a valid element object. *element* is an + element instance. Returns a true value if this is an element object. -.. function:: iterparse(source, events=None) +.. function:: iterparse(source, events=None, parser=None) Parses an XML section into an element tree incrementally, and reports what's going on to the user. *source* is a filename or :term:`file object` containing XML data. *events* is a list of events to report back. If omitted, only "end" - events are reported. Returns an :term:`iterator` providing ``(event, elem)`` - pairs. + events are reported. *parser* is an optional parser instance. If not + given, the standard :class:`XMLParser` parser is used. Returns an + :term:`iterator` providing ``(event, elem)`` pairs. .. note:: @@ -107,196 +114,273 @@ Functions .. function:: parse(source, parser=None) - Parses an XML section into an element tree. *source* is a filename or file - object containing XML data. *parser* is an optional parser instance. If not - given, the standard XMLTreeBuilder parser is used. Returns an ElementTree - instance. + Parses an XML section into an element tree. *source* is a filename or file + object containing XML data. *parser* is an optional parser instance. If + not given, the standard :class:`XMLParser` parser is used. Returns an + :class:`ElementTree` instance. .. function:: ProcessingInstruction(target, text=None) - PI element factory. This factory function creates a special element that will - be serialized as an XML processing instruction. *target* is a string containing - the PI target. *text* is a string containing the PI contents, if given. Returns - an element instance, representing a processing instruction. + PI element factory. This factory function creates a special element that + will be serialized as an XML processing instruction. *target* is a string + containing the PI target. *text* is a string containing the PI contents, if + given. Returns an element instance, representing a processing instruction. + + +.. function:: register_namespace(prefix, uri) + + Registers a namespace prefix. The registry is global, and any existing + mapping for either the given prefix or the namespace URI will be removed. + *prefix* is a namespace prefix. *uri* is a namespace uri. Tags and + attributes in this namespace will be serialized with the given prefix, if at + all possible. + + .. versionadded:: 3.2 .. function:: SubElement(parent, tag, attrib={}, **extra) - Subelement factory. This function creates an element instance, and appends it - to an existing element. + Subelement factory. This function creates an element instance, and appends + it to an existing element. + + The element name, attribute names, and attribute values can be either + bytestrings or Unicode strings. *parent* is the parent element. *tag* is + the subelement name. *attrib* is an optional dictionary, containing element + attributes. *extra* contains additional attributes, given as keyword + arguments. Returns an element instance. + + +.. function:: tostring(element, encoding="us-ascii", method="xml") + + Generates a string representation of an XML element, including all + subelements. *element* is an :class:`Element` instance. *encoding* [1]_ is + the output encoding (default is US-ASCII). Use ``encoding="unicode"`` to + generate a Unicode string. *method* is either ``"xml"``, + ``"html"`` or ``"text"`` (default is ``"xml"``). Returns an (optionally) + encoded string containing the XML data. - The element name, attribute names, and attribute values can be an ASCII-only - :class:`bytes` object or a :class:`str` object. *parent* is the parent - element. *tag* is the subelement name. *attrib* is an optional dictionary, - containing element attributes. *extra* contains additional attributes, given - as keyword arguments. Returns an element instance. +.. function:: tostringlist(element, encoding="us-ascii", method="xml") -.. function:: tostring(element, encoding=None) + Generates a string representation of an XML element, including all + subelements. *element* is an :class:`Element` instance. *encoding* [1]_ is + the output encoding (default is US-ASCII). Use ``encoding="unicode"`` to + generate a Unicode string. *method* is either ``"xml"``, + ``"html"`` or ``"text"`` (default is ``"xml"``). Returns a list of + (optionally) encoded strings containing the XML data. It does not guarantee + any specific sequence, except that ``"".join(tostringlist(element)) == + tostring(element)``. - Generates a string representation of an XML element, including all subelements. - *element* is an Element instance. *encoding* is the output encoding (default is - US-ASCII). Returns an encoded string containing the XML data. + .. versionadded:: 3.2 -.. function:: XML(text) +.. function:: XML(text, parser=None) Parses an XML section from a string constant. This function can be used to - embed "XML literals" in Python code. *text* is a string containing XML data. - Returns an Element instance. + embed "XML literals" in Python code. *text* is a string containing XML + data. *parser* is an optional parser instance. If not given, the standard + :class:`XMLParser` parser is used. Returns an :class:`Element` instance. -.. function:: XMLID(text) +.. function:: XMLID(text, parser=None) Parses an XML section from a string constant, and also returns a dictionary - which maps from element id:s to elements. *text* is a string containing XML - data. Returns a tuple containing an Element instance and a dictionary. + which maps from element id:s to elements. *text* is a string containing XML + data. *parser* is an optional parser instance. If not given, the standard + :class:`XMLParser` parser is used. Returns a tuple containing an + :class:`Element` instance and a dictionary. -.. _elementtree-element-interface: +.. _elementtree-element-objects: -The Element Interface ---------------------- +Element Objects +--------------- -Element objects returned by Element or SubElement have the following methods -and attributes. +.. class:: Element(tag, attrib={}, **extra) -.. attribute:: Element.tag + Element class. This class defines the Element interface, and provides a + reference implementation of this interface. - A string identifying what kind of data this element represents (the element - type, in other words). + The element name, attribute names, and attribute values can be either + bytestrings or Unicode strings. *tag* is the element name. *attrib* is + an optional dictionary, containing element attributes. *extra* contains + additional attributes, given as keyword arguments. -.. attribute:: Element.text + .. attribute:: tag - The *text* attribute can be used to hold additional data associated with the - element. As the name implies this attribute is usually a string but may be any - application-specific object. If the element is created from an XML file the - attribute will contain any text found between the element tags. + A string identifying what kind of data this element represents (the + element type, in other words). -.. attribute:: Element.tail + .. attribute:: text - The *tail* attribute can be used to hold additional data associated with the - element. This attribute is usually a string but may be any application-specific - object. If the element is created from an XML file the attribute will contain - any text found after the element's end tag and before the next tag. + The *text* attribute can be used to hold additional data associated with + the element. As the name implies this attribute is usually a string but + may be any application-specific object. If the element is created from + an XML file the attribute will contain any text found between the element + tags. -.. attribute:: Element.attrib + .. attribute:: tail - A dictionary containing the element's attributes. Note that while the *attrib* - value is always a real mutable Python dictionary, an ElementTree implementation - may choose to use another internal representation, and create the dictionary - only if someone asks for it. To take advantage of such implementations, use the - dictionary methods below whenever possible. + The *tail* attribute can be used to hold additional data associated with + the element. This attribute is usually a string but may be any + application-specific object. If the element is created from an XML file + the attribute will contain any text found after the element's end tag and + before the next tag. -The following dictionary-like methods work on the element attributes. + .. attribute:: attrib -.. method:: Element.clear() + A dictionary containing the element's attributes. Note that while the + *attrib* value is always a real mutable Python dictionary, an ElementTree + implementation may choose to use another internal representation, and + create the dictionary only if someone asks for it. To take advantage of + such implementations, use the dictionary methods below whenever possible. - Resets an element. This function removes all subelements, clears all - attributes, and sets the text and tail attributes to None. + The following dictionary-like methods work on the element attributes. -.. method:: Element.get(key, default=None) + .. method:: clear() - Gets the element attribute named *key*. + Resets an element. This function removes all subelements, clears all + attributes, and sets the text and tail attributes to None. - Returns the attribute value, or *default* if the attribute was not found. + .. method:: get(key, default=None) -.. method:: Element.items() + Gets the element attribute named *key*. - Returns the element attributes as a sequence of (name, value) pairs. The - attributes are returned in an arbitrary order. + Returns the attribute value, or *default* if the attribute was not found. -.. method:: Element.keys() + .. method:: items() - Returns the elements attribute names as a list. The names are returned in an - arbitrary order. + Returns the element attributes as a sequence of (name, value) pairs. The + attributes are returned in an arbitrary order. -.. method:: Element.set(key, value) + .. method:: keys() - Set the attribute *key* on the element to *value*. + Returns the elements attribute names as a list. The names are returned + in an arbitrary order. -The following methods work on the element's children (subelements). + .. method:: set(key, value) -.. method:: Element.append(subelement) + Set the attribute *key* on the element to *value*. - Adds the element *subelement* to the end of this elements internal list of - subelements. + The following methods work on the element's children (subelements). -.. method:: Element.find(match) + .. method:: append(subelement) - Finds the first subelement matching *match*. *match* may be a tag name or path. - Returns an element instance or ``None``. + Adds the element *subelement* to the end of this elements internal list + of subelements. -.. method:: Element.findall(match) + .. method:: extend(subelements) - Finds all subelements matching *match*. *match* may be a tag name or path. - Returns an iterable yielding all matching elements in document order. + Appends *subelements* from a sequence object with zero or more elements. + Raises :exc:`AssertionError` if a subelement is not a valid object. + .. versionadded:: 3.2 -.. method:: Element.findtext(condition, default=None) - Finds text for the first subelement matching *condition*. *condition* may be a - tag name or path. Returns the text content of the first matching element, or - *default* if no element was found. Note that if the matching element has no - text content an empty string is returned. + .. method:: find(match) + + Finds the first subelement matching *match*. *match* may be a tag name + or path. Returns an element instance or ``None``. + + + .. method:: findall(match) + + Finds all matching subelements, by tag name or path. Returns a list + containing all matching elements in document order. + + + .. method:: findtext(match, default=None) + + Finds text for the first subelement matching *match*. *match* may be + a tag name or path. Returns the text content of the first matching + element, or *default* if no element was found. Note that if the matching + element has no text content an empty string is returned. + + + .. method:: getchildren() + + .. deprecated:: 3.2 + Use ``list(elem)`` or iteration. + + + .. method:: getiterator(tag=None) + .. deprecated:: 3.2 + Use method :meth:`Element.iter` instead. -.. method:: Element.getchildren() - Returns all subelements. The elements are returned in document order. + .. method:: insert(index, element) + Inserts a subelement at the given position in this element. -.. method:: Element.getiterator(tag=None) - Creates a tree iterator with the current element as the root. The iterator - iterates over this element and all elements below it, in document (depth first) - order. If *tag* is not ``None`` or ``'*'``, only elements whose tag equals - *tag* are returned from the iterator. + .. method:: iter(tag=None) + Creates a tree :term:`iterator` with the current element as the root. + The iterator iterates over this element and all elements below it, in + document (depth first) order. If *tag* is not ``None`` or ``'*'``, only + elements whose tag equals *tag* are returned from the iterator. If the + tree structure is modified during iteration, the result is undefined. -.. method:: Element.insert(index, element) + .. versionadded:: 3.2 - Inserts a subelement at the given position in this element. + .. method:: iterfind(match) -.. method:: Element.makeelement(tag, attrib) + Finds all matching subelements, by tag name or path. Returns an iterable + yielding all matching elements in document order. - Creates a new element object of the same type as this element. Do not call this - method, use the SubElement factory function instead. + .. versionadded:: 3.2 -.. method:: Element.remove(subelement) + .. method:: itertext() - Removes *subelement* from the element. Unlike the findXYZ methods this method - compares elements based on the instance identity, not on tag value or contents. + Creates a text iterator. The iterator loops over this element and all + subelements, in document order, and returns all inner text. -Element objects also support the following sequence type methods for working -with subelements: :meth:`__delitem__`, :meth:`__getitem__`, :meth:`__setitem__`, -:meth:`__len__`. + .. versionadded:: 3.2 -Caution: Because Element objects do not define a :meth:`__bool__` method, -elements with no subelements will test as ``False``. :: - element = root.find('foo') + .. method:: makeelement(tag, attrib) - if not element: # careful! - print("element not found, or element has no subelements") + Creates a new element object of the same type as this element. Do not + call this method, use the :func:`SubElement` factory function instead. - if element is None: - print("element not found") + + .. method:: remove(subelement) + + Removes *subelement* from the element. Unlike the find\* methods this + method compares elements based on the instance identity, not on tag value + or contents. + + :class:`Element` objects also support the following sequence type methods + for working with subelements: :meth:`__delitem__`, :meth:`__getitem__`, + :meth:`__setitem__`, :meth:`__len__`. + + Caution: Elements with no subelements will test as ``False``. This behavior + will change in future versions. Use specific ``len(elem)`` or ``elem is + None`` test instead. :: + + element = root.find('foo') + + if not element: # careful! + print("element not found, or element has no subelements") + + if element is None: + print("element not found") .. _elementtree-elementtree-objects: @@ -307,50 +391,49 @@ ElementTree Objects .. class:: ElementTree(element=None, file=None) - ElementTree wrapper class. This class represents an entire element hierarchy, - and adds some extra support for serialization to and from standard XML. + ElementTree wrapper class. This class represents an entire element + hierarchy, and adds some extra support for serialization to and from + standard XML. - *element* is the root element. The tree is initialized with the contents of the - XML *file* if given. + *element* is the root element. The tree is initialized with the contents + of the XML *file* if given. .. method:: _setroot(element) Replaces the root element for this tree. This discards the current contents of the tree, and replaces it with the given element. Use with - care. *element* is an element instance. + care. *element* is an element instance. - .. method:: find(path) + .. method:: find(match) - Finds the first toplevel element with given tag. Same as - getroot().find(path). *path* is the element to look for. Returns the - first matching element, or ``None`` if no element was found. + Finds the first toplevel element matching *match*. *match* may be a tag + name or path. Same as getroot().find(match). Returns the first matching + element, or ``None`` if no element was found. - .. method:: findall(path) + .. method:: findall(match) - Finds all toplevel elements with the given tag. Same as - getroot().findall(path). *path* is the element to look for. Returns a - list or :term:`iterator` containing all matching elements, in document - order. + Finds all matching subelements, by tag name or path. Same as + getroot().findall(match). *match* may be a tag name or path. Returns a + list containing all matching elements, in document order. - .. method:: findtext(path, default=None) + .. method:: findtext(match, default=None) Finds the element text for the first toplevel element with given tag. - Same as getroot().findtext(path). *path* is the toplevel element to look - for. *default* is the value to return if the element was not - found. Returns the text content of the first matching element, or the - default value no element was found. Note that if the element has is - found, but has no text content, this method returns an empty string. + Same as getroot().findtext(match). *match* may be a tag name or path. + *default* is the value to return if the element was not found. Returns + the text content of the first matching element, or the default value no + element was found. Note that if the element is found, but has no text + content, this method returns an empty string. .. method:: getiterator(tag=None) - Creates and returns a tree iterator for the root element. The iterator - loops over all elements in this tree, in section order. *tag* is the tag - to look for (default is to return all elements) + .. deprecated:: 3.2 + Use method :meth:`ElementTree.iter` instead. .. method:: getroot() @@ -358,19 +441,40 @@ ElementTree Objects Returns the root element for this tree. + .. method:: iter(tag=None) + + Creates and returns a tree iterator for the root element. The iterator + loops over all elements in this tree, in section order. *tag* is the tag + to look for (default is to return all elements) + + + .. method:: iterfind(match) + + Finds all matching subelements, by tag name or path. Same as + getroot().iterfind(match). Returns an iterable yielding all matching + elements in document order. + + .. versionadded:: 3.2 + + .. method:: parse(source, parser=None) Loads an external XML section into this element tree. *source* is a file name or :term:`file object`. *parser* is an optional parser instance. - If not given, the standard XMLTreeBuilder parser is used. Returns the section + If not given, the standard XMLParser parser is used. Returns the section root element. - .. method:: write(file, encoding=None) + .. method:: write(file, encoding="us-ascii", xml_declaration=None, method="xml") Writes the element tree to a file, as XML. *file* is a file name, or a :term:`file object` opened for writing. *encoding* [1]_ is the output encoding - (default is US-ASCII). + (default is US-ASCII). Use ``encoding="unicode"`` to write a Unicode string. + *xml_declaration* controls if an XML declaration + should be added to the file. Use False for never, True for always, None + for only if not US-ASCII or UTF-8 or Unicode (default is None). *method* is + either ``"xml"``, ``"html"`` or ``"text"`` (default is ``"xml"``). + Returns an (optionally) encoded string. This is the XML file that is going to be manipulated:: @@ -389,13 +493,13 @@ Example of changing the attribute "target" of every link in first paragraph:: >>> from xml.etree.ElementTree import ElementTree >>> tree = ElementTree() >>> tree.parse("index.xhtml") - <Element html at b7d3f1ec> + <Element 'html' at 0xb77e6fac> >>> p = tree.find("body/p") # Finds first occurrence of tag p in body >>> p - <Element p at 8416e0c> - >>> links = p.getiterator("a") # Returns list of all links + <Element 'p' at 0xb77ec26c> + >>> links = list(p.iter("a")) # Returns list of all links >>> links - [<Element a at b7d4f9ec>, <Element a at b7d4fb0c>] + [<Element 'a' at 0xb77ec2ac>, <Element 'a' at 0xb77ec1cc>] >>> for i in links: # Iterates through all found links ... i.attrib["target"] = "blank" >>> tree.write("output.xhtml") @@ -408,12 +512,12 @@ QName Objects .. class:: QName(text_or_uri, tag=None) - QName wrapper. This can be used to wrap a QName attribute value, in order to - get proper namespace handling on output. *text_or_uri* is a string containing - the QName value, in the form {uri}local, or, if the tag argument is given, the - URI part of a QName. If *tag* is given, the first argument is interpreted as an - URI, and this argument is interpreted as a local name. :class:`QName` instances - are opaque. + QName wrapper. This can be used to wrap a QName attribute value, in order + to get proper namespace handling on output. *text_or_uri* is a string + containing the QName value, in the form {uri}local, or, if the tag argument + is given, the URI part of a QName. If *tag* is given, the first argument is + interpreted as an URI, and this argument is interpreted as a local name. + :class:`QName` instances are opaque. .. _elementtree-treebuilder-objects: @@ -424,74 +528,89 @@ TreeBuilder Objects .. class:: TreeBuilder(element_factory=None) - Generic element structure builder. This builder converts a sequence of start, - data, and end method calls to a well-formed element structure. You can use this - class to build an element structure using a custom XML parser, or a parser for - some other XML-like format. The *element_factory* is called to create new - Element instances when given. + Generic element structure builder. This builder converts a sequence of + start, data, and end method calls to a well-formed element structure. You + can use this class to build an element structure using a custom XML parser, + or a parser for some other XML-like format. The *element_factory* is called + to create new :class:`Element` instances when given. .. method:: close() - Flushes the parser buffers, and returns the toplevel document - element. Returns an Element instance. + Flushes the builder buffers, and returns the toplevel document + element. Returns an :class:`Element` instance. .. method:: data(data) - Adds text to the current element. *data* is a string. This should be - either an ASCII-only :class:`bytes` object or a :class:`str` object. + Adds text to the current element. *data* is a string. This should be + either a bytestring, or a Unicode string. .. method:: end(tag) - Closes the current element. *tag* is the element name. Returns the closed - element. + Closes the current element. *tag* is the element name. Returns the + closed element. .. method:: start(tag, attrs) - Opens a new element. *tag* is the element name. *attrs* is a dictionary - containing element attributes. Returns the opened element. + Opens a new element. *tag* is the element name. *attrs* is a dictionary + containing element attributes. Returns the opened element. + + In addition, a custom :class:`TreeBuilder` object can provide the + following method: -.. _elementtree-xmltreebuilder-objects: + .. method:: doctype(name, pubid, system) + + Handles a doctype declaration. *name* is the doctype name. *pubid* is + the public identifier. *system* is the system identifier. This method + does not exist on the default :class:`TreeBuilder` class. + + .. versionadded:: 3.2 -XMLTreeBuilder Objects ----------------------- +.. _elementtree-xmlparser-objects: -.. class:: XMLTreeBuilder(html=0, target=None) +XMLParser Objects +----------------- - Element structure builder for XML source data, based on the expat parser. *html* - are predefined HTML entities. This flag is not supported by the current - implementation. *target* is the target object. If omitted, the builder uses an - instance of the standard TreeBuilder class. + +.. class:: XMLParser(html=0, target=None, encoding=None) + + :class:`Element` structure builder for XML source data, based on the expat + parser. *html* are predefined HTML entities. This flag is not supported by + the current implementation. *target* is the target object. If omitted, the + builder uses an instance of the standard TreeBuilder class. *encoding* [1]_ + is optional. If given, the value overrides the encoding specified in the + XML file. .. method:: close() - Finishes feeding data to the parser. Returns an element structure. + Finishes feeding data to the parser. Returns an element structure. .. method:: doctype(name, pubid, system) - Handles a doctype declaration. *name* is the doctype name. *pubid* is the - public identifier. *system* is the system identifier. + .. deprecated:: 3.2 + Define the :meth:`TreeBuilder.doctype` method on a custom TreeBuilder + target. .. method:: feed(data) - Feeds data to the parser. *data* is encoded data. + Feeds data to the parser. *data* is encoded data. -:meth:`XMLTreeBuilder.feed` calls *target*\'s :meth:`start` method +:meth:`XMLParser.feed` calls *target*\'s :meth:`start` method for each opening tag, its :meth:`end` method for each closing tag, -and data is processed by method :meth:`data`. :meth:`XMLTreeBuilder.close` +and data is processed by method :meth:`data`. :meth:`XMLParser.close` calls *target*\'s method :meth:`close`. -:class:`XMLTreeBuilder` can be used not only for building a tree structure. +:class:`XMLParser` can be used not only for building a tree structure. This is an example of counting the maximum depth of an XML file:: - >>> from xml.etree.ElementTree import XMLTreeBuilder + >>> from xml.etree.ElementTree import XMLParser >>> class MaxDepth: # The target object of the parser ... maxDepth = 0 ... depth = 0 @@ -507,7 +626,7 @@ This is an example of counting the maximum depth of an XML file:: ... return self.maxDepth ... >>> target = MaxDepth() - >>> parser = XMLTreeBuilder(target=target) + >>> parser = XMLParser(target=target) >>> exampleXml = """ ... <a> ... <b> @@ -527,7 +646,6 @@ This is an example of counting the maximum depth of an XML file:: .. rubric:: Footnotes .. [#] The encoding string included in XML output should conform to the - appropriate standards. For example, "UTF-8" is valid, but "UTF8" is - not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl + appropriate standards. For example, "UTF-8" is valid, but "UTF8" is + not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl and http://www.iana.org/assignments/character-sets. - diff --git a/Doc/library/xml.etree.rst b/Doc/library/xml.etree.rst deleted file mode 100644 index a944765417..0000000000 --- a/Doc/library/xml.etree.rst +++ /dev/null @@ -1,23 +0,0 @@ -:mod:`xml.etree` --- The ElementTree API for XML -================================================ - -.. module:: xml.etree - :synopsis: Package containing common ElementTree modules. -.. moduleauthor:: Fredrik Lundh <fredrik@pythonware.com> - - -The ElementTree package is a simple, efficient, and quite popular library for -XML manipulation in Python. The :mod:`xml.etree` package contains the most -common components from the ElementTree API library. In the current release, -this package contains the :mod:`ElementTree`, :mod:`ElementPath`, and -:mod:`ElementInclude` modules from the full ElementTree distribution. - -.. XXX To be continued! - - -.. seealso:: - - `ElementTree Overview <http://effbot.org/tag/elementtree>`_ - The home page for :mod:`ElementTree`. This includes links to additional - documentation, alternative implementations, and other add-ons. - diff --git a/Doc/library/xml.sax.utils.rst b/Doc/library/xml.sax.utils.rst index 95099f6709..ff36fd89e8 100644 --- a/Doc/library/xml.sax.utils.rst +++ b/Doc/library/xml.sax.utils.rst @@ -50,13 +50,19 @@ or as base classes. using the reference concrete syntax. -.. class:: XMLGenerator(out=None, encoding='iso-8859-1') +.. class:: XMLGenerator(out=None, encoding='iso-8859-1', short_empty_elements=False) This class implements the :class:`ContentHandler` interface by writing SAX events back into an XML document. In other words, using an :class:`XMLGenerator` as the content handler will reproduce the original document being parsed. *out* should be a file-like object which will default to *sys.stdout*. *encoding* is the encoding of the output stream which defaults to ``'iso-8859-1'``. + *short_empty_elements* controls the formatting of elements that contain no + content: if *False* (the default) they are emitted as a pair of start/end + tags, if set to *True* they are emitted as a single self-closed tag. + + .. versionadded:: 3.2 + short_empty_elements .. class:: XMLFilterBase(base) diff --git a/Doc/library/xmlrpc.client.rst b/Doc/library/xmlrpc.client.rst index d25cbaf5a5..e72770ae93 100644 --- a/Doc/library/xmlrpc.client.rst +++ b/Doc/library/xmlrpc.client.rst @@ -10,6 +10,10 @@ .. XXX Not everything is documented yet. It might be good to describe Marshaller, Unmarshaller, getparser, dumps, loads, and Transport. +**Source code:** :source:`Lib/xmlrpc/client.py` + +-------------- + XML-RPC is a Remote Procedure Call method that uses XML passed via HTTP as a transport. With it, a client can call methods with parameters on a remote server (the server is named by a URI) and get back structured data. This module @@ -132,7 +136,7 @@ returning a value, which may be either returned data in a conformant type or a :class:`Fault` or :class:`ProtocolError` object indicating an error. Servers that support the XML introspection API support some common methods -grouped under the reserved :attr:`system` member: +grouped under the reserved :attr:`system` attribute: .. method:: ServerProxy.system.listMethods() @@ -306,7 +310,7 @@ Fault Objects ------------- A :class:`Fault` object encapsulates the content of an XML-RPC fault tag. Fault -objects have the following members: +objects have the following attributes: .. attribute:: Fault.faultCode @@ -355,7 +359,7 @@ ProtocolError Objects A :class:`ProtocolError` object describes a protocol error in the underlying transport layer (such as a 404 'not found' error if the server named by the URI -does not exist). It has the following members: +does not exist). It has the following attributes: .. attribute:: ProtocolError.url @@ -398,8 +402,8 @@ by providing an invalid URI:: MultiCall Objects ----------------- -In http://www.xmlrpc.com/discuss/msgReader%241208, an approach is presented to -encapsulate multiple calls to a remote server into a single request. +The :class:`MultiCall` object provides a way to encapsulate multiple calls to a +remote server into a single request [#]_. .. class:: MultiCall(server) @@ -530,3 +534,10 @@ Example of Client and Server Usage See :ref:`simplexmlrpcserver-example`. +.. rubric:: Footnotes + +.. [#] This approach has been first presented in `a discussion on xmlrpc.com + <http://web.archive.org/web/20060624230303/http://www.xmlrpc.com/discuss/msgReader$1208?mode=topic>`_. +.. the link now points to webarchive since the one at +.. http://www.xmlrpc.com/discuss/msgReader%241208 is broken (and webadmin +.. doesn't reply) diff --git a/Doc/library/xmlrpc.server.rst b/Doc/library/xmlrpc.server.rst index 3cb2c3aaa1..67feba6d1f 100644 --- a/Doc/library/xmlrpc.server.rst +++ b/Doc/library/xmlrpc.server.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Brian Quinlan <brianq@activestate.com> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/xmlrpc/server.py` + +-------------- The :mod:`xmlrpc.server` module provides a basic server framework for XML-RPC servers written in Python. Servers can either be free standing, using diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index 827b6b1e98..bcec1344a2 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -6,6 +6,10 @@ .. moduleauthor:: James C. Ahlstrom <jim@interet.com> .. sectionauthor:: James C. Ahlstrom <jim@interet.com> +**Source code:** :source:`Lib/zipfile.py` + +-------------- + The ZIP file format is a common archive and compression standard. This module provides tools to create, read, write, append, and list a ZIP file. Any advanced use of this module will require an understanding of the format, as @@ -19,14 +23,20 @@ decryption of encrypted files in ZIP archives, but it currently cannot create an encrypted file. Decryption is extremely slow as it is implemented in native Python rather than C. -For other archive formats, see the :mod:`bz2`, :mod:`gzip`, and -:mod:`tarfile` modules. - The module defines the following items: +.. exception:: BadZipFile + + The error raised for bad ZIP files. + + .. versionadded:: 3.2 + + .. exception:: BadZipfile - The error raised for bad ZIP files (old name: ``zipfile.error``). + Alias of :exc:`BadZipFile`, for compatibility with older Python versions. + + .. deprecated:: 3.2 .. exception:: LargeZipFile @@ -43,6 +53,7 @@ The module defines the following items: .. class:: PyZipFile + :noindex: Class for creating ZIP archives containing Python libraries. @@ -101,30 +112,37 @@ ZipFile Objects Open a ZIP file, where *file* can be either a path to a file (a string) or a file-like object. The *mode* parameter should be ``'r'`` to read an existing file, ``'w'`` to truncate and write a new file, or ``'a'`` to append to an - existing file. If *mode* is ``'a'`` and *file* refers to an existing ZIP file, - then additional files are added to it. If *file* does not refer to a ZIP file, - then a new ZIP archive is appended to the file. This is meant for adding a ZIP - archive to another file, such as :file:`python.exe`. Using :: - - cat myzip.zip >> python.exe - - also works, and at least :program:`WinZip` can read such files. If *mode* is - ``a`` and the file does not exist at all, it is created. *compression* is the - ZIP compression method to use when writing the archive, and should be - :const:`ZIP_STORED` or :const:`ZIP_DEFLATED`; unrecognized values will cause - :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED` is specified but the - :mod:`zlib` module is not available, :exc:`RuntimeError` is also raised. The - default is :const:`ZIP_STORED`. If *allowZip64* is ``True`` zipfile will create - ZIP files that use the ZIP64 extensions when the zipfile is larger than 2 GB. If - it is false (the default) :mod:`zipfile` will raise an exception when the ZIP - file would require ZIP64 extensions. ZIP64 extensions are disabled by default - because the default :program:`zip` and :program:`unzip` commands on Unix (the - InfoZIP utilities) don't support these extensions. + existing file. If *mode* is ``'a'`` and *file* refers to an existing ZIP + file, then additional files are added to it. If *file* does not refer to a + ZIP file, then a new ZIP archive is appended to the file. This is meant for + adding a ZIP archive to another file (such as :file:`python.exe`). If + *mode* is ``a`` and the file does not exist at all, it is created. + *compression* is the ZIP compression method to use when writing the archive, + and should be :const:`ZIP_STORED` or :const:`ZIP_DEFLATED`; unrecognized + values will cause :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED` + is specified but the :mod:`zlib` module is not available, :exc:`RuntimeError` + is also raised. The default is :const:`ZIP_STORED`. If *allowZip64* is + ``True`` zipfile will create ZIP files that use the ZIP64 extensions when + the zipfile is larger than 2 GB. If it is false (the default) :mod:`zipfile` + will raise an exception when the ZIP file would require ZIP64 extensions. + ZIP64 extensions are disabled by default because the default :program:`zip` + and :program:`unzip` commands on Unix (the InfoZIP utilities) don't support + these extensions. If the file is created with mode ``'a'`` or ``'w'`` and then :meth:`close`\ d without adding any files to the archive, the appropriate ZIP structures for an empty archive will be written to the file. + ZipFile is also a context manager and therefore supports the + :keyword:`with` statement. In the example, *myzip* is closed after the + :keyword:`with` statement's suite is finished---even if an exception occurs:: + + with ZipFile('spam.zip', 'w') as myzip: + myzip.write('eggs.txt') + + .. versionadded:: 3.2 + Added the ability to use :class:`ZipFile` as a context manager. + .. method:: ZipFile.close() @@ -263,7 +281,7 @@ ZipFile Objects byte, the name of the file in the archive will be truncated at the null byte. -.. method:: ZipFile.writestr(zinfo_or_arcname, bytes) +.. method:: ZipFile.writestr(zinfo_or_arcname, bytes[, compress_type]) Write the string *bytes* to the archive; *zinfo_or_arcname* is either the file name it will be given in the archive, or a :class:`ZipInfo` instance. If it's @@ -273,6 +291,10 @@ ZipFile Objects created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling :meth:`writestr` on a closed ZipFile will raise a :exc:`RuntimeError`. + If given, *compress_type* overrides the value given for the *compression* + parameter to the constructor for the new entry, or in the *zinfo_or_arcname* + (if that is a :class:`ZipInfo` instance). + .. note:: When passing a :class:`ZipInfo` instance as the *zinfo_or_arcname* parameter, @@ -280,6 +302,9 @@ ZipFile Objects member of the given :class:`ZipInfo` instance. By default, the :class:`ZipInfo` constructor sets this member to :const:`ZIP_STORED`. + .. versionchanged:: 3.2 + The *compression_type* argument. + The following data attributes are also available: @@ -296,37 +321,53 @@ The following data attributes are also available: string no longer than 65535 bytes. Comments longer than this will be truncated in the written archive when :meth:`ZipFile.close` is called. + .. _pyzipfile-objects: PyZipFile Objects ----------------- The :class:`PyZipFile` constructor takes the same parameters as the -:class:`ZipFile` constructor. Instances have one method in addition to those of -:class:`ZipFile` objects. +:class:`ZipFile` constructor, and one additional parameter, *optimize*. + +.. class:: PyZipFile(file, mode='r', compression=ZIP_STORED, allowZip64=False, \ + optimize=-1) + + .. versionadded:: 3.2 + The *optimize* parameter. + + Instances have one method in addition to those of :class:`ZipFile` objects: + .. method:: PyZipFile.writepy(pathname, basename='') -.. method:: PyZipFile.writepy(pathname, basename='') + Search for files :file:`\*.py` and add the corresponding file to the + archive. - Search for files :file:`\*.py` and add the corresponding file to the archive. - The corresponding file is a :file:`\*.pyo` file if available, else a - :file:`\*.pyc` file, compiling if necessary. If the pathname is a file, the - filename must end with :file:`.py`, and just the (corresponding - :file:`\*.py[co]`) file is added at the top level (no path information). If the - pathname is a file that does not end with :file:`.py`, a :exc:`RuntimeError` - will be raised. If it is a directory, and the directory is not a package - directory, then all the files :file:`\*.py[co]` are added at the top level. If - the directory is a package directory, then all :file:`\*.py[co]` are added under - the package name as a file path, and if any subdirectories are package - directories, all of these are added recursively. *basename* is intended for - internal use only. The :meth:`writepy` method makes archives with file names - like this:: + If the *optimize* parameter to :class:`PyZipFile` was not given or ``-1``, + the corresponding file is a :file:`\*.pyo` file if available, else a + :file:`\*.pyc` file, compiling if necessary. - string.pyc # Top level name - test/__init__.pyc # Package directory - test/testall.pyc # Module test.testall - test/bogus/__init__.pyc # Subpackage directory - test/bogus/myfile.pyc # Submodule test.bogus.myfile + If the *optimize* parameter to :class:`PyZipFile` was ``0``, ``1`` or + ``2``, only files with that optimization level (see :func:`compile`) are + added to the archive, compiling if necessary. + + If the pathname is a file, the filename must end with :file:`.py`, and + just the (corresponding :file:`\*.py[co]`) file is added at the top level + (no path information). If the pathname is a file that does not end with + :file:`.py`, a :exc:`RuntimeError` will be raised. If it is a directory, + and the directory is not a package directory, then all the files + :file:`\*.py[co]` are added at the top level. If the directory is a + package directory, then all :file:`\*.py[co]` are added under the package + name as a file path, and if any subdirectories are package directories, + all of these are added recursively. *basename* is intended for internal + use only. The :meth:`writepy` method makes archives with file names like + this:: + + string.pyc # Top level name + test/__init__.pyc # Package directory + test/testall.pyc # Module test.testall + test/bogus/__init__.pyc # Subpackage directory + test/bogus/myfile.pyc # Submodule test.bogus.myfile .. _zipinfo-objects: @@ -354,7 +395,7 @@ Instances have the following attributes: +-------+--------------------------+ | Index | Value | +=======+==========================+ - | ``0`` | Year | + | ``0`` | Year (>= 1980) | +-------+--------------------------+ | ``1`` | Month (one-based) | +-------+--------------------------+ @@ -367,6 +408,10 @@ Instances have the following attributes: | ``5`` | Seconds (zero-based) | +-------+--------------------------+ + .. note:: + + The ZIP file format does not support timestamps before 1980. + .. attribute:: ZipInfo.compress_type diff --git a/Doc/library/zipimport.rst b/Doc/library/zipimport.rst index 57ac1e4061..4f170924e9 100644 --- a/Doc/library/zipimport.rst +++ b/Doc/library/zipimport.rst @@ -9,11 +9,11 @@ This module adds the ability to import Python modules (:file:`\*.py`, :file:`\*.py[co]`) and packages from ZIP-format archives. It is usually not needed to use the :mod:`zipimport` module explicitly; it is automatically used -by the built-in :keyword:`import` mechanism for ``sys.path`` items that are paths +by the built-in :keyword:`import` mechanism for :data:`sys.path` items that are paths to ZIP archives. -Typically, ``sys.path`` is a list of directory names as strings. This module -also allows an item of ``sys.path`` to be a string naming a ZIP file archive. +Typically, :data:`sys.path` is a list of directory names as strings. This module +also allows an item of :data:`sys.path` to be a string naming a ZIP file archive. The ZIP archive can contain a subdirectory structure to support package imports, and a path within the archive can be specified to only import from a subdirectory. For example, the path :file:`/tmp/example.zip/lib/` would only @@ -26,18 +26,20 @@ Any files may be present in the ZIP archive, but only files :file:`.py` and corresponding :file:`.pyc` or :file:`.pyo` file, meaning that if a ZIP archive doesn't contain :file:`.pyc` files, importing may be rather slow. +ZIP archives with an archive comment are currently not supported. + .. seealso:: `PKZIP Application Note <http://www.pkware.com/documents/casestudies/APPNOTE.TXT>`_ Documentation on the ZIP file format by Phil Katz, the creator of the format and algorithms used. - :pep:`0273` - Import Modules from Zip Archives + :pep:`273` - Import Modules from Zip Archives Written by James C. Ahlstrom, who also provided an implementation. Python 2.3 follows the specification in PEP 273, but uses an implementation written by Just van Rossum that uses the import hooks described in PEP 302. - :pep:`0302` - New Import Hooks + :pep:`302` - New Import Hooks The PEP to add the import hooks that help this module work. diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst index 862cf9132a..897d919e31 100644 --- a/Doc/library/zlib.rst +++ b/Doc/library/zlib.rst @@ -18,9 +18,7 @@ order. This documentation doesn't attempt to cover all of the permutations; consult the zlib manual at http://www.zlib.net/manual.html for authoritative information. -For reading and writing ``.gz`` files see the :mod:`gzip` module. For -other archive formats, see the :mod:`bz2`, :mod:`zipfile`, and -:mod:`tarfile` modules. +For reading and writing ``.gz`` files see the :mod:`gzip` module. The available exception and functions in this module are: @@ -96,20 +94,24 @@ The available exception and functions in this module are: Decompresses the bytes in *data*, returning a bytes object containing the uncompressed data. The *wbits* parameter controls the size of the window - buffer. If *bufsize* is given, it is used as the initial size of the output + buffer, and is discussed further below. + If *bufsize* is given, it is used as the initial size of the output buffer. Raises the :exc:`error` exception if any error occurs. The absolute value of *wbits* is the base two logarithm of the size of the history buffer (the "window size") used when compressing data. Its absolute value should be between 8 and 15 for the most recent versions of the zlib library, larger values resulting in better compression at the expense of greater - memory usage. The default value is 15. When *wbits* is negative, the standard + memory usage. When decompressing a stream, *wbits* must not be smaller + than the size originally used to compress the stream; using a too-small + value will result in an exception. The default value is therefore the + highest value, 15. When *wbits* is negative, the standard :program:`gzip` header is suppressed. *bufsize* is the initial size of the buffer used to hold decompressed data. If more space is required, the buffer size will be increased as needed, so you don't have to get this value exactly right; tuning it will only save a few calls - to :cfunc:`malloc`. The default size is 16384. + to :c:func:`malloc`. The default size is 16384. .. function:: decompressobj([wbits]) diff --git a/Doc/license.rst b/Doc/license.rst index f45f0ae53c..41ae1fe51b 100644 --- a/Doc/license.rst +++ b/Doc/license.rst @@ -106,7 +106,18 @@ been GPL-compatible; the table below summarizes the various releases. +----------------+--------------+------------+------------+-----------------+ | 3.1.1 | 3.1 | 2009 | PSF | yes | +----------------+--------------+------------+------------+-----------------+ - +| 3.1.2 | 3.1.1 | 2010 | PSF | yes | ++----------------+--------------+------------+------------+-----------------+ +| 3.1.3 | 3.1.2 | 2010 | PSF | yes | ++----------------+--------------+------------+------------+-----------------+ +| 3.1.4 | 3.1.3 | 2011 | PSF | yes | ++----------------+--------------+------------+------------+-----------------+ +| 3.2 | 3.1 | 2011 | PSF | yes | ++----------------+--------------+------------+------------+-----------------+ +| 3.2.1 | 3.2 | 2011 | PSF | yes | ++----------------+--------------+------------+------------+-----------------+ +| 3.2.2 | 3.2.1 | 2011 | PSF | yes | ++----------------+--------------+------------+------------+-----------------+ .. note:: @@ -135,7 +146,7 @@ Terms and conditions for accessing or otherwise using Python analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python |release| alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of - copyright, i.e., "Copyright © 2001-2010 Python Software Foundation; All Rights + copyright, i.e., "Copyright © 2001-2012 Python Software Foundation; All Rights Reserved" are retained in Python |release| alone or in any derivative version prepared by Licensee. @@ -812,7 +823,7 @@ expat ----- The :mod:`pyexpat` extension is built using an included copy of the expat -sources unless the build is configured :option:`--with-system-expat`:: +sources unless the build is configured ``--with-system-expat``:: Copyright (c) 1998, 1999, 2000 Thai Open Source Software Center Ltd and Clark Cooper @@ -841,7 +852,7 @@ libffi ------ The :mod:`_ctypes` extension is built using an included copy of the libffi -sources unless the build is configured :option:`--with-system-libffi`:: +sources unless the build is configured ``--with-system-libffi``:: Copyright (c) 1996-2008 Red Hat, Inc and others. @@ -870,10 +881,10 @@ zlib ---- The :mod:`zlib` extension is built using an included copy of the zlib -sources unless the zlib version found on the system is too old to be +sources if the zlib version found on the system is too old to be used for the build:: - Copyright (C) 1995-2010 Jean-loup Gailly and Mark Adler + Copyright (C) 1995-2011 Jean-loup Gailly and Mark Adler This software is provided 'as-is', without any express or implied warranty. In no event will the authors be held liable for any damages diff --git a/Doc/make.bat b/Doc/make.bat index 6552aab65e..4ea2d519b2 100644 --- a/Doc/make.bat +++ b/Doc/make.bat @@ -34,7 +34,7 @@ echo. goto end :checkout -svn co %SVNROOT%/external/Sphinx-0.6.5/sphinx tools/sphinx +svn co %SVNROOT%/external/Sphinx-1.0.7/sphinx tools/sphinx svn co %SVNROOT%/external/docutils-0.6/docutils tools/docutils svn co %SVNROOT%/external/Jinja-2.3.1/jinja2 tools/jinja2 svn co %SVNROOT%/external/Pygments-1.3.1/pygments tools/pygments diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 31ae0ed45e..aea08e0805 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -310,8 +310,9 @@ not handled, the exception is temporarily saved. The :keyword:`finally` clause is executed. If there is a saved exception, it is re-raised at the end of the :keyword:`finally` clause. If the :keyword:`finally` clause raises another exception or executes a :keyword:`return` or :keyword:`break` statement, the -saved exception is lost. The exception information is not available to the -program during execution of the :keyword:`finally` clause. +saved exception is set as the context of the new exception. The exception +information is not available to the program during execution of the +:keyword:`finally` clause. .. index:: statement: return @@ -352,6 +353,8 @@ The execution of the :keyword:`with` statement with one "item" proceeds as follo #. The context expression (the expression given in the :token:`with_item`) is evaluated to obtain a context manager. +#. The context manager's :meth:`__exit__` is loaded for later use. + #. The context manager's :meth:`__enter__` method is invoked. #. If a target was included in the :keyword:`with` statement, the return value @@ -361,9 +364,9 @@ The execution of the :keyword:`with` statement with one "item" proceeds as follo The :keyword:`with` statement guarantees that if the :meth:`__enter__` method returns without an error, then :meth:`__exit__` will always be - called. Thus, if an error occurs during the assignment to the target - list, it will be treated the same as an error occurring within the suite - would be. See step 5 below. + called. Thus, if an error occurs during the assignment to the target list, + it will be treated the same as an error occurring within the suite would + be. See step 6 below. #. The suite is executed. @@ -425,7 +428,7 @@ A function definition defines a user-defined function object (see section .. productionlist:: funcdef: [`decorators`] "def" `funcname` "(" [`parameter_list`] ")" ["->" `expression`] ":" `suite` decorators: `decorator`+ - decorator: "@" `dotted_name` ["(" [`argument_list` [","]] ")"] NEWLINE + decorator: "@" `dotted_name` ["(" [`parameter_list` [","]] ")"] NEWLINE dotted_name: `identifier` ("." `identifier`)* parameter_list: (`defparameter` ",")* : ( "*" [`parameter`] ("," `defparameter`)* @@ -476,7 +479,7 @@ value --- this is a syntactic restriction that is not expressed by the grammar. **Default parameter values are evaluated when the function definition is executed.** This means that the expression is evaluated once, when the function -is defined, and that that same "pre-computed" value is used for each call. This +is defined, and that the same "pre-computed" value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified. @@ -552,7 +555,7 @@ A class definition defines a class object (see section :ref:`types`): .. productionlist:: classdef: [`decorators`] "class" `classname` [`inheritance`] ":" `suite` - inheritance: "(" [`argument_list` [","] | `comprehension`] ")" + inheritance: "(" [`parameter_list`] ")" classname: `identifier` A class definition is an executable statement. The inheritance list usually @@ -613,8 +616,9 @@ can be used to create instance variables with different implementation details. .. rubric:: Footnotes -.. [#] The exception is propagated to the invocation stack only if there is no - :keyword:`finally` clause that negates the exception. +.. [#] The exception is propagated to the invocation stack unless + there is a :keyword:`finally` clause which happens to raise another + exception. That new exception causes the old one to be lost. .. [#] Currently, control "flows off the end" except in the case of an exception or the execution of a :keyword:`return`, :keyword:`continue`, or diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 7ded851ab6..7dcd4596ac 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1157,6 +1157,14 @@ Basic customization .. XXX what about subclasses of string? +.. method:: object.__bytes__(self) + + .. index:: builtin: bytes + + Called by :func:`bytes` to compute a byte-string representation of an + object. This should return a ``bytes`` object. + + .. method:: object.__format__(self, format_spec) .. index:: @@ -1218,8 +1226,7 @@ Basic customization Arguments to rich comparison methods are never coerced. To automatically generate ordering operations from a single root operation, - see the `Total Ordering recipe in the ASPN cookbook - <http://code.activestate.com/recipes/576529/>`_\. + see :func:`functools.total_ordering`. .. method:: object.__hash__(self) @@ -1597,7 +1604,7 @@ attributes of *A* as they are defined within the body of the class statement. Once those definitions are executed, the ordered dictionary is fully populated and the metaclass's :meth:`__new__` method gets invoked. That method builds the new type and it saves the ordered dictionary keys in an attribute -called *members*. +called ``members``. Customizing instance and subclass checks diff --git a/Doc/reference/executionmodel.rst b/Doc/reference/executionmodel.rst index d11e74196a..82e37a21fa 100644 --- a/Doc/reference/executionmodel.rst +++ b/Doc/reference/executionmodel.rst @@ -94,9 +94,7 @@ module, except those beginning with an underscore. This form may only be used at the module level. A target occurring in a :keyword:`del` statement is also considered bound for -this purpose (though the actual semantics are to unbind the name). It is -illegal to unbind a name that is referenced by an enclosing scope; the compiler -will report a :exc:`SyntaxError`. +this purpose (though the actual semantics are to unbind the name). Each assignment or import statement occurs within a block defined by a class or function definition or at the module level (the top-level code block). diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 944c39f4aa..5b684684af 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -116,11 +116,11 @@ literals. See section :ref:`literals` for details. triple: immutable; data; type pair: immutable; object -With the exception of bytes literals, these all correspond to immutable data -types, and hence the object's identity is less important than its value. -Multiple evaluations of literals with the same value (either the same occurrence -in the program text or a different occurrence) may obtain the same object or a -different object with the same value. +All literals correspond to immutable data types, and hence the object's identity +is less important than its value. Multiple evaluations of literals with the +same value (either the same occurrence in the program text or a different +occurrence) may obtain the same object or a different object with the same +value. .. _parenthesized: @@ -651,7 +651,7 @@ the call. An implementation may provide built-in functions whose positional parameters do not have names, even if they are 'named' for the purpose of documentation, and which therefore cannot be supplied by keyword. In CPython, this is the - case for functions implemented in C that use :cfunc:`PyArg_ParseTuple` to + case for functions implemented in C that use :c:func:`PyArg_ParseTuple` to parse their arguments. If there are more positional arguments than there are formal parameter slots, a @@ -667,12 +667,15 @@ dictionary containing the excess keyword arguments (using the keywords as keys and the argument values as corresponding values), or a (new) empty dictionary if there were no excess keyword arguments. +.. index:: + single: *; in function calls + If the syntax ``*expression`` appears in the function call, ``expression`` must -evaluate to a sequence. Elements from this sequence are treated as if they were -additional positional arguments; if there are positional arguments *x1*,..., -*xN*, and ``expression`` evaluates to a sequence *y1*, ..., *yM*, this is -equivalent to a call with M+N positional arguments *x1*, ..., *xN*, *y1*, ..., -*yM*. +evaluate to an iterable. Elements from this iterable are treated as if they +were additional positional arguments; if there are positional arguments +*x1*, ..., *xN*, and ``expression`` evaluates to a sequence *y1*, ..., *yM*, +this is equivalent to a call with M+N positional arguments *x1*, ..., *xN*, +*y1*, ..., *yM*. A consequence of this is that although the ``*expression`` syntax may appear *after* some keyword arguments, it is processed *before* the keyword arguments @@ -693,6 +696,9 @@ A consequence of this is that although the ``*expression`` syntax may appear It is unusual for both keyword arguments and the ``*expression`` syntax to be used in the same call, so in practice this confusion does not arise. +.. index:: + single: **; in function calls + If the syntax ``**expression`` appears in the function call, ``expression`` must evaluate to a mapping, the contents of which are treated as additional keyword arguments. In the case of a keyword appearing in both ``expression`` and as an @@ -921,6 +927,11 @@ the left or right by the number of bits given by the second argument. A right shift by *n* bits is defined as division by ``pow(2,n)``. A left shift by *n* bits is defined as multiplication with ``pow(2,n)``. +.. note:: + + In the current implementation, the right-hand operand is required + to be at most :attr:`sys.maxsize`. If the right-hand operand is larger than + :attr:`sys.maxsize` an :exc:`OverflowError` exception is raised. .. _bitwise: @@ -958,9 +969,9 @@ must be integers. .. _comparisons: .. _is: -.. _isnot: +.. _is not: .. _in: -.. _notin: +.. _not in: Comparisons =========== @@ -1156,11 +1167,9 @@ not bother to return a value of the same type as its argument, so e.g., ``not 'foo'`` yields ``False``, not ``''``.) -Conditional Expressions +Conditional expressions ======================= -.. versionadded:: 2.5 - .. index:: pair: conditional; expression pair: ternary; operator @@ -1309,6 +1318,7 @@ groups from right to left). | ``(expressions...)``, | Binding or tuple display, | | ``[expressions...]``, | list display, | | ``{key:datum...}``, | dictionary display, | +| ``{expressions...}`` | set display | +-----------------------------------------------+-------------------------------------+ diff --git a/Doc/reference/introduction.rst b/Doc/reference/introduction.rst index 23ab88e04c..0ac57945db 100644 --- a/Doc/reference/introduction.rst +++ b/Doc/reference/introduction.rst @@ -66,7 +66,7 @@ IronPython An alternate Python for .NET. Unlike Python.NET, this is a complete Python implementation that generates IL, and compiles Python code directly to .NET assemblies. It was created by Jim Hugunin, the original creator of Jython. For - more information, see `the IronPython website <http://www.ironpython.com/>`_. + more information, see `the IronPython website <http://www.ironpython.net/>`_. PyPy An implementation of Python written completely in Python. It supports several diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 548e0a9ef7..34ed92fc7a 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -372,7 +372,7 @@ The :keyword:`del` statement del_stmt: "del" `target_list` Deletion is recursively defined very similar to the way assignment is defined. -Rather that spelling it out in full details, here are some hints. +Rather than spelling it out in full details, here are some hints. Deletion of a target list recursively deletes each target, from left to right. @@ -385,11 +385,6 @@ namespace, depending on whether the name occurs in a :keyword:`global` statement in the same code block. If the name is unbound, a :exc:`NameError` exception will be raised. -.. index:: pair: free; variable - -It is illegal to delete a name from the local namespace if it occurs as a free -variable in a nested block. - .. index:: pair: attribute; deletion Deletion of attribute references, subscriptions and slicings is passed to the @@ -397,6 +392,11 @@ primary object involved; deletion of a slicing is in general equivalent to assignment of an empty slice of the right type (but even this is determined by the sliced object). +.. versionchanged:: 3.2 + + Previously it was illegal to delete a name from the local namespace if it + occurs as a free variable in a nested block. + .. _return: @@ -495,8 +495,8 @@ The :keyword:`raise` statement If no expressions are present, :keyword:`raise` re-raises the last exception that was active in the current scope. If no exception is active in the current -scope, a :exc:`TypeError` exception is raised indicating that this is an error -(if running under IDLE, a :exc:`queue.Empty` exception is raised instead). +scope, a :exc:`RuntimeError` exception is raised indicating that this is an +error. Otherwise, :keyword:`raise` evaluates the first expression as the exception object. It must be either a subclass or an instance of :class:`BaseException`. @@ -674,7 +674,9 @@ Once the name of the module is known (unless otherwise specified, the term "module" will refer to both packages and modules), searching for the module or package can begin. The first place checked is :data:`sys.modules`, the cache of all modules that have been imported -previously. If the module is found there then it is used in step (2) of import. +previously. If the module is found there then it is used in step (2) of import +unless ``None`` is found in :data:`sys.modules`, in which case +:exc:`ImportError` is raised. .. index:: single: sys.meta_path @@ -691,7 +693,7 @@ within a package (as denoted by the existence of a dot in the name), then a second argument to :meth:`find_module` is given as the value of the :attr:`__path__` attribute from the parent package (everything up to the last dot in the name of the module being imported). If a finder can find the module -it returns a :term:`loader` (discussed later) or returns :keyword:`None`. +it returns a :term:`loader` (discussed later) or returns ``None``. .. index:: single: sys.path_hooks @@ -718,11 +720,11 @@ finder cached then :data:`sys.path_hooks` is searched by calling each object in the list with a single argument of the path, returning a finder or raises :exc:`ImportError`. If a finder is returned then it is cached in :data:`sys.path_importer_cache` and then used for that path entry. If no finder -can be found but the path exists then a value of :keyword:`None` is +can be found but the path exists then a value of ``None`` is stored in :data:`sys.path_importer_cache` to signify that an implicit, file-based finder that handles modules stored as individual files should be used for that path. If the path does not exist then a finder which always -returns :keyword:`None` is placed in the cache for the path. +returns ``None`` is placed in the cache for the path. .. index:: single: loader @@ -786,7 +788,7 @@ first form of :keyword:`import`, an alternate local name can be supplied by specifying ":keyword:`as` localname". If a name is not found, :exc:`ImportError` is raised. If the list of identifiers is replaced by a star (``'*'``), all public names defined in the module are bound in the local -namespace of the :keyword:`import` statement.. +namespace of the :keyword:`import` statement. .. index:: single: __all__ (optional module attribute) @@ -818,7 +820,7 @@ leading dot means the current package where the module making the import exists. Two dots means up one package level. Three dots is up two levels, etc. So if you execute ``from . import mod`` from a module in the ``pkg`` package then you will end up importing ``pkg.mod``. If you execute ``from ..subpkg2 -imprt mod`` from within ``pkg.subpkg1`` you will import ``pkg.subpkg2.mod``. +import mod`` from within ``pkg.subpkg1`` you will import ``pkg.subpkg2.mod``. The specification for relative imports is contained within :pep:`328`. :func:`importlib.import_module` is provided to support applications that @@ -944,7 +946,7 @@ definition, function definition, or :keyword:`import` statement. **Programmer's note:** the :keyword:`global` is a directive to the parser. It applies only to code parsed at the same time as the :keyword:`global` statement. In particular, a :keyword:`global` statement contained in a string or code -object supplied to the builtin :func:`exec` function does not affect the code +object supplied to the built-in :func:`exec` function does not affect the code block *containing* the function call, and code contained in such a string is unaffected by :keyword:`global` statements in the code containing the function call. The same applies to the :func:`eval` and :func:`compile` functions. diff --git a/Doc/reference/toplevel_components.rst b/Doc/reference/toplevel_components.rst index 21f801c5fa..f4bc71f07b 100644 --- a/Doc/reference/toplevel_components.rst +++ b/Doc/reference/toplevel_components.rst @@ -111,6 +111,6 @@ string argument to :func:`eval` must have the following form: single: input; raw single: readline() (file method) -Note: to read 'raw' input line without interpretation, you can use the the +Note: to read 'raw' input line without interpretation, you can use the :meth:`readline` method of file objects, including ``sys.stdin``. diff --git a/Doc/tools/rstlint.py b/Doc/tools/rstlint.py index e977f217af..2cc3d1277c 100755 --- a/Doc/tools/rstlint.py +++ b/Doc/tools/rstlint.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # -*- coding: utf-8 -*- # Check for stylistic and formal issues in .rst and .py diff --git a/Doc/tools/sphinxext/download.html b/Doc/tools/sphinxext/download.html index 4fca13882a..f89c4585f0 100644 --- a/Doc/tools/sphinxext/download.html +++ b/Doc/tools/sphinxext/download.html @@ -33,6 +33,10 @@ in the table are the size of the download files in megabytes.</p> <td><a href="{{ dlbase }}/python-{{ release }}-docs-text.zip">Download</a> (ca. 2 MB)</td> <td><a href="{{ dlbase }}/python-{{ release }}-docs-text.tar.bz2">Download</a> (ca. 1.5 MB)</td> </tr> + <tr><td>EPUB</td> + <td><a href="{{ dlbase }}/python-{{ release }}-docs-epub.zip">Download</a> (ca. 3.5 MB)</td> + <td><a href="{{ dlbase }}/python-{{ release }}-docs-epub.tar.bz2">Download</a> (ca. 3.5 MB)</td> + </tr> </table> diff --git a/Doc/tools/sphinxext/indexcontent.html b/Doc/tools/sphinxext/indexcontent.html index 30963c36dc..7f8547020f 100644 --- a/Doc/tools/sphinxext/indexcontent.html +++ b/Doc/tools/sphinxext/indexcontent.html @@ -24,8 +24,6 @@ <span class="linkdescr">information for installers & sys-admins</span></p> <p class="biglink"><a class="biglink" href="{{ pathto("distutils/index") }}">Distributing Python Modules</a><br/> <span class="linkdescr">sharing modules with others</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("documenting/index") }}">Documenting Python</a><br/> - <span class="linkdescr">guide for documentation authors</span></p> <p class="biglink"><a class="biglink" href="{{ pathto("faq/index") }}">FAQs</a><br/> <span class="linkdescr">frequently asked questions (with answers!)</span></p> </td></tr> @@ -34,7 +32,7 @@ <p><strong>Indices and tables:</strong></p> <table class="contentstable" align="center"><tr> <td width="50%"> - <p class="biglink"><a class="biglink" href="{{ pathto("modindex") }}">Global Module Index</a><br/> + <p class="biglink"><a class="biglink" href="{{ pathto("py-modindex") }}">Global Module Index</a><br/> <span class="linkdescr">quick access to all modules</span></p> <p class="biglink"><a class="biglink" href="{{ pathto("genindex") }}">General Index</a><br/> <span class="linkdescr">all functions, classes, terms</span></p> diff --git a/Doc/tools/sphinxext/indexsidebar.html b/Doc/tools/sphinxext/indexsidebar.html index 086d8834a6..672492e600 100644 --- a/Doc/tools/sphinxext/indexsidebar.html +++ b/Doc/tools/sphinxext/indexsidebar.html @@ -3,7 +3,7 @@ <h3>Docs for other versions</h3> <ul> <li><a href="http://docs.python.org/2.7/">Python 2.7 (stable)</a></li> - <li><a href="http://docs.python.org/dev/py3k/">Python 3.2 (in development)</a></li> + <li><a href="http://docs.python.org/3.1/">Python 3.1 (stable)</a></li> <li><a href="http://www.python.org/doc/versions/">Old versions</a></li> </ul> diff --git a/Doc/tools/sphinxext/layout.html b/Doc/tools/sphinxext/layout.html index a256181130..8659ef0364 100644 --- a/Doc/tools/sphinxext/layout.html +++ b/Doc/tools/sphinxext/layout.html @@ -6,6 +6,7 @@ {% endblock %} {% block extrahead %} <link rel="shortcut icon" type="image/png" href="{{ pathto('_static/py.png', 1) }}" /> + <script type="text/javascript" src="{{ pathto('_static/copybutton.js', 1) }}"></script> {{ super() }} {% endblock %} {% block footer %} diff --git a/Doc/tools/sphinxext/patchlevel.py b/Doc/tools/sphinxext/patchlevel.py index 082858e41c..b070d60a42 100644 --- a/Doc/tools/sphinxext/patchlevel.py +++ b/Doc/tools/sphinxext/patchlevel.py @@ -68,4 +68,4 @@ def get_version_info(): return version, release if __name__ == '__main__': - print get_header_version_info('.')[1] + print(get_header_version_info('.')[1]) diff --git a/Doc/tools/sphinxext/pyspecific.py b/Doc/tools/sphinxext/pyspecific.py index 7111c06c91..43292816a2 100644 --- a/Doc/tools/sphinxext/pyspecific.py +++ b/Doc/tools/sphinxext/pyspecific.py @@ -10,8 +10,10 @@ """ ISSUE_URI = 'http://bugs.python.org/issue%s' +SOURCE_URI = 'http://hg.python.org/cpython/file/3.2/%s' from docutils import nodes, utils +from sphinx.util.nodes import split_explicit_title # monkey-patch reST parser to disable alphabetic and roman enumerated lists from docutils.parsers.rst.states import Body @@ -44,6 +46,16 @@ def issue_role(typ, rawtext, text, lineno, inliner, options={}, content=[]): return [refnode], [] +# Support for linking to Python source files easily + +def source_role(typ, rawtext, text, lineno, inliner, options={}, content=[]): + has_t, title, target = split_explicit_title(text) + title = utils.unescape(title) + target = utils.unescape(target) + refnode = nodes.reference(title, title, refuri=SOURCE_URI % target) + return [refnode], [] + + # Support for marking up implementation details from sphinx.util.compat import Directive @@ -72,6 +84,67 @@ class ImplementationDetail(Directive): return [pnode] +# Support for documenting decorators + +from sphinx import addnodes +from sphinx.domains.python import PyModulelevel, PyClassmember + +class PyDecoratorMixin(object): + def handle_signature(self, sig, signode): + ret = super(PyDecoratorMixin, self).handle_signature(sig, signode) + signode.insert(0, addnodes.desc_addname('@', '@')) + return ret + + def needs_arglist(self): + return False + +class PyDecoratorFunction(PyDecoratorMixin, PyModulelevel): + def run(self): + # a decorator function is a function after all + self.name = 'py:function' + return PyModulelevel.run(self) + +class PyDecoratorMethod(PyDecoratorMixin, PyClassmember): + def run(self): + self.name = 'py:method' + return PyClassmember.run(self) + + +# Support for documenting version of removal in deprecations + +from sphinx.locale import versionlabels +from sphinx.util.compat import Directive + +versionlabels['deprecated-removed'] = \ + 'Deprecated since version %s, will be removed in version %s' + +class DeprecatedRemoved(Directive): + has_content = True + required_arguments = 2 + optional_arguments = 1 + final_argument_whitespace = True + option_spec = {} + + def run(self): + node = addnodes.versionmodified() + node.document = self.state.document + node['type'] = 'deprecated-removed' + version = (self.arguments[0], self.arguments[1]) + node['version'] = version + if len(self.arguments) == 3: + inodes, messages = self.state.inline_text(self.arguments[2], + self.lineno+1) + node.extend(inodes) + if self.content: + self.state.nested_parse(self.content, self.content_offset, node) + ret = [node] + messages + else: + ret = [node] + env = self.state.document.settings.env + env.note_versionchange('deprecated', version[0], node, self.lineno) + return ret + + # Support for building "topic help" for pydoc pydoc_topic_labels = [ @@ -119,10 +192,10 @@ class PydocTopicsBuilder(Builder): for label in self.status_iterator(pydoc_topic_labels, 'building topics... ', length=len(pydoc_topic_labels)): - if label not in self.env.labels: + if label not in self.env.domaindata['std']['labels']: self.warn('label %r not in documentation' % label) continue - docname, labelid, sectname = self.env.labels[label] + docname, labelid, sectname = self.env.domaindata['std']['labels'][label] doctree = self.env.get_and_resolve_doctree(docname, self) document = new_document('<section node>') document.append(doctree.ids[labelid]) @@ -147,7 +220,6 @@ import suspicious # Support for documenting Opcodes import re -from sphinx import addnodes opcode_sig_re = re.compile(r'(\w+(?:\+\d)?)(?:\s*\((.*)\))?') @@ -165,11 +237,41 @@ def parse_opcode_signature(env, sig, signode): return opname.strip() +# Support for documenting pdb commands + +pdbcmd_sig_re = re.compile(r'([a-z()!]+)\s*(.*)') + +# later... +#pdbargs_tokens_re = re.compile(r'''[a-zA-Z]+ | # identifiers +# [.,:]+ | # punctuation +# [\[\]()] | # parens +# \s+ # whitespace +# ''', re.X) + +def parse_pdb_command(env, sig, signode): + """Transform a pdb command signature into RST nodes.""" + m = pdbcmd_sig_re.match(sig) + if m is None: + raise ValueError + name, args = m.groups() + fullname = name.replace('(', '').replace(')', '') + signode += addnodes.desc_name(name, name) + if args: + signode += addnodes.desc_addname(' '+args, ' '+args) + return fullname + + def setup(app): app.add_role('issue', issue_role) + app.add_role('source', source_role) app.add_directive('impl-detail', ImplementationDetail) + app.add_directive('deprecated-removed', DeprecatedRemoved) app.add_builder(PydocTopicsBuilder) app.add_builder(suspicious.CheckSuspiciousMarkupBuilder) app.add_description_unit('opcode', 'opcode', '%s (opcode)', parse_opcode_signature) + app.add_description_unit('pdbcommand', 'pdbcmd', '%s (pdb command)', + parse_pdb_command) app.add_description_unit('2to3fixer', '2to3fixer', '%s (2to3 fixer)') + app.add_directive_to_domain('py', 'decorator', PyDecoratorFunction) + app.add_directive_to_domain('py', 'decoratormethod', PyDecoratorMethod) diff --git a/Doc/tools/sphinxext/static/basic.css b/Doc/tools/sphinxext/static/basic.css index 2b47622ab8..65aa5f1011 100644 --- a/Doc/tools/sphinxext/static/basic.css +++ b/Doc/tools/sphinxext/static/basic.css @@ -253,8 +253,8 @@ table.docutils { table.docutils td, table.docutils th { padding: 2px 5px 2px 5px; - border-left: 0; - background-color: #eef; + border-left: 0; + background-color: #eef; } table.docutils td p.last, table.docutils th p.last { @@ -270,7 +270,7 @@ table.footnote td, table.footnote th { } table.docutils th { - border-top: 1px solid #cac; + border-top: 1px solid #cac; background-color: #ede; } @@ -280,7 +280,7 @@ th { } th.head { - text-align: center; + text-align: center; } /* -- other body styles ----------------------------------------------------- */ @@ -333,7 +333,7 @@ dl.glossary dt { font-style: italic; } -p.deprecated { +p.deprecated, p.deprecated-removed { background-color: #ffe4e4; border: 1px solid #f66; padding: 7px diff --git a/Doc/tools/sphinxext/static/copybutton.js b/Doc/tools/sphinxext/static/copybutton.js new file mode 100644 index 0000000000..a3b1099e3a --- /dev/null +++ b/Doc/tools/sphinxext/static/copybutton.js @@ -0,0 +1,56 @@ +$(document).ready(function() { + /* Add a [>>>] button on the top-right corner of code samples to hide + * the >>> and ... prompts and the output and thus make the code + * copyable. */ + var div = $('.highlight-python .highlight,' + + '.highlight-python3 .highlight') + var pre = div.find('pre'); + + // get the styles from the current theme + pre.parent().parent().css('position', 'relative'); + var hide_text = 'Hide the prompts and output'; + var show_text = 'Show the prompts and output'; + var border_width = pre.css('border-top-width'); + var border_style = pre.css('border-top-style'); + var border_color = pre.css('border-top-color'); + var button_styles = { + 'cursor':'pointer', 'position': 'absolute', 'top': '0', 'right': '0', + 'border-color': border_color, 'border-style': border_style, + 'border-width': border_width, 'color': border_color, 'text-size': '75%', + 'font-family': 'monospace', 'padding-left': '0.2em', 'padding-right': '0.2em' + } + + // create and add the button to all the code blocks that contain >>> + div.each(function(index) { + var jthis = $(this); + if (jthis.find('.gp').length > 0) { + var button = $('<span class="copybutton">>>></span>'); + button.css(button_styles) + button.attr('title', hide_text); + jthis.prepend(button); + } + // tracebacks (.gt) contain bare text elements that need to be + // wrapped in a span to work with .nextUntil() (see later) + jthis.find('pre:has(.gt)').contents().filter(function() { + return ((this.nodeType == 3) && (this.data.trim().length > 0)); + }).wrap('<span>'); + }); + + // define the behavior of the button when it's clicked + $('.copybutton').toggle( + function() { + var button = $(this); + button.parent().find('.go, .gp, .gt').hide(); + button.next('pre').find('.gt').nextUntil('.gp, .go').css('visibility', 'hidden'); + button.css('text-decoration', 'line-through'); + button.attr('title', show_text); + }, + function() { + var button = $(this); + button.parent().find('.go, .gp, .gt').show(); + button.next('pre').find('.gt').nextUntil('.gp, .go').css('visibility', 'visible'); + button.css('text-decoration', 'none'); + button.attr('title', hide_text); + }); +}); + diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv index 4d4db94826..5076aed00e 100644 --- a/Doc/tools/sphinxext/susp-ignored.csv +++ b/Doc/tools/sphinxext/susp-ignored.csv @@ -5,7 +5,6 @@ c-api/sequence,,:i2,o[i1:i2] c-api/sequence,,:i2,o[i1:i2] = v c-api/sequence,,:i2,del o[i1:i2] c-api/unicode,,:end,str[start:end] -distutils/apiref,,:action,http://pypi.python.org/pypi?:action=list_classifiers distutils/setupscript,,::, extending/embedding,,:numargs,"if(!PyArg_ParseTuple(args, "":numargs""))" extending/extending,,:set,"if (PyArg_ParseTuple(args, ""O:set_callback"", &temp)) {" @@ -165,148 +164,112 @@ whatsnew/2.5,,:memory,:memory: whatsnew/2.5,,:step,[start:stop:step] whatsnew/2.5,,:stop,[start:stop:step] distutils/examples,267,`,This is the description of the ``foobar`` package. -documenting/fromlatex,39,:func,:func:`str(object)` -documenting/fromlatex,39,`,:func:`str(object)` -documenting/fromlatex,39,`,``str(object)`` -documenting/fromlatex,55,.. deprecated:,.. deprecated:: 2.5 -documenting/fromlatex,66,.. note:,.. note:: -documenting/fromlatex,76,:samp,":samp:`open({filename}, {mode})`" -documenting/fromlatex,76,`,":samp:`open({filename}, {mode})`" -documenting/fromlatex,80,`,``'c'`` -documenting/fromlatex,80,`,`Title <URL>`_ -documenting/fromlatex,80,`,``code`` -documenting/fromlatex,80,`,`Title <URL>`_ -documenting/fromlatex,99,:file,:file:`C:\\Temp\\my.tmp` -documenting/fromlatex,99,`,:file:`C:\\Temp\\my.tmp` -documenting/fromlatex,99,`,"``open(""C:\Temp\my.tmp"")``" -documenting/fromlatex,129,.. function:,.. function:: do_foo(bar) -documenting/fromlatex,141,.. function:,".. function:: open(filename[, mode[, buffering]])" -documenting/fromlatex,152,.. function:,.. function:: foo_* -documenting/fromlatex,152,:noindex,:noindex: -documenting/fromlatex,162,.. describe:,.. describe:: a == b -documenting/fromlatex,168,.. cmdoption:,.. cmdoption:: -O -documenting/fromlatex,168,.. envvar:,.. envvar:: PYTHONINSPECT -documenting/rest,33,`,``text`` -documenting/rest,47,:rolename,:rolename:`content` -documenting/rest,47,`,:rolename:`content` -documenting/rest,103,::,This is a normal text paragraph. The next paragraph is a code sample:: -documenting/rest,130,`,`Link text <http://target>`_ -documenting/rest,187,.. function:,.. function:: foo(x) -documenting/rest,187,:bar,:bar: no -documenting/rest,208,.. rubric:,.. rubric:: Footnotes -faq/programming,762,:reduce,"print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y," -faq/programming,762,:reduce,"Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro," -faq/programming,762,:chr,">=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr(" -faq/programming,1047,::,for x in sequence[::-1]: +faq/programming,,:reduce,"print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y," +faq/programming,,:reduce,"Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro," +faq/programming,,:chr,">=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr(" +faq/programming,,::,for x in sequence[::-1]: faq/windows,229,:EOF,@setlocal enableextensions & python -x %~f0 %* & goto :EOF faq/windows,393,:REG,.py :REG_SZ: c:\<path to python>\python.exe -u %s %s library/bisect,32,:hi,all(val >= x for val in a[i:hi]) library/bisect,42,:hi,all(val > x for val in a[i:hi]) library/http.client,52,:port,host:port -library/nntplib,272,:bytes,:bytes -library/nntplib,272,:lines,:lines -library/nntplib,272,:lines,"['xref', 'from', ':lines', ':bytes', 'references', 'date', 'message-id', 'subject']" -library/nntplib,272,:bytes,"['xref', 'from', ':lines', ':bytes', 'references', 'date', 'message-id', 'subject']" +library/nntplib,,:bytes,:bytes +library/nntplib,,:lines,:lines +library/nntplib,,:lines,"['xref', 'from', ':lines', ':bytes', 'references', 'date', 'message-id', 'subject']" +library/nntplib,,:bytes,"['xref', 'from', ':lines', ':bytes', 'references', 'date', 'message-id', 'subject']" library/pickle,,:memory,"conn = sqlite3.connect("":memory:"")" library/profile,,:lineno,"(sort by filename:lineno)," library/socket,,::,"(10, 1, 6, '', ('2001:888:2000:d::a2', 80, 0, 0))]" -library/stdtypes,1026,:end,s[start:end] -library/stdtypes,1195,:end,s[start:end] +library/stdtypes,,:end,s[start:end] +library/stdtypes,,:end,s[start:end] library/urllib.request,,:close,Connection:close library/urllib.request,,:password,"""joe:password@python.org""" library/urllib.request,,:lang,"xmlns=""http://www.w3.org/1999/xhtml"" xml:lang=""en"" lang=""en"">\n\n<head>\n" library/xmlrpc.client,103,:pass,http://user:pass@host:port/path library/xmlrpc.client,103,:port,http://user:pass@host:port/path library/xmlrpc.client,103,:pass,user:pass -license,717,`,* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY -license,717,`,* THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND -license,879,`,"``Software''), to deal in the Software without restriction, including" -license,879,`,"THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND," +license,,`,* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY +license,,`,* THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND +license,,`,"``Software''), to deal in the Software without restriction, including" +license,,`,"THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND," reference/lexical_analysis,704,`,$ ? ` whatsnew/2.7,735,:Sunday,'2009:4:Sunday' whatsnew/2.7,862,::,"export PYTHONWARNINGS=all,error:::Cookie:0" whatsnew/2.7,862,:Cookie,"export PYTHONWARNINGS=all,error:::Cookie:0" whatsnew/2.7,1619,::,>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') whatsnew/2.7,1619,::,"ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]'," -documenting/markup,33,.. sectionauthor:,.. sectionauthor:: Guido van Rossum <guido@python.org> -documenting/markup,42,:mod,:mod:`parrot` -- Dead parrot access -documenting/markup,42,`,:mod:`parrot` -- Dead parrot access -documenting/markup,42,.. module:,.. module:: parrot -documenting/markup,42,:platform,":platform: Unix, Windows" -documenting/markup,42,:synopsis,:synopsis: Analyze and reanimate dead parrots. -documenting/markup,42,.. moduleauthor:,.. moduleauthor:: Eric Cleese <eric@python.invalid> -documenting/markup,42,.. moduleauthor:,.. moduleauthor:: John Idle <john@python.invalid> -documenting/markup,88,:noindex,:noindex: -documenting/markup,95,.. function:,.. function:: spam(eggs) -documenting/markup,95,:noindex,:noindex: -documenting/markup,101,.. method:,.. method:: FileInput.input(...) -documenting/markup,121,.. cfunction:,".. cfunction:: PyObject* PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems)" -documenting/markup,131,.. cmember:,.. cmember:: PyObject* PyTypeObject.tp_bases -documenting/markup,150,.. cvar:,.. cvar:: PyObject* PyClass_Type -documenting/markup,179,.. function:,".. function:: Timer.repeat([repeat=3[, number=1000000]])" -documenting/markup,209,.. cmdoption:,.. cmdoption:: -m <module> -documenting/markup,227,.. describe:,.. describe:: opcode -documenting/markup,256,.. highlightlang:,.. highlightlang:: c -documenting/markup,276,.. literalinclude:,.. literalinclude:: example.py -documenting/markup,291,:rolename,:rolename:`content` -documenting/markup,291,`,:rolename:`content` -documenting/markup,296,:role,:role:`title <target>` -documenting/markup,296,`,:role:`title <target>` -documenting/markup,302,:meth,:meth:`~Queue.Queue.get` -documenting/markup,302,`,:meth:`~Queue.Queue.get` -documenting/markup,350,:func,:func:`filter` -documenting/markup,350,`,:func:`filter` -documenting/markup,350,:func,:func:`foo.filter` -documenting/markup,350,`,:func:`foo.filter` -documenting/markup,356,:func,:func:`open` -documenting/markup,356,`,:func:`open` -documenting/markup,356,:func,:func:`.open` -documenting/markup,356,`,:func:`.open` -documenting/markup,435,:file,... is installed in :file:`/usr/lib/python2.{x}/site-packages` ... -documenting/markup,435,`,... is installed in :file:`/usr/lib/python2.{x}/site-packages` ... -documenting/markup,454,:kbd,:kbd:`C-x C-f` -documenting/markup,454,`,:kbd:`C-x C-f` -documenting/markup,454,:kbd,:kbd:`Control-x Control-f` -documenting/markup,454,`,:kbd:`Control-x Control-f` -documenting/markup,468,:mailheader,:mailheader:`Content-Type` -documenting/markup,468,`,:mailheader:`Content-Type` -documenting/markup,477,:manpage,:manpage:`ls(1)` -documenting/markup,477,`,:manpage:`ls(1)` -documenting/markup,493,:menuselection,:menuselection:`Start --> Programs` -documenting/markup,493,`,:menuselection:`Start --> Programs` -documenting/markup,508,`,``code`` -documenting/markup,526,:file,:file: -documenting/markup,526,`,``code`` -documenting/markup,561,:ref,:ref:`label-name` -documenting/markup,561,`,:ref:`label-name` -documenting/markup,565,:ref,"It refers to the section itself, see :ref:`my-reference-label`." -documenting/markup,565,`,"It refers to the section itself, see :ref:`my-reference-label`." -documenting/markup,574,:ref,:ref: -documenting/markup,595,.. note:,.. note:: -documenting/markup,622,.. versionadded:,.. versionadded:: 3.1 -documenting/markup,647,::,.. impl-detail:: -documenting/markup,647,::,.. impl-detail:: This shortly mentions an implementation detail. -documenting/markup,667,.. seealso:,.. seealso:: -documenting/markup,667,:mod,Module :mod:`zipfile` -documenting/markup,667,`,Module :mod:`zipfile` -documenting/markup,667,:mod,Documentation of the :mod:`zipfile` standard module. -documenting/markup,667,`,Documentation of the :mod:`zipfile` standard module. -documenting/markup,667,`,"`GNU tar manual, Basic Tar Format <http://link>`_" -documenting/markup,681,.. centered:,.. centered:: -documenting/markup,726,.. toctree:,.. toctree:: -documenting/markup,726,:maxdepth,:maxdepth: 2 -documenting/markup,742,.. index:,.. index:: -documenting/markup,772,.. index:,".. index:: BNF, grammar, syntax, notation" -documenting/markup,803,`,"unaryneg ::= ""-"" `integer`" -documenting/markup,808,.. productionlist:,.. productionlist:: -documenting/markup,808,`,"try1_stmt: ""try"" "":"" `suite`" -documenting/markup,808,`,": (""except"" [`expression` ["","" `target`]] "":"" `suite`)+" -documenting/markup,808,`,": [""else"" "":"" `suite`]" -documenting/markup,808,`,": [""finally"" "":"" `suite`]" -documenting/markup,808,`,"try2_stmt: ""try"" "":"" `suite`" -documenting/markup,808,`,": ""finally"" "":"" `suite`" -library/importlib,396,`,The keys are the module names -- packages must end in ``.__init__``. -library/importlib,396,`,"name with ``.__init__`` appended to it.""""""" -library/importlib,396,`,# ``__init__``. -library/socket,249,::,"(10, 1, 6, '', ('2001:888:2000:d::a2', 80, 0, 0))]" -library/stdtypes,860,:end,s[start:end] +library/configparser,,`,# Set the optional `raw` argument of get() to True if you wish to disable +library/configparser,,`,# The optional `vars` argument is a dict with members that will take +library/configparser,,`,# The optional `fallback` argument can be used to provide a fallback value +library/configparser,,:option,${section:option} +library/configparser,,:system,path: ${Common:system_dir}/Library/Frameworks/ +library/configparser,,:home,my_dir: ${Common:home_dir}/twosheds +library/configparser,,:path,python_dir: ${Frameworks:path}/Python/Versions/${Frameworks:Python} +library/configparser,,:Python,python_dir: ${Frameworks:path}/Python/Versions/${Frameworks:Python} +library/pdb,,:lineno,[filename:lineno | bpnumber [bpnumber ...]] +library/pdb,,:lineno,filename:lineno +library/logging,,:Watch,WARNING:root:Watch out! +library/logging,,:So,INFO:root:So should this +library/logging,,:Started,INFO:root:Started +library/logging,,:Doing,INFO:root:Doing something +library/logging,,:Finished,INFO:root:Finished +library/logging,,:Look,WARNING:root:Look before you leap! +library/logging,,:So,INFO:So should this +library/logging,,:logger,severity:logger name:message +library/logging,,:message,severity:logger name:message +whatsnew/3.2,,:directory,... ${buildout:directory}/downloads/dist +whatsnew/3.2,,:location,... zope9-location = ${zope9:location} +whatsnew/3.2,,:prefix,... zope-conf = ${custom:prefix}/etc/zope.conf +howto/logging,,:root,WARNING:root:Watch out! +howto/logging,,:Watch,WARNING:root:Watch out! +howto/logging,,:root,DEBUG:root:This message should go to the log file +howto/logging,,:This,DEBUG:root:This message should go to the log file +howto/logging,,:root,INFO:root:So should this +howto/logging,,:So,INFO:root:So should this +howto/logging,,:root,"WARNING:root:And this, too" +howto/logging,,:And,"WARNING:root:And this, too" +howto/logging,,:root,INFO:root:Started +howto/logging,,:Started,INFO:root:Started +howto/logging,,:root,INFO:root:Doing something +howto/logging,,:Doing,INFO:root:Doing something +howto/logging,,:root,INFO:root:Finished +howto/logging,,:Finished,INFO:root:Finished +howto/logging,,:root,WARNING:root:Look before you leap! +howto/logging,,:Look,WARNING:root:Look before you leap! +howto/logging,,:This,DEBUG:This message should appear on the console +howto/logging,,:So,INFO:So should this +howto/logging,,:And,"WARNING:And this, too" +howto/logging,,:logger,severity:logger name:message +howto/logging,,:message,severity:logger name:message +library/logging.handlers,,:port,host:port +library/imaplib,116,:MM,"""DD-Mmm-YYYY HH:MM:SS" +library/imaplib,116,:SS,"""DD-Mmm-YYYY HH:MM:SS" +whatsnew/3.2,,::,"$ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::'" +howto/pyporting,75,::,# make sure to use :: Python *and* :: Python :: 3 so +howto/pyporting,75,::,"'Programming Language :: Python'," +howto/pyporting,75,::,'Programming Language :: Python :: 3' +whatsnew/3.2,,:gz,">>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:" +whatsnew/3.2,,:directory,${buildout:directory}/downloads/dist +whatsnew/3.2,,:location,zope9-location = ${zope9:location} +whatsnew/3.2,,:prefix,zope-conf = ${custom:prefix}/etc/zope.conf +whatsnew/3.2,,:beef,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') +whatsnew/3.2,,:cafe,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') +whatsnew/3.2,,:affe,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') +whatsnew/3.2,,:deaf,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') +whatsnew/3.2,,:feed,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') +whatsnew/3.2,,:beef,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," +whatsnew/3.2,,:cafe,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," +whatsnew/3.2,,:affe,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," +whatsnew/3.2,,:deaf,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," +whatsnew/3.2,,:feed,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," +library/pprint,209,::,"'classifiers': ['Development Status :: 4 - Beta'," +library/pprint,209,::,"'Intended Audience :: Developers'," +library/pprint,209,::,"'License :: OSI Approved :: MIT License'," +library/pprint,209,::,"'Natural Language :: English'," +library/pprint,209,::,"'Operating System :: OS Independent'," +library/pprint,209,::,"'Programming Language :: Python'," +library/pprint,209,::,"'Programming Language :: Python :: 2'," +library/pprint,209,::,"'Programming Language :: Python :: 2.6'," +library/pprint,209,::,"'Programming Language :: Python :: 2.7'," +library/pprint,209,::,"'Topic :: Software Development :: Libraries'," +library/pprint,209,::,"'Topic :: Software Development :: Libraries :: Python Modules']," diff --git a/Doc/tools/sphinxext/suspicious.py b/Doc/tools/sphinxext/suspicious.py index f15e931b90..888b2318b3 100644 --- a/Doc/tools/sphinxext/suspicious.py +++ b/Doc/tools/sphinxext/suspicious.py @@ -49,13 +49,15 @@ import sys from docutils import nodes from sphinx.builders import Builder -detect_all = re.compile(ur''' +detect_all = re.compile(r''' ::(?=[^=])| # two :: (but NOT ::=) :[a-zA-Z][a-zA-Z0-9]+| # :foo `| # ` (seldom used by itself) (?<!\.)\.\.[ \t]*\w+: # .. foo: (but NOT ... else:) ''', re.UNICODE | re.VERBOSE).finditer +py3 = sys.version_info >= (3, 0) + class Rule: def __init__(self, docname, lineno, issue, line): @@ -136,7 +138,11 @@ class CheckSuspiciousMarkupBuilder(Builder): if not self.any_issue: self.info() self.any_issue = True self.write_log_entry(lineno, issue, text) - self.warn('[%s:%d] "%s" found in "%-.120s"' % ( + if py3: + self.warn('[%s:%d] "%s" found in "%-.120s"' % + (self.docname, lineno, issue, text)) + else: + self.warn('[%s:%d] "%s" found in "%-.120s"' % ( self.docname.encode(sys.getdefaultencoding(),'replace'), lineno, issue.encode(sys.getdefaultencoding(),'replace'), @@ -144,13 +150,19 @@ class CheckSuspiciousMarkupBuilder(Builder): self.app.statuscode = 1 def write_log_entry(self, lineno, issue, text): - f = open(self.log_file_name, 'ab') - writer = csv.writer(f, dialect) - writer.writerow([self.docname.encode('utf-8'), - lineno, - issue.encode('utf-8'), - text.strip().encode('utf-8')]) - f.close() + if py3: + f = open(self.log_file_name, 'a') + writer = csv.writer(f, dialect) + writer.writerow([self.docname, lineno, issue, text.strip()]) + f.close() + else: + f = open(self.log_file_name, 'ab') + writer = csv.writer(f, dialect) + writer.writerow([self.docname.encode('utf-8'), + lineno, + issue.encode('utf-8'), + text.strip().encode('utf-8')]) + f.close() def load_rules(self, filename): """Load database of previously ignored issues. @@ -160,18 +172,26 @@ class CheckSuspiciousMarkupBuilder(Builder): """ self.info("loading ignore rules... ", nonl=1) self.rules = rules = [] - try: f = open(filename, 'rb') - except IOError: return + try: + if py3: + f = open(filename, 'r') + else: + f = open(filename, 'rb') + except IOError: + return for i, row in enumerate(csv.reader(f)): if len(row) != 4: raise ValueError( "wrong format in %s, line %d: %s" % (filename, i+1, row)) docname, lineno, issue, text = row - docname = docname.decode('utf-8') - if lineno: lineno = int(lineno) - else: lineno = None - issue = issue.decode('utf-8') - text = text.decode('utf-8') + if lineno: + lineno = int(lineno) + else: + lineno = None + if not py3: + docname = docname.decode('utf-8') + issue = issue.decode('utf-8') + text = text.decode('utf-8') rule = Rule(docname, lineno, issue, text) rules.append(rule) f.close() diff --git a/Doc/tutorial/classes.rst b/Doc/tutorial/classes.rst index 82735df4cb..68c4e5d886 100644 --- a/Doc/tutorial/classes.rst +++ b/Doc/tutorial/classes.rst @@ -458,8 +458,8 @@ argument:: self.add(x) Methods may reference global names in the same way as ordinary functions. The -global scope associated with a method is the module containing the class -definition. (The class itself is never used as a global scope.) While one +global scope associated with a method is the module containing its +definition. (A class is never used as a global scope.) While one rarely encounters a good reason for using global data in a method, there are many legitimate uses of the global scope: for one thing, functions and modules imported into the global scope can be used by methods, as well as functions and @@ -595,6 +595,28 @@ current class name with leading underscore(s) stripped. This mangling is done without regard to the syntactic position of the identifier, as long as it occurs within the definition of a class. +Name mangling is helpful for letting subclasses override methods without +breaking intraclass method calls. For example:: + + class Mapping: + def __init__(self, iterable): + self.items_list = [] + self.__update(iterable) + + def update(self, iterable): + for item in iterable: + self.items_list.append(item) + + __update = update # private copy of original update() method + + class MappingSubclass(Mapping): + + def update(self, keys, values): + # provides new signature for update() + # but does not break __init__() + for item in zip(keys, values): + self.items_list.append(item) + Note that the mangling rules are designed mostly to avoid accidents; it still is possible to access or modify a variable that is considered private. This can even be useful in special circumstances, such as in the debugger. @@ -717,7 +739,7 @@ object that defines the method :meth:`__next__` which accesses elements in the container one at a time. When there are no more elements, :meth:`__next__` raises a :exc:`StopIteration` exception which tells the :keyword:`for` loop to terminate. You can call the :meth:`__next__` method using the :func:`next` -builtin; this example shows how it all works:: +built-in function; this example shows how it all works:: >>> s = 'abc' >>> it = iter(s) @@ -730,7 +752,6 @@ builtin; this example shows how it all works:: >>> next(it) 'c' >>> next(it) - Traceback (most recent call last): File "<stdin>", line 1, in ? next(it) @@ -742,7 +763,7 @@ returns an object with a :meth:`__next__` method. If the class defines :meth:`__next__`, then :meth:`__iter__` can just return ``self``:: class Reverse: - "Iterator for looping over a sequence backwards" + """Iterator for looping over a sequence backwards.""" def __init__(self, data): self.data = data self.index = len(data) @@ -754,6 +775,8 @@ returns an object with a :meth:`__next__` method. If the class defines self.index = self.index - 1 return self.data[self.index] +:: + >>> rev = Reverse('spam') >>> iter(rev) <__main__.Reverse object at 0x00A1DB50> @@ -782,6 +805,8 @@ easy to create:: for index in range(len(data)-1, -1, -1): yield data[index] +:: + >>> for char in reverse('golf'): ... print(char) ... diff --git a/Doc/tutorial/controlflow.rst b/Doc/tutorial/controlflow.rst index e33a59619a..5ed5aea444 100644 --- a/Doc/tutorial/controlflow.rst +++ b/Doc/tutorial/controlflow.rst @@ -184,6 +184,9 @@ following loop, which searches for prime numbers:: 8 equals 2 * 4 9 equals 3 * 3 +(Yes, this is the correct code. Look closely: the ``else`` clause belongs to +the :keyword:`for` loop, **not** the :keyword:`if` statement.) + .. _tut-pass: @@ -409,8 +412,8 @@ write the function like this instead:: Keyword Arguments ----------------- -Functions can also be called using keyword arguments of the form ``keyword = -value``. For instance, the following function:: +Functions can also be called using :term:`keyword arguments <keyword argument>` +of the form ``kwarg=value``. For instance, the following function:: def parrot(voltage, state='a stiff', action='voom', type='Norwegian Blue'): print("-- This parrot wouldn't", action, end=' ') @@ -418,26 +421,31 @@ value``. For instance, the following function:: print("-- Lovely plumage, the", type) print("-- It's", state, "!") -could be called in any of the following ways:: +accepts one required argument (``voltage``) and three optional arguments +(``state``, ``action``, and ``type``). This function can be called in any +of the following ways:: - parrot(1000) - parrot(action = 'VOOOOOM', voltage = 1000000) - parrot('a thousand', state = 'pushing up the daisies') - parrot('a million', 'bereft of life', 'jump') + parrot(1000) # 1 positional argument + parrot(voltage=1000) # 1 keyword argument + parrot(voltage=1000000, action='VOOOOOM') # 2 keyword arguments + parrot(action='VOOOOOM', voltage=1000000) # 2 keyword arguments + parrot('a million', 'bereft of life', 'jump') # 3 positional arguments + parrot('a thousand', state='pushing up the daisies') # 1 positional, 1 keyword -but the following calls would all be invalid:: +but all the following calls would be invalid:: parrot() # required argument missing - parrot(voltage=5.0, 'dead') # non-keyword argument following keyword - parrot(110, voltage=220) # duplicate value for argument - parrot(actor='John Cleese') # unknown keyword - -In general, an argument list must have any positional arguments followed by any -keyword arguments, where the keywords must be chosen from the formal parameter -names. It's not important whether a formal parameter has a default value or -not. No argument may receive a value more than once --- formal parameter names -corresponding to positional arguments cannot be used as keywords in the same -calls. Here's an example that fails due to this restriction:: + parrot(voltage=5.0, 'dead') # non-keyword argument after a keyword argument + parrot(110, voltage=220) # duplicate value for the same argument + parrot(actor='John Cleese') # unknown keyword argument + +In a function call, keyword arguments must follow positional arguments. +All the keyword arguments passed must match one of the arguments +accepted by the function (e.g. ``actor`` is not a valid argument for the +``parrot`` function), and their order is not important. This also includes +non-optional arguments (e.g. ``parrot(voltage=1000)`` is valid too). +No argument may receive a value more than once. +Here's an example that fails due to this restriction:: >>> def function(a): ... pass diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst index defb47c72c..5fb72fdbaf 100644 --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -163,107 +163,137 @@ have fast appends and pops from both ends. For example:: List Comprehensions ------------------- -List comprehensions provide a concise way to create lists from sequences. -Common applications are to make lists where each element is the result of -some operations applied to each member of the sequence, or to create a -subsequence of those elements that satisfy a certain condition. +List comprehensions provide a concise way to create lists. +Common applications are to make new lists where each element is the result of +some operations applied to each member of another sequence or iterable, or to +create a subsequence of those elements that satisfy a certain condition. -A list comprehension consists of brackets containing an expression followed -by a :keyword:`for` clause, then zero or more :keyword:`for` or :keyword:`if` -clauses. The result will be a list resulting from evaluating the expression in -the context of the :keyword:`for` and :keyword:`if` clauses which follow it. If -the expression would evaluate to a tuple, it must be parenthesized. +For example, assume we want to create a list of squares, like:: + + >>> squares = [] + >>> for x in range(10): + ... squares.append(x**2) + ... + >>> squares + [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] + +We can obtain the same result with:: -Here we take a list of numbers and return a list of three times each number:: + squares = [x**2 for x in range(10)] - >>> vec = [2, 4, 6] - >>> [3*x for x in vec] - [6, 12, 18] +This is also equivalent to ``squares = map(lambda x: x**2, range(10))``, +but it's more concise and readable. -Now we get a little fancier:: +A list comprehension consists of brackets containing an expression followed +by a :keyword:`for` clause, then zero or more :keyword:`for` or :keyword:`if` +clauses. The result will be a new list resulting from evaluating the expression +in the context of the :keyword:`for` and :keyword:`if` clauses which follow it. +For example, this listcomp combines the elements of two lists if they are not +equal:: - >>> [[x, x**2] for x in vec] - [[2, 4], [4, 16], [6, 36]] + >>> [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y] + [(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)] -Here we apply a method call to each item in a sequence:: +and it's equivalent to:: + >>> combs = [] + >>> for x in [1,2,3]: + ... for y in [3,1,4]: + ... if x != y: + ... combs.append((x, y)) + ... + >>> combs + [(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)] + +Note how the order of the :keyword:`for` and :keyword:`if` statements is the +same in both these snippets. + +If the expression is a tuple (e.g. the ``(x, y)`` in the previous example), +it must be parenthesized. :: + + >>> vec = [-4, -2, 0, 2, 4] + >>> # create a new list with the values doubled + >>> [x*2 for x in vec] + [-8, -4, 0, 4, 8] + >>> # filter the list to exclude negative numbers + >>> [x for x in vec if x >= 0] + [0, 2, 4] + >>> # apply a function to all the elements + >>> [abs(x) for x in vec] + [4, 2, 0, 2, 4] + >>> # call a method on each element >>> freshfruit = [' banana', ' loganberry ', 'passion fruit '] >>> [weapon.strip() for weapon in freshfruit] ['banana', 'loganberry', 'passion fruit'] - -Using the :keyword:`if` clause we can filter the stream:: - - >>> [3*x for x in vec if x > 3] - [12, 18] - >>> [3*x for x in vec if x < 2] - [] - -Tuples can often be created without their parentheses, but not here:: - - >>> [x, x**2 for x in vec] # error - parens required for tuples + >>> # create a list of 2-tuples like (number, square) + >>> [(x, x**2) for x in range(6)] + [(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25)] + >>> # the tuple must be parenthesized, otherwise an error is raised + >>> [x, x**2 for x in range(6)] File "<stdin>", line 1, in ? - [x, x**2 for x in vec] + [x, x**2 for x in range(6)] ^ SyntaxError: invalid syntax - >>> [(x, x**2) for x in vec] - [(2, 4), (4, 16), (6, 36)] - -Here are some nested for loops and other fancy behavior:: - - >>> vec1 = [2, 4, 6] - >>> vec2 = [4, 3, -9] - >>> [x*y for x in vec1 for y in vec2] - [8, 6, -18, 16, 12, -36, 24, 18, -54] - >>> [x+y for x in vec1 for y in vec2] - [6, 5, -7, 8, 7, -5, 10, 9, -3] - >>> [vec1[i]*vec2[i] for i in range(len(vec1))] - [8, 12, -54] + >>> # flatten a list using a listcomp with two 'for' + >>> vec = [[1,2,3], [4,5,6], [7,8,9]] + >>> [num for elem in vec for num in elem] + [1, 2, 3, 4, 5, 6, 7, 8, 9] -List comprehensions can be applied to complex expressions and nested functions:: +List comprehensions can contain complex expressions and nested functions:: - >>> [str(round(355/113, i)) for i in range(1, 6)] + >>> from math import pi + >>> [str(round(pi, i)) for i in range(1, 6)] ['3.1', '3.14', '3.142', '3.1416', '3.14159'] - Nested List Comprehensions -------------------------- -If you've got the stomach for it, list comprehensions can be nested. They are a -powerful tool but -- like all powerful tools -- they need to be used carefully, -if at all. - -Consider the following example of a 3x3 matrix held as a list containing three -lists, one list per row:: - - >>> mat = [ - ... [1, 2, 3], - ... [4, 5, 6], - ... [7, 8, 9], - ... ] +The initial expression in a list comprehension can be any arbitrary expression, +including another list comprehension. -Now, if you wanted to swap rows and columns, you could use a list -comprehension:: +Consider the following example of a 3x4 matrix implemented as a list of +3 lists of length 4:: - >>> print([[row[i] for row in mat] for i in [0, 1, 2]]) - [[1, 4, 7], [2, 5, 8], [3, 6, 9]] + >>> matrix = [ + ... [1, 2, 3, 4], + ... [5, 6, 7, 8], + ... [9, 10, 11, 12], + ... ] -Special care has to be taken for the *nested* list comprehension: +The following list comprehension will transpose rows and columns:: - To avoid apprehension when nesting list comprehensions, read from right to - left. + >>> [[row[i] for row in matrix] for i in range(4)] + [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] -A more verbose version of this snippet shows the flow explicitly:: +As we saw in the previous section, the nested listcomp is evaluated in +the context of the :keyword:`for` that follows it, so this example is +equivalent to:: - for i in [0, 1, 2]: - for row in mat: - print(row[i], end="") - print() + >>> transposed = [] + >>> for i in range(4): + ... transposed.append([row[i] for row in matrix]) + ... + >>> transposed + [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] + +which, in turn, is the same as:: + + >>> transposed = [] + >>> for i in range(4): + ... # the following 3 lines implement the nested listcomp + ... transposed_row = [] + ... for row in matrix: + ... transposed_row.append(row[i]) + ... transposed.append(transposed_row) + ... + >>> transposed + [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] -In real world, you should prefer built-in functions to complex flow statements. +In the real world, you should prefer built-in functions to complex flow statements. The :func:`zip` function would do a great job for this use case:: - >>> list(zip(*mat)) - [(1, 4, 7), (2, 5, 8), (3, 6, 9)] + >>> zip(*matrix) + [(1, 5, 9), (2, 6, 10), (3, 7, 11), (4, 8, 12)] See :ref:`tut-unpacking-arguments` for details on the asterisk in this line. diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index c06568e39a..9c3c1435c3 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -92,18 +92,17 @@ thing in all languages that support your hardware's floating-point arithmetic (although some languages may not *display* the difference by default, or in all output modes). -Python's built-in :func:`str` function produces only 12 significant digits, and -you may wish to use that instead. It's unusual for ``eval(str(x))`` to -reproduce *x*, but the output may be more pleasant to look at:: +For more pleasant output, you may wish to use string formatting to produce a limited number of significant digits:: - >>> str(math.pi) + >>> format(math.pi, '.12g') # give 12 significant digits '3.14159265359' + >>> format(math.pi, '.2f') # give 2 digits after the point + '3.14' + >>> repr(math.pi) '3.141592653589793' - >>> format(math.pi, '.2f') - '3.14' It's important to realize that this is, in a real sense, an illusion: you're simply rounding the *display* of the true machine value. diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst index abe1ce09c0..00f5aea609 100644 --- a/Doc/tutorial/inputoutput.rst +++ b/Doc/tutorial/inputoutput.rst @@ -40,8 +40,8 @@ which can be read by the interpreter (or will force a :exc:`SyntaxError` if there is not equivalent syntax). For objects which don't have a particular representation for human consumption, :func:`str` will return the same value as :func:`repr`. Many values, such as numbers or structures like lists and -dictionaries, have the same representation using either function. Strings and -floating point numbers, in particular, have two distinct representations. +dictionaries, have the same representation using either function. Strings, in +particular, have two distinct representations. Some examples:: @@ -50,9 +50,7 @@ Some examples:: 'Hello, world.' >>> repr(s) "'Hello, world.'" - >>> str(1.0/7.0) - '0.142857142857' - >>> repr(1.0/7.0) + >>> str(1/7) '0.14285714285714285' >>> x = 10 * 3.25 >>> y = 200 * 200 @@ -162,7 +160,7 @@ Positional and keyword arguments can be arbitrarily combined:: An optional ``':'`` and format specifier can follow the field name. This allows greater control over how the value is formatted. The following example -truncates Pi to three places after the decimal. +rounds Pi to three places after the decimal. >>> import math >>> print('The value of PI is approximately {0:.3f}.'.format(math.pi)) @@ -207,7 +205,7 @@ Old string formatting --------------------- The ``%`` operator can also be used for string formatting. It interprets the -left argument much like a :cfunc:`sprintf`\ -style format string to be applied +left argument much like a :c:func:`sprintf`\ -style format string to be applied to the right argument, and returns the string resulting from this formatting operation. For example:: diff --git a/Doc/tutorial/interactive.rst b/Doc/tutorial/interactive.rst index 5faaf96e81..36acb06d29 100644 --- a/Doc/tutorial/interactive.rst +++ b/Doc/tutorial/interactive.rst @@ -156,17 +156,18 @@ symbol table. A command to check (or even suggest) matching parentheses, quotes, etc., would also be useful. One alternative enhanced interactive interpreter that has been around for quite -some time is `IPython`_, which features tab completion, object exploration and +some time is IPython_, which features tab completion, object exploration and advanced history management. It can also be thoroughly customized and embedded into other applications. Another similar enhanced interactive environment is -`bpython`_. +bpython_. .. rubric:: Footnotes .. [#] Python will execute the contents of a file identified by the :envvar:`PYTHONSTARTUP` environment variable when you start an interactive - interpreter. + interpreter. To customize Python even for non-interactive mode, see + :ref:`tut-customize`. .. _GNU Readline: http://tiswww.case.edu/php/chet/readline/rltop.html diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst index 94d7562a13..2338465d62 100644 --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -10,11 +10,11 @@ Using the Python Interpreter Invoking the Interpreter ======================== -The Python interpreter is usually installed as :file:`/usr/local/bin/python3.1` +The Python interpreter is usually installed as :file:`/usr/local/bin/python3.2` on those machines where it is available; putting :file:`/usr/local/bin` in your Unix shell's search path makes it possible to start it by typing the command :: - python3.1 + python3.2 to the shell. [#]_ Since the choice of the directory where the interpreter lives is an installation option, other places are possible; check with your local @@ -22,11 +22,11 @@ Python guru or system administrator. (E.g., :file:`/usr/local/python` is a popular alternative location.) On Windows machines, the Python installation is usually placed in -:file:`C:\\Python31`, though you can change this when you're running the +:file:`C:\\Python32`, though you can change this when you're running the installer. To add this directory to your path, you can type the following command into the command prompt in a DOS box:: - set path=%path%;C:\python31 + set path=%path%;C:\python32 Typing an end-of-file character (:kbd:`Control-D` on Unix, :kbd:`Control-Z` on Windows) at the primary prompt causes the interpreter to exit with a zero exit @@ -60,8 +60,7 @@ if you had spelled out its full name on the command line. When a script file is used, it is sometimes useful to be able to run the script and enter interactive mode afterwards. This can be done by passing :option:`-i` -before the script. (This does not work if the script is read from standard -input, for the same reason as explained in the previous paragraph.) +before the script. .. _tut-argpassing: @@ -94,8 +93,8 @@ with the *secondary prompt*, by default three dots (``...``). The interpreter prints a welcome message stating its version number and a copyright notice before printing the first prompt:: - $ python3.1 - Python 3.1 (py3k, Sep 12 2007, 12:21:02) + $ python3.2 + Python 3.2 (py3k, Sep 12 2007, 12:21:02) [GCC 3.4.6 20060404 (Red Hat 3.4.6-8)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> @@ -148,7 +147,7 @@ Executable Python Scripts On BSD'ish Unix systems, Python scripts can be made directly executable, like shell scripts, by putting the line :: - #! /usr/bin/env python3.1 + #! /usr/bin/env python3.2 (assuming that the interpreter is on the user's :envvar:`PATH`) at the beginning of the script and giving the file an executable mode. The ``#!`` must be the @@ -169,6 +168,8 @@ also be ``.pyw``, in that case, the console window that normally appears is suppressed. +.. _tut-source-encoding: + Source Code Encoding -------------------- @@ -234,6 +235,29 @@ in the script:: exec(open(filename).read()) +.. _tut-customize: + +The Customization Modules +------------------------- + +Python provides two hooks to let you customize it: :mod:`sitecustomize` and +:mod:`usercustomize`. To see how it works, you need first to find the location +of your user site-packages directory. Start Python and run this code: + + >>> import site + >>> site.getusersitepackages() + '/home/user/.local/lib/python3.2/site-packages' + +Now you can create a file named :file:`usercustomize.py` in that directory and +put anything you want in it. It will affect every invocation of Python, unless +it is started with the :option:`-s` option to disable the automatic import. + +:mod:`sitecustomize` works in the same way, but is typically created by an +administrator of the computer in the global site-packages directory, and is +imported before :mod:`usercustomize`. See the documentation of the :mod:`site` +module for more details. + + .. rubric:: Footnotes .. [#] On Unix, the Python 3.x interpreter is by default not installed with the @@ -241,4 +265,3 @@ in the script:: simultaneously installed Python 2.x executable. .. [#] A problem with the GNU Readline package may prevent this. - diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index 3e42cee174..4d67677ace 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -483,6 +483,12 @@ concatenated and so on:: >>> 3*a[:3] + ['Boo!'] ['spam', 'eggs', 100, 'spam', 'eggs', 100, 'spam', 'eggs', 100, 'Boo!'] +All slice operations return a new list containing the requested elements. This +means that the following slice returns a shallow copy of the list *a*:: + + >>> a[:] + ['spam', 'eggs', 100, 1234] + Unlike strings, which are *immutable*, it is possible to change individual elements of a list:: @@ -587,13 +593,13 @@ This example introduces several new features. and ``!=`` (not equal to). * The *body* of the loop is *indented*: indentation is Python's way of grouping - statements. Python does not (yet!) provide an intelligent input line editing - facility, so you have to type a tab or space(s) for each indented line. In - practice you will prepare more complicated input for Python with a text editor; - most text editors have an auto-indent facility. When a compound statement is - entered interactively, it must be followed by a blank line to indicate - completion (since the parser cannot guess when you have typed the last line). - Note that each line within a basic block must be indented by the same amount. + statements. At the interactive prompt, you have to type a tab or space(s) for + each indented line. In practice you will prepare more complicated input + for Python with a text editor; all decent text editors have an auto-indent + facility. When a compound statement is entered interactively, it must be + followed by a blank line to indicate completion (since the parser cannot + guess when you have typed the last line). Note that each line within a basic + block must be indented by the same amount. * The :func:`print` function writes the value of the expression(s) it is given. It differs from just writing the expression you want to write (as we did diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index d4bfbda811..e5adb05b85 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -159,23 +159,22 @@ The Module Search Path .. index:: triple: module; search; path -When a module named :mod:`spam` is imported, the interpreter searches for a file -named :file:`spam.py` in the current directory, and then in the list of -directories specified by the environment variable :envvar:`PYTHONPATH`. This -has the same syntax as the shell variable :envvar:`PATH`, that is, a list of -directory names. When :envvar:`PYTHONPATH` is not set, or when the file is not -found there, the search continues in an installation-dependent default path; on -Unix, this is usually :file:`.:/usr/local/lib/python`. - -Actually, modules are searched in the list of directories given by the variable -``sys.path`` which is initialized from the directory containing the input script -(or the current directory), :envvar:`PYTHONPATH` and the installation- dependent -default. This allows Python programs that know what they're doing to modify or -replace the module search path. Note that because the directory containing the -script being run is on the search path, it is important that the script not have -the same name as a standard module, or Python will attempt to load the script as -a module when that module is imported. This will generally be an error. See -section :ref:`tut-standardmodules` for more information. +When a module named :mod:`spam` is imported, the interpreter first searches for +a built-in module with that name. If not found, it then searches for a file +named :file:`spam.py` in a list of directories given by the variable +:data:`sys.path`. :data:`sys.path` is initialized from these locations: + +* the directory containing the input script (or the current directory). +* :envvar:`PYTHONPATH` (a list of directory names, with the same syntax as the + shell variable :envvar:`PATH`). +* the installation-dependent default. + +After initialization, Python programs can modify :data:`sys.path`. The +directory containing the script being run is placed at the beginning of the +search path, ahead of the standard library path. This means that scripts in that +directory will be loaded instead of modules of the same name in the library +directory. This is an error unless the replacement is intended. See section +:ref:`tut-standardmodules` for more information. .. % Do we need stuff on zip files etc. ? DUBOIS diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst index b138f65a7e..97297434ed 100644 --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -72,7 +72,7 @@ three`` at the command line:: The :mod:`getopt` module processes *sys.argv* using the conventions of the Unix :func:`getopt` function. More powerful and flexible command line processing is -provided by the :mod:`optparse` module. +provided by the :mod:`argparse` module. .. _tut-stderr: diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 11e2d7d9c0..d0b330d9d5 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1,5 +1,8 @@ .. highlightlang:: none +.. ATTENTION: You probably should update Misc/python.man, too, if you modify +.. this file. + .. _using-on-general: Command line and environment @@ -21,7 +24,7 @@ Command line When invoking Python, you may specify any of these options:: - python [-bBdEhiORsSuvVWx?] [-c command | -m module-name | script | - ] [args] + python [-bBdEhiORqsSuvVWx?] [-c command | -m module-name | script | - ] [args] The most common use case is, of course, a simple invocation of a script:: @@ -92,8 +95,9 @@ source. file is not available. If this option is given, the first element of :data:`sys.argv` will be the - full path to the module file. As with the :option:`-c` option, the current - directory will be added to the start of :data:`sys.path`. + full path to the module file (while the module file is being located, the + first element will be set to ``"-m"``). As with the :option:`-c` option, + the current directory will be added to the start of :data:`sys.path`. Many standard library modules contain code that is invoked on their execution as a script. An example is the :mod:`timeit` module:: @@ -111,6 +115,7 @@ source. .. versionchanged:: 3.1 Supply the package name to run a ``__main__`` submodule. + .. describe:: - Read commands from standard input (:data:`sys.stdin`). If standard input is @@ -215,6 +220,13 @@ Miscellaneous options Discard docstrings in addition to the :option:`-O` optimizations. +.. cmdoption:: -q + + Don't display the copyright and version messages even in interactive mode. + + .. versionadded:: 3.2 + + .. cmdoption:: -R Turn on hash randomization, so that the :meth:`__hash__` values of str, bytes @@ -235,12 +247,13 @@ Miscellaneous options See also :envvar:`PYTHONHASHSEED`. - .. versionadded:: 3.1.5 + .. versionadded:: 3.2.3 .. cmdoption:: -s - Don't add user site directory to sys.path + Don't add the :data:`user site-packages directory <site.USER_SITE>` to + :data:`sys.path`. .. seealso:: @@ -257,7 +270,8 @@ Miscellaneous options Force the binary layer of the stdin, stdout and stderr streams (which is available as their ``buffer`` attribute) to be unbuffered. The text I/O - layer will still be line-buffered. + layer will still be line-buffered if writing to the console, or + block-buffered if redirected to a non-interactive file. See also :envvar:`PYTHONUNBUFFERED`. @@ -317,7 +331,7 @@ Miscellaneous options the remaining fields. Empty fields match all values; trailing empty fields may be omitted. The *message* field matches the start of the warning message printed; this match is case-insensitive. The *category* field matches the - warning category. This must be a class name; the match test whether the + warning category. This must be a class name; the match tests whether the actual warning category of the message is a subclass of the specified warning category. The full class name must be given. The *module* field matches the (fully-qualified) module name; this match is case-sensitive. The *line* @@ -329,6 +343,8 @@ Miscellaneous options :pep:`230` -- Warning framework + :envvar:`PYTHONWARNINGS` + .. cmdoption:: -x @@ -338,6 +354,16 @@ Miscellaneous options .. note:: The line numbers in error messages will be off by one. +.. cmdoption:: -X + + Reserved for various implementation-specific options. CPython currently + defines none of them, but allows to pass arbitrary values and retrieve + them through the :data:`sys._xoptions` dictionary. + + .. versionchanged:: 3.2 + It is now allowed to pass :option:`-X` with CPython. + + Options you shouldn't use ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -347,11 +373,6 @@ Options you shouldn't use .. _Jython: http://jython.org -.. cmdoption:: -X - - Reserved for alternative implementations of Python to use for their own - purposes. - .. _using-on-envvars: @@ -451,13 +472,14 @@ These environment variables influence Python's behavior. .. envvar:: PYTHONCASEOK If this is set, Python ignores case in :keyword:`import` statements. This - only works on Windows. + only works on Windows, OS X, and OS/2. .. envvar:: PYTHONDONTWRITEBYTECODE If this is set, Python won't try to write ``.pyc`` or ``.pyo`` files on the - import of source modules. + import of source modules. This is equivalent to specifying the :option:`-B` + option. .. envvar:: PYTHONHASHSEED @@ -478,14 +500,15 @@ These environment variables influence Python's behavior. the value 0 will lead to the same hash values as when hash randomization is disabled. - .. versionadded:: 3.1.5 + .. versionadded:: 3.2.3 .. envvar:: PYTHONIOENCODING - Overrides the encoding used for stdin/stdout/stderr, in the syntax - ``encodingname:errorhandler``. The ``:errorhandler`` part is optional and - has the same meaning as in :func:`str.encode`. + If this is set before running the interpreter, it overrides the encoding used + for stdin/stdout/stderr, in the syntax ``encodingname:errorhandler``. The + ``:errorhandler`` part is optional and has the same meaning as in + :func:`str.encode`. For stderr, the ``:errorhandler`` part is ignored; the handler will always be ``'backslashreplace'``. @@ -493,7 +516,8 @@ These environment variables influence Python's behavior. .. envvar:: PYTHONNOUSERSITE - If this is set, Python won't add the user site directory to sys.path + If this is set, Python won't add the :data:`user site-packages directory + <site.USER_SITE>` to :data:`sys.path`. .. seealso:: @@ -502,7 +526,10 @@ These environment variables influence Python's behavior. .. envvar:: PYTHONUSERBASE - Sets the base directory for the user site directory + Defines the :data:`user base directory <site.USER_BASE>`, which is used to + compute the path of the :data:`user site-packages directory <site.USER_SITE>` + and :ref:`Distutils installation paths <inst-alt-install-user>` for ``python + setup.py install --user``. .. seealso:: @@ -515,12 +542,18 @@ These environment variables influence Python's behavior. value instead of the value got through the C runtime. Only works on Mac OS X. +.. envvar:: PYTHONWARNINGS + + This is equivalent to the :option:`-W` option. If set to a comma + separated string, it is equivalent to specifying :option:`-W` multiple + times. + Debug-mode variables ~~~~~~~~~~~~~~~~~~~~ Setting these variables only has an effect in a debug build of Python, that is, -if Python was configured with the :option:`--with-pydebug` build option. +if Python was configured with the ``--with-pydebug`` build option. .. envvar:: PYTHONTHREADDEBUG diff --git a/Doc/using/unix.rst b/Doc/using/unix.rst index 61e707b11c..8bbc31269a 100644 --- a/Doc/using/unix.rst +++ b/Doc/using/unix.rst @@ -1,4 +1,4 @@ -.. highlightlang:: none +.. highlightlang:: sh .. _using-on-unix: @@ -26,11 +26,11 @@ following links: .. seealso:: - http://www.linux.com/articles/60383 + http://www.debian.org/doc/manuals/maint-guide/first.en.html for Debian users http://linuxmafia.com/pub/linux/suse-linux-internals/chapter35.html for OpenSuse users - http://docs.fedoraproject.org/drafts/rpm-guide-en/ch-creating-rpms.html + http://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch-creating-rpms.html for Fedora users http://www.slackbook.org/html/package-management-making-packages.html for Slackware users @@ -55,8 +55,8 @@ On FreeBSD and OpenBSD On OpenSolaris -------------- -To install the newest Python versions on OpenSolaris, install blastwave -(http://www.blastwave.org/howto.html) and type "pkg_get -i python" at the +To install the newest Python versions on OpenSolaris, install `blastwave +<http://www.blastwave.org/howto.html>`_ and type ``pkg_get -i python`` at the prompt. @@ -65,22 +65,23 @@ Building Python If you want to compile CPython yourself, first thing you should do is get the `source <http://python.org/download/source/>`_. You can download either the -latest release's source or just grab a fresh `checkout -<http://www.python.org/dev/faq/#how-do-i-get-a-checkout-of-the-repository-read-only-and-read-write>`_. +latest release's source or just grab a fresh `clone +<http://docs.python.org/devguide/setup#getting-the-source-code>`_. (If you want +to contribute patches, you will need a clone.) -The build process consists the usual :: +The build process consists in the usual :: ./configure make make install invocations. Configuration options and caveats for specific Unix platforms are -extensively documented in the :file:`README` file in the root of the Python +extensively documented in the :source:`README` file in the root of the Python source tree. .. warning:: - ``make install`` can overwrite or masquerade the :file:`python` binary. + ``make install`` can overwrite or masquerade the :file:`python3` binary. ``make altinstall`` is therefore recommended instead of ``make install`` since it only installs :file:`{exec_prefix}/bin/python{version}`. @@ -98,7 +99,7 @@ For example, on most Linux systems, the default for both is :file:`/usr`. +-----------------------------------------------+------------------------------------------+ | File/directory | Meaning | +===============================================+==========================================+ -| :file:`{exec_prefix}/bin/python` | Recommended location of the interpreter. | +| :file:`{exec_prefix}/bin/python3` | Recommended location of the interpreter. | +-----------------------------------------------+------------------------------------------+ | :file:`{prefix}/lib/python{version}`, | Recommended locations of the directories | | :file:`{exec_prefix}/lib/python{version}` | containing the standard modules. | @@ -108,10 +109,6 @@ For example, on most Linux systems, the default for both is :file:`/usr`. | | developing Python extensions and | | | embedding the interpreter. | +-----------------------------------------------+------------------------------------------+ -| :file:`~/.pythonrc.py` | User-specific initialization file loaded | -| | by the user module; not used by default | -| | or by most applications. | -+-----------------------------------------------+------------------------------------------+ Miscellaneous @@ -125,11 +122,11 @@ e.g. with :: and put an appropriate Shebang line at the top of the script. A good choice is usually :: - #!/usr/bin/env python + #!/usr/bin/env python3 which searches for the Python interpreter in the whole :envvar:`PATH`. However, some Unices may not have the :program:`env` command, so you may need to hardcode -``/usr/bin/python`` as the interpreter path. +``/usr/bin/python3`` as the interpreter path. To use shell commands in your Python scripts, look at the :mod:`subprocess` module. diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 12378b3233..742a2900a8 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -45,9 +45,9 @@ for detailed information about platforms with precompiled installers. "7 Minutes to "Hello World!"" by Richard Dooling, 2006 - `Installing on Windows <http://diveintopython.org/installing_python/windows.html>`_ + `Installing on Windows <http://diveintopython.net/installing_python/windows.html>`_ in "`Dive into Python: Python from novice to pro - <http://diveintopython.org/index.html>`_" + <http://diveintopython.net/index.html>`_" by Mark Pilgrim, 2004, ISBN 1-59059-356-1 @@ -290,7 +290,7 @@ Compiling Python on Windows If you want to compile CPython yourself, first thing you should do is get the `source <http://python.org/download/source/>`_. You can download either the latest release's source or just grab a fresh `checkout -<http://www.python.org/dev/faq/#how-do-i-get-a-checkout-of-the-repository-read-only-and-read-write>`_. +<http://docs.python.org/devguide/setup#checking-out-the-code>`_. For Microsoft Visual C++, which is the compiler with which official Python releases are built, the source tree contains solutions/project files. View the diff --git a/Doc/whatsnew/2.0.rst b/Doc/whatsnew/2.0.rst index 3bb12b666c..850e57d4a6 100644 --- a/Doc/whatsnew/2.0.rst +++ b/Doc/whatsnew/2.0.rst @@ -189,7 +189,7 @@ support Unicode: ignored and ``'replace'`` uses U+FFFD, the official replacement character, in case of any problems. -* The :keyword:`exec` statement, and various built-ins such as ``eval()``, +* The ``exec`` statement, and various built-ins such as ``eval()``, ``getattr()``, and ``setattr()`` will also accept Unicode strings as well as regular strings. (It's possible that the process of fixing this missed some built-ins; if you find a built-in function that accepts strings but doesn't @@ -515,11 +515,11 @@ functions:: # kw is a dictionary of keyword args ... -The :keyword:`print` statement can now have its output directed to a file-like -object by following the :keyword:`print` with ``>> file``, similar to the +The ``print`` statement can now have its output directed to a file-like +object by following the ``print`` with ``>> file``, similar to the redirection operator in Unix shells. Previously you'd either have to use the :meth:`write` method of the file-like object, which lacks the convenience and -simplicity of :keyword:`print`, or you could assign a new value to +simplicity of ``print``, or you could assign a new value to ``sys.stdout`` and then restore the old value. For sending output to standard error, it's much easier to write this:: @@ -581,7 +581,7 @@ Consult the README in the Python source distribution for more instructions. An attempt has been made to alleviate one of Python's warts, the often-confusing :exc:`NameError` exception when code refers to a local variable before the variable has been assigned a value. For example, the following code raises an -exception on the :keyword:`print` statement in both 1.5.2 and 2.0; in 1.5.2 a +exception on the ``print`` statement in both 1.5.2 and 2.0; in 1.5.2 a :exc:`NameError` exception is raised, while 2.0 raises a new :exc:`UnboundLocalError` exception. :exc:`UnboundLocalError` is a subclass of :exc:`NameError`, so any existing code that expects :exc:`NameError` to be diff --git a/Doc/whatsnew/2.1.rst b/Doc/whatsnew/2.1.rst index 7f405213db..117af10c4f 100644 --- a/Doc/whatsnew/2.1.rst +++ b/Doc/whatsnew/2.1.rst @@ -81,13 +81,13 @@ though, since such code would have been pretty confusing to read in the first place. One side effect of the change is that the ``from module import *`` and -:keyword:`exec` statements have been made illegal inside a function scope under +``exec`` statements have been made illegal inside a function scope under certain conditions. The Python reference manual has said all along that ``from module import *`` is only legal at the top level of a module, but the CPython interpreter has never enforced this before. As part of the implementation of nested scopes, the compiler which turns Python source into bytecodes has to generate different code to access variables in a containing scope. ``from -module import *`` and :keyword:`exec` make it impossible for the compiler to +module import *`` and ``exec`` make it impossible for the compiler to figure this out, because they add names to the local namespace that are unknowable at compile time. Therefore, if a function contains function definitions or :keyword:`lambda` expressions with free variables, the compiler @@ -102,11 +102,11 @@ To make the preceding explanation a bit clearer, here's an example:: def g(): return x -Line 4 containing the :keyword:`exec` statement is a syntax error, since -:keyword:`exec` would define a new local variable named ``x`` whose value should +Line 4 containing the ``exec`` statement is a syntax error, since +``exec`` would define a new local variable named ``x`` whose value should be accessed by :func:`g`. -This shouldn't be much of a limitation, since :keyword:`exec` is rarely used in +This shouldn't be much of a limitation, since ``exec`` is rarely used in most Python code (and when it is used, it's often a sign of a poor design anyway). diff --git a/Doc/whatsnew/2.2.rst b/Doc/whatsnew/2.2.rst index 89e4d76bef..1db1ee7dfd 100644 --- a/Doc/whatsnew/2.2.rst +++ b/Doc/whatsnew/2.2.rst @@ -754,7 +754,7 @@ Here are the changes 2.2 introduces: * Classes can define methods called :meth:`__truediv__` and :meth:`__floordiv__` to overload the two division operators. At the C level, there are also slots in - the :ctype:`PyNumberMethods` structure so extension types can define the two + the :c:type:`PyNumberMethods` structure so extension types can define the two operators. * Python 2.2 supports some command-line arguments for testing whether code will @@ -892,13 +892,13 @@ though, since such code would have been pretty confusing to read in the first place. One side effect of the change is that the ``from module import *`` and -:keyword:`exec` statements have been made illegal inside a function scope under +``exec`` statements have been made illegal inside a function scope under certain conditions. The Python reference manual has said all along that ``from module import *`` is only legal at the top level of a module, but the CPython interpreter has never enforced this before. As part of the implementation of nested scopes, the compiler which turns Python source into bytecodes has to generate different code to access variables in a containing scope. ``from -module import *`` and :keyword:`exec` make it impossible for the compiler to +module import *`` and ``exec`` make it impossible for the compiler to figure this out, because they add names to the local namespace that are unknowable at compile time. Therefore, if a function contains function definitions or :keyword:`lambda` expressions with free variables, the compiler @@ -913,11 +913,11 @@ To make the preceding explanation a bit clearer, here's an example:: def g(): return x -Line 4 containing the :keyword:`exec` statement is a syntax error, since -:keyword:`exec` would define a new local variable named ``x`` whose value should +Line 4 containing the ``exec`` statement is a syntax error, since +``exec`` would define a new local variable named ``x`` whose value should be accessed by :func:`g`. -This shouldn't be much of a limitation, since :keyword:`exec` is rarely used in +This shouldn't be much of a limitation, since ``exec`` is rarely used in most Python code (and when it is used, it's often a sign of a poor design anyway). @@ -983,7 +983,7 @@ New and Improved Modules Jun-ichiro "itojun" Hagino.) * Two new format characters were added to the :mod:`struct` module for 64-bit - integers on platforms that support the C :ctype:`long long` type. ``q`` is for + integers on platforms that support the C :c:type:`long long` type. ``q`` is for a signed 64-bit integer, and ``Q`` is for an unsigned one. The value is returned in Python's long integer type. (Contributed by Tim Peters.) @@ -1057,16 +1057,16 @@ code, none of the changes described here will affect you very much. at much higher speeds than Python-based functions and should reduce the overhead of profiling and tracing. This will be of interest to authors of development environments for Python. Two new C functions were added to Python's API, - :cfunc:`PyEval_SetProfile` and :cfunc:`PyEval_SetTrace`. The existing + :c:func:`PyEval_SetProfile` and :c:func:`PyEval_SetTrace`. The existing :func:`sys.setprofile` and :func:`sys.settrace` functions still exist, and have simply been changed to use the new C-level interface. (Contributed by Fred L. Drake, Jr.) * Another low-level API, primarily of interest to implementors of Python - debuggers and development tools, was added. :cfunc:`PyInterpreterState_Head` and - :cfunc:`PyInterpreterState_Next` let a caller walk through all the existing - interpreter objects; :cfunc:`PyInterpreterState_ThreadHead` and - :cfunc:`PyThreadState_Next` allow looping over all the thread states for a given + debuggers and development tools, was added. :c:func:`PyInterpreterState_Head` and + :c:func:`PyInterpreterState_Next` let a caller walk through all the existing + interpreter objects; :c:func:`PyInterpreterState_ThreadHead` and + :c:func:`PyThreadState_Next` allow looping over all the thread states for a given interpreter. (Contributed by David Beazley.) * The C-level interface to the garbage collector has been changed to make it @@ -1078,19 +1078,19 @@ code, none of the changes described here will affect you very much. To upgrade an extension module to the new API, perform the following steps: -* Rename :cfunc:`Py_TPFLAGS_GC` to :cfunc:`PyTPFLAGS_HAVE_GC`. +* Rename :c:func:`Py_TPFLAGS_GC` to :c:func:`PyTPFLAGS_HAVE_GC`. -* Use :cfunc:`PyObject_GC_New` or :cfunc:`PyObject_GC_NewVar` to allocate - objects, and :cfunc:`PyObject_GC_Del` to deallocate them. +* Use :c:func:`PyObject_GC_New` or :c:func:`PyObject_GC_NewVar` to allocate + objects, and :c:func:`PyObject_GC_Del` to deallocate them. -* Rename :cfunc:`PyObject_GC_Init` to :cfunc:`PyObject_GC_Track` and - :cfunc:`PyObject_GC_Fini` to :cfunc:`PyObject_GC_UnTrack`. +* Rename :c:func:`PyObject_GC_Init` to :c:func:`PyObject_GC_Track` and + :c:func:`PyObject_GC_Fini` to :c:func:`PyObject_GC_UnTrack`. -* Remove :cfunc:`PyGC_HEAD_SIZE` from object size calculations. +* Remove :c:func:`PyGC_HEAD_SIZE` from object size calculations. -* Remove calls to :cfunc:`PyObject_AS_GC` and :cfunc:`PyObject_FROM_GC`. +* Remove calls to :c:func:`PyObject_AS_GC` and :c:func:`PyObject_FROM_GC`. -* A new ``et`` format sequence was added to :cfunc:`PyArg_ParseTuple`; ``et`` +* A new ``et`` format sequence was added to :c:func:`PyArg_ParseTuple`; ``et`` takes both a parameter and an encoding name, and converts the parameter to the given encoding if the parameter turns out to be a Unicode string, or leaves it alone if it's an 8-bit string, assuming it to already be in the desired @@ -1099,10 +1099,10 @@ code, none of the changes described here will affect you very much. specified new encoding. (Contributed by M.-A. Lemburg, and used for the MBCS support on Windows described in the following section.) -* A different argument parsing function, :cfunc:`PyArg_UnpackTuple`, has been +* A different argument parsing function, :c:func:`PyArg_UnpackTuple`, has been added that's simpler and presumably faster. Instead of specifying a format string, the caller simply gives the minimum and maximum number of arguments - expected, and a set of pointers to :ctype:`PyObject\*` variables that will be + expected, and a set of pointers to :c:type:`PyObject\*` variables that will be filled in with argument values. * Two new flags :const:`METH_NOARGS` and :const:`METH_O` are available in method @@ -1111,14 +1111,14 @@ code, none of the changes described here will affect you very much. corresponding method that uses :const:`METH_VARARGS`. Also, the old :const:`METH_OLDARGS` style of writing C methods is now officially deprecated. -* Two new wrapper functions, :cfunc:`PyOS_snprintf` and :cfunc:`PyOS_vsnprintf` +* Two new wrapper functions, :c:func:`PyOS_snprintf` and :c:func:`PyOS_vsnprintf` were added to provide cross-platform implementations for the relatively new - :cfunc:`snprintf` and :cfunc:`vsnprintf` C lib APIs. In contrast to the standard - :cfunc:`sprintf` and :cfunc:`vsprintf` functions, the Python versions check the + :c:func:`snprintf` and :c:func:`vsnprintf` C lib APIs. In contrast to the standard + :c:func:`sprintf` and :c:func:`vsprintf` functions, the Python versions check the bounds of the buffer used to protect against buffer overruns. (Contributed by M.-A. Lemburg.) -* The :cfunc:`_PyTuple_Resize` function has lost an unused parameter, so now it +* The :c:func:`_PyTuple_Resize` function has lost an unused parameter, so now it takes 2 parameters instead of 3. The third argument was never used, and can simply be discarded when porting code from earlier versions to Python 2.2. @@ -1219,7 +1219,7 @@ Some of the more notable changes are: operator, but these features were rarely used and therefore buggy. The :meth:`tolist` method and the :attr:`start`, :attr:`stop`, and :attr:`step` attributes are also being deprecated. At the C level, the fourth argument to - the :cfunc:`PyRange_New` function, ``repeat``, has also been deprecated. + the :c:func:`PyRange_New` function, ``repeat``, has also been deprecated. * There were a bunch of patches to the dictionary implementation, mostly to fix potential core dumps if a dictionary contains objects that sneakily changed @@ -1242,8 +1242,8 @@ Some of the more notable changes are: up to display the output. This patch makes it possible to import such scripts, in case they're also usable as modules. (Implemented by David Bolen.) -* On platforms where Python uses the C :cfunc:`dlopen` function to load - extension modules, it's now possible to set the flags used by :cfunc:`dlopen` +* On platforms where Python uses the C :c:func:`dlopen` function to load + extension modules, it's now possible to set the flags used by :c:func:`dlopen` using the :func:`sys.getdlopenflags` and :func:`sys.setdlopenflags` functions. (Contributed by Bram Stolk.) diff --git a/Doc/whatsnew/2.3.rst b/Doc/whatsnew/2.3.rst index 3894f8744a..0cc29f608b 100644 --- a/Doc/whatsnew/2.3.rst +++ b/Doc/whatsnew/2.3.rst @@ -1797,8 +1797,8 @@ Pymalloc: A Specialized Object Allocator Pymalloc, a specialized object allocator written by Vladimir Marangozov, was a feature added to Python 2.1. Pymalloc is intended to be faster than the system -:cfunc:`malloc` and to have less memory overhead for allocation patterns typical -of Python programs. The allocator uses C's :cfunc:`malloc` function to get large +:c:func:`malloc` and to have less memory overhead for allocation patterns typical +of Python programs. The allocator uses C's :c:func:`malloc` function to get large pools of memory and then fulfills smaller memory requests from these pools. In 2.1 and 2.2, pymalloc was an experimental feature and wasn't enabled by @@ -1814,13 +1814,13 @@ runtime. There's one particularly common error that causes problems. There are a number of memory allocation functions in Python's C API that have previously just been -aliases for the C library's :cfunc:`malloc` and :cfunc:`free`, meaning that if +aliases for the C library's :c:func:`malloc` and :c:func:`free`, meaning that if you accidentally called mismatched functions the error wouldn't be noticeable. When the object allocator is enabled, these functions aren't aliases of -:cfunc:`malloc` and :cfunc:`free` any more, and calling the wrong function to +:c:func:`malloc` and :c:func:`free` any more, and calling the wrong function to free memory may get you a core dump. For example, if memory was allocated using -:cfunc:`PyObject_Malloc`, it has to be freed using :cfunc:`PyObject_Free`, not -:cfunc:`free`. A few modules included with Python fell afoul of this and had to +:c:func:`PyObject_Malloc`, it has to be freed using :c:func:`PyObject_Free`, not +:c:func:`free`. A few modules included with Python fell afoul of this and had to be fixed; doubtless there are more third-party modules that will have the same problem. @@ -1831,14 +1831,14 @@ one family for allocating chunks of memory and another family of functions specifically for allocating Python objects. * To allocate and free an undistinguished chunk of memory use the "raw memory" - family: :cfunc:`PyMem_Malloc`, :cfunc:`PyMem_Realloc`, and :cfunc:`PyMem_Free`. + family: :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, and :c:func:`PyMem_Free`. * The "object memory" family is the interface to the pymalloc facility described above and is biased towards a large number of "small" allocations: - :cfunc:`PyObject_Malloc`, :cfunc:`PyObject_Realloc`, and :cfunc:`PyObject_Free`. + :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`, and :c:func:`PyObject_Free`. * To allocate and free Python objects, use the "object" family - :cfunc:`PyObject_New`, :cfunc:`PyObject_NewVar`, and :cfunc:`PyObject_Del`. + :c:func:`PyObject_New`, :c:func:`PyObject_NewVar`, and :c:func:`PyObject_Del`. Thanks to lots of work by Tim Peters, pymalloc in 2.3 also provides debugging features to catch memory overwrites and doubled frees in both extension modules @@ -1877,10 +1877,10 @@ Changes to Python's build process and to the C API include: (:file:`libpython2.3.so`) by supplying :option:`--enable-shared` when running Python's :program:`configure` script. (Contributed by Ondrej Palkovsky.) -* The :cmacro:`DL_EXPORT` and :cmacro:`DL_IMPORT` macros are now deprecated. +* The :c:macro:`DL_EXPORT` and :c:macro:`DL_IMPORT` macros are now deprecated. Initialization functions for Python extension modules should now be declared - using the new macro :cmacro:`PyMODINIT_FUNC`, while the Python core will - generally use the :cmacro:`PyAPI_FUNC` and :cmacro:`PyAPI_DATA` macros. + using the new macro :c:macro:`PyMODINIT_FUNC`, while the Python core will + generally use the :c:macro:`PyAPI_FUNC` and :c:macro:`PyAPI_DATA` macros. * The interpreter can be compiled without any docstrings for the built-in functions and modules by supplying :option:`--without-doc-strings` to the @@ -1888,19 +1888,19 @@ Changes to Python's build process and to the C API include: but will also mean that you can't get help for Python's built-ins. (Contributed by Gustavo Niemeyer.) -* The :cfunc:`PyArg_NoArgs` macro is now deprecated, and code that uses it +* The :c:func:`PyArg_NoArgs` macro is now deprecated, and code that uses it should be changed. For Python 2.2 and later, the method definition table can specify the :const:`METH_NOARGS` flag, signalling that there are no arguments, and the argument checking can then be removed. If compatibility with pre-2.2 versions of Python is important, the code could use ``PyArg_ParseTuple(args, "")`` instead, but this will be slower than using :const:`METH_NOARGS`. -* :cfunc:`PyArg_ParseTuple` accepts new format characters for various sizes of - unsigned integers: ``B`` for :ctype:`unsigned char`, ``H`` for :ctype:`unsigned - short int`, ``I`` for :ctype:`unsigned int`, and ``K`` for :ctype:`unsigned +* :c:func:`PyArg_ParseTuple` accepts new format characters for various sizes of + unsigned integers: ``B`` for :c:type:`unsigned char`, ``H`` for :c:type:`unsigned + short int`, ``I`` for :c:type:`unsigned int`, and ``K`` for :c:type:`unsigned long long`. -* A new function, :cfunc:`PyObject_DelItemString(mapping, char \*key)` was added +* A new function, :c:func:`PyObject_DelItemString(mapping, char \*key)` was added as shorthand for ``PyObject_DelItem(mapping, PyString_New(key))``. * File objects now manage their internal string buffer differently, increasing @@ -1910,7 +1910,7 @@ Changes to Python's build process and to the C API include: * It's now possible to define class and static methods for a C extension type by setting either the :const:`METH_CLASS` or :const:`METH_STATIC` flags in a - method's :ctype:`PyMethodDef` structure. + method's :c:type:`PyMethodDef` structure. * Python now includes a copy of the Expat XML parser's source code, removing any dependence on a system version or local installation of Expat. diff --git a/Doc/whatsnew/2.4.rst b/Doc/whatsnew/2.4.rst index fadde50184..faf92704a5 100644 --- a/Doc/whatsnew/2.4.rst +++ b/Doc/whatsnew/2.4.rst @@ -469,7 +469,7 @@ PEP 327: Decimal Data Type ========================== Python has always supported floating-point (FP) numbers, based on the underlying -C :ctype:`double` type, as a data type. However, while most programming +C :c:type:`double` type, as a data type. However, while most programming languages provide a floating-point type, many people (even programmers) are unaware that floating-point numbers don't represent certain decimal fractions accurately. The new :class:`Decimal` type can represent these fractions @@ -498,7 +498,7 @@ mantissa is multiplied by 4 (2 to the power of the exponent 2); 1.25 \* 4 equals 5. Modern systems usually provide floating-point support that conforms to a -standard called IEEE 754. C's :ctype:`double` type is usually implemented as a +standard called IEEE 754. C's :c:type:`double` type is usually implemented as a 64-bit IEEE 754 number, which uses 52 bits of space for the mantissa. This means that numbers can only be specified to 52 bits of precision. If you're trying to represent numbers whose expansion repeats endlessly, the expansion is @@ -736,7 +736,7 @@ display conventions that are localized to a particular country or language. However, the module was careful to not change the numeric locale because various functions in Python's implementation required that the numeric locale remain set to the ``'C'`` locale. Often this was because the code was using the C -library's :cfunc:`atof` function. +library's :c:func:`atof` function. Not setting the numeric locale caused trouble for extensions that used third- party C libraries, however, because they wouldn't have the correct locale set. @@ -746,11 +746,11 @@ numbers in the current locale. The solution described in the PEP is to add three new functions to the Python API that perform ASCII-only conversions, ignoring the locale setting: -* :cfunc:`PyOS_ascii_strtod(str, ptr)` and :cfunc:`PyOS_ascii_atof(str, ptr)` - both convert a string to a C :ctype:`double`. +* :c:func:`PyOS_ascii_strtod(str, ptr)` and :c:func:`PyOS_ascii_atof(str, ptr)` + both convert a string to a C :c:type:`double`. -* :cfunc:`PyOS_ascii_formatd(buffer, buf_len, format, d)` converts a - :ctype:`double` to an ASCII string. +* :c:func:`PyOS_ascii_formatd(buffer, buf_len, format, d)` converts a + :c:type:`double` to an ASCII string. The code for these functions came from the GLib library (http://library.gnome.org/devel/glib/stable/), whose developers kindly @@ -896,7 +896,7 @@ Here are all of the changes that Python 2.4 makes to the core Python language. (Contributed by Nick Coghlan.) * The :func:`eval(expr, globals, locals)` and :func:`execfile(filename, globals, - locals)` functions and the :keyword:`exec` statement now accept any mapping type + locals)` functions and the ``exec`` statement now accept any mapping type for the *locals* parameter. Previously this had to be a regular Python dictionary. (Contributed by Raymond Hettinger.) @@ -938,7 +938,7 @@ Optimizations * The machinery for growing and shrinking lists was optimized for speed and for space efficiency. Appending and popping from lists now runs faster due to more efficient code paths and less frequent use of the underlying system - :cfunc:`realloc`. List comprehensions also benefit. :meth:`list.extend` was + :c:func:`realloc`. List comprehensions also benefit. :meth:`list.extend` was also optimized and no longer converts its argument into a temporary list before extending the base list. (Contributed by Raymond Hettinger.) @@ -947,7 +947,7 @@ Optimizations :meth:`__len__` method. (Contributed by Raymond Hettinger.) * The methods :meth:`list.__getitem__`, :meth:`dict.__getitem__`, and - :meth:`dict.__contains__` are are now implemented as :class:`method_descriptor` + :meth:`dict.__contains__` are now implemented as :class:`method_descriptor` objects rather than :class:`wrapper_descriptor` objects. This form of access doubles their performance and makes them more suitable for use as arguments to functionals: ``map(mydict.__getitem__, keylist)``. (Contributed by Raymond @@ -1445,34 +1445,34 @@ Build and C API Changes Some of the changes to Python's build process and to the C API are: * Three new convenience macros were added for common return values from - extension functions: :cmacro:`Py_RETURN_NONE`, :cmacro:`Py_RETURN_TRUE`, and - :cmacro:`Py_RETURN_FALSE`. (Contributed by Brett Cannon.) + extension functions: :c:macro:`Py_RETURN_NONE`, :c:macro:`Py_RETURN_TRUE`, and + :c:macro:`Py_RETURN_FALSE`. (Contributed by Brett Cannon.) -* Another new macro, :cmacro:`Py_CLEAR(obj)`, decreases the reference count of +* Another new macro, :c:macro:`Py_CLEAR(obj)`, decreases the reference count of *obj* and sets *obj* to the null pointer. (Contributed by Jim Fulton.) -* A new function, :cfunc:`PyTuple_Pack(N, obj1, obj2, ..., objN)`, constructs +* A new function, :c:func:`PyTuple_Pack(N, obj1, obj2, ..., objN)`, constructs tuples from a variable length argument list of Python objects. (Contributed by Raymond Hettinger.) -* A new function, :cfunc:`PyDict_Contains(d, k)`, implements fast dictionary +* A new function, :c:func:`PyDict_Contains(d, k)`, implements fast dictionary lookups without masking exceptions raised during the look-up process. (Contributed by Raymond Hettinger.) -* The :cmacro:`Py_IS_NAN(X)` macro returns 1 if its float or double argument +* The :c:macro:`Py_IS_NAN(X)` macro returns 1 if its float or double argument *X* is a NaN. (Contributed by Tim Peters.) * C code can avoid unnecessary locking by using the new - :cfunc:`PyEval_ThreadsInitialized` function to tell if any thread operations + :c:func:`PyEval_ThreadsInitialized` function to tell if any thread operations have been performed. If this function returns false, no lock operations are needed. (Contributed by Nick Coghlan.) -* A new function, :cfunc:`PyArg_VaParseTupleAndKeywords`, is the same as - :cfunc:`PyArg_ParseTupleAndKeywords` but takes a :ctype:`va_list` instead of a +* A new function, :c:func:`PyArg_VaParseTupleAndKeywords`, is the same as + :c:func:`PyArg_ParseTupleAndKeywords` but takes a :c:type:`va_list` instead of a number of arguments. (Contributed by Greg Chapman.) * A new method flag, :const:`METH_COEXISTS`, allows a function defined in slots - to co-exist with a :ctype:`PyCFunction` having the same name. This can halve + to co-exist with a :c:type:`PyCFunction` having the same name. This can halve the access time for a method such as :meth:`set.__contains__`. (Contributed by Raymond Hettinger.) @@ -1486,8 +1486,8 @@ Some of the changes to Python's build process and to the C API are: though that processor architecture doesn't call that register "the TSC register". (Contributed by Jeremy Hylton.) -* The :ctype:`tracebackobject` type has been renamed to - :ctype:`PyTracebackObject`. +* The :c:type:`tracebackobject` type has been renamed to + :c:type:`PyTracebackObject`. .. ====================================================================== @@ -1554,7 +1554,7 @@ Acknowledgements ================ The author would like to thank the following people for offering suggestions, -corrections and assistance with various drafts of this article: Koray Can, Hye- -Shik Chang, Michael Dyck, Raymond Hettinger, Brian Hurt, Hamish Lawson, Fredrik -Lundh, Sean Reifschneider, Sadruddin Rejeb. +corrections and assistance with various drafts of this article: Koray Can, +Hye-Shik Chang, Michael Dyck, Raymond Hettinger, Brian Hurt, Hamish Lawson, +Fredrik Lundh, Sean Reifschneider, Sadruddin Rejeb. diff --git a/Doc/whatsnew/2.5.rst b/Doc/whatsnew/2.5.rst index c7f7d586ab..ff599c8d8a 100644 --- a/Doc/whatsnew/2.5.rst +++ b/Doc/whatsnew/2.5.rst @@ -870,31 +870,31 @@ to be able to remove the string-exception feature in a few releases. PEP 353: Using ssize_t as the index type ======================================== -A wide-ranging change to Python's C API, using a new :ctype:`Py_ssize_t` type -definition instead of :ctype:`int`, will permit the interpreter to handle more +A wide-ranging change to Python's C API, using a new :c:type:`Py_ssize_t` type +definition instead of :c:type:`int`, will permit the interpreter to handle more data on 64-bit platforms. This change doesn't affect Python's capacity on 32-bit platforms. -Various pieces of the Python interpreter used C's :ctype:`int` type to store +Various pieces of the Python interpreter used C's :c:type:`int` type to store sizes or counts; for example, the number of items in a list or tuple were stored -in an :ctype:`int`. The C compilers for most 64-bit platforms still define -:ctype:`int` as a 32-bit type, so that meant that lists could only hold up to +in an :c:type:`int`. The C compilers for most 64-bit platforms still define +:c:type:`int` as a 32-bit type, so that meant that lists could only hold up to ``2**31 - 1`` = 2147483647 items. (There are actually a few different programming models that 64-bit C compilers can use -- see http://www.unix.org/version2/whatsnew/lp64_wp.html for a discussion -- but the -most commonly available model leaves :ctype:`int` as 32 bits.) +most commonly available model leaves :c:type:`int` as 32 bits.) A limit of 2147483647 items doesn't really matter on a 32-bit platform because you'll run out of memory before hitting the length limit. Each list item requires space for a pointer, which is 4 bytes, plus space for a -:ctype:`PyObject` representing the item. 2147483647\*4 is already more bytes +:c:type:`PyObject` representing the item. 2147483647\*4 is already more bytes than a 32-bit address space can contain. It's possible to address that much memory on a 64-bit platform, however. The pointers for a list that size would only require 16 GiB of space, so it's not unreasonable that Python programmers might construct lists that large. Therefore, the Python interpreter had to be changed to use some type other than -:ctype:`int`, and this will be a 64-bit type on 64-bit platforms. The change +:c:type:`int`, and this will be a 64-bit type on 64-bit platforms. The change will cause incompatibilities on 64-bit machines, so it was deemed worth making the transition now, while the number of 64-bit users is still relatively small. (In 5 or 10 years, we may *all* be on 64-bit machines, and the transition would @@ -902,15 +902,15 @@ be more painful then.) This change most strongly affects authors of C extension modules. Python strings and container types such as lists and tuples now use -:ctype:`Py_ssize_t` to store their size. Functions such as -:cfunc:`PyList_Size` now return :ctype:`Py_ssize_t`. Code in extension modules -may therefore need to have some variables changed to :ctype:`Py_ssize_t`. +:c:type:`Py_ssize_t` to store their size. Functions such as +:c:func:`PyList_Size` now return :c:type:`Py_ssize_t`. Code in extension modules +may therefore need to have some variables changed to :c:type:`Py_ssize_t`. -The :cfunc:`PyArg_ParseTuple` and :cfunc:`Py_BuildValue` functions have a new -conversion code, ``n``, for :ctype:`Py_ssize_t`. :cfunc:`PyArg_ParseTuple`'s -``s#`` and ``t#`` still output :ctype:`int` by default, but you can define the -macro :cmacro:`PY_SSIZE_T_CLEAN` before including :file:`Python.h` to make -them return :ctype:`Py_ssize_t`. +The :c:func:`PyArg_ParseTuple` and :c:func:`Py_BuildValue` functions have a new +conversion code, ``n``, for :c:type:`Py_ssize_t`. :c:func:`PyArg_ParseTuple`'s +``s#`` and ``t#`` still output :c:type:`int` by default, but you can define the +macro :c:macro:`PY_SSIZE_T_CLEAN` before including :file:`Python.h` to make +them return :c:type:`Py_ssize_t`. :pep:`353` has a section on conversion guidelines that extension authors should read to learn about supporting 64-bit platforms. @@ -954,8 +954,8 @@ interpreter will check that the type returned is correct, and raises a :exc:`TypeError` if this requirement isn't met. A corresponding :attr:`nb_index` slot was added to the C-level -:ctype:`PyNumberMethods` structure to let C extensions implement this protocol. -:cfunc:`PyNumber_Index(obj)` can be used in extension code to call the +:c:type:`PyNumberMethods` structure to let C extensions implement this protocol. +:c:func:`PyNumber_Index(obj)` can be used in extension code to call the :meth:`__index__` function and retrieve its result. @@ -1179,7 +1179,7 @@ marked in the following list. (Contributed by Bob Ippolito at the NeedForSpeed sprint.) * The :mod:`re` module got a 1 or 2% speedup by switching to Python's allocator - functions instead of the system's :cfunc:`malloc` and :cfunc:`free`. + functions instead of the system's :c:func:`malloc` and :c:func:`free`. (Contributed by Jack Diederich at the NeedForSpeed sprint.) * The code generator's peephole optimizer now performs simple constant folding @@ -1203,7 +1203,7 @@ marked in the following list. Sean Reifschneider at the NeedForSpeed sprint.) * Importing now caches the paths tried, recording whether they exist or not so - that the interpreter makes fewer :cfunc:`open` and :cfunc:`stat` calls on + that the interpreter makes fewer :c:func:`open` and :c:func:`stat` calls on startup. (Contributed by Martin von Löwis and Georg Brandl.) .. Patch 921466 @@ -1459,7 +1459,7 @@ complete list of changes, or look through the SVN logs for all the details. On FreeBSD, the :func:`os.stat` function now returns times with nanosecond resolution, and the returned object now has :attr:`st_gen` and - :attr:`st_birthtime`. The :attr:`st_flags` member is also available, if the + :attr:`st_birthtime`. The :attr:`st_flags` attribute is also available, if the platform supports it. (Contributed by Antti Louko and Diego Pettenò.) .. (Patch 1180695, 1212117) @@ -1568,7 +1568,7 @@ complete list of changes, or look through the SVN logs for all the details. reporting ``('CPython', 'trunk', '45313:45315')``. This information is also available to C extensions via the - :cfunc:`Py_GetBuildInfo` function that returns a string of build information + :c:func:`Py_GetBuildInfo` function that returns a string of build information like this: ``"trunk:45355:45356M, Apr 13 2006, 07:42:19"``. (Contributed by Barry Warsaw.) @@ -1690,7 +1690,7 @@ attributes of the :class:`CDLL` object. :: result = libc.printf("Line of output\n") Type constructors for the various C types are provided: :func:`c_int`, -:func:`c_float`, :func:`c_double`, :func:`c_char_p` (equivalent to :ctype:`char +:func:`c_float`, :func:`c_double`, :func:`c_char_p` (equivalent to :c:type:`char \*`), and so forth. Unlike Python's types, the C versions are all mutable; you can assign to their :attr:`value` attribute to change the wrapped value. Python integers and strings will be automatically converted to the corresponding C @@ -1720,7 +1720,7 @@ attribute of the function object to change this:: ``ctypes.pythonapi`` object. This object does *not* release the global interpreter lock before calling a function, because the lock must be held when calling into the interpreter's code. There's a :class:`py_object()` type -constructor that will create a :ctype:`PyObject \*` pointer. A simple usage:: +constructor that will create a :c:type:`PyObject \*` pointer. A simple usage:: import ctypes @@ -2087,8 +2087,8 @@ Changes to Python's build process and to the C API include: http://scan.coverity.com for the statistics. * The largest change to the C API came from :pep:`353`, which modifies the - interpreter to use a :ctype:`Py_ssize_t` type definition instead of - :ctype:`int`. See the earlier section :ref:`pep-353` for a discussion of this + interpreter to use a :c:type:`Py_ssize_t` type definition instead of + :c:type:`int`. See the earlier section :ref:`pep-353` for a discussion of this change. * The design of the bytecode compiler has changed a great deal, no longer @@ -2113,10 +2113,10 @@ Changes to Python's build process and to the C API include: discusses the design. To start learning about the code, read the definition of the various AST nodes in :file:`Parser/Python.asdl`. A Python script reads this file and generates a set of C structure definitions in - :file:`Include/Python-ast.h`. The :cfunc:`PyParser_ASTFromString` and - :cfunc:`PyParser_ASTFromFile`, defined in :file:`Include/pythonrun.h`, take + :file:`Include/Python-ast.h`. The :c:func:`PyParser_ASTFromString` and + :c:func:`PyParser_ASTFromFile`, defined in :file:`Include/pythonrun.h`, take Python source as input and return the root of an AST representing the contents. - This AST can then be turned into a code object by :cfunc:`PyAST_Compile`. For + This AST can then be turned into a code object by :c:func:`PyAST_Compile`. For more information, read the source code, and then ask questions on python-dev. The AST code was developed under Jeremy Hylton's management, and implemented by @@ -2138,55 +2138,55 @@ Changes to Python's build process and to the C API include: Note that this change means extension modules must be more careful when allocating memory. Python's API has many different functions for allocating - memory that are grouped into families. For example, :cfunc:`PyMem_Malloc`, - :cfunc:`PyMem_Realloc`, and :cfunc:`PyMem_Free` are one family that allocates - raw memory, while :cfunc:`PyObject_Malloc`, :cfunc:`PyObject_Realloc`, and - :cfunc:`PyObject_Free` are another family that's supposed to be used for + memory that are grouped into families. For example, :c:func:`PyMem_Malloc`, + :c:func:`PyMem_Realloc`, and :c:func:`PyMem_Free` are one family that allocates + raw memory, while :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`, and + :c:func:`PyObject_Free` are another family that's supposed to be used for creating Python objects. Previously these different families all reduced to the platform's - :cfunc:`malloc` and :cfunc:`free` functions. This meant it didn't matter if - you got things wrong and allocated memory with the :cfunc:`PyMem` function but - freed it with the :cfunc:`PyObject` function. With 2.5's changes to obmalloc, + :c:func:`malloc` and :c:func:`free` functions. This meant it didn't matter if + you got things wrong and allocated memory with the :c:func:`PyMem` function but + freed it with the :c:func:`PyObject` function. With 2.5's changes to obmalloc, these families now do different things and mismatches will probably result in a segfault. You should carefully test your C extension modules with Python 2.5. -* The built-in set types now have an official C API. Call :cfunc:`PySet_New` - and :cfunc:`PyFrozenSet_New` to create a new set, :cfunc:`PySet_Add` and - :cfunc:`PySet_Discard` to add and remove elements, and :cfunc:`PySet_Contains` - and :cfunc:`PySet_Size` to examine the set's state. (Contributed by Raymond +* The built-in set types now have an official C API. Call :c:func:`PySet_New` + and :c:func:`PyFrozenSet_New` to create a new set, :c:func:`PySet_Add` and + :c:func:`PySet_Discard` to add and remove elements, and :c:func:`PySet_Contains` + and :c:func:`PySet_Size` to examine the set's state. (Contributed by Raymond Hettinger.) * C code can now obtain information about the exact revision of the Python - interpreter by calling the :cfunc:`Py_GetBuildInfo` function that returns a + interpreter by calling the :c:func:`Py_GetBuildInfo` function that returns a string of build information like this: ``"trunk:45355:45356M, Apr 13 2006, 07:42:19"``. (Contributed by Barry Warsaw.) * Two new macros can be used to indicate C functions that are local to the current file so that a faster calling convention can be used. - :cfunc:`Py_LOCAL(type)` declares the function as returning a value of the + :c:func:`Py_LOCAL(type)` declares the function as returning a value of the specified *type* and uses a fast-calling qualifier. - :cfunc:`Py_LOCAL_INLINE(type)` does the same thing and also requests the - function be inlined. If :cfunc:`PY_LOCAL_AGGRESSIVE` is defined before + :c:func:`Py_LOCAL_INLINE(type)` does the same thing and also requests the + function be inlined. If :c:func:`PY_LOCAL_AGGRESSIVE` is defined before :file:`python.h` is included, a set of more aggressive optimizations are enabled for the module; you should benchmark the results to find out if these optimizations actually make the code faster. (Contributed by Fredrik Lundh at the NeedForSpeed sprint.) -* :cfunc:`PyErr_NewException(name, base, dict)` can now accept a tuple of base +* :c:func:`PyErr_NewException(name, base, dict)` can now accept a tuple of base classes as its *base* argument. (Contributed by Georg Brandl.) -* The :cfunc:`PyErr_Warn` function for issuing warnings is now deprecated in - favour of :cfunc:`PyErr_WarnEx(category, message, stacklevel)` which lets you +* The :c:func:`PyErr_Warn` function for issuing warnings is now deprecated in + favour of :c:func:`PyErr_WarnEx(category, message, stacklevel)` which lets you specify the number of stack frames separating this function and the caller. A - *stacklevel* of 1 is the function calling :cfunc:`PyErr_WarnEx`, 2 is the + *stacklevel* of 1 is the function calling :c:func:`PyErr_WarnEx`, 2 is the function above that, and so forth. (Added by Neal Norwitz.) * The CPython interpreter is still written in C, but the code can now be compiled with a C++ compiler without errors. (Implemented by Anthony Baxter, Martin von Löwis, Skip Montanaro.) -* The :cfunc:`PyRange_New` function was removed. It was never documented, never +* The :c:func:`PyRange_New` function was removed. It was never documented, never used in the core code, and had dangerously lax error checking. In the unlikely case that your extensions were using it, you can replace it by something like the following:: @@ -2203,7 +2203,7 @@ Port-Specific Changes --------------------- * MacOS X (10.3 and higher): dynamic loading of modules now uses the - :cfunc:`dlopen` function instead of MacOS-specific functions. + :c:func:`dlopen` function instead of MacOS-specific functions. * MacOS X: an :option:`--enable-universalsdk` switch was added to the :program:`configure` script that compiles the interpreter as a universal binary @@ -2259,15 +2259,15 @@ code: Setting :attr:`rpc_paths` to ``None`` or an empty tuple disables this path checking. -* C API: Many functions now use :ctype:`Py_ssize_t` instead of :ctype:`int` to +* C API: Many functions now use :c:type:`Py_ssize_t` instead of :c:type:`int` to allow processing more data on 64-bit machines. Extension code may need to make the same change to avoid warnings and to support 64-bit machines. See the earlier section :ref:`pep-353` for a discussion of this change. * C API: The obmalloc changes mean that you must be careful to not mix usage - of the :cfunc:`PyMem_\*` and :cfunc:`PyObject_\*` families of functions. Memory - allocated with one family's :cfunc:`\*_Malloc` must be freed with the - corresponding family's :cfunc:`\*_Free` function. + of the :c:func:`PyMem_\*` and :c:func:`PyObject_\*` families of functions. Memory + allocated with one family's :c:func:`\*_Malloc` must be freed with the + corresponding family's :c:func:`\*_Free` function. .. ====================================================================== diff --git a/Doc/whatsnew/2.6.rst b/Doc/whatsnew/2.6.rst index 1af78c2276..68dcc254bd 100644 --- a/Doc/whatsnew/2.6.rst +++ b/Doc/whatsnew/2.6.rst @@ -123,7 +123,7 @@ about features that will be removed in Python 3.0. You can run code with this switch to see how much work will be necessary to port code to 3.0. The value of this switch is available to Python code as the boolean variable :data:`sys.py3kwarning`, -and to C extension code as :cdata:`Py_Py3kWarningFlag`. +and to C extension code as :c:data:`Py_Py3kWarningFlag`. .. seealso:: @@ -234,7 +234,7 @@ have adopted Sphinx as their documentation tool. .. seealso:: - :ref:`documenting-index` + `Documenting Python <http://docs.python.org/devguide/documenting.html>`__ Describes how to write for Python's documentation. `Sphinx <http://sphinx.pocoo.org/>`__ @@ -615,8 +615,8 @@ multiple of 4. result = queue.get() print 'Factorial', N, '=', result -A :class:`Queue` is used to communicate the input parameter *N* and -the result. The :class:`Queue` object is stored in a global variable. +A :class:`Queue` is used to communicate the result of the factorial. +The :class:`Queue` object is stored in a global variable. The child process will use the value of the variable when the child was created; because it's a :class:`Queue`, parent and child can use the object to communicate. (If the parent were to change the value of @@ -979,10 +979,10 @@ can be used to include Unicode characters:: print len(s) # 12 Unicode characters At the C level, Python 3.0 will rename the existing 8-bit -string type, called :ctype:`PyStringObject` in Python 2.x, -to :ctype:`PyBytesObject`. Python 2.6 uses ``#define`` -to support using the names :cfunc:`PyBytesObject`, -:cfunc:`PyBytes_Check`, :cfunc:`PyBytes_FromStringAndSize`, +string type, called :c:type:`PyStringObject` in Python 2.x, +to :c:type:`PyBytesObject`. Python 2.6 uses ``#define`` +to support using the names :c:func:`PyBytesObject`, +:c:func:`PyBytes_Check`, :c:func:`PyBytes_FromStringAndSize`, and all the other functions and macros used with strings. Instances of the :class:`bytes` type are immutable just @@ -1014,8 +1014,8 @@ and some of the methods of lists, such as :meth:`append`, bytearray(b'ABCde') There's also a corresponding C API, with -:cfunc:`PyByteArray_FromObject`, -:cfunc:`PyByteArray_FromStringAndSize`, +:c:func:`PyByteArray_FromObject`, +:c:func:`PyByteArray_FromStringAndSize`, and various other functions. .. seealso:: @@ -1134,7 +1134,7 @@ indicate that the external caller is done. .. XXX PyObject_GetBuffer not documented in c-api -The *flags* argument to :cfunc:`PyObject_GetBuffer` specifies +The *flags* argument to :c:func:`PyObject_GetBuffer` specifies constraints upon the memory returned. Some examples are: * :const:`PyBUF_WRITABLE` indicates that the memory must be writable. @@ -1145,7 +1145,7 @@ constraints upon the memory returned. Some examples are: requests a C-contiguous (last dimension varies the fastest) or Fortran-contiguous (first dimension varies the fastest) array layout. -Two new argument codes for :cfunc:`PyArg_ParseTuple`, +Two new argument codes for :c:func:`PyArg_ParseTuple`, ``s*`` and ``z*``, return locked buffer objects for a parameter. .. seealso:: @@ -1485,6 +1485,13 @@ Other Language Changes Some smaller changes made to the core Python language are: +* Directories and zip archives containing a :file:`__main__.py` file + can now be executed directly by passing their name to the + interpreter. The directory or zip archive is automatically inserted + as the first entry in sys.path. (Suggestion and initial patch by + Andy Chu, subsequently revised by Phillip J. Eby and Nick Coghlan; + :issue:`1739468`.) + * The :func:`hasattr` function was catching and ignoring all errors, under the assumption that they meant a :meth:`__getattr__` method was failing somehow and the return value of :func:`hasattr` would @@ -1632,7 +1639,7 @@ Some smaller changes made to the core Python language are: :meth:`__hash__` method inherited from a parent class, so assigning ``None`` was implemented as an override. At the C level, extensions can set ``tp_hash`` to - :cfunc:`PyObject_HashNotImplemented`. + :c:func:`PyObject_HashNotImplemented`. (Fixed by Nick Coghlan and Amaury Forgeot d'Arc; :issue:`2235`.) * The :exc:`GeneratorExit` exception now subclasses @@ -1702,7 +1709,7 @@ Optimizations By default, this change is only applied to types that are included with the Python core. Extension modules may not necessarily be compatible with this cache, - so they must explicitly add :cmacro:`Py_TPFLAGS_HAVE_VERSION_TAG` + so they must explicitly add :c:macro:`Py_TPFLAGS_HAVE_VERSION_TAG` to the module's ``tp_flags`` field to enable the method cache. (To be compatible with the method cache, the extension module's code must not directly access and modify the ``tp_dict`` member of @@ -1785,7 +1792,7 @@ changes, or look through the Subversion logs for all the details. were applied. (Maintained by Josiah Carlson; see :issue:`1736190` for one patch.) -* The :mod:`bsddb` module also has a new maintainer, Jesús Cea, and the package +* The :mod:`bsddb` module also has a new maintainer, Jesús Cea Avion, and the package is now available as a standalone package. The web page for the package is `www.jcea.es/programacion/pybsddb.htm <http://www.jcea.es/programacion/pybsddb.htm>`__. @@ -2281,7 +2288,7 @@ changes, or look through the Subversion logs for all the details. (Contributed by Raymond Hettinger; :issue:`1861`.) * The :mod:`select` module now has wrapper functions - for the Linux :cfunc:`epoll` and BSD :cfunc:`kqueue` system calls. + for the Linux :c:func:`epoll` and BSD :c:func:`kqueue` system calls. :meth:`modify` method was added to the existing :class:`poll` objects; ``pollobj.modify(fd, eventmask)`` takes a file descriptor or file object and an event mask, modifying the recorded event mask @@ -2314,13 +2321,13 @@ changes, or look through the Subversion logs for all the details. Calling ``signal.set_wakeup_fd(fd)`` sets a file descriptor to be used; when a signal is received, a byte is written to that file descriptor. There's also a C-level function, - :cfunc:`PySignal_SetWakeupFd`, for setting the descriptor. + :c:func:`PySignal_SetWakeupFd`, for setting the descriptor. Event loops will use this by opening a pipe to create two descriptors, one for reading and one for writing. The writable descriptor will be passed to :func:`set_wakeup_fd`, and the readable descriptor will be added to the list of descriptors monitored by the event loop via - :cfunc:`select` or :cfunc:`poll`. + :c:func:`select` or :c:func:`poll`. On receiving a signal, a byte will be written and the main event loop will be woken up, avoiding the need to poll. @@ -2381,7 +2388,7 @@ changes, or look through the Subversion logs for all the details. has been updated from version 2.3.2 in Python 2.5 to version 2.4.1. -* The :mod:`struct` module now supports the C99 :ctype:`_Bool` type, +* The :mod:`struct` module now supports the C99 :c:type:`_Bool` type, using the format character ``'?'``. (Contributed by David Remahl.) @@ -2389,7 +2396,7 @@ changes, or look through the Subversion logs for all the details. now have :meth:`terminate`, :meth:`kill`, and :meth:`send_signal` methods. On Windows, :meth:`send_signal` only supports the :const:`SIGTERM` signal, and all these methods are aliases for the Win32 API function - :cfunc:`TerminateProcess`. + :c:func:`TerminateProcess`. (Contributed by Christian Heimes.) * A new variable in the :mod:`sys` module, :attr:`float_info`, is an @@ -2974,7 +2981,7 @@ Changes to Python's build process and to the C API include: * Python now must be compiled with C89 compilers (after 19 years!). This means that the Python source tree has dropped its - own implementations of :cfunc:`memmove` and :cfunc:`strerror`, which + own implementations of :c:func:`memmove` and :c:func:`strerror`, which are in the C89 standard library. * Python 2.6 can be built with Microsoft Visual Studio 2008 (version @@ -2996,7 +3003,7 @@ Changes to Python's build process and to the C API include: * The new buffer interface, previously described in `the PEP 3118 section <#pep-3118-revised-buffer-protocol>`__, - adds :cfunc:`PyObject_GetBuffer` and :cfunc:`PyBuffer_Release`, + adds :c:func:`PyObject_GetBuffer` and :c:func:`PyBuffer_Release`, as well as a few other functions. * Python's use of the C stdio library is now thread-safe, or at least @@ -3004,27 +3011,27 @@ Changes to Python's build process and to the C API include: bug occurred if one thread closed a file object while another thread was reading from or writing to the object. In 2.6 file objects have a reference count, manipulated by the - :cfunc:`PyFile_IncUseCount` and :cfunc:`PyFile_DecUseCount` + :c:func:`PyFile_IncUseCount` and :c:func:`PyFile_DecUseCount` functions. File objects can't be closed unless the reference count - is zero. :cfunc:`PyFile_IncUseCount` should be called while the GIL + is zero. :c:func:`PyFile_IncUseCount` should be called while the GIL is still held, before carrying out an I/O operation using the - ``FILE *`` pointer, and :cfunc:`PyFile_DecUseCount` should be called + ``FILE *`` pointer, and :c:func:`PyFile_DecUseCount` should be called immediately after the GIL is re-acquired. (Contributed by Antoine Pitrou and Gregory P. Smith.) * Importing modules simultaneously in two different threads no longer deadlocks; it will now raise an :exc:`ImportError`. A new API - function, :cfunc:`PyImport_ImportModuleNoBlock`, will look for a + function, :c:func:`PyImport_ImportModuleNoBlock`, will look for a module in ``sys.modules`` first, then try to import it after acquiring an import lock. If the import lock is held by another thread, an :exc:`ImportError` is raised. (Contributed by Christian Heimes.) * Several functions return information about the platform's - floating-point support. :cfunc:`PyFloat_GetMax` returns + floating-point support. :c:func:`PyFloat_GetMax` returns the maximum representable floating point value, - and :cfunc:`PyFloat_GetMin` returns the minimum - positive value. :cfunc:`PyFloat_GetInfo` returns an object + and :c:func:`PyFloat_GetMin` returns the minimum + positive value. :c:func:`PyFloat_GetInfo` returns an object containing more information from the :file:`float.h` file, such as ``"mant_dig"`` (number of digits in the mantissa), ``"epsilon"`` (smallest difference between 1.0 and the next largest value @@ -3032,7 +3039,7 @@ Changes to Python's build process and to the C API include: (Contributed by Christian Heimes; :issue:`1534`.) * C functions and methods that use - :cfunc:`PyComplex_AsCComplex` will now accept arguments that + :c:func:`PyComplex_AsCComplex` will now accept arguments that have a :meth:`__complex__` method. In particular, the functions in the :mod:`cmath` module will now accept objects with this method. This is a backport of a Python 3.0 change. @@ -3046,15 +3053,15 @@ Changes to Python's build process and to the C API include: * Many C extensions define their own little macro for adding integers and strings to the module's dictionary in the ``init*`` function. Python 2.6 finally defines standard macros - for adding values to a module, :cmacro:`PyModule_AddStringMacro` - and :cmacro:`PyModule_AddIntMacro()`. (Contributed by + for adding values to a module, :c:macro:`PyModule_AddStringMacro` + and :c:macro:`PyModule_AddIntMacro()`. (Contributed by Christian Heimes.) * Some macros were renamed in both 3.0 and 2.6 to make it clearer that they are macros, - not functions. :cmacro:`Py_Size()` became :cmacro:`Py_SIZE()`, - :cmacro:`Py_Type()` became :cmacro:`Py_TYPE()`, and - :cmacro:`Py_Refcnt()` became :cmacro:`Py_REFCNT()`. + not functions. :c:macro:`Py_Size()` became :c:macro:`Py_SIZE()`, + :c:macro:`Py_Type()` became :c:macro:`Py_TYPE()`, and + :c:macro:`Py_Refcnt()` became :c:macro:`Py_REFCNT()`. The mixed-case macros are still available in Python 2.6 for backward compatibility. (:issue:`1629`) @@ -3112,7 +3119,7 @@ Port-Specific Changes: Windows * The :mod:`socket` module's socket objects now have an :meth:`ioctl` method that provides a limited interface to the - :cfunc:`WSAIoctl` system interface. + :c:func:`WSAIoctl` system interface. * The :mod:`_winreg` module now has a function, :func:`ExpandEnvironmentStrings`, @@ -3258,13 +3265,13 @@ that may require changes to your code: the implementation now explicitly checks for this case and raises an :exc:`ImportError`. -* C API: the :cfunc:`PyImport_Import` and :cfunc:`PyImport_ImportModule` +* C API: the :c:func:`PyImport_Import` and :c:func:`PyImport_ImportModule` functions now default to absolute imports, not relative imports. This will affect C extensions that import other modules. * C API: extension data types that shouldn't be hashable should define their ``tp_hash`` slot to - :cfunc:`PyObject_HashNotImplemented`. + :c:func:`PyObject_HashNotImplemented`. * The :mod:`socket` module exception :exc:`socket.error` now inherits from :exc:`IOError`. Previously it wasn't a subclass of diff --git a/Doc/whatsnew/2.7.rst b/Doc/whatsnew/2.7.rst new file mode 100644 index 0000000000..eb6372a72d --- /dev/null +++ b/Doc/whatsnew/2.7.rst @@ -0,0 +1,2484 @@ +**************************** + What's New in Python 2.7 +**************************** + +:Author: A.M. Kuchling (amk at amk.ca) +:Release: |release| +:Date: |today| + +.. hyperlink all the methods & functions. + +.. T_STRING_INPLACE not described in main docs +.. "Format String Syntax" in string.rst could use many more examples. + +.. $Id$ + Rules for maintenance: + + * Anyone can add text to this document. Do not spend very much time + on the wording of your changes, because your text will probably + get rewritten to some degree. + + * The maintainer will go through Misc/NEWS periodically and add + changes; it's therefore more important to add your changes to + Misc/NEWS than to this file. + + * This is not a complete list of every single change; completeness + is the purpose of Misc/NEWS. Some changes I consider too small + or esoteric to include. If such a change is added to the text, + I'll just remove it. (This is another reason you shouldn't spend + too much time on writing your addition.) + + * If you want to draw your new text to the attention of the + maintainer, add 'XXX' to the beginning of the paragraph or + section. + + * It's OK to just add a fragmentary note about a change. For + example: "XXX Describe the transmogrify() function added to the + socket module." The maintainer will research the change and + write the necessary text. + + * You can comment out your additions if you like, but it's not + necessary (especially when a final release is some months away). + + * Credit the author of a patch or bugfix. Just the name is + sufficient; the e-mail address isn't necessary. + + * It's helpful to add the bug/patch number in a parenthetical comment. + + XXX Describe the transmogrify() function added to the socket + module. + (Contributed by P.Y. Developer; :issue:`12345`.) + + This saves the maintainer some effort going through the SVN logs + when researching a change. + +This article explains the new features in Python 2.7. The final +release of 2.7 is currently scheduled for July 2010; the detailed +schedule is described in :pep:`373`. + +Numeric handling has been improved in many ways, for both +floating-point numbers and for the :class:`Decimal` class. There are +some useful additions to the standard library, such as a greatly +enhanced :mod:`unittest` module, the :mod:`argparse` module for +parsing command-line options, convenient ordered-dictionary and +:class:`Counter` classes in the :mod:`collections` module, and many +other improvements. + +Python 2.7 is planned to be the last of the 2.x releases, so we worked +on making it a good release for the long term. To help with porting +to Python 3, several new features from the Python 3.x series have been +included in 2.7. + +This article doesn't attempt to provide a complete specification of +the new features, but instead provides a convenient overview. For +full details, you should refer to the documentation for Python 2.7 at +http://docs.python.org. If you want to understand the rationale for +the design and implementation, refer to the PEP for a particular new +feature or the issue on http://bugs.python.org in which a change was +discussed. Whenever possible, "What's New in Python" links to the +bug/patch item for each change. + +.. _whatsnew27-python31: + +The Future for Python 2.x +========================= + +Python 2.7 is intended to be the last major release in the 2.x series. +The Python maintainers are planning to focus their future efforts on +the Python 3.x series. + +This means that 2.7 will remain in place for a long time, running +production systems that have not been ported to Python 3.x. +Two consequences of the long-term significance of 2.7 are: + +* It's very likely the 2.7 release will have a longer period of + maintenance compared to earlier 2.x versions. Python 2.7 will + continue to be maintained while the transition to 3.x continues, and + the developers are planning to support Python 2.7 with bug-fix + releases beyond the typical two years. + +* A policy decision was made to silence warnings only of interest to + developers. :exc:`DeprecationWarning` and its + descendants are now ignored unless otherwise requested, preventing + users from seeing warnings triggered by an application. This change + was also made in the branch that will become Python 3.2. (Discussed + on stdlib-sig and carried out in :issue:`7319`.) + + In previous releases, :exc:`DeprecationWarning` messages were + enabled by default, providing Python developers with a clear + indication of where their code may break in a future major version + of Python. + + However, there are increasingly many users of Python-based + applications who are not directly involved in the development of + those applications. :exc:`DeprecationWarning` messages are + irrelevant to such users, making them worry about an application + that's actually working correctly and burdening application developers + with responding to these concerns. + + You can re-enable display of :exc:`DeprecationWarning` messages by + running Python with the :option:`-Wdefault` (short form: + :option:`-Wd`) switch, or by setting the :envvar:`PYTHONWARNINGS` + environment variable to ``"default"`` (or ``"d"``) before running + Python. Python code can also re-enable them + by calling ``warnings.simplefilter('default')``. + + +Python 3.1 Features +======================= + +Much as Python 2.6 incorporated features from Python 3.0, +version 2.7 incorporates some of the new features +in Python 3.1. The 2.x series continues to provide tools +for migrating to the 3.x series. + +A partial list of 3.1 features that were backported to 2.7: + +* The syntax for set literals (``{1,2,3}`` is a mutable set). +* Dictionary and set comprehensions (``{ i: i*2 for i in range(3)}``). +* Multiple context managers in a single :keyword:`with` statement. +* A new version of the :mod:`io` library, rewritten in C for performance. +* The ordered-dictionary type described in :ref:`pep-0372`. +* The new ``","`` format specifier described in :ref:`pep-0378`. +* The :class:`memoryview` object. +* A small subset of the :mod:`importlib` module, + `described below <#importlib-section>`__. +* The :func:`repr` of a float ``x`` is shorter in many cases: it's now + based on the shortest decimal string that's guaranteed to round back + to ``x``. As in previous versions of Python, it's guaranteed that + ``float(repr(x))`` recovers ``x``. +* Float-to-string and string-to-float conversions are correctly rounded. + The :func:`round` function is also now correctly rounded. +* The :c:type:`PyCapsule` type, used to provide a C API for extension modules. +* The :c:func:`PyLong_AsLongAndOverflow` C API function. + +Other new Python3-mode warnings include: + +* :func:`operator.isCallable` and :func:`operator.sequenceIncludes`, + which are not supported in 3.x, now trigger warnings. +* The :option:`-3` switch now automatically + enables the :option:`-Qwarn` switch that causes warnings + about using classic division with integers and long integers. + + + +.. ======================================================================== +.. Large, PEP-level features and changes should be described here. +.. ======================================================================== + +.. _pep-0372: + +PEP 372: Adding an Ordered Dictionary to collections +==================================================== + +Regular Python dictionaries iterate over key/value pairs in arbitrary order. +Over the years, a number of authors have written alternative implementations +that remember the order that the keys were originally inserted. Based on +the experiences from those implementations, 2.7 introduces a new +:class:`~collections.OrderedDict` class in the :mod:`collections` module. + +The :class:`~collections.OrderedDict` API provides the same interface as regular +dictionaries but iterates over keys and values in a guaranteed order +depending on when a key was first inserted:: + + >>> from collections import OrderedDict + >>> d = OrderedDict([('first', 1), + ... ('second', 2), + ... ('third', 3)]) + >>> d.items() + [('first', 1), ('second', 2), ('third', 3)] + +If a new entry overwrites an existing entry, the original insertion +position is left unchanged:: + + >>> d['second'] = 4 + >>> d.items() + [('first', 1), ('second', 4), ('third', 3)] + +Deleting an entry and reinserting it will move it to the end:: + + >>> del d['second'] + >>> d['second'] = 5 + >>> d.items() + [('first', 1), ('third', 3), ('second', 5)] + +The :meth:`~collections.OrderedDict.popitem` method has an optional *last* +argument that defaults to True. If *last* is True, the most recently +added key is returned and removed; if it's False, the +oldest key is selected:: + + >>> od = OrderedDict([(x,0) for x in range(20)]) + >>> od.popitem() + (19, 0) + >>> od.popitem() + (18, 0) + >>> od.popitem(last=False) + (0, 0) + >>> od.popitem(last=False) + (1, 0) + +Comparing two ordered dictionaries checks both the keys and values, +and requires that the insertion order was the same:: + + >>> od1 = OrderedDict([('first', 1), + ... ('second', 2), + ... ('third', 3)]) + >>> od2 = OrderedDict([('third', 3), + ... ('first', 1), + ... ('second', 2)]) + >>> od1 == od2 + False + >>> # Move 'third' key to the end + >>> del od2['third']; od2['third'] = 3 + >>> od1 == od2 + True + +Comparing an :class:`~collections.OrderedDict` with a regular dictionary +ignores the insertion order and just compares the keys and values. + +How does the :class:`~collections.OrderedDict` work? It maintains a +doubly-linked list of keys, appending new keys to the list as they're inserted. +A secondary dictionary maps keys to their corresponding list node, so +deletion doesn't have to traverse the entire linked list and therefore +remains O(1). + +The standard library now supports use of ordered dictionaries in several +modules. + +* The :mod:`ConfigParser` module uses them by default, meaning that + configuration files can now be read, modified, and then written back + in their original order. + +* The :meth:`~collections.somenamedtuple._asdict()` method for + :func:`collections.namedtuple` now returns an ordered dictionary with the + values appearing in the same order as the underlying tuple indices. + +* The :mod:`json` module's :class:`~json.JSONDecoder` class + constructor was extended with an *object_pairs_hook* parameter to + allow :class:`OrderedDict` instances to be built by the decoder. + Support was also added for third-party tools like + `PyYAML <http://pyyaml.org/>`_. + +.. seealso:: + + :pep:`372` - Adding an ordered dictionary to collections + PEP written by Armin Ronacher and Raymond Hettinger; + implemented by Raymond Hettinger. + +.. _pep-0378: + +PEP 378: Format Specifier for Thousands Separator +================================================= + +To make program output more readable, it can be useful to add +separators to large numbers, rendering them as +18,446,744,073,709,551,616 instead of 18446744073709551616. + +The fully general solution for doing this is the :mod:`locale` module, +which can use different separators ("," in North America, "." in +Europe) and different grouping sizes, but :mod:`locale` is complicated +to use and unsuitable for multi-threaded applications where different +threads are producing output for different locales. + +Therefore, a simple comma-grouping mechanism has been added to the +mini-language used by the :meth:`str.format` method. When +formatting a floating-point number, simply include a comma between the +width and the precision:: + + >>> '{:20,.2f}'.format(18446744073709551616.0) + '18,446,744,073,709,551,616.00' + +When formatting an integer, include the comma after the width: + + >>> '{:20,d}'.format(18446744073709551616) + '18,446,744,073,709,551,616' + +This mechanism is not adaptable at all; commas are always used as the +separator and the grouping is always into three-digit groups. The +comma-formatting mechanism isn't as general as the :mod:`locale` +module, but it's easier to use. + +.. seealso:: + + :pep:`378` - Format Specifier for Thousands Separator + PEP written by Raymond Hettinger; implemented by Eric Smith. + +PEP 389: The argparse Module for Parsing Command Lines +====================================================== + +The :mod:`argparse` module for parsing command-line arguments was +added as a more powerful replacement for the +:mod:`optparse` module. + +This means Python now supports three different modules for parsing +command-line arguments: :mod:`getopt`, :mod:`optparse`, and +:mod:`argparse`. The :mod:`getopt` module closely resembles the C +library's :c:func:`getopt` function, so it remains useful if you're writing a +Python prototype that will eventually be rewritten in C. +:mod:`optparse` becomes redundant, but there are no plans to remove it +because there are many scripts still using it, and there's no +automated way to update these scripts. (Making the :mod:`argparse` +API consistent with :mod:`optparse`'s interface was discussed but +rejected as too messy and difficult.) + +In short, if you're writing a new script and don't need to worry +about compatibility with earlier versions of Python, use +:mod:`argparse` instead of :mod:`optparse`. + +Here's an example:: + + import argparse + + parser = argparse.ArgumentParser(description='Command-line example.') + + # Add optional switches + parser.add_argument('-v', action='store_true', dest='is_verbose', + help='produce verbose output') + parser.add_argument('-o', action='store', dest='output', + metavar='FILE', + help='direct output to FILE instead of stdout') + parser.add_argument('-C', action='store', type=int, dest='context', + metavar='NUM', default=0, + help='display NUM lines of added context') + + # Allow any number of additional arguments. + parser.add_argument(nargs='*', action='store', dest='inputs', + help='input filenames (default is stdin)') + + args = parser.parse_args() + print args.__dict__ + +Unless you override it, :option:`-h` and :option:`--help` switches +are automatically added, and produce neatly formatted output:: + + -> ./python.exe argparse-example.py --help + usage: argparse-example.py [-h] [-v] [-o FILE] [-C NUM] [inputs [inputs ...]] + + Command-line example. + + positional arguments: + inputs input filenames (default is stdin) + + optional arguments: + -h, --help show this help message and exit + -v produce verbose output + -o FILE direct output to FILE instead of stdout + -C NUM display NUM lines of added context + +As with :mod:`optparse`, the command-line switches and arguments +are returned as an object with attributes named by the *dest* parameters:: + + -> ./python.exe argparse-example.py -v + {'output': None, + 'is_verbose': True, + 'context': 0, + 'inputs': []} + + -> ./python.exe argparse-example.py -v -o /tmp/output -C 4 file1 file2 + {'output': '/tmp/output', + 'is_verbose': True, + 'context': 4, + 'inputs': ['file1', 'file2']} + +:mod:`argparse` has much fancier validation than :mod:`optparse`; you +can specify an exact number of arguments as an integer, 0 or more +arguments by passing ``'*'``, 1 or more by passing ``'+'``, or an +optional argument with ``'?'``. A top-level parser can contain +sub-parsers to define subcommands that have different sets of +switches, as in ``svn commit``, ``svn checkout``, etc. You can +specify an argument's type as :class:`~argparse.FileType`, which will +automatically open files for you and understands that ``'-'`` means +standard input or output. + +.. seealso:: + + `argparse module documentation <http://docs.python.org/dev/library/argparse.html>`__ + + `Upgrading optparse code to use argparse <http://docs.python.org/dev/library/argparse.html#upgrading-optparse-code>`__ + Part of the Python documentation, describing how to convert + code that uses :mod:`optparse`. + + :pep:`389` - argparse - New Command Line Parsing Module + PEP written and implemented by Steven Bethard. + +PEP 391: Dictionary-Based Configuration For Logging +==================================================== + +.. XXX not documented in library reference yet; add link here once it's added. + +The :mod:`logging` module is very flexible; applications can define +a tree of logging subsystems, and each logger in this tree can filter +out certain messages, format them differently, and direct messages to +a varying number of handlers. + +All this flexibility can require a lot of configuration. You can +write Python statements to create objects and set their properties, +but a complex set-up requires verbose but boring code. +:mod:`logging` also supports a :func:`~logging.config.fileConfig` +function that parses a file, but the file format doesn't support +configuring filters, and it's messier to generate programmatically. + +Python 2.7 adds a :func:`~logging.config.dictConfig` function that +uses a dictionary to configure logging. There are many ways to +produce a dictionary from different sources: construct one with code; +parse a file containing JSON; or use a YAML parsing library if one is +installed. + +The following example configures two loggers, the root logger and a +logger named "network". Messages sent to the root logger will be +sent to the system log using the syslog protocol, and messages +to the "network" logger will be written to a :file:`network.log` file +that will be rotated once the log reaches 1Mb. + +:: + + import logging + import logging.config + + configdict = { + 'version': 1, # Configuration schema in use; must be 1 for now + 'formatters': { + 'standard': { + 'format': ('%(asctime)s %(name)-15s ' + '%(levelname)-8s %(message)s')}}, + + 'handlers': {'netlog': {'backupCount': 10, + 'class': 'logging.handlers.RotatingFileHandler', + 'filename': '/logs/network.log', + 'formatter': 'standard', + 'level': 'INFO', + 'maxBytes': 1024*1024}, + 'syslog': {'class': 'logging.handlers.SysLogHandler', + 'formatter': 'standard', + 'level': 'ERROR'}}, + + # Specify all the subordinate loggers + 'loggers': { + 'network': { + 'handlers': ['netlog'] + } + }, + # Specify properties of the root logger + 'root': { + 'handlers': ['syslog'] + }, + } + + # Set up configuration + logging.config.dictConfig(configdict) + + # As an example, log two error messages + logger = logging.getLogger('/') + logger.error('Database not found') + + netlogger = logging.getLogger('network') + netlogger.error('Connection failed') + +Three smaller enhancements to the :mod:`logging` module, all +implemented by Vinay Sajip, are: + +.. rev79293 + +* The :class:`~logging.handlers.SysLogHandler` class now supports + syslogging over TCP. The constructor has a *socktype* parameter + giving the type of socket to use, either :const:`socket.SOCK_DGRAM` + for UDP or :const:`socket.SOCK_STREAM` for TCP. The default + protocol remains UDP. + +* :class:`Logger` instances gained a :meth:`getChild` method that retrieves a + descendant logger using a relative path. For example, + once you retrieve a logger by doing ``log = getLogger('app')``, + calling ``log.getChild('network.listen')`` is equivalent to + ``getLogger('app.network.listen')``. + +* The :class:`LoggerAdapter` class gained a :meth:`isEnabledFor` method + that takes a *level* and returns whether the underlying logger would + process a message of that level of importance. + +.. seealso:: + + :pep:`391` - Dictionary-Based Configuration For Logging + PEP written and implemented by Vinay Sajip. + +PEP 3106: Dictionary Views +==================================================== + +The dictionary methods :meth:`keys`, :meth:`values`, and :meth:`items` +are different in Python 3.x. They return an object called a :dfn:`view` +instead of a fully materialized list. + +It's not possible to change the return values of :meth:`keys`, +:meth:`values`, and :meth:`items` in Python 2.7 because too much code +would break. Instead the 3.x versions were added under the new names +:meth:`viewkeys`, :meth:`viewvalues`, and :meth:`viewitems`. + +:: + + >>> d = dict((i*10, chr(65+i)) for i in range(26)) + >>> d + {0: 'A', 130: 'N', 10: 'B', 140: 'O', 20: ..., 250: 'Z'} + >>> d.viewkeys() + dict_keys([0, 130, 10, 140, 20, 150, 30, ..., 250]) + +Views can be iterated over, but the key and item views also behave +like sets. The ``&`` operator performs intersection, and ``|`` +performs a union:: + + >>> d1 = dict((i*10, chr(65+i)) for i in range(26)) + >>> d2 = dict((i**.5, i) for i in range(1000)) + >>> d1.viewkeys() & d2.viewkeys() + set([0.0, 10.0, 20.0, 30.0]) + >>> d1.viewkeys() | range(0, 30) + set([0, 1, 130, 3, 4, 5, 6, ..., 120, 250]) + +The view keeps track of the dictionary and its contents change as the +dictionary is modified:: + + >>> vk = d.viewkeys() + >>> vk + dict_keys([0, 130, 10, ..., 250]) + >>> d[260] = '&' + >>> vk + dict_keys([0, 130, 260, 10, ..., 250]) + +However, note that you can't add or remove keys while you're iterating +over the view:: + + >>> for k in vk: + ... d[k*2] = k + ... + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + RuntimeError: dictionary changed size during iteration + +You can use the view methods in Python 2.x code, and the 2to3 +converter will change them to the standard :meth:`keys`, +:meth:`values`, and :meth:`items` methods. + +.. seealso:: + + :pep:`3106` - Revamping dict.keys(), .values() and .items() + PEP written by Guido van Rossum. + Backported to 2.7 by Alexandre Vassalotti; :issue:`1967`. + + +PEP 3137: The memoryview Object +==================================================== + +The :class:`memoryview` object provides a view of another object's +memory content that matches the :class:`bytes` type's interface. + + >>> import string + >>> m = memoryview(string.letters) + >>> m + <memory at 0x37f850> + >>> len(m) # Returns length of underlying object + 52 + >>> m[0], m[25], m[26] # Indexing returns one byte + ('a', 'z', 'A') + >>> m2 = m[0:26] # Slicing returns another memoryview + >>> m2 + <memory at 0x37f080> + +The content of the view can be converted to a string of bytes or +a list of integers: + + >>> m2.tobytes() + 'abcdefghijklmnopqrstuvwxyz' + >>> m2.tolist() + [97, 98, 99, 100, 101, 102, 103, ... 121, 122] + >>> + +:class:`memoryview` objects allow modifying the underlying object if +it's a mutable object. + + >>> m2[0] = 75 + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: cannot modify read-only memory + >>> b = bytearray(string.letters) # Creating a mutable object + >>> b + bytearray(b'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ') + >>> mb = memoryview(b) + >>> mb[0] = '*' # Assign to view, changing the bytearray. + >>> b[0:5] # The bytearray has been changed. + bytearray(b'*bcde') + >>> + +.. seealso:: + + :pep:`3137` - Immutable Bytes and Mutable Buffer + PEP written by Guido van Rossum. + Implemented by Travis Oliphant, Antoine Pitrou and others. + Backported to 2.7 by Antoine Pitrou; :issue:`2396`. + + + +Other Language Changes +====================== + +Some smaller changes made to the core Python language are: + +* The syntax for set literals has been backported from Python 3.x. + Curly brackets are used to surround the contents of the resulting + mutable set; set literals are + distinguished from dictionaries by not containing colons and values. + ``{}`` continues to represent an empty dictionary; use + ``set()`` for an empty set. + + >>> {1,2,3,4,5} + set([1, 2, 3, 4, 5]) + >>> set() # empty set + set([]) + >>> {} # empty dict + {} + + Backported by Alexandre Vassalotti; :issue:`2335`. + +* Dictionary and set comprehensions are another feature backported from + 3.x, generalizing list/generator comprehensions to use + the literal syntax for sets and dictionaries. + + >>> {x: x*x for x in range(6)} + {0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25} + >>> {('a'*x) for x in range(6)} + set(['', 'a', 'aa', 'aaa', 'aaaa', 'aaaaa']) + + Backported by Alexandre Vassalotti; :issue:`2333`. + +* The :keyword:`with` statement can now use multiple context managers + in one statement. Context managers are processed from left to right + and each one is treated as beginning a new :keyword:`with` statement. + This means that:: + + with A() as a, B() as b: + ... suite of statements ... + + is equivalent to:: + + with A() as a: + with B() as b: + ... suite of statements ... + + The :func:`contextlib.nested` function provides a very similar + function, so it's no longer necessary and has been deprecated. + + (Proposed in http://codereview.appspot.com/53094; implemented by + Georg Brandl.) + +* Conversions between floating-point numbers and strings are + now correctly rounded on most platforms. These conversions occur + in many different places: :func:`str` on + floats and complex numbers; the :class:`float` and :class:`complex` + constructors; + numeric formatting; serializing and + deserializing floats and complex numbers using the + :mod:`marshal`, :mod:`pickle` + and :mod:`json` modules; + parsing of float and imaginary literals in Python code; + and :class:`~decimal.Decimal`-to-float conversion. + + Related to this, the :func:`repr` of a floating-point number *x* + now returns a result based on the shortest decimal string that's + guaranteed to round back to *x* under correct rounding (with + round-half-to-even rounding mode). Previously it gave a string + based on rounding x to 17 decimal digits. + + .. maybe add an example? + + The rounding library responsible for this improvement works on + Windows and on Unix platforms using the gcc, icc, or suncc + compilers. There may be a small number of platforms where correct + operation of this code cannot be guaranteed, so the code is not + used on such systems. You can find out which code is being used + by checking :data:`sys.float_repr_style`, which will be ``short`` + if the new code is in use and ``legacy`` if it isn't. + + Implemented by Eric Smith and Mark Dickinson, using David Gay's + :file:`dtoa.c` library; :issue:`7117`. + +* Conversions from long integers and regular integers to floating + point now round differently, returning the floating-point number + closest to the number. This doesn't matter for small integers that + can be converted exactly, but for large numbers that will + unavoidably lose precision, Python 2.7 now approximates more + closely. For example, Python 2.6 computed the following:: + + >>> n = 295147905179352891391 + >>> float(n) + 2.9514790517935283e+20 + >>> n - long(float(n)) + 65535L + + Python 2.7's floating-point result is larger, but much closer to the + true value:: + + >>> n = 295147905179352891391 + >>> float(n) + 2.9514790517935289e+20 + >>> n - long(float(n)) + -1L + + (Implemented by Mark Dickinson; :issue:`3166`.) + + Integer division is also more accurate in its rounding behaviours. (Also + implemented by Mark Dickinson; :issue:`1811`.) + +* Implicit coercion for complex numbers has been removed; the interpreter + will no longer ever attempt to call a :meth:`__coerce__` method on complex + objects. (Removed by Meador Inge and Mark Dickinson; :issue:`5211`.) + +* The :meth:`str.format` method now supports automatic numbering of the replacement + fields. This makes using :meth:`str.format` more closely resemble using + ``%s`` formatting:: + + >>> '{}:{}:{}'.format(2009, 04, 'Sunday') + '2009:4:Sunday' + >>> '{}:{}:{day}'.format(2009, 4, day='Sunday') + '2009:4:Sunday' + + The auto-numbering takes the fields from left to right, so the first ``{...}`` + specifier will use the first argument to :meth:`str.format`, the next + specifier will use the next argument, and so on. You can't mix auto-numbering + and explicit numbering -- either number all of your specifier fields or none + of them -- but you can mix auto-numbering and named fields, as in the second + example above. (Contributed by Eric Smith; :issue:`5237`.) + + Complex numbers now correctly support usage with :func:`format`, + and default to being right-aligned. + Specifying a precision or comma-separation applies to both the real + and imaginary parts of the number, but a specified field width and + alignment is applied to the whole of the resulting ``1.5+3j`` + output. (Contributed by Eric Smith; :issue:`1588` and :issue:`7988`.) + + The 'F' format code now always formats its output using uppercase characters, + so it will now produce 'INF' and 'NAN'. + (Contributed by Eric Smith; :issue:`3382`.) + + A low-level change: the :meth:`object.__format__` method now triggers + a :exc:`PendingDeprecationWarning` if it's passed a format string, + because the :meth:`__format__` method for :class:`object` converts + the object to a string representation and formats that. Previously + the method silently applied the format string to the string + representation, but that could hide mistakes in Python code. If + you're supplying formatting information such as an alignment or + precision, presumably you're expecting the formatting to be applied + in some object-specific way. (Fixed by Eric Smith; :issue:`7994`.) + +* The :func:`int` and :func:`long` types gained a ``bit_length`` + method that returns the number of bits necessary to represent + its argument in binary:: + + >>> n = 37 + >>> bin(n) + '0b100101' + >>> n.bit_length() + 6 + >>> n = 2**123-1 + >>> n.bit_length() + 123 + >>> (n+1).bit_length() + 124 + + (Contributed by Fredrik Johansson and Victor Stinner; :issue:`3439`.) + +* The :keyword:`import` statement will no longer try an absolute import + if a relative import (e.g. ``from .os import sep``) fails. This + fixes a bug, but could possibly break certain :keyword:`import` + statements that were only working by accident. (Fixed by Meador Inge; + :issue:`7902`.) + +* It's now possible for a subclass of the built-in :class:`unicode` type + to override the :meth:`__unicode__` method. (Implemented by + Victor Stinner; :issue:`1583863`.) + +* The :class:`bytearray` type's :meth:`~bytearray.translate` method now accepts + ``None`` as its first argument. (Fixed by Georg Brandl; + :issue:`4759`.) + + .. bytearray doesn't seem to be documented + +* When using ``@classmethod`` and ``@staticmethod`` to wrap + methods as class or static methods, the wrapper object now + exposes the wrapped function as their :attr:`__func__` attribute. + (Contributed by Amaury Forgeot d'Arc, after a suggestion by + George Sakkis; :issue:`5982`.) + +* When a restricted set of attributes were set using ``__slots__``, + deleting an unset attribute would not raise :exc:`AttributeError` + as you would expect. Fixed by Benjamin Peterson; :issue:`7604`.) + +* Two new encodings are now supported: "cp720", used primarily for + Arabic text; and "cp858", a variant of CP 850 that adds the euro + symbol. (CP720 contributed by Alexander Belchenko and Amaury + Forgeot d'Arc in :issue:`1616979`; CP858 contributed by Tim Hatch in + :issue:`8016`.) + +* The :class:`file` object will now set the :attr:`filename` attribute + on the :exc:`IOError` exception when trying to open a directory + on POSIX platforms (noted by Jan Kaliszewski; :issue:`4764`), and + now explicitly checks for and forbids writing to read-only file objects + instead of trusting the C library to catch and report the error + (fixed by Stefan Krah; :issue:`5677`). + +* The Python tokenizer now translates line endings itself, so the + :func:`compile` built-in function now accepts code using any + line-ending convention. Additionally, it no longer requires that the + code end in a newline. + +* Extra parentheses in function definitions are illegal in Python 3.x, + meaning that you get a syntax error from ``def f((x)): pass``. In + Python3-warning mode, Python 2.7 will now warn about this odd usage. + (Noted by James Lingard; :issue:`7362`.) + +* It's now possible to create weak references to old-style class + objects. New-style classes were always weak-referenceable. (Fixed + by Antoine Pitrou; :issue:`8268`.) + +* When a module object is garbage-collected, the module's dictionary is + now only cleared if no one else is holding a reference to the + dictionary (:issue:`7140`). + +.. ====================================================================== + +.. _new-27-interpreter: + +Interpreter Changes +------------------------------- + +A new environment variable, :envvar:`PYTHONWARNINGS`, +allows controlling warnings. It should be set to a string +containing warning settings, equivalent to those +used with the :option:`-W` switch, separated by commas. +(Contributed by Brian Curtin; :issue:`7301`.) + +For example, the following setting will print warnings every time +they occur, but turn warnings from the :mod:`Cookie` module into an +error. (The exact syntax for setting an environment variable varies +across operating systems and shells.) + +:: + + export PYTHONWARNINGS=all,error:::Cookie:0 + +.. ====================================================================== + + +Optimizations +------------- + +Several performance enhancements have been added: + +.. * A new :program:`configure` option, :option:`--with-computed-gotos`, + compiles the main bytecode interpreter loop using a new dispatch + mechanism that gives speedups of up to 20%, depending on the system + and benchmark. The new mechanism is only supported on certain + compilers, such as gcc, SunPro, and icc. + +* A new opcode was added to perform the initial setup for + :keyword:`with` statements, looking up the :meth:`__enter__` and + :meth:`__exit__` methods. (Contributed by Benjamin Peterson.) + +* The garbage collector now performs better for one common usage + pattern: when many objects are being allocated without deallocating + any of them. This would previously take quadratic + time for garbage collection, but now the number of full garbage collections + is reduced as the number of objects on the heap grows. + The new logic only performs a full garbage collection pass when + the middle generation has been collected 10 times and when the + number of survivor objects from the middle generation exceeds 10% of + the number of objects in the oldest generation. (Suggested by Martin + von Löwis and implemented by Antoine Pitrou; :issue:`4074`.) + +* The garbage collector tries to avoid tracking simple containers + which can't be part of a cycle. In Python 2.7, this is now true for + tuples and dicts containing atomic types (such as ints, strings, + etc.). Transitively, a dict containing tuples of atomic types won't + be tracked either. This helps reduce the cost of each + garbage collection by decreasing the number of objects to be + considered and traversed by the collector. + (Contributed by Antoine Pitrou; :issue:`4688`.) + +* Long integers are now stored internally either in base 2**15 or in base + 2**30, the base being determined at build time. Previously, they + were always stored in base 2**15. Using base 2**30 gives + significant performance improvements on 64-bit machines, but + benchmark results on 32-bit machines have been mixed. Therefore, + the default is to use base 2**30 on 64-bit machines and base 2**15 + on 32-bit machines; on Unix, there's a new configure option + :option:`--enable-big-digits` that can be used to override this default. + + Apart from the performance improvements this change should be + invisible to end users, with one exception: for testing and + debugging purposes there's a new structseq :data:`sys.long_info` that + provides information about the internal format, giving the number of + bits per digit and the size in bytes of the C type used to store + each digit:: + + >>> import sys + >>> sys.long_info + sys.long_info(bits_per_digit=30, sizeof_digit=4) + + (Contributed by Mark Dickinson; :issue:`4258`.) + + Another set of changes made long objects a few bytes smaller: 2 bytes + smaller on 32-bit systems and 6 bytes on 64-bit. + (Contributed by Mark Dickinson; :issue:`5260`.) + +* The division algorithm for long integers has been made faster + by tightening the inner loop, doing shifts instead of multiplications, + and fixing an unnecessary extra iteration. + Various benchmarks show speedups of between 50% and 150% for long + integer divisions and modulo operations. + (Contributed by Mark Dickinson; :issue:`5512`.) + Bitwise operations are also significantly faster (initial patch by + Gregory Smith; :issue:`1087418`). + +* The implementation of ``%`` checks for the left-side operand being + a Python string and special-cases it; this results in a 1-3% + performance increase for applications that frequently use ``%`` + with strings, such as templating libraries. + (Implemented by Collin Winter; :issue:`5176`.) + +* List comprehensions with an ``if`` condition are compiled into + faster bytecode. (Patch by Antoine Pitrou, back-ported to 2.7 + by Jeffrey Yasskin; :issue:`4715`.) + +* Converting an integer or long integer to a decimal string was made + faster by special-casing base 10 instead of using a generalized + conversion function that supports arbitrary bases. + (Patch by Gawain Bolton; :issue:`6713`.) + +* The :meth:`split`, :meth:`replace`, :meth:`rindex`, + :meth:`rpartition`, and :meth:`rsplit` methods of string-like types + (strings, Unicode strings, and :class:`bytearray` objects) now use a + fast reverse-search algorithm instead of a character-by-character + scan. This is sometimes faster by a factor of 10. (Added by + Florent Xicluna; :issue:`7462` and :issue:`7622`.) + +* The :mod:`pickle` and :mod:`cPickle` modules now automatically + intern the strings used for attribute names, reducing memory usage + of the objects resulting from unpickling. (Contributed by Jake + McGuire; :issue:`5084`.) + +* The :mod:`cPickle` module now special-cases dictionaries, + nearly halving the time required to pickle them. + (Contributed by Collin Winter; :issue:`5670`.) + +.. ====================================================================== + +New and Improved Modules +======================== + +As in every release, Python's standard library received a number of +enhancements and bug fixes. Here's a partial list of the most notable +changes, sorted alphabetically by module name. Consult the +:file:`Misc/NEWS` file in the source tree for a more complete list of +changes, or look through the Subversion logs for all the details. + +* The :mod:`bdb` module's base debugging class :class:`~bdb.Bdb` + gained a feature for skipping modules. The constructor + now takes an iterable containing glob-style patterns such as + ``django.*``; the debugger will not step into stack frames + from a module that matches one of these patterns. + (Contributed by Maru Newby after a suggestion by + Senthil Kumaran; :issue:`5142`.) + +* The :mod:`binascii` module now supports the buffer API, so it can be + used with :class:`memoryview` instances and other similar buffer objects. + (Backported from 3.x by Florent Xicluna; :issue:`7703`.) + +* Updated module: the :mod:`bsddb` module has been updated from 4.7.2devel9 + to version 4.8.4 of + `the pybsddb package <http://www.jcea.es/programacion/pybsddb.htm>`__. + The new version features better Python 3.x compatibility, various bug fixes, + and adds several new BerkeleyDB flags and methods. + (Updated by Jesús Cea Avión; :issue:`8156`. The pybsddb + changelog can be read at http://hg.jcea.es/pybsddb/file/tip/ChangeLog.) + +* The :mod:`bz2` module's :class:`~bz2.BZ2File` now supports the context + management protocol, so you can write ``with bz2.BZ2File(...) as f:``. + (Contributed by Hagen Fürstenau; :issue:`3860`.) + +* New class: the :class:`~collections.Counter` class in the :mod:`collections` + module is useful for tallying data. :class:`~collections.Counter` instances + behave mostly like dictionaries but return zero for missing keys instead of + raising a :exc:`KeyError`: + + .. doctest:: + :options: +NORMALIZE_WHITESPACE + + >>> from collections import Counter + >>> c = Counter() + >>> for letter in 'here is a sample of english text': + ... c[letter] += 1 + ... + >>> c + Counter({' ': 6, 'e': 5, 's': 3, 'a': 2, 'i': 2, 'h': 2, + 'l': 2, 't': 2, 'g': 1, 'f': 1, 'm': 1, 'o': 1, 'n': 1, + 'p': 1, 'r': 1, 'x': 1}) + >>> c['e'] + 5 + >>> c['z'] + 0 + + There are three additional :class:`~collections.Counter` methods. + :meth:`~collections.Counter.most_common` returns the N most common + elements and their counts. :meth:`~collections.Counter.elements` + returns an iterator over the contained elements, repeating each + element as many times as its count. + :meth:`~collections.Counter.subtract` takes an iterable and + subtracts one for each element instead of adding; if the argument is + a dictionary or another :class:`Counter`, the counts are + subtracted. :: + + >>> c.most_common(5) + [(' ', 6), ('e', 5), ('s', 3), ('a', 2), ('i', 2)] + >>> c.elements() -> + 'a', 'a', ' ', ' ', ' ', ' ', ' ', ' ', + 'e', 'e', 'e', 'e', 'e', 'g', 'f', 'i', 'i', + 'h', 'h', 'm', 'l', 'l', 'o', 'n', 'p', 's', + 's', 's', 'r', 't', 't', 'x' + >>> c['e'] + 5 + >>> c.subtract('very heavy on the letter e') + >>> c['e'] # Count is now lower + -1 + + Contributed by Raymond Hettinger; :issue:`1696199`. + + .. revision 79660 + + New class: :class:`~collections.OrderedDict` is described in the earlier + section :ref:`pep-0372`. + + New method: The :class:`~collections.deque` data type now has a + :meth:`~collections.deque.count` method that returns the number of + contained elements equal to the supplied argument *x*, and a + :meth:`~collections.deque.reverse` method that reverses the elements + of the deque in-place. :class:`deque` also exposes its maximum + length as the read-only :attr:`~collections.deque.maxlen` attribute. + (Both features added by Raymond Hettinger.) + + The :class:`~collections.namedtuple` class now has an optional *rename* parameter. + If *rename* is true, field names that are invalid because they've + been repeated or aren't legal Python identifiers will be + renamed to legal names that are derived from the field's + position within the list of fields: + + >>> from collections import namedtuple + >>> T = namedtuple('T', ['field1', '$illegal', 'for', 'field2'], rename=True) + >>> T._fields + ('field1', '_1', '_2', 'field2') + + (Added by Raymond Hettinger; :issue:`1818`.) + + Finally, the :class:`~collections.Mapping` abstract base class now + returns :const:`NotImplemented` if a mapping is compared to + another type that isn't a :class:`Mapping`. + (Fixed by Daniel Stutzbach; :issue:`8729`.) + +* Constructors for the parsing classes in the :mod:`ConfigParser` module now + take a *allow_no_value* parameter, defaulting to false; if true, + options without values will be allowed. For example:: + + >>> import ConfigParser, StringIO + >>> sample_config = """ + ... [mysqld] + ... user = mysql + ... pid-file = /var/run/mysqld/mysqld.pid + ... skip-bdb + ... """ + >>> config = ConfigParser.RawConfigParser(allow_no_value=True) + >>> config.readfp(StringIO.StringIO(sample_config)) + >>> config.get('mysqld', 'user') + 'mysql' + >>> print config.get('mysqld', 'skip-bdb') + None + >>> print config.get('mysqld', 'unknown') + Traceback (most recent call last): + ... + NoOptionError: No option 'unknown' in section: 'mysqld' + + (Contributed by Mats Kindahl; :issue:`7005`.) + +* Deprecated function: :func:`contextlib.nested`, which allows + handling more than one context manager with a single :keyword:`with` + statement, has been deprecated, because the :keyword:`with` statement + now supports multiple context managers. + +* The :mod:`cookielib` module now ignores cookies that have an invalid + version field, one that doesn't contain an integer value. (Fixed by + John J. Lee; :issue:`3924`.) + +* The :mod:`copy` module's :func:`~copy.deepcopy` function will now + correctly copy bound instance methods. (Implemented by + Robert Collins; :issue:`1515`.) + +* The :mod:`ctypes` module now always converts ``None`` to a C NULL + pointer for arguments declared as pointers. (Changed by Thomas + Heller; :issue:`4606`.) The underlying `libffi library + <http://sourceware.org/libffi/>`__ has been updated to version + 3.0.9, containing various fixes for different platforms. (Updated + by Matthias Klose; :issue:`8142`.) + +* New method: the :mod:`datetime` module's :class:`~datetime.timedelta` class + gained a :meth:`~datetime.timedelta.total_seconds` method that returns the + number of seconds in the duration. (Contributed by Brian Quinlan; :issue:`5788`.) + +* New method: the :class:`~decimal.Decimal` class gained a + :meth:`~decimal.Decimal.from_float` class method that performs an exact + conversion of a floating-point number to a :class:`~decimal.Decimal`. + This exact conversion strives for the + closest decimal approximation to the floating-point representation's value; + the resulting decimal value will therefore still include the inaccuracy, + if any. + For example, ``Decimal.from_float(0.1)`` returns + ``Decimal('0.1000000000000000055511151231257827021181583404541015625')``. + (Implemented by Raymond Hettinger; :issue:`4796`.) + + Comparing instances of :class:`Decimal` with floating-point + numbers now produces sensible results based on the numeric values + of the operands. Previously such comparisons would fall back to + Python's default rules for comparing objects, which produced arbitrary + results based on their type. Note that you still cannot combine + :class:`Decimal` and floating-point in other operations such as addition, + since you should be explicitly choosing how to convert between float and + :class:`Decimal`. + (Fixed by Mark Dickinson; :issue:`2531`.) + + The constructor for :class:`~decimal.Decimal` now accepts + floating-point numbers (added by Raymond Hettinger; :issue:`8257`) + and non-European Unicode characters such as Arabic-Indic digits + (contributed by Mark Dickinson; :issue:`6595`). + + Most of the methods of the :class:`~decimal.Context` class now accept integers + as well as :class:`~decimal.Decimal` instances; the only exceptions are the + :meth:`~decimal.Context.canonical` and :meth:`~decimal.Context.is_canonical` + methods. (Patch by Juan José Conti; :issue:`7633`.) + + When using :class:`~decimal.Decimal` instances with a string's + :meth:`~str.format` method, the default alignment was previously + left-alignment. This has been changed to right-alignment, which is + more sensible for numeric types. (Changed by Mark Dickinson; :issue:`6857`.) + + Comparisons involving a signaling NaN value (or ``sNAN``) now signal + :const:`InvalidOperation` instead of silently returning a true or + false value depending on the comparison operator. Quiet NaN values + (or ``NaN``) are now hashable. (Fixed by Mark Dickinson; + :issue:`7279`.) + +* The :mod:`difflib` module now produces output that is more + compatible with modern :command:`diff`/:command:`patch` tools + through one small change, using a tab character instead of spaces as + a separator in the header giving the filename. (Fixed by Anatoly + Techtonik; :issue:`7585`.) + +* The Distutils ``sdist`` command now always regenerates the + :file:`MANIFEST` file, since even if the :file:`MANIFEST.in` or + :file:`setup.py` files haven't been modified, the user might have + created some new files that should be included. + (Fixed by Tarek Ziadé; :issue:`8688`.) + +* The :mod:`doctest` module's :const:`IGNORE_EXCEPTION_DETAIL` flag + will now ignore the name of the module containing the exception + being tested. (Patch by Lennart Regebro; :issue:`7490`.) + +* The :mod:`email` module's :class:`~email.message.Message` class will + now accept a Unicode-valued payload, automatically converting the + payload to the encoding specified by :attr:`output_charset`. + (Added by R. David Murray; :issue:`1368247`.) + +* The :class:`~fractions.Fraction` class now accepts a single float or + :class:`~decimal.Decimal` instance, or two rational numbers, as + arguments to its constructor. (Implemented by Mark Dickinson; + rationals added in :issue:`5812`, and float/decimal in + :issue:`8294`.) + + Ordering comparisons (``<``, ``<=``, ``>``, ``>=``) between + fractions and complex numbers now raise a :exc:`TypeError`. + This fixes an oversight, making the :class:`Fraction` match the other + numeric types. + + .. revision 79455 + +* New class: :class:`~ftplib.FTP_TLS` in + the :mod:`ftplib` module provides secure FTP + connections using TLS encapsulation of authentication as well as + subsequent control and data transfers. + (Contributed by Giampaolo Rodola; :issue:`2054`.) + + The :meth:`~ftplib.FTP.storbinary` method for binary uploads can now restart + uploads thanks to an added *rest* parameter (patch by Pablo Mouzo; + :issue:`6845`.) + +* New class decorator: :func:`total_ordering` in the :mod:`functools` + module takes a class that defines an :meth:`__eq__` method and one of + :meth:`__lt__`, :meth:`__le__`, :meth:`__gt__`, or :meth:`__ge__`, + and generates the missing comparison methods. Since the + :meth:`__cmp__` method is being deprecated in Python 3.x, + this decorator makes it easier to define ordered classes. + (Added by Raymond Hettinger; :issue:`5479`.) + + New function: :func:`cmp_to_key` will take an old-style comparison + function that expects two arguments and return a new callable that + can be used as the *key* parameter to functions such as + :func:`sorted`, :func:`min` and :func:`max`, etc. The primary + intended use is to help with making code compatible with Python 3.x. + (Added by Raymond Hettinger.) + +* New function: the :mod:`gc` module's :func:`~gc.is_tracked` returns + true if a given instance is tracked by the garbage collector, false + otherwise. (Contributed by Antoine Pitrou; :issue:`4688`.) + +* The :mod:`gzip` module's :class:`~gzip.GzipFile` now supports the context + management protocol, so you can write ``with gzip.GzipFile(...) as f:`` + (contributed by Hagen Fürstenau; :issue:`3860`), and it now implements + the :class:`io.BufferedIOBase` ABC, so you can wrap it with + :class:`io.BufferedReader` for faster processing + (contributed by Nir Aides; :issue:`7471`). + It's also now possible to override the modification time + recorded in a gzipped file by providing an optional timestamp to + the constructor. (Contributed by Jacques Frechet; :issue:`4272`.) + + Files in gzip format can be padded with trailing zero bytes; the + :mod:`gzip` module will now consume these trailing bytes. (Fixed by + Tadek Pietraszek and Brian Curtin; :issue:`2846`.) + +* New attribute: the :mod:`hashlib` module now has an :attr:`~hashlib.hashlib.algorithms` + attribute containing a tuple naming the supported algorithms. + In Python 2.7, ``hashlib.algorithms`` contains + ``('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512')``. + (Contributed by Carl Chenet; :issue:`7418`.) + +* The default :class:`~httplib.HTTPResponse` class used by the :mod:`httplib` module now + supports buffering, resulting in much faster reading of HTTP responses. + (Contributed by Kristján Valur Jónsson; :issue:`4879`.) + + The :class:`~httplib.HTTPConnection` and :class:`~httplib.HTTPSConnection` classes + now support a *source_address* parameter, a ``(host, port)`` 2-tuple + giving the source address that will be used for the connection. + (Contributed by Eldon Ziegler; :issue:`3972`.) + +* The :mod:`ihooks` module now supports relative imports. Note that + :mod:`ihooks` is an older module for customizing imports, + superseded by the :mod:`imputil` module added in Python 2.0. + (Relative import support added by Neil Schemenauer.) + + .. revision 75423 + +* The :mod:`imaplib` module now supports IPv6 addresses. + (Contributed by Derek Morr; :issue:`1655`.) + +* New function: the :mod:`inspect` module's :func:`~inspect.getcallargs` + takes a callable and its positional and keyword arguments, + and figures out which of the callable's parameters will receive each argument, + returning a dictionary mapping argument names to their values. For example:: + + >>> from inspect import getcallargs + >>> def f(a, b=1, *pos, **named): + ... pass + >>> getcallargs(f, 1, 2, 3) + {'a': 1, 'b': 2, 'pos': (3,), 'named': {}} + >>> getcallargs(f, a=2, x=4) + {'a': 2, 'b': 1, 'pos': (), 'named': {'x': 4}} + >>> getcallargs(f) + Traceback (most recent call last): + ... + TypeError: f() takes at least 1 argument (0 given) + + Contributed by George Sakkis; :issue:`3135`. + +* Updated module: The :mod:`io` library has been upgraded to the version shipped with + Python 3.1. For 3.1, the I/O library was entirely rewritten in C + and is 2 to 20 times faster depending on the task being performed. The + original Python version was renamed to the :mod:`_pyio` module. + + One minor resulting change: the :class:`io.TextIOBase` class now + has an :attr:`errors` attribute giving the error setting + used for encoding and decoding errors (one of ``'strict'``, ``'replace'``, + ``'ignore'``). + + The :class:`io.FileIO` class now raises an :exc:`OSError` when passed + an invalid file descriptor. (Implemented by Benjamin Peterson; + :issue:`4991`.) The :meth:`~io.IOBase.truncate` method now preserves the + file position; previously it would change the file position to the + end of the new file. (Fixed by Pascal Chambon; :issue:`6939`.) + +* New function: ``itertools.compress(data, selectors)`` takes two + iterators. Elements of *data* are returned if the corresponding + value in *selectors* is true:: + + itertools.compress('ABCDEF', [1,0,1,0,1,1]) => + A, C, E, F + + .. maybe here is better to use >>> list(itertools.compress(...)) instead + + New function: ``itertools.combinations_with_replacement(iter, r)`` + returns all the possible *r*-length combinations of elements from the + iterable *iter*. Unlike :func:`~itertools.combinations`, individual elements + can be repeated in the generated combinations:: + + itertools.combinations_with_replacement('abc', 2) => + ('a', 'a'), ('a', 'b'), ('a', 'c'), + ('b', 'b'), ('b', 'c'), ('c', 'c') + + Note that elements are treated as unique depending on their position + in the input, not their actual values. + + The :func:`itertools.count` function now has a *step* argument that + allows incrementing by values other than 1. :func:`~itertools.count` also + now allows keyword arguments, and using non-integer values such as + floats or :class:`~decimal.Decimal` instances. (Implemented by Raymond + Hettinger; :issue:`5032`.) + + :func:`itertools.combinations` and :func:`itertools.product` + previously raised :exc:`ValueError` for values of *r* larger than + the input iterable. This was deemed a specification error, so they + now return an empty iterator. (Fixed by Raymond Hettinger; :issue:`4816`.) + +* Updated module: The :mod:`json` module was upgraded to version 2.0.9 of the + simplejson package, which includes a C extension that makes + encoding and decoding faster. + (Contributed by Bob Ippolito; :issue:`4136`.) + + To support the new :class:`collections.OrderedDict` type, :func:`json.load` + now has an optional *object_pairs_hook* parameter that will be called + with any object literal that decodes to a list of pairs. + (Contributed by Raymond Hettinger; :issue:`5381`.) + +* The :mod:`mailbox` module's :class:`Maildir` class now records the + timestamp on the directories it reads, and only re-reads them if the + modification time has subsequently changed. This improves + performance by avoiding unneeded directory scans. (Fixed by + A.M. Kuchling and Antoine Pitrou; :issue:`1607951`, :issue:`6896`.) + +* New functions: the :mod:`math` module gained + :func:`~math.erf` and :func:`~math.erfc` for the error function and the complementary error function, + :func:`~math.expm1` which computes ``e**x - 1`` with more precision than + using :func:`~math.exp` and subtracting 1, + :func:`~math.gamma` for the Gamma function, and + :func:`~math.lgamma` for the natural log of the Gamma function. + (Contributed by Mark Dickinson and nirinA raseliarison; :issue:`3366`.) + +* The :mod:`multiprocessing` module's :class:`Manager*` classes + can now be passed a callable that will be called whenever + a subprocess is started, along with a set of arguments that will be + passed to the callable. + (Contributed by lekma; :issue:`5585`.) + + The :class:`~multiprocessing.Pool` class, which controls a pool of worker processes, + now has an optional *maxtasksperchild* parameter. Worker processes + will perform the specified number of tasks and then exit, causing the + :class:`~multiprocessing.Pool` to start a new worker. This is useful if tasks may leak + memory or other resources, or if some tasks will cause the worker to + become very large. + (Contributed by Charles Cazabon; :issue:`6963`.) + +* The :mod:`nntplib` module now supports IPv6 addresses. + (Contributed by Derek Morr; :issue:`1664`.) + +* New functions: the :mod:`os` module wraps the following POSIX system + calls: :func:`~os.getresgid` and :func:`~os.getresuid`, which return the + real, effective, and saved GIDs and UIDs; + :func:`~os.setresgid` and :func:`~os.setresuid`, which set + real, effective, and saved GIDs and UIDs to new values; + :func:`~os.initgroups`, which initialize the group access list + for the current process. (GID/UID functions + contributed by Travis H.; :issue:`6508`. Support for initgroups added + by Jean-Paul Calderone; :issue:`7333`.) + + The :func:`os.fork` function now re-initializes the import lock in + the child process; this fixes problems on Solaris when :func:`~os.fork` + is called from a thread. (Fixed by Zsolt Cserna; :issue:`7242`.) + +* In the :mod:`os.path` module, the :func:`~os.path.normpath` and + :func:`~os.path.abspath` functions now preserve Unicode; if their input path + is a Unicode string, the return value is also a Unicode string. + (:meth:`~os.path.normpath` fixed by Matt Giuca in :issue:`5827`; + :meth:`~os.path.abspath` fixed by Ezio Melotti in :issue:`3426`.) + +* The :mod:`pydoc` module now has help for the various symbols that Python + uses. You can now do ``help('<<')`` or ``help('@')``, for example. + (Contributed by David Laban; :issue:`4739`.) + +* The :mod:`re` module's :func:`~re.split`, :func:`~re.sub`, and :func:`~re.subn` + now accept an optional *flags* argument, for consistency with the + other functions in the module. (Added by Gregory P. Smith.) + +* New function: :func:`~runpy.run_path` in the :mod:`runpy` module + will execute the code at a provided *path* argument. *path* can be + the path of a Python source file (:file:`example.py`), a compiled + bytecode file (:file:`example.pyc`), a directory + (:file:`./package/`), or a zip archive (:file:`example.zip`). If a + directory or zip path is provided, it will be added to the front of + ``sys.path`` and the module :mod:`__main__` will be imported. It's + expected that the directory or zip contains a :file:`__main__.py`; + if it doesn't, some other :file:`__main__.py` might be imported from + a location later in ``sys.path``. This makes more of the machinery + of :mod:`runpy` available to scripts that want to mimic the way + Python's command line processes an explicit path name. + (Added by Nick Coghlan; :issue:`6816`.) + +* New function: in the :mod:`shutil` module, :func:`~shutil.make_archive` + takes a filename, archive type (zip or tar-format), and a directory + path, and creates an archive containing the directory's contents. + (Added by Tarek Ziadé.) + + :mod:`shutil`'s :func:`~shutil.copyfile` and :func:`~shutil.copytree` + functions now raise a :exc:`~shutil.SpecialFileError` exception when + asked to copy a named pipe. Previously the code would treat + named pipes like a regular file by opening them for reading, and + this would block indefinitely. (Fixed by Antoine Pitrou; :issue:`3002`.) + +* The :mod:`signal` module no longer re-installs the signal handler + unless this is truly necessary, which fixes a bug that could make it + impossible to catch the EINTR signal robustly. (Fixed by + Charles-Francois Natali; :issue:`8354`.) + +* New functions: in the :mod:`site` module, three new functions + return various site- and user-specific paths. + :func:`~site.getsitepackages` returns a list containing all + global site-packages directories, + :func:`~site.getusersitepackages` returns the path of the user's + site-packages directory, and + :func:`~site.getuserbase` returns the value of the :envvar:`USER_BASE` + environment variable, giving the path to a directory that can be used + to store data. + (Contributed by Tarek Ziadé; :issue:`6693`.) + + The :mod:`site` module now reports exceptions occurring + when the :mod:`sitecustomize` module is imported, and will no longer + catch and swallow the :exc:`KeyboardInterrupt` exception. (Fixed by + Victor Stinner; :issue:`3137`.) + +* The :func:`~socket.create_connection` function + gained a *source_address* parameter, a ``(host, port)`` 2-tuple + giving the source address that will be used for the connection. + (Contributed by Eldon Ziegler; :issue:`3972`.) + + The :meth:`~socket.socket.recv_into` and :meth:`~socket.socket.recvfrom_into` + methods will now write into objects that support the buffer API, most usefully + the :class:`bytearray` and :class:`memoryview` objects. (Implemented by + Antoine Pitrou; :issue:`8104`.) + +* The :mod:`SocketServer` module's :class:`~SocketServer.TCPServer` class now + supports socket timeouts and disabling the Nagle algorithm. + The :attr:`~SocketServer.TCPServer.disable_nagle_algorithm` class attribute + defaults to False; if overridden to be True, + new request connections will have the TCP_NODELAY option set to + prevent buffering many small sends into a single TCP packet. + The :attr:`~SocketServer.TCPServer.timeout` class attribute can hold + a timeout in seconds that will be applied to the request socket; if + no request is received within that time, :meth:`handle_timeout` + will be called and :meth:`handle_request` will return. + (Contributed by Kristján Valur Jónsson; :issue:`6192` and :issue:`6267`.) + +* Updated module: the :mod:`sqlite3` module has been updated to + version 2.6.0 of the `pysqlite package <http://code.google.com/p/pysqlite/>`__. Version 2.6.0 includes a number of bugfixes, and adds + the ability to load SQLite extensions from shared libraries. + Call the ``enable_load_extension(True)`` method to enable extensions, + and then call :meth:`~sqlite3.Connection.load_extension` to load a particular shared library. + (Updated by Gerhard Häring.) + +* The :mod:`ssl` module's :class:`ssl.SSLSocket` objects now support the + buffer API, which fixed a test suite failure (fix by Antoine Pitrou; + :issue:`7133`) and automatically set + OpenSSL's :c:macro:`SSL_MODE_AUTO_RETRY`, which will prevent an error + code being returned from :meth:`recv` operations that trigger an SSL + renegotiation (fix by Antoine Pitrou; :issue:`8222`). + + The :func:`ssl.wrap_socket` constructor function now takes a + *ciphers* argument that's a string listing the encryption algorithms + to be allowed; the format of the string is described + `in the OpenSSL documentation + <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`__. + (Added by Antoine Pitrou; :issue:`8322`.) + + Another change makes the extension load all of OpenSSL's ciphers and + digest algorithms so that they're all available. Some SSL + certificates couldn't be verified, reporting an "unknown algorithm" + error. (Reported by Beda Kosata, and fixed by Antoine Pitrou; + :issue:`8484`.) + + The version of OpenSSL being used is now available as the module + attributes :data:`ssl.OPENSSL_VERSION` (a string), + :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and + :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer). (Added by Antoine + Pitrou; :issue:`8321`.) + +* The :mod:`struct` module will no longer silently ignore overflow + errors when a value is too large for a particular integer format + code (one of ``bBhHiIlLqQ``); it now always raises a + :exc:`struct.error` exception. (Changed by Mark Dickinson; + :issue:`1523`.) The :func:`~struct.pack` function will also + attempt to use :meth:`__index__` to convert and pack non-integers + before trying the :meth:`__int__` method or reporting an error. + (Changed by Mark Dickinson; :issue:`8300`.) + +* New function: the :mod:`subprocess` module's + :func:`~subprocess.check_output` runs a command with a specified set of arguments + and returns the command's output as a string when the command runs without + error, or raises a :exc:`~subprocess.CalledProcessError` exception otherwise. + + :: + + >>> subprocess.check_output(['df', '-h', '.']) + 'Filesystem Size Used Avail Capacity Mounted on\n + /dev/disk0s2 52G 49G 3.0G 94% /\n' + + >>> subprocess.check_output(['df', '-h', '/bogus']) + ... + subprocess.CalledProcessError: Command '['df', '-h', '/bogus']' returned non-zero exit status 1 + + (Contributed by Gregory P. Smith.) + + The :mod:`subprocess` module will now retry its internal system calls + on receiving an :const:`EINTR` signal. (Reported by several people; final + patch by Gregory P. Smith in :issue:`1068268`.) + +* New function: :func:`~symtable.is_declared_global` in the :mod:`symtable` module + returns true for variables that are explicitly declared to be global, + false for ones that are implicitly global. + (Contributed by Jeremy Hylton.) + +* The :mod:`syslog` module will now use the value of ``sys.argv[0]`` as the + identifier instead of the previous default value of ``'python'``. + (Changed by Sean Reifschneider; :issue:`8451`.) + +* The ``sys.version_info`` value is now a named tuple, with attributes + named :attr:`major`, :attr:`minor`, :attr:`micro`, + :attr:`releaselevel`, and :attr:`serial`. (Contributed by Ross + Light; :issue:`4285`.) + + :func:`sys.getwindowsversion` also returns a named tuple, + with attributes named :attr:`major`, :attr:`minor`, :attr:`build`, + :attr:`platform`, :attr:`service_pack`, :attr:`service_pack_major`, + :attr:`service_pack_minor`, :attr:`suite_mask`, and + :attr:`product_type`. (Contributed by Brian Curtin; :issue:`7766`.) + +* The :mod:`tarfile` module's default error handling has changed, to + no longer suppress fatal errors. The default error level was previously 0, + which meant that errors would only result in a message being written to the + debug log, but because the debug log is not activated by default, + these errors go unnoticed. The default error level is now 1, + which raises an exception if there's an error. + (Changed by Lars Gustäbel; :issue:`7357`.) + + :mod:`tarfile` now supports filtering the :class:`~tarfile.TarInfo` + objects being added to a tar file. When you call :meth:`~tarfile.TarFile.add`, + you may supply an optional *filter* argument + that's a callable. The *filter* callable will be passed the + :class:`~tarfile.TarInfo` for every file being added, and can modify and return it. + If the callable returns ``None``, the file will be excluded from the + resulting archive. This is more powerful than the existing + *exclude* argument, which has therefore been deprecated. + (Added by Lars Gustäbel; :issue:`6856`.) + The :class:`~tarfile.TarFile` class also now supports the context manager protocol. + (Added by Lars Gustäbel; :issue:`7232`.) + +* The :meth:`~threading.Event.wait` method of the :class:`threading.Event` class + now returns the internal flag on exit. This means the method will usually + return true because :meth:`~threading.Event.wait` is supposed to block until the + internal flag becomes true. The return value will only be false if + a timeout was provided and the operation timed out. + (Contributed by Tim Lesher; :issue:`1674032`.) + +* The Unicode database provided by the :mod:`unicodedata` module is + now used internally to determine which characters are numeric, + whitespace, or represent line breaks. The database also + includes information from the :file:`Unihan.txt` data file (patch + by Anders Chrigström and Amaury Forgeot d'Arc; :issue:`1571184`) + and has been updated to version 5.2.0 (updated by + Florent Xicluna; :issue:`8024`). + +* The :mod:`urlparse` module's :func:`~urlparse.urlsplit` now handles + unknown URL schemes in a fashion compliant with :rfc:`3986`: if the + URL is of the form ``"<something>://..."``, the text before the + ``://`` is treated as the scheme, even if it's a made-up scheme that + the module doesn't know about. This change may break code that + worked around the old behaviour. For example, Python 2.6.4 or 2.5 + will return the following: + + >>> import urlparse + >>> urlparse.urlsplit('invented://host/filename?query') + ('invented', '', '//host/filename?query', '', '') + + Python 2.7 (and Python 2.6.5) will return: + + >>> import urlparse + >>> urlparse.urlsplit('invented://host/filename?query') + ('invented', 'host', '/filename?query', '', '') + + (Python 2.7 actually produces slightly different output, since it + returns a named tuple instead of a standard tuple.) + + The :mod:`urlparse` module also supports IPv6 literal addresses as defined by + :rfc:`2732` (contributed by Senthil Kumaran; :issue:`2987`). :: + + >>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') + ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]', + path='/foo', params='', query='', fragment='') + +* New class: the :class:`~weakref.WeakSet` class in the :mod:`weakref` + module is a set that only holds weak references to its elements; elements + will be removed once there are no references pointing to them. + (Originally implemented in Python 3.x by Raymond Hettinger, and backported + to 2.7 by Michael Foord.) + +* The ElementTree library, :mod:`xml.etree`, no longer escapes + ampersands and angle brackets when outputting an XML processing + instruction (which looks like ``<?xml-stylesheet href="#style1"?>``) + or comment (which looks like ``<!-- comment -->``). + (Patch by Neil Muller; :issue:`2746`.) + +* The XML-RPC client and server, provided by the :mod:`xmlrpclib` and + :mod:`SimpleXMLRPCServer` modules, have improved performance by + supporting HTTP/1.1 keep-alive and by optionally using gzip encoding + to compress the XML being exchanged. The gzip compression is + controlled by the :attr:`encode_threshold` attribute of + :class:`SimpleXMLRPCRequestHandler`, which contains a size in bytes; + responses larger than this will be compressed. + (Contributed by Kristján Valur Jónsson; :issue:`6267`.) + +* The :mod:`zipfile` module's :class:`~zipfile.ZipFile` now supports the context + management protocol, so you can write ``with zipfile.ZipFile(...) as f:``. + (Contributed by Brian Curtin; :issue:`5511`.) + + :mod:`zipfile` now also supports archiving empty directories and + extracts them correctly. (Fixed by Kuba Wieczorek; :issue:`4710`.) + Reading files out of an archive is faster, and interleaving + :meth:`~zipfile.ZipFile.read` and :meth:`~zipfile.ZipFile.readline` now works correctly. + (Contributed by Nir Aides; :issue:`7610`.) + + The :func:`~zipfile.is_zipfile` function now + accepts a file object, in addition to the path names accepted in earlier + versions. (Contributed by Gabriel Genellina; :issue:`4756`.) + + The :meth:`~zipfile.ZipFile.writestr` method now has an optional *compress_type* parameter + that lets you override the default compression method specified in the + :class:`~zipfile.ZipFile` constructor. (Contributed by Ronald Oussoren; + :issue:`6003`.) + + +.. ====================================================================== +.. whole new modules get described in subsections here + + +.. _importlib-section: + +New module: importlib +------------------------------ + +Python 3.1 includes the :mod:`importlib` package, a re-implementation +of the logic underlying Python's :keyword:`import` statement. +:mod:`importlib` is useful for implementors of Python interpreters and +to users who wish to write new importers that can participate in the +import process. Python 2.7 doesn't contain the complete +:mod:`importlib` package, but instead has a tiny subset that contains +a single function, :func:`~importlib.import_module`. + +``import_module(name, package=None)`` imports a module. *name* is +a string containing the module or package's name. It's possible to do +relative imports by providing a string that begins with a ``.`` +character, such as ``..utils.errors``. For relative imports, the +*package* argument must be provided and is the name of the package that +will be used as the anchor for +the relative import. :func:`~importlib.import_module` both inserts the imported +module into ``sys.modules`` and returns the module object. + +Here are some examples:: + + >>> from importlib import import_module + >>> anydbm = import_module('anydbm') # Standard absolute import + >>> anydbm + <module 'anydbm' from '/p/python/Lib/anydbm.py'> + >>> # Relative import + >>> file_util = import_module('..file_util', 'distutils.command') + >>> file_util + <module 'distutils.file_util' from '/python/Lib/distutils/file_util.pyc'> + +:mod:`importlib` was implemented by Brett Cannon and introduced in +Python 3.1. + + +New module: sysconfig +--------------------------------- + +The :mod:`sysconfig` module has been pulled out of the Distutils +package, becoming a new top-level module in its own right. +:mod:`sysconfig` provides functions for getting information about +Python's build process: compiler switches, installation paths, the +platform name, and whether Python is running from its source +directory. + +Some of the functions in the module are: + +* :func:`~sysconfig.get_config_var` returns variables from Python's + Makefile and the :file:`pyconfig.h` file. +* :func:`~sysconfig.get_config_vars` returns a dictionary containing + all of the configuration variables. +* :func:`~sysconfig.getpath` returns the configured path for + a particular type of module: the standard library, + site-specific modules, platform-specific modules, etc. +* :func:`~sysconfig.is_python_build` returns true if you're running a + binary from a Python source tree, and false otherwise. + +Consult the :mod:`sysconfig` documentation for more details and for +a complete list of functions. + +The Distutils package and :mod:`sysconfig` are now maintained by Tarek +Ziadé, who has also started a Distutils2 package (source repository at +http://hg.python.org/distutils2/) for developing a next-generation +version of Distutils. + + +ttk: Themed Widgets for Tk +-------------------------- + +Tcl/Tk 8.5 includes a set of themed widgets that re-implement basic Tk +widgets but have a more customizable appearance and can therefore more +closely resemble the native platform's widgets. This widget +set was originally called Tile, but was renamed to Ttk (for "themed Tk") +on being added to Tcl/Tck release 8.5. + +To learn more, read the :mod:`ttk` module documentation. You may also +wish to read the Tcl/Tk manual page describing the +Ttk theme engine, available at +http://www.tcl.tk/man/tcl8.5/TkCmd/ttk_intro.htm. Some +screenshots of the Python/Ttk code in use are at +http://code.google.com/p/python-ttk/wiki/Screenshots. + +The :mod:`ttk` module was written by Guilherme Polo and added in +:issue:`2983`. An alternate version called ``Tile.py``, written by +Martin Franklin and maintained by Kevin Walzer, was proposed for +inclusion in :issue:`2618`, but the authors argued that Guilherme +Polo's work was more comprehensive. + + +.. _unittest-section: + +Updated module: unittest +--------------------------------- + +The :mod:`unittest` module was greatly enhanced; many +new features were added. Most of these features were implemented +by Michael Foord, unless otherwise noted. The enhanced version of +the module is downloadable separately for use with Python versions 2.4 to 2.6, +packaged as the :mod:`unittest2` package, from +http://pypi.python.org/pypi/unittest2. + +When used from the command line, the module can automatically discover +tests. It's not as fancy as `py.test <http://pytest.org>`__ or +`nose <http://code.google.com/p/python-nose/>`__, but provides a simple way +to run tests kept within a set of package directories. For example, +the following command will search the :file:`test/` subdirectory for +any importable test files named ``test*.py``:: + + python -m unittest discover -s test + +Consult the :mod:`unittest` module documentation for more details. +(Developed in :issue:`6001`.) + +The :func:`main` function supports some other new options: + +* :option:`-b` or :option:`--buffer` will buffer the standard output + and standard error streams during each test. If the test passes, + any resulting output will be discarded; on failure, the buffered + output will be displayed. + +* :option:`-c` or :option:`--catch` will cause the control-C interrupt + to be handled more gracefully. Instead of interrupting the test + process immediately, the currently running test will be completed + and then the partial results up to the interruption will be reported. + If you're impatient, a second press of control-C will cause an immediate + interruption. + + This control-C handler tries to avoid causing problems when the code + being tested or the tests being run have defined a signal handler of + their own, by noticing that a signal handler was already set and + calling it. If this doesn't work for you, there's a + :func:`removeHandler` decorator that can be used to mark tests that + should have the control-C handling disabled. + +* :option:`-f` or :option:`--failfast` makes + test execution stop immediately when a test fails instead of + continuing to execute further tests. (Suggested by Cliff Dyer and + implemented by Michael Foord; :issue:`8074`.) + +The progress messages now show 'x' for expected failures +and 'u' for unexpected successes when run in verbose mode. +(Contributed by Benjamin Peterson.) + +Test cases can raise the :exc:`~unittest.SkipTest` exception to skip a +test (:issue:`1034053`). + +The error messages for :meth:`~unittest.TestCase.assertEqual`, +:meth:`~unittest.TestCase.assertTrue`, and :meth:`~unittest.TestCase.assertFalse` +failures now provide more information. If you set the +:attr:`~unittest.TestCase.longMessage` attribute of your :class:`~unittest.TestCase` classes to +True, both the standard error message and any additional message you +provide will be printed for failures. (Added by Michael Foord; :issue:`5663`.) + +The :meth:`~unittest.TestCase.assertRaises` method now +returns a context handler when called without providing a callable +object to run. For example, you can write this:: + + with self.assertRaises(KeyError): + {}['foo'] + +(Implemented by Antoine Pitrou; :issue:`4444`.) + +.. rev 78774 + +Module- and class-level setup and teardown fixtures are now supported. +Modules can contain :func:`~unittest.setUpModule` and :func:`~unittest.tearDownModule` +functions. Classes can have :meth:`~unittest.TestCase.setUpClass` and +:meth:`~unittest.TestCase.tearDownClass` methods that must be defined as class methods +(using ``@classmethod`` or equivalent). These functions and +methods are invoked when the test runner switches to a test case in a +different module or class. + +The methods :meth:`~unittest.TestCase.addCleanup` and +:meth:`~unittest.TestCase.doCleanups` were added. +:meth:`~unittest.TestCase.addCleanup` lets you add cleanup functions that +will be called unconditionally (after :meth:`~unittest.TestCase.setUp` if +:meth:`~unittest.TestCase.setUp` fails, otherwise after :meth:`~unittest.TestCase.tearDown`). This allows +for much simpler resource allocation and deallocation during tests +(:issue:`5679`). + +A number of new methods were added that provide more specialized +tests. Many of these methods were written by Google engineers +for use in their test suites; Gregory P. Smith, Michael Foord, and +GvR worked on merging them into Python's version of :mod:`unittest`. + +* :meth:`~unittest.TestCase.assertIsNone` and :meth:`~unittest.TestCase.assertIsNotNone` take one + expression and verify that the result is or is not ``None``. + +* :meth:`~unittest.TestCase.assertIs` and :meth:`~unittest.TestCase.assertIsNot` + take two values and check whether the two values evaluate to the same object or not. + (Added by Michael Foord; :issue:`2578`.) + +* :meth:`~unittest.TestCase.assertIsInstance` and + :meth:`~unittest.TestCase.assertNotIsInstance` check whether + the resulting object is an instance of a particular class, or of + one of a tuple of classes. (Added by Georg Brandl; :issue:`7031`.) + +* :meth:`~unittest.TestCase.assertGreater`, :meth:`~unittest.TestCase.assertGreaterEqual`, + :meth:`~unittest.TestCase.assertLess`, and :meth:`~unittest.TestCase.assertLessEqual` compare + two quantities. + +* :meth:`~unittest.TestCase.assertMultiLineEqual` compares two strings, and if they're + not equal, displays a helpful comparison that highlights the + differences in the two strings. This comparison is now used by + default when Unicode strings are compared with :meth:`~unittest.TestCase.assertEqual`. + +* :meth:`~unittest.TestCase.assertRegexpMatches` and + :meth:`~unittest.TestCase.assertNotRegexpMatches` checks whether the + first argument is a string matching or not matching the regular + expression provided as the second argument (:issue:`8038`). + +* :meth:`~unittest.TestCase.assertRaisesRegexp` checks whether a particular exception + is raised, and then also checks that the string representation of + the exception matches the provided regular expression. + +* :meth:`~unittest.TestCase.assertIn` and :meth:`~unittest.TestCase.assertNotIn` + tests whether *first* is or is not in *second*. + +* :meth:`~unittest.TestCase.assertItemsEqual` tests whether two provided sequences + contain the same elements. + +* :meth:`~unittest.TestCase.assertSetEqual` compares whether two sets are equal, and + only reports the differences between the sets in case of error. + +* Similarly, :meth:`~unittest.TestCase.assertListEqual` and :meth:`~unittest.TestCase.assertTupleEqual` + compare the specified types and explain any differences without necessarily + printing their full values; these methods are now used by default + when comparing lists and tuples using :meth:`~unittest.TestCase.assertEqual`. + More generally, :meth:`~unittest.TestCase.assertSequenceEqual` compares two sequences + and can optionally check whether both sequences are of a + particular type. + +* :meth:`~unittest.TestCase.assertDictEqual` compares two dictionaries and reports the + differences; it's now used by default when you compare two dictionaries + using :meth:`~unittest.TestCase.assertEqual`. :meth:`~unittest.TestCase.assertDictContainsSubset` checks whether + all of the key/value pairs in *first* are found in *second*. + +* :meth:`~unittest.TestCase.assertAlmostEqual` and :meth:`~unittest.TestCase.assertNotAlmostEqual` test + whether *first* and *second* are approximately equal. This method + can either round their difference to an optionally-specified number + of *places* (the default is 7) and compare it to zero, or require + the difference to be smaller than a supplied *delta* value. + +* :meth:`~unittest.TestLoader.loadTestsFromName` properly honors the + :attr:`~unittest.TestLoader.suiteClass` attribute of + the :class:`~unittest.TestLoader`. (Fixed by Mark Roddy; :issue:`6866`.) + +* A new hook lets you extend the :meth:`~unittest.TestCase.assertEqual` method to handle + new data types. The :meth:`~unittest.TestCase.addTypeEqualityFunc` method takes a type + object and a function. The function will be used when both of the + objects being compared are of the specified type. This function + should compare the two objects and raise an exception if they don't + match; it's a good idea for the function to provide additional + information about why the two objects aren't matching, much as the new + sequence comparison methods do. + +:func:`unittest.main` now takes an optional ``exit`` argument. If +False, :func:`~unittest.main` doesn't call :func:`sys.exit`, allowing +:func:`main` to be used from the interactive interpreter. +(Contributed by J. Pablo Fernández; :issue:`3379`.) + +:class:`~unittest.TestResult` has new :meth:`~unittest.TestResult.startTestRun` and +:meth:`~unittest.TestResult.stopTestRun` methods that are called immediately before +and after a test run. (Contributed by Robert Collins; :issue:`5728`.) + +With all these changes, the :file:`unittest.py` was becoming awkwardly +large, so the module was turned into a package and the code split into +several files (by Benjamin Peterson). This doesn't affect how the +module is imported or used. + +.. seealso:: + + http://www.voidspace.org.uk/python/articles/unittest2.shtml + Describes the new features, how to use them, and the + rationale for various design decisions. (By Michael Foord.) + +.. _elementtree-section: + +Updated module: ElementTree 1.3 +--------------------------------- + +The version of the ElementTree library included with Python was updated to +version 1.3. Some of the new features are: + +* The various parsing functions now take a *parser* keyword argument + giving an :class:`~xml.etree.ElementTree.XMLParser` instance that will + be used. This makes it possible to override the file's internal encoding:: + + p = ET.XMLParser(encoding='utf-8') + t = ET.XML("""<root/>""", parser=p) + + Errors in parsing XML now raise a :exc:`ParseError` exception, whose + instances have a :attr:`position` attribute + containing a (*line*, *column*) tuple giving the location of the problem. + +* ElementTree's code for converting trees to a string has been + significantly reworked, making it roughly twice as fast in many + cases. The :meth:`ElementTree.write() <xml.etree.ElementTree.ElementTree.write>` + and :meth:`Element.write` methods now have a *method* parameter that can be + "xml" (the default), "html", or "text". HTML mode will output empty + elements as ``<empty></empty>`` instead of ``<empty/>``, and text + mode will skip over elements and only output the text chunks. If + you set the :attr:`tag` attribute of an element to ``None`` but + leave its children in place, the element will be omitted when the + tree is written out, so you don't need to do more extensive rearrangement + to remove a single element. + + Namespace handling has also been improved. All ``xmlns:<whatever>`` + declarations are now output on the root element, not scattered throughout + the resulting XML. You can set the default namespace for a tree + by setting the :attr:`default_namespace` attribute and can + register new prefixes with :meth:`~xml.etree.ElementTree.register_namespace`. In XML mode, + you can use the true/false *xml_declaration* parameter to suppress the + XML declaration. + +* New :class:`~xml.etree.ElementTree.Element` method: + :meth:`~xml.etree.ElementTree.Element.extend` appends the items from a + sequence to the element's children. Elements themselves behave like + sequences, so it's easy to move children from one element to + another:: + + from xml.etree import ElementTree as ET + + t = ET.XML("""<list> + <item>1</item> <item>2</item> <item>3</item> + </list>""") + new = ET.XML('<root/>') + new.extend(t) + + # Outputs <root><item>1</item>...</root> + print ET.tostring(new) + +* New :class:`Element` method: + :meth:`~xml.etree.ElementTree.Element.iter` yields the children of the + element as a generator. It's also possible to write ``for child in + elem:`` to loop over an element's children. The existing method + :meth:`getiterator` is now deprecated, as is :meth:`getchildren` + which constructs and returns a list of children. + +* New :class:`Element` method: + :meth:`~xml.etree.ElementTree.Element.itertext` yields all chunks of + text that are descendants of the element. For example:: + + t = ET.XML("""<list> + <item>1</item> <item>2</item> <item>3</item> + </list>""") + + # Outputs ['\n ', '1', ' ', '2', ' ', '3', '\n'] + print list(t.itertext()) + +* Deprecated: using an element as a Boolean (i.e., ``if elem:``) would + return true if the element had any children, or false if there were + no children. This behaviour is confusing -- ``None`` is false, but + so is a childless element? -- so it will now trigger a + :exc:`FutureWarning`. In your code, you should be explicit: write + ``len(elem) != 0`` if you're interested in the number of children, + or ``elem is not None``. + +Fredrik Lundh develops ElementTree and produced the 1.3 version; +you can read his article describing 1.3 at +http://effbot.org/zone/elementtree-13-intro.htm. +Florent Xicluna updated the version included with +Python, after discussions on python-dev and in :issue:`6472`.) + +.. ====================================================================== + + +Build and C API Changes +======================= + +Changes to Python's build process and to the C API include: + +* The latest release of the GNU Debugger, GDB 7, can be `scripted + using Python + <http://sourceware.org/gdb/current/onlinedocs/gdb/Python.html>`__. + When you begin debugging an executable program P, GDB will look for + a file named ``P-gdb.py`` and automatically read it. Dave Malcolm + contributed a :file:`python-gdb.py` that adds a number of + commands useful when debugging Python itself. For example, + ``py-up`` and ``py-down`` go up or down one Python stack frame, + which usually corresponds to several C stack frames. ``py-print`` + prints the value of a Python variable, and ``py-bt`` prints the + Python stack trace. (Added as a result of :issue:`8032`.) + +* If you use the :file:`.gdbinit` file provided with Python, + the "pyo" macro in the 2.7 version now works correctly when the thread being + debugged doesn't hold the GIL; the macro now acquires it before printing. + (Contributed by Victor Stinner; :issue:`3632`.) + +* :c:func:`Py_AddPendingCall` is now thread-safe, letting any + worker thread submit notifications to the main Python thread. This + is particularly useful for asynchronous IO operations. + (Contributed by Kristján Valur Jónsson; :issue:`4293`.) + +* New function: :c:func:`PyCode_NewEmpty` creates an empty code object; + only the filename, function name, and first line number are required. + This is useful for extension modules that are attempting to + construct a more useful traceback stack. Previously such + extensions needed to call :c:func:`PyCode_New`, which had many + more arguments. (Added by Jeffrey Yasskin.) + +* New function: :c:func:`PyErr_NewExceptionWithDoc` creates a new + exception class, just as the existing :c:func:`PyErr_NewException` does, + but takes an extra ``char *`` argument containing the docstring for the + new exception class. (Added by 'lekma' on the Python bug tracker; + :issue:`7033`.) + +* New function: :c:func:`PyFrame_GetLineNumber` takes a frame object + and returns the line number that the frame is currently executing. + Previously code would need to get the index of the bytecode + instruction currently executing, and then look up the line number + corresponding to that address. (Added by Jeffrey Yasskin.) + +* New functions: :c:func:`PyLong_AsLongAndOverflow` and + :c:func:`PyLong_AsLongLongAndOverflow` approximates a Python long + integer as a C :c:type:`long` or :c:type:`long long`. + If the number is too large to fit into + the output type, an *overflow* flag is set and returned to the caller. + (Contributed by Case Van Horsen; :issue:`7528` and :issue:`7767`.) + +* New function: stemming from the rewrite of string-to-float conversion, + a new :c:func:`PyOS_string_to_double` function was added. The old + :c:func:`PyOS_ascii_strtod` and :c:func:`PyOS_ascii_atof` functions + are now deprecated. + +* New function: :c:func:`PySys_SetArgvEx` sets the value of + ``sys.argv`` and can optionally update ``sys.path`` to include the + directory containing the script named by ``sys.argv[0]`` depending + on the value of an *updatepath* parameter. + + This function was added to close a security hole for applications + that embed Python. The old function, :c:func:`PySys_SetArgv`, would + always update ``sys.path``, and sometimes it would add the current + directory. This meant that, if you ran an application embedding + Python in a directory controlled by someone else, attackers could + put a Trojan-horse module in the directory (say, a file named + :file:`os.py`) that your application would then import and run. + + If you maintain a C/C++ application that embeds Python, check + whether you're calling :c:func:`PySys_SetArgv` and carefully consider + whether the application should be using :c:func:`PySys_SetArgvEx` + with *updatepath* set to false. + + Security issue reported as `CVE-2008-5983 + <http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983>`_; + discussed in :issue:`5753`, and fixed by Antoine Pitrou. + +* New macros: the Python header files now define the following macros: + :c:macro:`Py_ISALNUM`, + :c:macro:`Py_ISALPHA`, + :c:macro:`Py_ISDIGIT`, + :c:macro:`Py_ISLOWER`, + :c:macro:`Py_ISSPACE`, + :c:macro:`Py_ISUPPER`, + :c:macro:`Py_ISXDIGIT`, + and :c:macro:`Py_TOLOWER`, :c:macro:`Py_TOUPPER`. + All of these functions are analogous to the C + standard macros for classifying characters, but ignore the current + locale setting, because in + several places Python needs to analyze characters in a + locale-independent way. (Added by Eric Smith; + :issue:`5793`.) + + .. XXX these macros don't seem to be described in the c-api docs. + +* Removed function: :c:macro:`PyEval_CallObject` is now only available + as a macro. A function version was being kept around to preserve + ABI linking compatibility, but that was in 1997; it can certainly be + deleted by now. (Removed by Antoine Pitrou; :issue:`8276`.) + +* New format codes: the :c:func:`PyFormat_FromString`, + :c:func:`PyFormat_FromStringV`, and :c:func:`PyErr_Format` functions now + accept ``%lld`` and ``%llu`` format codes for displaying + C's :c:type:`long long` types. + (Contributed by Mark Dickinson; :issue:`7228`.) + +* The complicated interaction between threads and process forking has + been changed. Previously, the child process created by + :func:`os.fork` might fail because the child is created with only a + single thread running, the thread performing the :func:`os.fork`. + If other threads were holding a lock, such as Python's import lock, + when the fork was performed, the lock would still be marked as + "held" in the new process. But in the child process nothing would + ever release the lock, since the other threads weren't replicated, + and the child process would no longer be able to perform imports. + + Python 2.7 acquires the import lock before performing an + :func:`os.fork`, and will also clean up any locks created using the + :mod:`threading` module. C extension modules that have internal + locks, or that call :c:func:`fork()` themselves, will not benefit + from this clean-up. + + (Fixed by Thomas Wouters; :issue:`1590864`.) + +* The :c:func:`Py_Finalize` function now calls the internal + :func:`threading._shutdown` function; this prevents some exceptions from + being raised when an interpreter shuts down. + (Patch by Adam Olsen; :issue:`1722344`.) + +* When using the :c:type:`PyMemberDef` structure to define attributes + of a type, Python will no longer let you try to delete or set a + :const:`T_STRING_INPLACE` attribute. + + .. rev 79644 + +* Global symbols defined by the :mod:`ctypes` module are now prefixed + with ``Py``, or with ``_ctypes``. (Implemented by Thomas + Heller; :issue:`3102`.) + +* New configure option: the :option:`--with-system-expat` switch allows + building the :mod:`pyexpat` module to use the system Expat library. + (Contributed by Arfrever Frehtes Taifersar Arahesis; :issue:`7609`.) + +* New configure option: the + :option:`--with-valgrind` option will now disable the pymalloc + allocator, which is difficult for the Valgrind memory-error detector + to analyze correctly. + Valgrind will therefore be better at detecting memory leaks and + overruns. (Contributed by James Henstridge; :issue:`2422`.) + +* New configure option: you can now supply an empty string to + :option:`--with-dbmliborder=` in order to disable all of the various + DBM modules. (Added by Arfrever Frehtes Taifersar Arahesis; + :issue:`6491`.) + +* The :program:`configure` script now checks for floating-point rounding bugs + on certain 32-bit Intel chips and defines a :c:macro:`X87_DOUBLE_ROUNDING` + preprocessor definition. No code currently uses this definition, + but it's available if anyone wishes to use it. + (Added by Mark Dickinson; :issue:`2937`.) + + :program:`configure` also now sets a :envvar:`LDCXXSHARED` Makefile + variable for supporting C++ linking. (Contributed by Arfrever + Frehtes Taifersar Arahesis; :issue:`1222585`.) + +* The build process now creates the necessary files for pkg-config + support. (Contributed by Clinton Roy; :issue:`3585`.) + +* The build process now supports Subversion 1.7. (Contributed by + Arfrever Frehtes Taifersar Arahesis; :issue:`6094`.) + + +.. _whatsnew27-capsules: + +Capsules +------------------- + +Python 3.1 adds a new C datatype, :c:type:`PyCapsule`, for providing a +C API to an extension module. A capsule is essentially the holder of +a C ``void *`` pointer, and is made available as a module attribute; for +example, the :mod:`socket` module's API is exposed as ``socket.CAPI``, +and :mod:`unicodedata` exposes ``ucnhash_CAPI``. Other extensions +can import the module, access its dictionary to get the capsule +object, and then get the ``void *`` pointer, which will usually point +to an array of pointers to the module's various API functions. + +There is an existing data type already used for this, +:c:type:`PyCObject`, but it doesn't provide type safety. Evil code +written in pure Python could cause a segmentation fault by taking a +:c:type:`PyCObject` from module A and somehow substituting it for the +:c:type:`PyCObject` in module B. Capsules know their own name, +and getting the pointer requires providing the name:: + + void *vtable; + + if (!PyCapsule_IsValid(capsule, "mymodule.CAPI") { + PyErr_SetString(PyExc_ValueError, "argument type invalid"); + return NULL; + } + + vtable = PyCapsule_GetPointer(capsule, "mymodule.CAPI"); + +You are assured that ``vtable`` points to whatever you're expecting. +If a different capsule was passed in, :c:func:`PyCapsule_IsValid` would +detect the mismatched name and return false. Refer to +:ref:`using-capsules` for more information on using these objects. + +Python 2.7 now uses capsules internally to provide various +extension-module APIs, but the :c:func:`PyCObject_AsVoidPtr` was +modified to handle capsules, preserving compile-time compatibility +with the :c:type:`CObject` interface. Use of +:c:func:`PyCObject_AsVoidPtr` will signal a +:exc:`PendingDeprecationWarning`, which is silent by default. + +Implemented in Python 3.1 and backported to 2.7 by Larry Hastings; +discussed in :issue:`5630`. + + +.. ====================================================================== + +Port-Specific Changes: Windows +----------------------------------- + +* The :mod:`msvcrt` module now contains some constants from + the :file:`crtassem.h` header file: + :data:`CRT_ASSEMBLY_VERSION`, + :data:`VC_ASSEMBLY_PUBLICKEYTOKEN`, + and :data:`LIBRARIES_ASSEMBLY_NAME_PREFIX`. + (Contributed by David Cournapeau; :issue:`4365`.) + +* The :mod:`_winreg` module for accessing the registry now implements + the :func:`CreateKeyEx` and :func:`DeleteKeyEx` functions, extended + versions of previously-supported functions that take several extra + arguments. The :func:`DisableReflectionKey`, + :func:`EnableReflectionKey`, and :func:`QueryReflectionKey` were also + tested and documented. + (Implemented by Brian Curtin: :issue:`7347`.) + +* The new :c:func:`_beginthreadex` API is used to start threads, and + the native thread-local storage functions are now used. + (Contributed by Kristján Valur Jónsson; :issue:`3582`.) + +* The :func:`os.kill` function now works on Windows. The signal value + can be the constants :const:`CTRL_C_EVENT`, + :const:`CTRL_BREAK_EVENT`, or any integer. The first two constants + will send Control-C and Control-Break keystroke events to + subprocesses; any other value will use the :c:func:`TerminateProcess` + API. (Contributed by Miki Tebeka; :issue:`1220212`.) + +* The :func:`os.listdir` function now correctly fails + for an empty path. (Fixed by Hirokazu Yamamoto; :issue:`5913`.) + +* The :mod:`mimelib` module will now read the MIME database from + the Windows registry when initializing. + (Patch by Gabriel Genellina; :issue:`4969`.) + +.. ====================================================================== + +Port-Specific Changes: Mac OS X +----------------------------------- + +* The path ``/Library/Python/2.7/site-packages`` is now appended to + ``sys.path``, in order to share added packages between the system + installation and a user-installed copy of the same version. + (Changed by Ronald Oussoren; :issue:`4865`.) + +Port-Specific Changes: FreeBSD +----------------------------------- + +* FreeBSD 7.1's :const:`SO_SETFIB` constant, used with + :func:`~socket.getsockopt`/:func:`~socket.setsockopt` to select an + alternate routing table, is now available in the :mod:`socket` + module. (Added by Kyle VanderBeek; :issue:`8235`.) + +Other Changes and Fixes +======================= + +* Two benchmark scripts, :file:`iobench` and :file:`ccbench`, were + added to the :file:`Tools` directory. :file:`iobench` measures the + speed of the built-in file I/O objects returned by :func:`open` + while performing various operations, and :file:`ccbench` is a + concurrency benchmark that tries to measure computing throughput, + thread switching latency, and IO processing bandwidth when + performing several tasks using a varying number of threads. + +* The :file:`Tools/i18n/msgfmt.py` script now understands plural + forms in :file:`.po` files. (Fixed by Martin von Löwis; + :issue:`5464`.) + +* When importing a module from a :file:`.pyc` or :file:`.pyo` file + with an existing :file:`.py` counterpart, the :attr:`co_filename` + attributes of the resulting code objects are overwritten when the + original filename is obsolete. This can happen if the file has been + renamed, moved, or is accessed through different paths. (Patch by + Ziga Seilnacht and Jean-Paul Calderone; :issue:`1180193`.) + +* The :file:`regrtest.py` script now takes a :option:`--randseed=` + switch that takes an integer that will be used as the random seed + for the :option:`-r` option that executes tests in random order. + The :option:`-r` option also reports the seed that was used + (Added by Collin Winter.) + +* Another :file:`regrtest.py` switch is :option:`-j`, which + takes an integer specifying how many tests run in parallel. This + allows reducing the total runtime on multi-core machines. + This option is compatible with several other options, including the + :option:`-R` switch which is known to produce long runtimes. + (Added by Antoine Pitrou, :issue:`6152`.) This can also be used + with a new :option:`-F` switch that runs selected tests in a loop + until they fail. (Added by Antoine Pitrou; :issue:`7312`.) + +* When executed as a script, the :file:`py_compile.py` module now + accepts ``'-'`` as an argument, which will read standard input for + the list of filenames to be compiled. (Contributed by Piotr + Ożarowski; :issue:`8233`.) + +.. ====================================================================== + +Porting to Python 2.7 +===================== + +This section lists previously described changes and other bugfixes +that may require changes to your code: + +* The :func:`range` function processes its arguments more + consistently; it will now call :meth:`__int__` on non-float, + non-integer arguments that are supplied to it. (Fixed by Alexander + Belopolsky; :issue:`1533`.) + +* The string :meth:`format` method changed the default precision used + for floating-point and complex numbers from 6 decimal + places to 12, which matches the precision used by :func:`str`. + (Changed by Eric Smith; :issue:`5920`.) + +* Because of an optimization for the :keyword:`with` statement, the special + methods :meth:`__enter__` and :meth:`__exit__` must belong to the object's + type, and cannot be directly attached to the object's instance. This + affects new-style classes (derived from :class:`object`) and C extension + types. (:issue:`6101`.) + +* Due to a bug in Python 2.6, the *exc_value* parameter to + :meth:`__exit__` methods was often the string representation of the + exception, not an instance. This was fixed in 2.7, so *exc_value* + will be an instance as expected. (Fixed by Florent Xicluna; + :issue:`7853`.) + +* When a restricted set of attributes were set using ``__slots__``, + deleting an unset attribute would not raise :exc:`AttributeError` + as you would expect. Fixed by Benjamin Peterson; :issue:`7604`.) + +In the standard library: + +* Operations with :class:`datetime` instances that resulted in a year + falling outside the supported range didn't always raise + :exc:`OverflowError`. Such errors are now checked more carefully + and will now raise the exception. (Reported by Mark Leander, patch + by Anand B. Pillai and Alexander Belopolsky; :issue:`7150`.) + +* When using :class:`Decimal` instances with a string's + :meth:`format` method, the default alignment was previously + left-alignment. This has been changed to right-alignment, which might + change the output of your programs. + (Changed by Mark Dickinson; :issue:`6857`.) + + Comparisons involving a signaling NaN value (or ``sNAN``) now signal + :const:`InvalidOperation` instead of silently returning a true or + false value depending on the comparison operator. Quiet NaN values + (or ``NaN``) are now hashable. (Fixed by Mark Dickinson; + :issue:`7279`.) + +* The ElementTree library, :mod:`xml.etree`, no longer escapes + ampersands and angle brackets when outputting an XML processing + instruction (which looks like `<?xml-stylesheet href="#style1"?>`) + or comment (which looks like `<!-- comment -->`). + (Patch by Neil Muller; :issue:`2746`.) + +* The :meth:`readline` method of :class:`StringIO` objects now does + nothing when a negative length is requested, as other file-like + objects do. (:issue:`7348`). + +* The :mod:`syslog` module will now use the value of ``sys.argv[0]`` as the + identifier instead of the previous default value of ``'python'``. + (Changed by Sean Reifschneider; :issue:`8451`.) + +* The :mod:`tarfile` module's default error handling has changed, to + no longer suppress fatal errors. The default error level was previously 0, + which meant that errors would only result in a message being written to the + debug log, but because the debug log is not activated by default, + these errors go unnoticed. The default error level is now 1, + which raises an exception if there's an error. + (Changed by Lars Gustäbel; :issue:`7357`.) + +* The :mod:`urlparse` module's :func:`~urlparse.urlsplit` now handles + unknown URL schemes in a fashion compliant with :rfc:`3986`: if the + URL is of the form ``"<something>://..."``, the text before the + ``://`` is treated as the scheme, even if it's a made-up scheme that + the module doesn't know about. This change may break code that + worked around the old behaviour. For example, Python 2.6.4 or 2.5 + will return the following: + + >>> import urlparse + >>> urlparse.urlsplit('invented://host/filename?query') + ('invented', '', '//host/filename?query', '', '') + + Python 2.7 (and Python 2.6.5) will return: + + >>> import urlparse + >>> urlparse.urlsplit('invented://host/filename?query') + ('invented', 'host', '/filename?query', '', '') + + (Python 2.7 actually produces slightly different output, since it + returns a named tuple instead of a standard tuple.) + +For C extensions: + +* C extensions that use integer format codes with the ``PyArg_Parse*`` + family of functions will now raise a :exc:`TypeError` exception + instead of triggering a :exc:`DeprecationWarning` (:issue:`5080`). + +* Use the new :c:func:`PyOS_string_to_double` function instead of the old + :c:func:`PyOS_ascii_strtod` and :c:func:`PyOS_ascii_atof` functions, + which are now deprecated. + +For applications that embed Python: + +* The :c:func:`PySys_SetArgvEx` function was added, letting + applications close a security hole when the existing + :c:func:`PySys_SetArgv` function was used. Check whether you're + calling :c:func:`PySys_SetArgv` and carefully consider whether the + application should be using :c:func:`PySys_SetArgvEx` with + *updatepath* set to false. + +.. ====================================================================== + + +.. _acks27: + +Acknowledgements +================ + +The author would like to thank the following people for offering +suggestions, corrections and assistance with various drafts of this +article: Nick Coghlan, Philip Jenvey, Ryan Lovett, R. David Murray, +Hugh Secker-Walker. + diff --git a/Doc/whatsnew/3.0.rst b/Doc/whatsnew/3.0.rst index 549f314e5c..55b65bae21 100644 --- a/Doc/whatsnew/3.0.rst +++ b/Doc/whatsnew/3.0.rst @@ -96,9 +96,9 @@ up if you're used to Python 2.5. Print Is A Function ------------------- -The :keyword:`print` statement has been replaced with a :func:`print` +The ``print`` statement has been replaced with a :func:`print` function, with keyword arguments to replace most of the special syntax -of the old :keyword:`print` statement (:pep:`3105`). Examples:: +of the old ``print`` statement (:pep:`3105`). Examples:: Old: print "The answer is", 2*2 New: print("The answer is", 2*2) @@ -126,7 +126,7 @@ which produces:: Note: * The :func:`print` function doesn't support the "softspace" feature of - the old :keyword:`print` statement. For example, in Python 2.x, + the old ``print`` statement. For example, in Python 2.x, ``print "A\n", "B"`` would write ``"A\nB\n"``; but in Python 3.0, ``print("A\n", "B")`` writes ``"A\n B\n"``. @@ -135,7 +135,7 @@ Note: ``print(x)`` instead! * When using the ``2to3`` source-to-source conversion tool, all - :keyword:`print` statements are automatically converted to + ``print`` statements are automatically converted to :func:`print` function calls, so this is mostly a non-issue for larger projects. @@ -154,7 +154,9 @@ Some well-known APIs no longer return lists: :meth:`dict.itervalues` methods are no longer supported. * :func:`map` and :func:`filter` return iterators. If you really need - a list, a quick fix is e.g. ``list(map(...))``, but a better fix is + a list and the input sequences are all of equal length, a quick + fix is to wrap :func:`map` in :func:`list`, e.g. ``list(map(...))``, + but a better fix is often to use a list comprehension (especially when the original code uses :keyword:`lambda`), or rewriting the code so it doesn't need a list at all. Particularly tricky is :func:`map` invoked for the @@ -162,6 +164,12 @@ Some well-known APIs no longer return lists: regular :keyword:`for` loop (since creating a list would just be wasteful). + If the input sequences are not of equal length, :func:`map` will + stop at the termination of the shortest of the sequences. For full + compatibility with `map` from Python 2.x, also wrap the sequences in + :func:`itertools.zip_longest`, e.g. ``map(func, *sequences)`` becomes + ``list(map(func, itertools.zip_longest(*sequences)))``. + * :func:`range` now behaves like :func:`xrange` used to behave, except it works with values of arbitrary size. The latter no longer exists. @@ -178,7 +186,7 @@ Python 3.0 has simplified the rules for ordering comparisons: meaningful natural ordering. Thus, expressions like ``1 < ''``, ``0 > None`` or ``len <= len`` are no longer valid, and e.g. ``None < None`` raises :exc:`TypeError` instead of returning - :keyword:`False`. A corollary is that sorting a heterogeneous list + ``False``. A corollary is that sorting a heterogeneous list no longer makes sense -- all the elements must be comparable to each other. Note that this does not apply to the ``==`` and ``!=`` operators: objects of different incomparable types always compare @@ -270,7 +278,7 @@ changed. single "euro" character. (Of course, this change only affects raw string literals; the euro character is ``'\u20ac'`` in Python 3.0.) -* The builtin :class:`basestring` abstract type was removed. Use +* The built-in :class:`basestring` abstract type was removed. Use :class:`str` instead. The :class:`str` and :class:`bytes` types don't have functionality enough in common to warrant a shared base class. The ``2to3`` tool (see below) replaces every occurrence of @@ -293,6 +301,12 @@ changed. There is no longer any need for using the encoding-aware streams in the :mod:`codecs` module. +* The initial values of :data:`sys.stdin`, :data:`sys.stdout` and + :data:`sys.stderr` are now unicode-only text files (i.e., they are + instances of :class:`io.TextIOBase`). To read and write bytes data + with these streams, you need to use their :data:`io.TextIOBase.buffer` + attribute. + * Filenames are passed to and returned from APIs as (Unicode) strings. This can present platform-specific problems because on some platforms filenames are arbitrary byte strings. (On the other hand, @@ -383,10 +397,10 @@ New Syntax literals (``0720``) are gone. * New binary literals, e.g. ``0b1010`` (already in 2.6), and - there is a new corresponding builtin function, :func:`bin`. + there is a new corresponding built-in function, :func:`bin`. * Bytes literals are introduced with a leading ``b`` or ``B``, and - there is a new corresponding builtin function, :func:`bytes`. + there is a new corresponding built-in function, :func:`bytes`. Changed Syntax -------------- @@ -397,9 +411,8 @@ Changed Syntax * :keyword:`as` and :keyword:`with` are now reserved words. (Since 2.6, actually.) -* :keyword:`True`, :keyword:`False`, and :keyword:`None` are reserved - words. (2.6 partially enforced the restrictions on :keyword:`None` - already.) +* ``True``, ``False``, and ``None`` are reserved words. (2.6 partially enforced + the restrictions on ``None`` already.) * Change from :keyword:`except` *exc*, *var* to :keyword:`except` *exc* :keyword:`as` *var*. See :pep:`3110`. @@ -504,9 +517,7 @@ consulted for longer descriptions. produces a literal of type :class:`bytes`. * :ref:`pep-3116`. The :mod:`io` module is now the standard way of - doing file I/O, and the initial values of :data:`sys.stdin`, - :data:`sys.stdout` and :data:`sys.stderr` are now instances of - :class:`io.TextIOBase`. The builtin :func:`open` function is now an + doing file I/O. The built-in :func:`open` function is now an alias for :func:`io.open` and has additional keyword arguments *encoding*, *errors*, *newline* and *closefd*. Also note that an invalid *mode* argument now raises :exc:`ValueError`, not @@ -521,7 +532,7 @@ consulted for longer descriptions. * :ref:`pep-3119`. The :mod:`abc` module and the ABCs defined in the :mod:`collections` module plays a somewhat more prominent role in - the language now, and builtin collection types like :class:`dict` + the language now, and built-in collection types like :class:`dict` and :class:`list` conform to the :class:`collections.MutableMapping` and :class:`collections.MutableSequence` ABCs, respectively. @@ -615,7 +626,7 @@ review: Some other changes to standard library modules, not covered by :pep:`3108`: -* Killed :mod:`sets`. Use the builtin :func:`set` function. +* Killed :mod:`sets`. Use the built-in :func:`set` class. * Cleanup of the :mod:`sys` module: removed :func:`sys.exitfunc`, :func:`sys.exc_clear`, :data:`sys.exc_type`, :data:`sys.exc_value`, @@ -795,8 +806,8 @@ Builtins It raises :exc:`EOFError` if the input is terminated prematurely. To get the old behavior of :func:`input`, use ``eval(input())``. -* A new builtin :func:`next` was added to call the :meth:`__next__` - method on an object. +* A new built-in function :func:`next` was added to call the + :meth:`__next__` method on an object. * The :func:`round` function rounding strategy and return type have changed. Exact halfway cases are now rounded to the nearest even @@ -850,21 +861,21 @@ to the C API. * :pep:`3121`: Extension Module Initialization & Finalization. -* :pep:`3123`: Making :cmacro:`PyObject_HEAD` conform to standard C. +* :pep:`3123`: Making :c:macro:`PyObject_HEAD` conform to standard C. * No more C API support for restricted execution. -* :cfunc:`PyNumber_Coerce`, :cfunc:`PyNumber_CoerceEx`, - :cfunc:`PyMember_Get`, and :cfunc:`PyMember_Set` C APIs are removed. +* :c:func:`PyNumber_Coerce`, :c:func:`PyNumber_CoerceEx`, + :c:func:`PyMember_Get`, and :c:func:`PyMember_Set` C APIs are removed. -* New C API :cfunc:`PyImport_ImportModuleNoBlock`, works like - :cfunc:`PyImport_ImportModule` but won't block on the import lock +* New C API :c:func:`PyImport_ImportModuleNoBlock`, works like + :c:func:`PyImport_ImportModule` but won't block on the import lock (returning an error instead). * Renamed the boolean conversion C-level slot and method: ``nb_nonzero`` is now ``nb_bool``. -* Removed :cmacro:`METH_OLDARGS` and :cmacro:`WITH_CYCLE_GC` from the C API. +* Removed :c:macro:`METH_OLDARGS` and :c:macro:`WITH_CYCLE_GC` from the C API. .. ====================================================================== @@ -906,7 +917,7 @@ best strategy is the following: It is not recommended to try to write source code that runs unchanged under both Python 2.6 and 3.0; you'd have to use a very contorted -coding style, e.g. avoiding :keyword:`print` statements, metaclasses, +coding style, e.g. avoiding ``print`` statements, metaclasses, and much more. If you are maintaining a library that needs to support both Python 2.6 and Python 3.0, the best approach is to modify step 3 above by editing the 2.6 version of the source code and running the diff --git a/Doc/whatsnew/3.1.rst b/Doc/whatsnew/3.1.rst index 1b6b3ce11f..64ae1c1936 100644 --- a/Doc/whatsnew/3.1.rst +++ b/Doc/whatsnew/3.1.rst @@ -81,33 +81,11 @@ Support was also added for third-party tools like `PyYAML <http://pyyaml.org/>`_ PEP written by Armin Ronacher and Raymond Hettinger. Implementation written by Raymond Hettinger. -Since an ordered dictionary remembers its insertion order, it can be used -in conjuction with sorting to make a sorted dictionary:: - - >>> # regular unsorted dictionary - >>> d = {'banana': 3, 'apple':4, 'pear': 1, 'orange': 2} - - >>> # dictionary sorted by key - >>> OrderedDict(sorted(d.items(), key=lambda t: t[0])) - OrderedDict([('apple', 4), ('banana', 3), ('orange', 2), ('pear', 1)]) - - >>> # dictionary sorted by value - >>> OrderedDict(sorted(d.items(), key=lambda t: t[1])) - OrderedDict([('pear', 1), ('orange', 2), ('banana', 3), ('apple', 4)]) - - >>> # dictionary sorted by length of the key string - >>> OrderedDict(sorted(d.items(), key=lambda t: len(t[0]))) - OrderedDict([('pear', 1), ('apple', 4), ('orange', 2), ('banana', 3)]) - -The new sorted dictionaries maintain their sort order when entries -are deleted. But when new keys are added, the keys are appended -to the end and the sort is not maintained. - PEP 378: Format Specifier for Thousands Separator ================================================= -The builtin :func:`format` function and the :meth:`str.format` method use +The built-in :func:`format` function and the :meth:`str.format` method use a mini-language that now includes a simple, non-locale aware way to format a number with a thousands separator. That provides a way to humanize a program's output, improving its professional appearance and readability:: @@ -519,21 +497,21 @@ Changes to Python's build process and to the C API include: (Contributed by Mark Dickinson; :issue:`4258`.) -* The :cfunc:`PyLong_AsUnsignedLongLong()` function now handles a negative +* The :c:func:`PyLong_AsUnsignedLongLong()` function now handles a negative *pylong* by raising :exc:`OverflowError` instead of :exc:`TypeError`. (Contributed by Mark Dickinson and Lisandro Dalcrin; :issue:`5175`.) -* Deprecated :cfunc:`PyNumber_Int`. Use :cfunc:`PyNumber_Long` instead. +* Deprecated :c:func:`PyNumber_Int`. Use :c:func:`PyNumber_Long` instead. (Contributed by Mark Dickinson; :issue:`4910`.) -* Added a new :cfunc:`PyOS_string_to_double` function to replace the - deprecated functions :cfunc:`PyOS_ascii_strtod` and :cfunc:`PyOS_ascii_atof`. +* Added a new :c:func:`PyOS_string_to_double` function to replace the + deprecated functions :c:func:`PyOS_ascii_strtod` and :c:func:`PyOS_ascii_atof`. (Contributed by Mark Dickinson; :issue:`5914`.) -* Added :ctype:`PyCapsule` as a replacement for the :ctype:`PyCObject` API. +* Added :c:type:`PyCapsule` as a replacement for the :c:type:`PyCObject` API. The principal difference is that the new type has a well defined interface for passing typing safety information and a less complicated signature for calling a destructor. The old type had a problematic API and is now diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst new file mode 100644 index 0000000000..0553ec35fe --- /dev/null +++ b/Doc/whatsnew/3.2.rst @@ -0,0 +1,2704 @@ +**************************** + What's New In Python 3.2 +**************************** + +:Author: Raymond Hettinger +:Release: |release| +:Date: |today| + +.. $Id$ + Rules for maintenance: + + * Anyone can add text to this document. Do not spend very much time + on the wording of your changes, because your text will probably + get rewritten. (Note, during release candidate phase or just before + a beta release, please use the tracker instead -- this helps avoid + merge conflicts. If you must add a suggested entry directly, + please put it in an XXX comment and the maintainer will take notice). + + * The maintainer will go through Misc/NEWS periodically and add + changes; it's therefore more important to add your changes to + Misc/NEWS than to this file. + + * This is not a complete list of every single change; completeness + is the purpose of Misc/NEWS. Some changes I consider too small + or esoteric to include. If such a change is added to the text, + I'll just remove it. (This is another reason you shouldn't spend + too much time on writing your addition.) + + * If you want to draw your new text to the attention of the + maintainer, add 'XXX' to the beginning of the paragraph or + section. + + * It's OK to just add a fragmentary note about a change. For + example: "XXX Describe the transmogrify() function added to the + socket module." The maintainer will research the change and + write the necessary text. + + * You can comment out your additions if you like, but it's not + necessary (especially when a final release is some months away). + + * Credit the author of a patch or bugfix. Just the name is + sufficient; the e-mail address isn't necessary. It's helpful to + add the issue number: + + XXX Describe the transmogrify() function added to the socket + module. + + (Contributed by P.Y. Developer; :issue:`12345`.) + + This saves the maintainer the effort of going through the SVN log + when researching a change. + +This article explains the new features in Python 3.2 as compared to 3.1. It +focuses on a few highlights and gives a few examples. For full details, see the +:source:`Misc/NEWS <Misc/NEWS>` file. + +.. seealso:: + + :pep:`392` - Python 3.2 Release Schedule + + +PEP 384: Defining a Stable ABI +============================== + +In the past, extension modules built for one Python version were often +not usable with other Python versions. Particularly on Windows, every +feature release of Python required rebuilding all extension modules that +one wanted to use. This requirement was the result of the free access to +Python interpreter internals that extension modules could use. + +With Python 3.2, an alternative approach becomes available: extension +modules which restrict themselves to a limited API (by defining +Py_LIMITED_API) cannot use many of the internals, but are constrained +to a set of API functions that are promised to be stable for several +releases. As a consequence, extension modules built for 3.2 in that +mode will also work with 3.3, 3.4, and so on. Extension modules that +make use of details of memory structures can still be built, but will +need to be recompiled for every feature release. + +.. seealso:: + + :pep:`384` - Defining a Stable ABI + PEP written by Martin von Löwis. + + +PEP 389: Argparse Command Line Parsing Module +============================================= + +A new module for command line parsing, :mod:`argparse`, was introduced to +overcome the limitations of :mod:`optparse` which did not provide support for +positional arguments (not just options), subcommands, required options and other +common patterns of specifying and validating options. + +This module has already had widespread success in the community as a +third-party module. Being more fully featured than its predecessor, the +:mod:`argparse` module is now the preferred module for command-line processing. +The older module is still being kept available because of the substantial amount +of legacy code that depends on it. + +Here's an annotated example parser showing features like limiting results to a +set of choices, specifying a *metavar* in the help screen, validating that one +or more positional arguments is present, and making a required option:: + + import argparse + parser = argparse.ArgumentParser( + description = 'Manage servers', # main description for help + epilog = 'Tested on Solaris and Linux') # displayed after help + parser.add_argument('action', # argument name + choices = ['deploy', 'start', 'stop'], # three allowed values + help = 'action on each target') # help msg + parser.add_argument('targets', + metavar = 'HOSTNAME', # var name used in help msg + nargs = '+', # require one or more targets + help = 'url for target machines') # help msg explanation + parser.add_argument('-u', '--user', # -u or --user option + required = True, # make it a required argument + help = 'login as user') + +Example of calling the parser on a command string:: + + >>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain' + >>> result = parser.parse_args(cmd.split()) + >>> result.action + 'deploy' + >>> result.targets + ['sneezy.example.com', 'sleepy.example.com'] + >>> result.user + 'skycaptain' + +Example of the parser's automatically generated help:: + + >>> parser.parse_args('-h'.split()) + + usage: manage_cloud.py [-h] -u USER + {deploy,start,stop} HOSTNAME [HOSTNAME ...] + + Manage servers + + positional arguments: + {deploy,start,stop} action on each target + HOSTNAME url for target machines + + optional arguments: + -h, --help show this help message and exit + -u USER, --user USER login as user + + Tested on Solaris and Linux + +An especially nice :mod:`argparse` feature is the ability to define subparsers, +each with their own argument patterns and help displays:: + + import argparse + parser = argparse.ArgumentParser(prog='HELM') + subparsers = parser.add_subparsers() + + parser_l = subparsers.add_parser('launch', help='Launch Control') # first subgroup + parser_l.add_argument('-m', '--missiles', action='store_true') + parser_l.add_argument('-t', '--torpedos', action='store_true') + + parser_m = subparsers.add_parser('move', help='Move Vessel', # second subgroup + aliases=('steer', 'turn')) # equivalent names + parser_m.add_argument('-c', '--course', type=int, required=True) + parser_m.add_argument('-s', '--speed', type=int, default=0) + + $ ./helm.py --help # top level help (launch and move) + $ ./helm.py launch --help # help for launch options + $ ./helm.py launch --missiles # set missiles=True and torpedos=False + $ ./helm.py steer --course 180 --speed 5 # set movement parameters + +.. seealso:: + + :pep:`389` - New Command Line Parsing Module + PEP written by Steven Bethard. + + :ref:`upgrading-optparse-code` for details on the differences from :mod:`optparse`. + + +PEP 391: Dictionary Based Configuration for Logging +==================================================== + +The :mod:`logging` module provided two kinds of configuration, one style with +function calls for each option or another style driven by an external file saved +in a :mod:`ConfigParser` format. Those options did not provide the flexibility +to create configurations from JSON or YAML files, nor did they support +incremental configuration, which is needed for specifying logger options from a +command line. + +To support a more flexible style, the module now offers +:func:`logging.config.dictConfig` for specifying logging configuration with +plain Python dictionaries. The configuration options include formatters, +handlers, filters, and loggers. Here's a working example of a configuration +dictionary:: + + {"version": 1, + "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"}, + "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"} + }, + "handlers": {"console": { + "class": "logging.StreamHandler", + "formatter": "brief", + "level": "INFO", + "stream": "ext://sys.stdout"}, + "console_priority": { + "class": "logging.StreamHandler", + "formatter": "full", + "level": "ERROR", + "stream": "ext://sys.stderr"} + }, + "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}} + + +If that dictionary is stored in a file called :file:`conf.json`, it can be +loaded and called with code like this:: + + >>> import json, logging.config + >>> with open('conf.json') as f: + conf = json.load(f) + >>> logging.config.dictConfig(conf) + >>> logging.info("Transaction completed normally") + INFO : root : Transaction completed normally + >>> logging.critical("Abnormal termination") + 2011-02-17 11:14:36,694 root CRITICAL Abnormal termination + +.. seealso:: + + :pep:`391` - Dictionary Based Configuration for Logging + PEP written by Vinay Sajip. + + +PEP 3148: The ``concurrent.futures`` module +============================================ + +Code for creating and managing concurrency is being collected in a new top-level +namespace, *concurrent*. Its first member is a *futures* package which provides +a uniform high-level interface for managing threads and processes. + +The design for :mod:`concurrent.futures` was inspired by the +*java.util.concurrent* package. In that model, a running call and its result +are represented by a :class:`~concurrent.futures.Future` object that abstracts +features common to threads, processes, and remote procedure calls. That object +supports status checks (running or done), timeouts, cancellations, adding +callbacks, and access to results or exceptions. + +The primary offering of the new module is a pair of executor classes for +launching and managing calls. The goal of the executors is to make it easier to +use existing tools for making parallel calls. They save the effort needed to +setup a pool of resources, launch the calls, create a results queue, add +time-out handling, and limit the total number of threads, processes, or remote +procedure calls. + +Ideally, each application should share a single executor across multiple +components so that process and thread limits can be centrally managed. This +solves the design challenge that arises when each component has its own +competing strategy for resource management. + +Both classes share a common interface with three methods: +:meth:`~concurrent.futures.Executor.submit` for scheduling a callable and +returning a :class:`~concurrent.futures.Future` object; +:meth:`~concurrent.futures.Executor.map` for scheduling many asynchronous calls +at a time, and :meth:`~concurrent.futures.Executor.shutdown` for freeing +resources. The class is a :term:`context manager` and can be used in a +:keyword:`with` statement to assure that resources are automatically released +when currently pending futures are done executing. + +A simple of example of :class:`~concurrent.futures.ThreadPoolExecutor` is a +launch of four parallel threads for copying files:: + + import concurrent.futures, shutil + with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e: + e.submit(shutil.copy, 'src1.txt', 'dest1.txt') + e.submit(shutil.copy, 'src2.txt', 'dest2.txt') + e.submit(shutil.copy, 'src3.txt', 'dest3.txt') + e.submit(shutil.copy, 'src4.txt', 'dest4.txt') + +.. seealso:: + + :pep:`3148` - Futures -- Execute Computations Asynchronously + PEP written by Brian Quinlan. + + :ref:`Code for Threaded Parallel URL reads<threadpoolexecutor-example>`, an + example using threads to fetch multiple web pages in parallel. + + :ref:`Code for computing prime numbers in + parallel<processpoolexecutor-example>`, an example demonstrating + :class:`~concurrent.futures.ProcessPoolExecutor`. + + +PEP 3147: PYC Repository Directories +===================================== + +Python's scheme for caching bytecode in *.pyc* files did not work well in +environments with multiple Python interpreters. If one interpreter encountered +a cached file created by another interpreter, it would recompile the source and +overwrite the cached file, thus losing the benefits of caching. + +The issue of "pyc fights" has become more pronounced as it has become +commonplace for Linux distributions to ship with multiple versions of Python. +These conflicts also arise with CPython alternatives such as Unladen Swallow. + +To solve this problem, Python's import machinery has been extended to use +distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and +Unladen Swallow each competing for a file called "mymodule.pyc", they will now +look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and +"mymodule.unladen10.pyc". And to prevent all of these new files from +cluttering source directories, the *pyc* files are now collected in a +"__pycache__" directory stored under the package directory. + +Aside from the filenames and target directories, the new scheme has a few +aspects that are visible to the programmer: + +* Imported modules now have a :attr:`__cached__` attribute which stores the name + of the actual file that was imported: + + >>> import collections + >>> collections.__cached__ + 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' + +* The tag that is unique to each interpreter is accessible from the :mod:`imp` + module: + + >>> import imp + >>> imp.get_tag() + 'cpython-32' + +* Scripts that try to deduce source filename from the imported file now need to + be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc" + filename. Instead, use the new functions in the :mod:`imp` module: + + >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc') + 'c:/py32/lib/collections.py' + >>> imp.cache_from_source('c:/py32/lib/collections.py') + 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' + +* The :mod:`py_compile` and :mod:`compileall` modules have been updated to + reflect the new naming convention and target directory. The command-line + invocation of *compileall* has new options: ``-i`` for + specifying a list of files and directories to compile and ``-b`` which causes + bytecode files to be written to their legacy location rather than + *__pycache__*. + +* The :mod:`importlib.abc` module has been updated with new :term:`abstract base + classes <abstract base class>` for loading bytecode files. The obsolete + ABCs, :class:`~importlib.abc.PyLoader` and + :class:`~importlib.abc.PyPycLoader`, have been deprecated (instructions on how + to stay Python 3.1 compatible are included with the documentation). + +.. seealso:: + + :pep:`3147` - PYC Repository Directories + PEP written by Barry Warsaw. + + +PEP 3149: ABI Version Tagged .so Files +====================================== + +The PYC repository directory allows multiple bytecode cache files to be +co-located. This PEP implements a similar mechanism for shared object files by +giving them a common directory and distinct names for each version. + +The common directory is "pyshared" and the file names are made distinct by +identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the +major and minor version numbers, and optional build flags (such as "d" for +debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo", +you may see these files when the distribution package is installed:: + + /usr/share/pyshared/foo.cpython-32m.so + /usr/share/pyshared/foo.cpython-33md.so + +In Python itself, the tags are accessible from functions in the :mod:`sysconfig` +module:: + + >>> import sysconfig + >>> sysconfig.get_config_var('SOABI') # find the version tag + 'cpython-32mu' + >>> sysconfig.get_config_var('SO') # find the full filename extension + '.cpython-32mu.so' + +.. seealso:: + + :pep:`3149` - ABI Version Tagged .so Files + PEP written by Barry Warsaw. + + +PEP 3333: Python Web Server Gateway Interface v1.0.1 +===================================================== + +This informational PEP clarifies how bytes/text issues are to be handled by the +WSGI protocol. The challenge is that string handling in Python 3 is most +conveniently handled with the :class:`str` type even though the HTTP protocol +is itself bytes oriented. + +The PEP differentiates so-called *native strings* that are used for +request/response headers and metadata versus *byte strings* which are used for +the bodies of requests and responses. + +The *native strings* are always of type :class:`str` but are restricted to code +points between *U+0000* through *U+00FF* which are translatable to bytes using +*Latin-1* encoding. These strings are used for the keys and values in the +environment dictionary and for response headers and statuses in the +:func:`start_response` function. They must follow :rfc:`2616` with respect to +encoding. That is, they must either be *ISO-8859-1* characters or use +:rfc:`2047` MIME encoding. + +For developers porting WSGI applications from Python 2, here are the salient +points: + +* If the app already used strings for headers in Python 2, no change is needed. + +* If instead, the app encoded output headers or decoded input headers, then the + headers will need to be re-encoded to Latin-1. For example, an output header + encoded in utf-8 was using ``h.encode('utf-8')`` now needs to convert from + bytes to native strings using ``h.encode('utf-8').decode('latin-1')``. + +* Values yielded by an application or sent using the :meth:`write` method + must be byte strings. The :func:`start_response` function and environ + must use native strings. The two cannot be mixed. + +For server implementers writing CGI-to-WSGI pathways or other CGI-style +protocols, the users must to be able access the environment using native strings +even though the underlying platform may have a different convention. To bridge +this gap, the :mod:`wsgiref` module has a new function, +:func:`wsgiref.handlers.read_environ` for transcoding CGI variables from +:attr:`os.environ` into native strings and returning a new dictionary. + +.. seealso:: + + :pep:`3333` - Python Web Server Gateway Interface v1.0.1 + PEP written by Phillip Eby. + + +Other Language Changes +====================== + +Some smaller changes made to the core Python language are: + +* String formatting for :func:`format` and :meth:`str.format` gained new + capabilities for the format character **#**. Previously, for integers in + binary, octal, or hexadecimal, it caused the output to be prefixed with '0b', + '0o', or '0x' respectively. Now it can also handle floats, complex, and + Decimal, causing the output to always have a decimal point even when no digits + follow it. + + >>> format(20, '#o') + '0o24' + >>> format(12.34, '#5.0f') + ' 12.' + + (Suggested by Mark Dickinson and implemented by Eric Smith in :issue:`7094`.) + +* There is also a new :meth:`str.format_map` method that extends the + capabilities of the existing :meth:`str.format` method by accepting arbitrary + :term:`mapping` objects. This new method makes it possible to use string + formatting with any of Python's many dictionary-like objects such as + :class:`~collections.defaultdict`, :class:`~shelve.Shelf`, + :class:`~configparser.ConfigParser`, or :mod:`dbm`. It is also useful with + custom :class:`dict` subclasses that normalize keys before look-up or that + supply a :meth:`__missing__` method for unknown keys:: + + >>> import shelve + >>> d = shelve.open('tmp.shl') + >>> 'The {project_name} status is {status} as of {date}'.format_map(d) + 'The testing project status is green as of February 15, 2011' + + >>> class LowerCasedDict(dict): + def __getitem__(self, key): + return dict.__getitem__(self, key.lower()) + >>> lcd = LowerCasedDict(part='widgets', quantity=10) + >>> 'There are {QUANTITY} {Part} in stock'.format_map(lcd) + 'There are 10 widgets in stock' + + >>> class PlaceholderDict(dict): + def __missing__(self, key): + return '<{}>'.format(key) + >>> 'Hello {name}, welcome to {location}'.format_map(PlaceholderDict()) + 'Hello <name>, welcome to <location>' + + (Suggested by Raymond Hettinger and implemented by Eric Smith in + :issue:`6081`.) + +* The interpreter can now be started with a quiet option, ``-q``, to prevent + the copyright and version information from being displayed in the interactive + mode. The option can be introspected using the :attr:`sys.flags` attribute:: + + $ python -q + >>> sys.flags + sys.flags(debug=0, division_warning=0, inspect=0, interactive=0, + optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, + ignore_environment=0, verbose=0, bytes_warning=0, quiet=1) + + (Contributed by Marcin Wojdyr in :issue:`1772833`). + +* The :func:`hasattr` function works by calling :func:`getattr` and detecting + whether an exception is raised. This technique allows it to detect methods + created dynamically by :meth:`__getattr__` or :meth:`__getattribute__` which + would otherwise be absent from the class dictionary. Formerly, *hasattr* + would catch any exception, possibly masking genuine errors. Now, *hasattr* + has been tightened to only catch :exc:`AttributeError` and let other + exceptions pass through:: + + >>> class A: + @property + def f(self): + return 1 // 0 + + >>> a = A() + >>> hasattr(a, 'f') + Traceback (most recent call last): + ... + ZeroDivisionError: integer division or modulo by zero + + (Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.) + +* The :func:`str` of a float or complex number is now the same as its + :func:`repr`. Previously, the :func:`str` form was shorter but that just + caused confusion and is no longer needed now that the shortest possible + :func:`repr` is displayed by default: + + >>> import math + >>> repr(math.pi) + '3.141592653589793' + >>> str(math.pi) + '3.141592653589793' + + (Proposed and implemented by Mark Dickinson; :issue:`9337`.) + +* :class:`memoryview` objects now have a :meth:`~memoryview.release()` method + and they also now support the context manager protocol. This allows timely + release of any resources that were acquired when requesting a buffer from the + original object. + + >>> with memoryview(b'abcdefgh') as v: + print(v.tolist()) + [97, 98, 99, 100, 101, 102, 103, 104] + + (Added by Antoine Pitrou; :issue:`9757`.) + +* Previously it was illegal to delete a name from the local namespace if it + occurs as a free variable in a nested block:: + + def outer(x): + def inner(): + return x + inner() + del x + + This is now allowed. Remember that the target of an :keyword:`except` clause + is cleared, so this code which used to work with Python 2.6, raised a + :exc:`SyntaxError` with Python 3.1 and now works again:: + + def f(): + def print_error(): + print(e) + try: + something + except Exception as e: + print_error() + # implicit "del e" here + + (See :issue:`4617`.) + +* The internal :c:type:`structsequence` tool now creates subclasses of tuple. + This means that C structures like those returned by :func:`os.stat`, + :func:`time.gmtime`, and :attr:`sys.version_info` now work like a + :term:`named tuple` and now work with functions and methods that + expect a tuple as an argument. This is a big step forward in making the C + structures as flexible as their pure Python counterparts: + + >>> isinstance(sys.version_info, tuple) + True + >>> 'Version %d.%d.%d %s(%d)' % sys.version_info + 'Version 3.2.0 final(0)' + + (Suggested by Arfrever Frehtes Taifersar Arahesis and implemented + by Benjamin Peterson in :issue:`8413`.) + +* Warnings are now easier to control using the :envvar:`PYTHONWARNINGS` + environment variable as an alternative to using ``-W`` at the command line:: + + $ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::' + + (Suggested by Barry Warsaw and implemented by Philip Jenvey in :issue:`7301`.) + +* A new warning category, :exc:`ResourceWarning`, has been added. It is + emitted when potential issues with resource consumption or cleanup + are detected. It is silenced by default in normal release builds but + can be enabled through the means provided by the :mod:`warnings` + module, or on the command line. + + A :exc:`ResourceWarning` is issued at interpreter shutdown if the + :data:`gc.garbage` list isn't empty, and if :attr:`gc.DEBUG_UNCOLLECTABLE` is + set, all uncollectable objects are printed. This is meant to make the + programmer aware that their code contains object finalization issues. + + A :exc:`ResourceWarning` is also issued when a :term:`file object` is destroyed + without having been explicitly closed. While the deallocator for such + object ensures it closes the underlying operating system resource + (usually, a file descriptor), the delay in deallocating the object could + produce various issues, especially under Windows. Here is an example + of enabling the warning from the command line:: + + $ python -q -Wdefault + >>> f = open("foo", "wb") + >>> del f + __main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'> + + (Added by Antoine Pitrou and Georg Brandl in :issue:`10093` and :issue:`477863`.) + +* :class:`range` objects now support *index* and *count* methods. This is part + of an effort to make more objects fully implement the + :class:`collections.Sequence` :term:`abstract base class`. As a result, the + language will have a more uniform API. In addition, :class:`range` objects + now support slicing and negative indices, even with values larger than + :attr:`sys.maxsize`. This makes *range* more interoperable with lists:: + + >>> range(0, 100, 2).count(10) + 1 + >>> range(0, 100, 2).index(10) + 5 + >>> range(0, 100, 2)[5] + 10 + >>> range(0, 100, 2)[0:5] + range(0, 10, 2) + + (Contributed by Daniel Stutzbach in :issue:`9213`, by Alexander Belopolsky + in :issue:`2690`, and by Nick Coghlan in :issue:`10889`.) + +* The :func:`callable` builtin function from Py2.x was resurrected. It provides + a concise, readable alternative to using an :term:`abstract base class` in an + expression like ``isinstance(x, collections.Callable)``: + + >>> callable(max) + True + >>> callable(20) + False + + (See :issue:`10518`.) + +* Python's import mechanism can now load modules installed in directories with + non-ASCII characters in the path name. This solved an aggravating problem + with home directories for users with non-ASCII characters in their usernames. + + (Required extensive work by Victor Stinner in :issue:`9425`.) + + +New, Improved, and Deprecated Modules +===================================== + +Python's standard library has undergone significant maintenance efforts and +quality improvements. + +The biggest news for Python 3.2 is that the :mod:`email` package, :mod:`mailbox` +module, and :mod:`nntplib` modules now work correctly with the bytes/text model +in Python 3. For the first time, there is correct handling of messages with +mixed encodings. + +Throughout the standard library, there has been more careful attention to +encodings and text versus bytes issues. In particular, interactions with the +operating system are now better able to exchange non-ASCII data using the +Windows MBCS encoding, locale-aware encodings, or UTF-8. + +Another significant win is the addition of substantially better support for +*SSL* connections and security certificates. + +In addition, more classes now implement a :term:`context manager` to support +convenient and reliable resource clean-up using a :keyword:`with` statement. + +email +----- + +The usability of the :mod:`email` package in Python 3 has been mostly fixed by +the extensive efforts of R. David Murray. The problem was that emails are +typically read and stored in the form of :class:`bytes` rather than :class:`str` +text, and they may contain multiple encodings within a single email. So, the +email package had to be extended to parse and generate email messages in bytes +format. + +* New functions :func:`~email.message_from_bytes` and + :func:`~email.message_from_binary_file`, and new classes + :class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser` + allow binary message data to be parsed into model objects. + +* Given bytes input to the model, :meth:`~email.message.Message.get_payload` + will by default decode a message body that has a + :mailheader:`Content-Transfer-Encoding` of *8bit* using the charset + specified in the MIME headers and return the resulting string. + +* Given bytes input to the model, :class:`~email.generator.Generator` will + convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of + *8bit* to instead have a *7bit* :mailheader:`Content-Transfer-Encoding`. + + Headers with unencoded non-ASCII bytes are deemed to be :rfc:`2047`\ -encoded + using the *unknown-8bit* character set. + +* A new class :class:`~email.generator.BytesGenerator` produces bytes as output, + preserving any unchanged non-ASCII data that was present in the input used to + build the model, including message bodies with a + :mailheader:`Content-Transfer-Encoding` of *8bit*. + +* The :mod:`smtplib` :class:`~smtplib.SMTP` class now accepts a byte string + for the *msg* argument to the :meth:`~smtplib.SMTP.sendmail` method, + and a new method, :meth:`~smtplib.SMTP.send_message` accepts a + :class:`~email.message.Message` object and can optionally obtain the + *from_addr* and *to_addrs* addresses directly from the object. + +(Proposed and implemented by R. David Murray, :issue:`4661` and :issue:`10321`.) + +elementtree +----------- + +The :mod:`xml.etree.ElementTree` package and its :mod:`xml.etree.cElementTree` +counterpart have been updated to version 1.3. + +Several new and useful functions and methods have been added: + +* :func:`xml.etree.ElementTree.fromstringlist` which builds an XML document + from a sequence of fragments +* :func:`xml.etree.ElementTree.register_namespace` for registering a global + namespace prefix +* :func:`xml.etree.ElementTree.tostringlist` for string representation + including all sublists +* :meth:`xml.etree.ElementTree.Element.extend` for appending a sequence of zero + or more elements +* :meth:`xml.etree.ElementTree.Element.iterfind` searches an element and + subelements +* :meth:`xml.etree.ElementTree.Element.itertext` creates a text iterator over + an element and its subelements +* :meth:`xml.etree.ElementTree.TreeBuilder.end` closes the current element +* :meth:`xml.etree.ElementTree.TreeBuilder.doctype` handles a doctype + declaration + +Two methods have been deprecated: + +* :meth:`xml.etree.ElementTree.getchildren` use ``list(elem)`` instead. +* :meth:`xml.etree.ElementTree.getiterator` use ``Element.iter`` instead. + +For details of the update, see `Introducing ElementTree +<http://effbot.org/zone/elementtree-13-intro.htm>`_ on Fredrik Lundh's website. + +(Contributed by Florent Xicluna and Fredrik Lundh, :issue:`6472`.) + +functools +--------- + +* The :mod:`functools` module includes a new decorator for caching function + calls. :func:`functools.lru_cache` can save repeated queries to an external + resource whenever the results are expected to be the same. + + For example, adding a caching decorator to a database query function can save + database accesses for popular searches: + + >>> import functools + >>> @functools.lru_cache(maxsize=300) + >>> def get_phone_number(name): + c = conn.cursor() + c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,)) + return c.fetchone()[0] + + >>> for name in user_requests: + get_phone_number(name) # cached lookup + + To help with choosing an effective cache size, the wrapped function is + instrumented for tracking cache statistics: + + >>> get_phone_number.cache_info() + CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300) + + If the phonelist table gets updated, the outdated contents of the cache can be + cleared with: + + >>> get_phone_number.cache_clear() + + (Contributed by Raymond Hettinger and incorporating design ideas from Jim + Baker, Miki Tebeka, and Nick Coghlan; see `recipe 498245 + <http://code.activestate.com/recipes/498245>`_\, `recipe 577479 + <http://code.activestate.com/recipes/577479>`_\, :issue:`10586`, and + :issue:`10593`.) + +* The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute + pointing to the original callable function. This allows wrapped functions to + be introspected. It also copies :attr:`__annotations__` if defined. And now + it also gracefully skips over missing attributes such as :attr:`__doc__` which + might not be defined for the wrapped callable. + + In the above example, the cache can be removed by recovering the original + function: + + >>> get_phone_number = get_phone_number.__wrapped__ # uncached function + + (By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and + :issue:`8814`.) + +* To help write classes with rich comparison methods, a new decorator + :func:`functools.total_ordering` will use a existing equality and inequality + methods to fill in the remaining methods. + + For example, supplying *__eq__* and *__lt__* will enable + :func:`~functools.total_ordering` to fill-in *__le__*, *__gt__* and *__ge__*:: + + @total_ordering + class Student: + def __eq__(self, other): + return ((self.lastname.lower(), self.firstname.lower()) == + (other.lastname.lower(), other.firstname.lower())) + def __lt__(self, other): + return ((self.lastname.lower(), self.firstname.lower()) < + (other.lastname.lower(), other.firstname.lower())) + + With the *total_ordering* decorator, the remaining comparison methods + are filled in automatically. + + (Contributed by Raymond Hettinger.) + +* To aid in porting programs from Python 2, the :func:`functools.cmp_to_key` + function converts an old-style comparison function to + modern :term:`key function`: + + >>> # locale-aware sort order + >>> sorted(iterable, key=cmp_to_key(locale.strcoll)) + + For sorting examples and a brief sorting tutorial, see the `Sorting HowTo + <http://wiki.python.org/moin/HowTo/Sorting/>`_ tutorial. + + (Contributed by Raymond Hettinger.) + +itertools +--------- + +* The :mod:`itertools` module has a new :func:`~itertools.accumulate` function + modeled on APL's *scan* operator and Numpy's *accumulate* function: + + >>> from itertools import accumulate + >>> list(accumulate([8, 2, 50])) + [8, 10, 60] + + >>> prob_dist = [0.1, 0.4, 0.2, 0.3] + >>> list(accumulate(prob_dist)) # cumulative probability distribution + [0.1, 0.5, 0.7, 1.0] + + For an example using :func:`~itertools.accumulate`, see the :ref:`examples for + the random module <random-examples>`. + + (Contributed by Raymond Hettinger and incorporating design suggestions + from Mark Dickinson.) + +collections +----------- + +* The :class:`collections.Counter` class now has two forms of in-place + subtraction, the existing *-=* operator for `saturating subtraction + <http://en.wikipedia.org/wiki/Saturation_arithmetic>`_ and the new + :meth:`~collections.Counter.subtract` method for regular subtraction. The + former is suitable for `multisets <http://en.wikipedia.org/wiki/Multiset>`_ + which only have positive counts, and the latter is more suitable for use cases + that allow negative counts: + + >>> tally = Counter(dogs=5, cat=3) + >>> tally -= Counter(dogs=2, cats=8) # saturating subtraction + >>> tally + Counter({'dogs': 3}) + + >>> tally = Counter(dogs=5, cats=3) + >>> tally.subtract(dogs=2, cats=8) # regular subtraction + >>> tally + Counter({'dogs': 3, 'cats': -5}) + + (Contributed by Raymond Hettinger.) + +* The :class:`collections.OrderedDict` class has a new method + :meth:`~collections.OrderedDict.move_to_end` which takes an existing key and + moves it to either the first or last position in the ordered sequence. + + The default is to move an item to the last position. This is equivalent of + renewing an entry with ``od[k] = od.pop(k)``. + + A fast move-to-end operation is useful for resequencing entries. For example, + an ordered dictionary can be used to track order of access by aging entries + from the oldest to the most recently accessed. + + >>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e']) + >>> list(d) + ['a', 'b', 'X', 'd', 'e'] + >>> d.move_to_end('X') + >>> list(d) + ['a', 'b', 'd', 'e', 'X'] + + (Contributed by Raymond Hettinger.) + +* The :class:`collections.deque` class grew two new methods + :meth:`~collections.deque.count` and :meth:`~collections.deque.reverse` that + make them more substitutable for :class:`list` objects: + + >>> d = deque('simsalabim') + >>> d.count('s') + 2 + >>> d.reverse() + >>> d + deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's']) + + (Contributed by Raymond Hettinger.) + +threading +--------- + +The :mod:`threading` module has a new :class:`~threading.Barrier` +synchronization class for making multiple threads wait until all of them have +reached a common barrier point. Barriers are useful for making sure that a task +with multiple preconditions does not run until all of the predecessor tasks are +complete. + +Barriers can work with an arbitrary number of threads. This is a generalization +of a `Rendezvous <http://en.wikipedia.org/wiki/Synchronous_rendezvous>`_ which +is defined for only two threads. + +Implemented as a two-phase cyclic barrier, :class:`~threading.Barrier` objects +are suitable for use in loops. The separate *filling* and *draining* phases +assure that all threads get released (drained) before any one of them can loop +back and re-enter the barrier. The barrier fully resets after each cycle. + +Example of using barriers:: + + from threading import Barrier, Thread + + def get_votes(site): + ballots = conduct_election(site) + all_polls_closed.wait() # do not count until all polls are closed + totals = summarize(ballots) + publish(site, totals) + + all_polls_closed = Barrier(len(sites)) + for site in sites: + Thread(target=get_votes, args=(site,)).start() + +In this example, the barrier enforces a rule that votes cannot be counted at any +polling site until all polls are closed. Notice how a solution with a barrier +is similar to one with :meth:`threading.Thread.join`, but the threads stay alive +and continue to do work (summarizing ballots) after the barrier point is +crossed. + +If any of the predecessor tasks can hang or be delayed, a barrier can be created +with an optional *timeout* parameter. Then if the timeout period elapses before +all the predecessor tasks reach the barrier point, all waiting threads are +released and a :exc:`~threading.BrokenBarrierError` exception is raised:: + + def get_votes(site): + ballots = conduct_election(site) + try: + all_polls_closed.wait(timeout = midnight - time.now()) + except BrokenBarrierError: + lockbox = seal_ballots(ballots) + queue.put(lockbox) + else: + totals = summarize(ballots) + publish(site, totals) + +In this example, the barrier enforces a more robust rule. If some election +sites do not finish before midnight, the barrier times-out and the ballots are +sealed and deposited in a queue for later handling. + +See `Barrier Synchronization Patterns +<http://parlab.eecs.berkeley.edu/wiki/_media/patterns/paraplop_g1_3.pdf>`_ for +more examples of how barriers can be used in parallel computing. Also, there is +a simple but thorough explanation of barriers in `The Little Book of Semaphores +<http://greenteapress.com/semaphores/downey08semaphores.pdf>`_, *section 3.6*. + +(Contributed by Kristján Valur Jónsson with an API review by Jeffrey Yasskin in +:issue:`8777`.) + +datetime and time +----------------- + +* The :mod:`datetime` module has a new type :class:`~datetime.timezone` that + implements the :class:`~datetime.tzinfo` interface by returning a fixed UTC + offset and timezone name. This makes it easier to create timezone-aware + datetime objects:: + + >>> from datetime import datetime, timezone + + >>> datetime.now(timezone.utc) + datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc) + + >>> datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z") + datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc) + +* Also, :class:`~datetime.timedelta` objects can now be multiplied by + :class:`float` and divided by :class:`float` and :class:`int` objects. + And :class:`~datetime.timedelta` objects can now divide one another. + +* The :meth:`datetime.date.strftime` method is no longer restricted to years + after 1900. The new supported year range is from 1000 to 9999 inclusive. + +* Whenever a two-digit year is used in a time tuple, the interpretation has been + governed by :attr:`time.accept2dyear`. The default is *True* which means that + for a two-digit year, the century is guessed according to the POSIX rules + governing the ``%y`` strptime format. + + Starting with Py3.2, use of the century guessing heuristic will emit a + :exc:`DeprecationWarning`. Instead, it is recommended that + :attr:`time.accept2dyear` be set to *False* so that large date ranges + can be used without guesswork:: + + >>> import time, warnings + >>> warnings.resetwarnings() # remove the default warning filters + + >>> time.accept2dyear = True # guess whether 11 means 11 or 2011 + >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0)) + Warning (from warnings module): + ... + DeprecationWarning: Century info guessed for a 2-digit year. + 'Fri Jan 1 12:34:56 2011' + + >>> time.accept2dyear = False # use the full range of allowable dates + >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0)) + 'Fri Jan 1 12:34:56 11' + + Several functions now have significantly expanded date ranges. When + :attr:`time.accept2dyear` is false, the :func:`time.asctime` function will + accept any year that fits in a C int, while the :func:`time.mktime` and + :func:`time.strftime` functions will accept the full range supported by the + corresponding operating system functions. + +(Contributed by Alexander Belopolsky and Victor Stinner in :issue:`1289118`, +:issue:`5094`, :issue:`6641`, :issue:`2706`, :issue:`1777412`, :issue:`8013`, +and :issue:`10827`.) + +.. XXX http://bugs.python.org/issue?%40search_text=datetime&%40sort=-activity + +math +---- + +The :mod:`math` module has been updated with six new functions inspired by the +C99 standard. + +The :func:`~math.isfinite` function provides a reliable and fast way to detect +special values. It returns *True* for regular numbers and *False* for *Nan* or +*Infinity*: + +>>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))] +[True, True, False, False] + +The :func:`~math.expm1` function computes ``e**x-1`` for small values of *x* +without incurring the loss of precision that usually accompanies the subtraction +of nearly equal quantities: + +>>> expm1(0.013671875) # more accurate way to compute e**x-1 for a small x +0.013765762467652909 + +The :func:`~math.erf` function computes a probability integral or `Gaussian +error function <http://en.wikipedia.org/wiki/Error_function>`_. The +complementary error function, :func:`~math.erfc`, is ``1 - erf(x)``: + +>>> erf(1.0/sqrt(2.0)) # portion of normal distribution within 1 standard deviation +0.682689492137086 +>>> erfc(1.0/sqrt(2.0)) # portion of normal distribution outside 1 standard deviation +0.31731050786291404 +>>> erf(1.0/sqrt(2.0)) + erfc(1.0/sqrt(2.0)) +1.0 + +The :func:`~math.gamma` function is a continuous extension of the factorial +function. See http://en.wikipedia.org/wiki/Gamma_function for details. Because +the function is related to factorials, it grows large even for small values of +*x*, so there is also a :func:`~math.lgamma` function for computing the natural +logarithm of the gamma function: + +>>> gamma(7.0) # six factorial +720.0 +>>> lgamma(801.0) # log(800 factorial) +4551.950730698041 + +(Contributed by Mark Dickinson.) + +abc +--- + +The :mod:`abc` module now supports :func:`~abc.abstractclassmethod` and +:func:`~abc.abstractstaticmethod`. + +These tools make it possible to define an :term:`abstract base class` that +requires a particular :func:`classmethod` or :func:`staticmethod` to be +implemented:: + + class Temperature(metaclass=abc.ABCMeta): + @abc.abstractclassmethod + def from_fahrenheit(cls, t): + ... + @abc.abstractclassmethod + def from_celsius(cls, t): + ... + +(Patch submitted by Daniel Urban; :issue:`5867`.) + +io +-- + +The :class:`io.BytesIO` has a new method, :meth:`~io.BytesIO.getbuffer`, which +provides functionality similar to :func:`memoryview`. It creates an editable +view of the data without making a copy. The buffer's random access and support +for slice notation are well-suited to in-place editing:: + + >>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11 + + >>> def change_location(buffer, record_number, location): + start = record_number * REC_LEN + LOC_START + buffer[start: start+LOC_LEN] = location + + >>> import io + + >>> byte_stream = io.BytesIO( + b'G3805 storeroom Main chassis ' + b'X7899 shipping Reserve cog ' + b'L6988 receiving Primary sprocket' + ) + >>> buffer = byte_stream.getbuffer() + >>> change_location(buffer, 1, b'warehouse ') + >>> change_location(buffer, 0, b'showroom ') + >>> print(byte_stream.getvalue()) + b'G3805 showroom Main chassis ' + b'X7899 warehouse Reserve cog ' + b'L6988 receiving Primary sprocket' + +(Contributed by Antoine Pitrou in :issue:`5506`.) + +reprlib +------- + +When writing a :meth:`__repr__` method for a custom container, it is easy to +forget to handle the case where a member refers back to the container itself. +Python's builtin objects such as :class:`list` and :class:`set` handle +self-reference by displaying "..." in the recursive part of the representation +string. + +To help write such :meth:`__repr__` methods, the :mod:`reprlib` module has a new +decorator, :func:`~reprlib.recursive_repr`, for detecting recursive calls to +:meth:`__repr__` and substituting a placeholder string instead:: + + >>> class MyList(list): + @recursive_repr() + def __repr__(self): + return '<' + '|'.join(map(repr, self)) + '>' + + >>> m = MyList('abc') + >>> m.append(m) + >>> m.append('x') + >>> print(m) + <'a'|'b'|'c'|...|'x'> + +(Contributed by Raymond Hettinger in :issue:`9826` and :issue:`9840`.) + +logging +------- + +In addition to dictionary-based configuration described above, the +:mod:`logging` package has many other improvements. + +The logging documentation has been augmented by a :ref:`basic tutorial +<logging-basic-tutorial>`\, an :ref:`advanced tutorial +<logging-advanced-tutorial>`\, and a :ref:`cookbook <logging-cookbook>` of +logging recipes. These documents are the fastest way to learn about logging. + +The :func:`logging.basicConfig` set-up function gained a *style* argument to +support three different types of string formatting. It defaults to "%" for +traditional %-formatting, can be set to "{" for the new :meth:`str.format` style, or +can be set to "$" for the shell-style formatting provided by +:class:`string.Template`. The following three configurations are equivalent:: + + >>> from logging import basicConfig + >>> basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s") + >>> basicConfig(style='{', format="{name} -> {levelname} {message}") + >>> basicConfig(style='$', format="$name -> $levelname: $message") + +If no configuration is set-up before a logging event occurs, there is now a +default configuration using a :class:`~logging.StreamHandler` directed to +:attr:`sys.stderr` for events of ``WARNING`` level or higher. Formerly, an +event occurring before a configuration was set-up would either raise an +exception or silently drop the event depending on the value of +:attr:`logging.raiseExceptions`. The new default handler is stored in +:attr:`logging.lastResort`. + +The use of filters has been simplified. Instead of creating a +:class:`~logging.Filter` object, the predicate can be any Python callable that +returns *True* or *False*. + +There were a number of other improvements that add flexibility and simplify +configuration. See the module documentation for a full listing of changes in +Python 3.2. + +csv +--- + +The :mod:`csv` module now supports a new dialect, :class:`~csv.unix_dialect`, +which applies quoting for all fields and a traditional Unix style with ``'\n'`` as +the line terminator. The registered dialect name is ``unix``. + +The :class:`csv.DictWriter` has a new method, +:meth:`~csv.DictWriter.writeheader` for writing-out an initial row to document +the field names:: + + >>> import csv, sys + >>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix') + >>> w.writeheader() + "name","dept" + >>> w.writerows([ + {'name': 'tom', 'dept': 'accounting'}, + {'name': 'susan', 'dept': 'Salesl'}]) + "tom","accounting" + "susan","sales" + +(New dialect suggested by Jay Talbot in :issue:`5975`, and the new method +suggested by Ed Abraham in :issue:`1537721`.) + +contextlib +---------- + +There is a new and slightly mind-blowing tool +:class:`~contextlib.ContextDecorator` that is helpful for creating a +:term:`context manager` that does double duty as a function decorator. + +As a convenience, this new functionality is used by +:func:`~contextlib.contextmanager` so that no extra effort is needed to support +both roles. + +The basic idea is that both context managers and function decorators can be used +for pre-action and post-action wrappers. Context managers wrap a group of +statements using a :keyword:`with` statement, and function decorators wrap a +group of statements enclosed in a function. So, occasionally there is a need to +write a pre-action or post-action wrapper that can be used in either role. + +For example, it is sometimes useful to wrap functions or groups of statements +with a logger that can track the time of entry and time of exit. Rather than +writing both a function decorator and a context manager for the task, the +:func:`~contextlib.contextmanager` provides both capabilities in a single +definition:: + + from contextlib import contextmanager + import logging + + logging.basicConfig(level=logging.INFO) + + @contextmanager + def track_entry_and_exit(name): + logging.info('Entering: {}'.format(name)) + yield + logging.info('Exiting: {}'.format(name)) + +Formerly, this would have only been usable as a context manager:: + + with track_entry_and_exit('widget loader'): + print('Some time consuming activity goes here') + load_widget() + +Now, it can be used as a decorator as well:: + + @track_entry_and_exit('widget loader') + def activity(): + print('Some time consuming activity goes here') + load_widget() + +Trying to fulfill two roles at once places some limitations on the technique. +Context managers normally have the flexibility to return an argument usable by +a :keyword:`with` statement, but there is no parallel for function decorators. + +In the above example, there is not a clean way for the *track_entry_and_exit* +context manager to return a logging instance for use in the body of enclosed +statements. + +(Contributed by Michael Foord in :issue:`9110`.) + +decimal and fractions +--------------------- + +Mark Dickinson crafted an elegant and efficient scheme for assuring that +different numeric datatypes will have the same hash value whenever their actual +values are equal (:issue:`8188`):: + + assert hash(Fraction(3, 2)) == hash(1.5) == \ + hash(Decimal("1.5")) == hash(complex(1.5, 0)) + +Some of the hashing details are exposed through a new attribute, +:attr:`sys.hash_info`, which describes the bit width of the hash value, the +prime modulus, the hash values for *infinity* and *nan*, and the multiplier +used for the imaginary part of a number: + +>>> sys.hash_info +sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003) + +An early decision to limit the inter-operability of various numeric types has +been relaxed. It is still unsupported (and ill-advised) to have implicit +mixing in arithmetic expressions such as ``Decimal('1.1') + float('1.1')`` +because the latter loses information in the process of constructing the binary +float. However, since existing floating point value can be converted losslessly +to either a decimal or rational representation, it makes sense to add them to +the constructor and to support mixed-type comparisons. + +* The :class:`decimal.Decimal` constructor now accepts :class:`float` objects + directly so there in no longer a need to use the :meth:`~decimal.Decimal.from_float` + method (:issue:`8257`). + +* Mixed type comparisons are now fully supported so that + :class:`~decimal.Decimal` objects can be directly compared with :class:`float` + and :class:`fractions.Fraction` (:issue:`2531` and :issue:`8188`). + +Similar changes were made to :class:`fractions.Fraction` so that the +:meth:`~fractions.Fraction.from_float()` and :meth:`~fractions.Fraction.from_decimal` +methods are no longer needed (:issue:`8294`): + +>>> Decimal(1.1) +Decimal('1.100000000000000088817841970012523233890533447265625') +>>> Fraction(1.1) +Fraction(2476979795053773, 2251799813685248) + +Another useful change for the :mod:`decimal` module is that the +:attr:`Context.clamp` attribute is now public. This is useful in creating +contexts that correspond to the decimal interchange formats specified in IEEE +754 (see :issue:`8540`). + +(Contributed by Mark Dickinson and Raymond Hettinger.) + +ftp +--- + +The :class:`ftplib.FTP` class now supports the context manager protocol to +unconditionally consume :exc:`socket.error` exceptions and to close the FTP +connection when done:: + + >>> from ftplib import FTP + >>> with FTP("ftp1.at.proftpd.org") as ftp: + ftp.login() + ftp.dir() + + '230 Anonymous login ok, restrictions apply.' + dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 . + dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .. + dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS + dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora + +Other file-like objects such as :class:`mmap.mmap` and :func:`fileinput.input` +also grew auto-closing context managers:: + + with fileinput.input(files=('log1.txt', 'log2.txt')) as f: + for line in f: + process(line) + +(Contributed by Tarek Ziadé and Giampaolo Rodolà in :issue:`4972`, and +by Georg Brandl in :issue:`8046` and :issue:`1286`.) + +The :class:`~ftplib.FTP_TLS` class now accepts a *context* parameter, which is a +:class:`ssl.SSLContext` object allowing bundling SSL configuration options, +certificates and private keys into a single (potentially long-lived) structure. + +(Contributed by Giampaolo Rodolà; :issue:`8806`.) + +popen +----- + +The :func:`os.popen` and :func:`subprocess.Popen` functions now support +:keyword:`with` statements for auto-closing of the file descriptors. + +(Contributed by Antoine Pitrou and Brian Curtin in :issue:`7461` and +:issue:`10554`.) + +select +------ + +The :mod:`select` module now exposes a new, constant attribute, +:attr:`~select.PIPE_BUF`, which gives the minimum number of bytes which are +guaranteed not to block when :func:`select.select` says a pipe is ready +for writing. + +>>> import select +>>> select.PIPE_BUF +512 + +(Available on Unix systems. Patch by Sébastien Sablé in :issue:`9862`) + +gzip and zipfile +---------------- + +:class:`gzip.GzipFile` now implements the :class:`io.BufferedIOBase` +:term:`abstract base class` (except for ``truncate()``). It also has a +:meth:`~gzip.GzipFile.peek` method and supports unseekable as well as +zero-padded file objects. + +The :mod:`gzip` module also gains the :func:`~gzip.compress` and +:func:`~gzip.decompress` functions for easier in-memory compression and +decompression. Keep in mind that text needs to be encoded as :class:`bytes` +before compressing and decompressing: + +>>> s = 'Three shall be the number thou shalt count, ' +>>> s += 'and the number of the counting shall be three' +>>> b = s.encode() # convert to utf-8 +>>> len(b) +89 +>>> c = gzip.compress(b) +>>> len(c) +77 +>>> gzip.decompress(c).decode()[:42] # decompress and convert to text +'Three shall be the number thou shalt count,' + +(Contributed by Anand B. Pillai in :issue:`3488`; and by Antoine Pitrou, Nir +Aides and Brian Curtin in :issue:`9962`, :issue:`1675951`, :issue:`7471` and +:issue:`2846`.) + +Also, the :class:`zipfile.ZipExtFile` class was reworked internally to represent +files stored inside an archive. The new implementation is significantly faster +and can be wrapped in a :class:`io.BufferedReader` object for more speedups. It +also solves an issue where interleaved calls to *read* and *readline* gave the +wrong results. + +(Patch submitted by Nir Aides in :issue:`7610`.) + +tarfile +------- + +The :class:`~tarfile.TarFile` class can now be used as a context manager. In +addition, its :meth:`~tarfile.TarFile.add` method has a new option, *filter*, +that controls which files are added to the archive and allows the file metadata +to be edited. + +The new *filter* option replaces the older, less flexible *exclude* parameter +which is now deprecated. If specified, the optional *filter* parameter needs to +be a :term:`keyword argument`. The user-supplied filter function accepts a +:class:`~tarfile.TarInfo` object and returns an updated +:class:`~tarfile.TarInfo` object, or if it wants the file to be excluded, the +function can return *None*:: + + >>> import tarfile, glob + + >>> def myfilter(tarinfo): + if tarinfo.isfile(): # only save real files + tarinfo.uname = 'monty' # redact the user name + return tarinfo + + >>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf: + for filename in glob.glob('*.txt'): + tf.add(filename, filter=myfilter) + tf.list() + -rw-r--r-- monty/501 902 2011-01-26 17:59:11 annotations.txt + -rw-r--r-- monty/501 123 2011-01-26 17:59:11 general_questions.txt + -rw-r--r-- monty/501 3514 2011-01-26 17:59:11 prion.txt + -rw-r--r-- monty/501 124 2011-01-26 17:59:11 py_todo.txt + -rw-r--r-- monty/501 1399 2011-01-26 17:59:11 semaphore_notes.txt + +(Proposed by Tarek Ziadé and implemented by Lars Gustäbel in :issue:`6856`.) + +hashlib +------- + +The :mod:`hashlib` module has two new constant attributes listing the hashing +algorithms guaranteed to be present in all implementations and those available +on the current implementation:: + + >>> import hashlib + + >>> hashlib.algorithms_guaranteed + {'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'} + + >>> hashlib.algorithms_available + {'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256', + 'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2', + 'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224', + 'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'} + +(Suggested by Carl Chenet in :issue:`7418`.) + +ast +--- + +The :mod:`ast` module has a wonderful a general-purpose tool for safely +evaluating expression strings using the Python literal +syntax. The :func:`ast.literal_eval` function serves as a secure alternative to +the builtin :func:`eval` function which is easily abused. Python 3.2 adds +:class:`bytes` and :class:`set` literals to the list of supported types: +strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None. + +:: + + >>> from ast import literal_eval + + >>> request = "{'req': 3, 'func': 'pow', 'args': (2, 0.5)}" + >>> literal_eval(request) + {'args': (2, 0.5), 'req': 3, 'func': 'pow'} + + >>> request = "os.system('do something harmful')" + >>> literal_eval(request) + Traceback (most recent call last): + ... + ValueError: malformed node or string: <_ast.Call object at 0x101739a10> + +(Implemented by Benjamin Peterson and Georg Brandl.) + +os +-- + +Different operating systems use various encodings for filenames and environment +variables. The :mod:`os` module provides two new functions, +:func:`~os.fsencode` and :func:`~os.fsdecode`, for encoding and decoding +filenames: + +>>> filename = 'Sehenswürdigkeiten' +>>> os.fsencode(filename) +b'Sehensw\xc3\xbcrdigkeiten' + +Some operating systems allow direct access to encoded bytes in the +environment. If so, the :attr:`os.supports_bytes_environ` constant will be +true. + +For direct access to encoded environment variables (if available), +use the new :func:`os.getenvb` function or use :data:`os.environb` +which is a bytes version of :data:`os.environ`. + +(Contributed by Victor Stinner.) + +shutil +------ + +The :func:`shutil.copytree` function has two new options: + +* *ignore_dangling_symlinks*: when ``symlinks=False`` so that the function + copies a file pointed to by a symlink, not the symlink itself. This option + will silence the error raised if the file doesn't exist. + +* *copy_function*: is a callable that will be used to copy files. + :func:`shutil.copy2` is used by default. + +(Contributed by Tarek Ziadé.) + +In addition, the :mod:`shutil` module now supports :ref:`archiving operations +<archiving-operations>` for zipfiles, uncompressed tarfiles, gzipped tarfiles, +and bzipped tarfiles. And there are functions for registering additional +archiving file formats (such as xz compressed tarfiles or custom formats). + +The principal functions are :func:`~shutil.make_archive` and +:func:`~shutil.unpack_archive`. By default, both operate on the current +directory (which can be set by :func:`os.chdir`) and on any sub-directories. +The archive filename needs to be specified with a full pathname. The archiving +step is non-destructive (the original files are left unchanged). + +:: + + >>> import shutil, pprint + + >>> os.chdir('mydata') # change to the source directory + >>> f = shutil.make_archive('/var/backup/mydata', + 'zip') # archive the current directory + >>> f # show the name of archive + '/var/backup/mydata.zip' + >>> os.chdir('tmp') # change to an unpacking + >>> shutil.unpack_archive('/var/backup/mydata.zip') # recover the data + + >>> pprint.pprint(shutil.get_archive_formats()) # display known formats + [('bztar', "bzip2'ed tar-file"), + ('gztar', "gzip'ed tar-file"), + ('tar', 'uncompressed tar file'), + ('zip', 'ZIP file')] + + >>> shutil.register_archive_format( # register a new archive format + name = 'xz', + function = xz.compress, # callable archiving function + extra_args = [('level', 8)], # arguments to the function + description = 'xz compression' + ) + +(Contributed by Tarek Ziadé.) + +sqlite3 +------- + +The :mod:`sqlite3` module was updated to pysqlite version 2.6.0. It has two new capabilities. + +* The :attr:`sqlite3.Connection.in_transit` attribute is true if there is an + active transaction for uncommitted changes. + +* The :meth:`sqlite3.Connection.enable_load_extension` and + :meth:`sqlite3.Connection.load_extension` methods allows you to load SQLite + extensions from ".so" files. One well-known extension is the fulltext-search + extension distributed with SQLite. + +(Contributed by R. David Murray and Shashwat Anand; :issue:`8845`.) + +html +---- + +A new :mod:`html` module was introduced with only a single function, +:func:`~html.escape`, which is used for escaping reserved characters from HTML +markup: + +>>> import html +>>> html.escape('x > 2 && x < 7') +'x > 2 && x < 7' + +socket +------ + +The :mod:`socket` module has two new improvements. + +* Socket objects now have a :meth:`~socket.socket.detach()` method which puts + the socket into closed state without actually closing the underlying file + descriptor. The latter can then be reused for other purposes. + (Added by Antoine Pitrou; :issue:`8524`.) + +* :func:`socket.create_connection` now supports the context manager protocol + to unconditionally consume :exc:`socket.error` exceptions and to close the + socket when done. + (Contributed by Giampaolo Rodolà; :issue:`9794`.) + +ssl +--- + +The :mod:`ssl` module added a number of features to satisfy common requirements +for secure (encrypted, authenticated) internet connections: + +* A new class, :class:`~ssl.SSLContext`, serves as a container for persistent + SSL data, such as protocol settings, certificates, private keys, and various + other options. It includes a :meth:`~ssl.SSLContext.wrap_socket` for creating + an SSL socket from an SSL context. + +* A new function, :func:`ssl.match_hostname`, supports server identity + verification for higher-level protocols by implementing the rules of HTTPS + (from :rfc:`2818`) which are also suitable for other protocols. + +* The :func:`ssl.wrap_socket` constructor function now takes a *ciphers* + argument. The *ciphers* string lists the allowed encryption algorithms using + the format described in the `OpenSSL documentation + <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`__. + +* When linked against recent versions of OpenSSL, the :mod:`ssl` module now + supports the Server Name Indication extension to the TLS protocol, allowing + multiple "virtual hosts" using different certificates on a single IP port. + This extension is only supported in client mode, and is activated by passing + the *server_hostname* argument to :meth:`ssl.SSLContext.wrap_socket`. + +* Various options have been added to the :mod:`ssl` module, such as + :data:`~ssl.OP_NO_SSLv2` which disables the insecure and obsolete SSLv2 + protocol. + +* The extension now loads all the OpenSSL ciphers and digest algorithms. If + some SSL certificates cannot be verified, they are reported as an "unknown + algorithm" error. + +* The version of OpenSSL being used is now accessible using the module + attributes :data:`ssl.OPENSSL_VERSION` (a string), + :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and + :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer). + +(Contributed by Antoine Pitrou in :issue:`8850`, :issue:`1589`, :issue:`8322`, +:issue:`5639`, :issue:`4870`, :issue:`8484`, and :issue:`8321`.) + +nntp +---- + +The :mod:`nntplib` module has a revamped implementation with better bytes and +text semantics as well as more practical APIs. These improvements break +compatibility with the nntplib version in Python 3.1, which was partly +dysfunctional in itself. + +Support for secure connections through both implicit (using +:class:`nntplib.NNTP_SSL`) and explicit (using :meth:`nntplib.NNTP.starttls`) +TLS has also been added. + +(Contributed by Antoine Pitrou in :issue:`9360` and Andrew Vant in :issue:`1926`.) + +certificates +------------ + +:class:`http.client.HTTPSConnection`, :class:`urllib.request.HTTPSHandler` +and :func:`urllib.request.urlopen` now take optional arguments to allow for +server certificate checking against a set of Certificate Authorities, +as recommended in public uses of HTTPS. + +(Added by Antoine Pitrou, :issue:`9003`.) + +imaplib +------- + +Support for explicit TLS on standard IMAP4 connections has been added through +the new :mod:`imaplib.IMAP4.starttls` method. + +(Contributed by Lorenzo M. Catucci and Antoine Pitrou, :issue:`4471`.) + +http.client +----------- + +There were a number of small API improvements in the :mod:`http.client` module. +The old-style HTTP 0.9 simple responses are no longer supported and the *strict* +parameter is deprecated in all classes. + +The :class:`~http.client.HTTPConnection` and +:class:`~http.client.HTTPSConnection` classes now have a *source_address* +parameter for a (host, port) tuple indicating where the HTTP connection is made +from. + +Support for certificate checking and HTTPS virtual hosts were added to +:class:`~http.client.HTTPSConnection`. + +The :meth:`~http.client.HTTPConnection.request` method on connection objects +allowed an optional *body* argument so that a :term:`file object` could be used +to supply the content of the request. Conveniently, the *body* argument now +also accepts an :term:`iterable` object so long as it includes an explicit +``Content-Length`` header. This extended interface is much more flexible than +before. + +To establish an HTTPS connection through a proxy server, there is a new +:meth:`~http.client.HTTPConnection.set_tunnel` method that sets the host and +port for HTTP Connect tunneling. + +To match the behavior of :mod:`http.server`, the HTTP client library now also +encodes headers with ISO-8859-1 (Latin-1) encoding. It was already doing that +for incoming headers, so now the behavior is consistent for both incoming and +outgoing traffic. (See work by Armin Ronacher in :issue:`10980`.) + +unittest +-------- + +The unittest module has a number of improvements supporting test discovery for +packages, easier experimentation at the interactive prompt, new testcase +methods, improved diagnostic messages for test failures, and better method +names. + +* The command-line call ``python -m unittest`` can now accept file paths + instead of module names for running specific tests (:issue:`10620`). The new + test discovery can find tests within packages, locating any test importable + from the top-level directory. The top-level directory can be specified with + the `-t` option, a pattern for matching files with ``-p``, and a directory to + start discovery with ``-s``:: + + $ python -m unittest discover -s my_proj_dir -p _test.py + + (Contributed by Michael Foord.) + +* Experimentation at the interactive prompt is now easier because the + :class:`unittest.case.TestCase` class can now be instantiated without + arguments: + + >>> TestCase().assertEqual(pow(2, 3), 8) + + (Contributed by Michael Foord.) + +* The :mod:`unittest` module has two new methods, + :meth:`~unittest.TestCase.assertWarns` and + :meth:`~unittest.TestCase.assertWarnsRegex` to verify that a given warning type + is triggered by the code under test:: + + with self.assertWarns(DeprecationWarning): + legacy_function('XYZ') + + (Contributed by Antoine Pitrou, :issue:`9754`.) + + Another new method, :meth:`~unittest.TestCase.assertCountEqual` is used to + compare two iterables to determine if their element counts are equal (whether + the same elements are present with the same number of occurrences regardless + of order):: + + def test_anagram(self): + self.assertCountEqual('algorithm', 'logarithm') + + (Contributed by Raymond Hettinger.) + +* A principal feature of the unittest module is an effort to produce meaningful + diagnostics when a test fails. When possible, the failure is recorded along + with a diff of the output. This is especially helpful for analyzing log files + of failed test runs. However, since diffs can sometime be voluminous, there is + a new :attr:`~unittest.TestCase.maxDiff` attribute that sets maximum length of + diffs displayed. + +* In addition, the method names in the module have undergone a number of clean-ups. + + For example, :meth:`~unittest.TestCase.assertRegex` is the new name for + :meth:`~unittest.TestCase.assertRegexpMatches` which was misnamed because the + test uses :func:`re.search`, not :func:`re.match`. Other methods using + regular expressions are now named using short form "Regex" in preference to + "Regexp" -- this matches the names used in other unittest implementations, + matches Python's old name for the :mod:`re` module, and it has unambiguous + camel-casing. + + (Contributed by Raymond Hettinger and implemented by Ezio Melotti.) + +* To improve consistency, some long-standing method aliases are being + deprecated in favor of the preferred names: + + =============================== ============================== + Old Name Preferred Name + =============================== ============================== + :meth:`assert_` :meth:`.assertTrue` + :meth:`assertEquals` :meth:`.assertEqual` + :meth:`assertNotEquals` :meth:`.assertNotEqual` + :meth:`assertAlmostEquals` :meth:`.assertAlmostEqual` + :meth:`assertNotAlmostEquals` :meth:`.assertNotAlmostEqual` + =============================== ============================== + + Likewise, the ``TestCase.fail*`` methods deprecated in Python 3.1 are expected + to be removed in Python 3.3. Also see the :ref:`deprecated-aliases` section in + the :mod:`unittest` documentation. + + (Contributed by Ezio Melotti; :issue:`9424`.) + +* The :meth:`~unittest.TestCase.assertDictContainsSubset` method was deprecated + because it was misimplemented with the arguments in the wrong order. This + created hard-to-debug optical illusions where tests like + ``TestCase().assertDictContainsSubset({'a':1, 'b':2}, {'a':1})`` would fail. + + (Contributed by Raymond Hettinger.) + +random +------ + +The integer methods in the :mod:`random` module now do a better job of producing +uniform distributions. Previously, they computed selections with +``int(n*random())`` which had a slight bias whenever *n* was not a power of two. +Now, multiple selections are made from a range up to the next power of two and a +selection is kept only when it falls within the range ``0 <= x < n``. The +functions and methods affected are :func:`~random.randrange`, +:func:`~random.randint`, :func:`~random.choice`, :func:`~random.shuffle` and +:func:`~random.sample`. + +(Contributed by Raymond Hettinger; :issue:`9025`.) + +poplib +------ + +:class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a +:class:`ssl.SSLContext` object allowing bundling SSL configuration options, +certificates and private keys into a single (potentially long-lived) +structure. + +(Contributed by Giampaolo Rodolà; :issue:`8807`.) + +asyncore +-------- + +:class:`asyncore.dispatcher` now provides a +:meth:`~asyncore.dispatcher.handle_accepted()` method +returning a `(sock, addr)` pair which is called when a connection has actually +been established with a new remote endpoint. This is supposed to be used as a +replacement for old :meth:`~asyncore.dispatcher.handle_accept()` and avoids +the user to call :meth:`~asyncore.dispatcher.accept()` directly. + +(Contributed by Giampaolo Rodolà; :issue:`6706`.) + +tempfile +-------- + +The :mod:`tempfile` module has a new context manager, +:class:`~tempfile.TemporaryDirectory` which provides easy deterministic +cleanup of temporary directories:: + + with tempfile.TemporaryDirectory() as tmpdirname: + print('created temporary dir:', tmpdirname) + +(Contributed by Neil Schemenauer and Nick Coghlan; :issue:`5178`.) + +inspect +------- + +* The :mod:`inspect` module has a new function + :func:`~inspect.getgeneratorstate` to easily identify the current state of a + generator-iterator:: + + >>> from inspect import getgeneratorstate + >>> def gen(): + yield 'demo' + >>> g = gen() + >>> getgeneratorstate(g) + 'GEN_CREATED' + >>> next(g) + 'demo' + >>> getgeneratorstate(g) + 'GEN_SUSPENDED' + >>> next(g, None) + >>> getgeneratorstate(g) + 'GEN_CLOSED' + + (Contributed by Rodolpho Eckhardt and Nick Coghlan, :issue:`10220`.) + +* To support lookups without the possibility of activating a dynamic attribute, + the :mod:`inspect` module has a new function, :func:`~inspect.getattr_static`. + Unlike :func:`hasattr`, this is a true read-only search, guaranteed not to + change state while it is searching:: + + >>> class A: + @property + def f(self): + print('Running') + return 10 + + >>> a = A() + >>> getattr(a, 'f') + Running + 10 + >>> inspect.getattr_static(a, 'f') + <property object at 0x1022bd788> + + (Contributed by Michael Foord.) + +pydoc +----- + +The :mod:`pydoc` module now provides a much-improved Web server interface, as +well as a new command-line option ``-b`` to automatically open a browser window +to display that server:: + + $ pydoc3.2 -b + +(Contributed by Ron Adam; :issue:`2001`.) + +dis +--- + +The :mod:`dis` module gained two new functions for inspecting code, +:func:`~dis.code_info` and :func:`~dis.show_code`. Both provide detailed code +object information for the supplied function, method, source code string or code +object. The former returns a string and the latter prints it:: + + >>> import dis, random + >>> dis.show_code(random.choice) + Name: choice + Filename: /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py + Argument count: 2 + Kw-only arguments: 0 + Number of locals: 3 + Stack size: 11 + Flags: OPTIMIZED, NEWLOCALS, NOFREE + Constants: + 0: 'Choose a random element from a non-empty sequence.' + 1: 'Cannot choose from an empty sequence' + Names: + 0: _randbelow + 1: len + 2: ValueError + 3: IndexError + Variable names: + 0: self + 1: seq + 2: i + +In addition, the :func:`~dis.dis` function now accepts string arguments +so that the common idiom ``dis(compile(s, '', 'eval'))`` can be shortened +to ``dis(s)``:: + + >>> dis('3*x+1 if x%2==1 else x//2') + 1 0 LOAD_NAME 0 (x) + 3 LOAD_CONST 0 (2) + 6 BINARY_MODULO + 7 LOAD_CONST 1 (1) + 10 COMPARE_OP 2 (==) + 13 POP_JUMP_IF_FALSE 28 + 16 LOAD_CONST 2 (3) + 19 LOAD_NAME 0 (x) + 22 BINARY_MULTIPLY + 23 LOAD_CONST 1 (1) + 26 BINARY_ADD + 27 RETURN_VALUE + >> 28 LOAD_NAME 0 (x) + 31 LOAD_CONST 0 (2) + 34 BINARY_FLOOR_DIVIDE + 35 RETURN_VALUE + +Taken together, these improvements make it easier to explore how CPython is +implemented and to see for yourself what the language syntax does +under-the-hood. + +(Contributed by Nick Coghlan in :issue:`9147`.) + +dbm +--- + +All database modules now support the :meth:`get` and :meth:`setdefault` methods. + +(Suggested by Ray Allen in :issue:`9523`.) + +ctypes +------ + +A new type, :class:`ctypes.c_ssize_t` represents the C :c:type:`ssize_t` datatype. + +site +---- + +The :mod:`site` module has three new functions useful for reporting on the +details of a given Python installation. + +* :func:`~site.getsitepackages` lists all global site-packages directories. + +* :func:`~site.getuserbase` reports on the user's base directory where data can + be stored. + +* :func:`~site.getusersitepackages` reveals the user-specific site-packages + directory path. + +:: + + >>> import site + >>> site.getsitepackages() + ['/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages', + '/Library/Frameworks/Python.framework/Versions/3.2/lib/site-python', + '/Library/Python/3.2/site-packages'] + >>> site.getuserbase() + '/Users/raymondhettinger/Library/Python/3.2' + >>> site.getusersitepackages() + '/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages' + +Conveniently, some of site's functionality is accessible directly from the +command-line:: + + $ python -m site --user-base + /Users/raymondhettinger/.local + $ python -m site --user-site + /Users/raymondhettinger/.local/lib/python3.2/site-packages + +(Contributed by Tarek Ziadé in :issue:`6693`.) + +sysconfig +--------- + +The new :mod:`sysconfig` module makes it straightforward to discover +installation paths and configuration variables that vary across platforms and +installations. + +The module offers access simple access functions for platform and version +information: + +* :func:`~sysconfig.get_platform` returning values like *linux-i586* or + *macosx-10.6-ppc*. +* :func:`~sysconfig.get_python_version` returns a Python version string + such as "3.2". + +It also provides access to the paths and variables corresponding to one of +seven named schemes used by :mod:`distutils`. Those include *posix_prefix*, +*posix_home*, *posix_user*, *nt*, *nt_user*, *os2*, *os2_home*: + +* :func:`~sysconfig.get_paths` makes a dictionary containing installation paths + for the current installation scheme. +* :func:`~sysconfig.get_config_vars` returns a dictionary of platform specific + variables. + +There is also a convenient command-line interface:: + + C:\Python32>python -m sysconfig + Platform: "win32" + Python version: "3.2" + Current installation scheme: "nt" + + Paths: + data = "C:\Python32" + include = "C:\Python32\Include" + platinclude = "C:\Python32\Include" + platlib = "C:\Python32\Lib\site-packages" + platstdlib = "C:\Python32\Lib" + purelib = "C:\Python32\Lib\site-packages" + scripts = "C:\Python32\Scripts" + stdlib = "C:\Python32\Lib" + + Variables: + BINDIR = "C:\Python32" + BINLIBDEST = "C:\Python32\Lib" + EXE = ".exe" + INCLUDEPY = "C:\Python32\Include" + LIBDEST = "C:\Python32\Lib" + SO = ".pyd" + VERSION = "32" + abiflags = "" + base = "C:\Python32" + exec_prefix = "C:\Python32" + platbase = "C:\Python32" + prefix = "C:\Python32" + projectbase = "C:\Python32" + py_version = "3.2" + py_version_nodot = "32" + py_version_short = "3.2" + srcdir = "C:\Python32" + userbase = "C:\Documents and Settings\Raymond\Application Data\Python" + +(Moved out of Distutils by Tarek Ziadé.) + +pdb +--- + +The :mod:`pdb` debugger module gained a number of usability improvements: + +* :file:`pdb.py` now has a ``-c`` option that executes commands as given in a + :file:`.pdbrc` script file. +* A :file:`.pdbrc` script file can contain ``continue`` and ``next`` commands + that continue debugging. +* The :class:`Pdb` class constructor now accepts a *nosigint* argument. +* New commands: ``l(list)``, ``ll(long list)`` and ``source`` for + listing source code. +* New commands: ``display`` and ``undisplay`` for showing or hiding + the value of an expression if it has changed. +* New command: ``interact`` for starting an interactive interpreter containing + the global and local names found in the current scope. +* Breakpoints can be cleared by breakpoint number. + +(Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.) + +configparser +------------ + +The :mod:`configparser` module was modified to improve usability and +predictability of the default parser and its supported INI syntax. The old +:class:`ConfigParser` class was removed in favor of :class:`SafeConfigParser` +which has in turn been renamed to :class:`~configparser.ConfigParser`. Support +for inline comments is now turned off by default and section or option +duplicates are not allowed in a single configuration source. + +Config parsers gained a new API based on the mapping protocol:: + + >>> parser = ConfigParser() + >>> parser.read_string(""" + [DEFAULT] + location = upper left + visible = yes + editable = no + color = blue + + [main] + title = Main Menu + color = green + + [options] + title = Options + """) + >>> parser['main']['color'] + 'green' + >>> parser['main']['editable'] + 'no' + >>> section = parser['options'] + >>> section['title'] + 'Options' + >>> section['title'] = 'Options (editable: %(editable)s)' + >>> section['title'] + 'Options (editable: no)' + +The new API is implemented on top of the classical API, so custom parser +subclasses should be able to use it without modifications. + +The INI file structure accepted by config parsers can now be customized. Users +can specify alternative option/value delimiters and comment prefixes, change the +name of the *DEFAULT* section or switch the interpolation syntax. + +There is support for pluggable interpolation including an additional interpolation +handler :class:`~configparser.ExtendedInterpolation`:: + + >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) + >>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'}, + 'custom': {'prefix': '/usr/local'}}) + >>> parser.read_string(""" + [buildout] + parts = + zope9 + instance + find-links = + ${buildout:directory}/downloads/dist + + [zope9] + recipe = plone.recipe.zope9install + location = /opt/zope + + [instance] + recipe = plone.recipe.zope9instance + zope9-location = ${zope9:location} + zope-conf = ${custom:prefix}/etc/zope.conf + """) + >>> parser['buildout']['find-links'] + '\n/home/ambv/zope9/downloads/dist' + >>> parser['instance']['zope-conf'] + '/usr/local/etc/zope.conf' + >>> instance = parser['instance'] + >>> instance['zope-conf'] + '/usr/local/etc/zope.conf' + >>> instance['zope9-location'] + '/opt/zope' + +A number of smaller features were also introduced, like support for specifying +encoding in read operations, specifying fallback values for get-functions, or +reading directly from dictionaries and strings. + +(All changes contributed by Łukasz Langa.) + +.. XXX consider showing a difflib example + +urllib.parse +------------ + +A number of usability improvements were made for the :mod:`urllib.parse` module. + +The :func:`~urllib.parse.urlparse` function now supports `IPv6 +<http://en.wikipedia.org/wiki/IPv6>`_ addresses as described in :rfc:`2732`: + + >>> import urllib.parse + >>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') + ParseResult(scheme='http', + netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]', + path='/foo/', + params='', + query='', + fragment='') + +The :func:`~urllib.parse.urldefrag` function now returns a :term:`named tuple`:: + + >>> r = urllib.parse.urldefrag('http://python.org/about/#target') + >>> r + DefragResult(url='http://python.org/about/', fragment='target') + >>> r[0] + 'http://python.org/about/' + >>> r.fragment + 'target' + +And, the :func:`~urllib.parse.urlencode` function is now much more flexible, +accepting either a string or bytes type for the *query* argument. If it is a +string, then the *safe*, *encoding*, and *error* parameters are sent to +:func:`~urllib.parse.quote_plus` for encoding:: + + >>> urllib.parse.urlencode([ + ('type', 'telenovela'), + ('name', '¿Dónde Está Elisa?')], + encoding='latin-1') + 'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F' + +As detailed in :ref:`parsing-ascii-encoded-bytes`, all the :mod:`urllib.parse` +functions now accept ASCII-encoded byte strings as input, so long as they are +not mixed with regular strings. If ASCII-encoded byte strings are given as +parameters, the return types will also be an ASCII-encoded byte strings: + + >>> urllib.parse.urlparse(b'http://www.python.org:80/about/') + ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80', + path=b'/about/', params=b'', query=b'', fragment=b'') + +(Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in :issue:`2987`, +:issue:`5468`, and :issue:`9873`.) + +mailbox +------- + +Thanks to a concerted effort by R. David Murray, the :mod:`mailbox` module has +been fixed for Python 3.2. The challenge was that mailbox had been originally +designed with a text interface, but email messages are best represented with +:class:`bytes` because various parts of a message may have different encodings. + +The solution harnessed the :mod:`email` package's binary support for parsing +arbitrary email messages. In addition, the solution required a number of API +changes. + +As expected, the :meth:`~mailbox.Mailbox.add` method for +:class:`mailbox.Mailbox` objects now accepts binary input. + +:class:`~io.StringIO` and text file input are deprecated. Also, string input +will fail early if non-ASCII characters are used. Previously it would fail when +the email was processed in a later step. + +There is also support for binary output. The :meth:`~mailbox.Mailbox.get_file` +method now returns a file in the binary mode (where it used to incorrectly set +the file to text-mode). There is also a new :meth:`~mailbox.Mailbox.get_bytes` +method that returns a :class:`bytes` representation of a message corresponding +to a given *key*. + +It is still possible to get non-binary output using the old API's +:meth:`~mailbox.Mailbox.get_string` method, but that approach +is not very useful. Instead, it is best to extract messages from +a :class:`~mailbox.Message` object or to load them from binary input. + +(Contributed by R. David Murray, with efforts from Steffen Daode Nurpmeso and an +initial patch by Victor Stinner in :issue:`9124`.) + +turtledemo +---------- + +The demonstration code for the :mod:`turtle` module was moved from the *Demo* +directory to main library. It includes over a dozen sample scripts with +lively displays. Being on :attr:`sys.path`, it can now be run directly +from the command-line:: + + $ python -m turtledemo + +(Moved from the Demo directory by Alexander Belopolsky in :issue:`10199`.) + +Multi-threading +=============== + +* The mechanism for serializing execution of concurrently running Python threads + (generally known as the :term:`GIL` or :term:`Global Interpreter Lock`) has + been rewritten. Among the objectives were more predictable switching + intervals and reduced overhead due to lock contention and the number of + ensuing system calls. The notion of a "check interval" to allow thread + switches has been abandoned and replaced by an absolute duration expressed in + seconds. This parameter is tunable through :func:`sys.setswitchinterval()`. + It currently defaults to 5 milliseconds. + + Additional details about the implementation can be read from a `python-dev + mailing-list message + <http://mail.python.org/pipermail/python-dev/2009-October/093321.html>`_ + (however, "priority requests" as exposed in this message have not been kept + for inclusion). + + (Contributed by Antoine Pitrou.) + +* Regular and recursive locks now accept an optional *timeout* argument to their + :meth:`~threading.Lock.acquire` method. (Contributed by Antoine Pitrou; + :issue:`7316`.) + +* Similarly, :meth:`threading.Semaphore.acquire` also gained a *timeout* + argument. (Contributed by Torsten Landschoff; :issue:`850728`.) + +* Regular and recursive lock acquisitions can now be interrupted by signals on + platforms using Pthreads. This means that Python programs that deadlock while + acquiring locks can be successfully killed by repeatedly sending SIGINT to the + process (by pressing :kbd:`Ctrl+C` in most shells). + (Contributed by Reid Kleckner; :issue:`8844`.) + + +Optimizations +============= + +A number of small performance enhancements have been added: + +* Python's peephole optimizer now recognizes patterns such ``x in {1, 2, 3}`` as + being a test for membership in a set of constants. The optimizer recasts the + :class:`set` as a :class:`frozenset` and stores the pre-built constant. + + Now that the speed penalty is gone, it is practical to start writing + membership tests using set-notation. This style is both semantically clear + and operationally fast:: + + extension = name.rpartition('.')[2] + if extension in {'xml', 'html', 'xhtml', 'css'}: + handle(name) + + (Patch and additional tests contributed by Dave Malcolm; :issue:`6690`). + +* Serializing and unserializing data using the :mod:`pickle` module is now + several times faster. + + (Contributed by Alexandre Vassalotti, Antoine Pitrou + and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.) + +* The `Timsort algorithm <http://en.wikipedia.org/wiki/Timsort>`_ used in + :meth:`list.sort` and :func:`sorted` now runs faster and uses less memory + when called with a :term:`key function`. Previously, every element of + a list was wrapped with a temporary object that remembered the key value + associated with each element. Now, two arrays of keys and values are + sorted in parallel. This saves the memory consumed by the sort wrappers, + and it saves time lost to delegating comparisons. + + (Patch by Daniel Stutzbach in :issue:`9915`.) + +* JSON decoding performance is improved and memory consumption is reduced + whenever the same string is repeated for multiple keys. Also, JSON encoding + now uses the C speedups when the ``sort_keys`` argument is true. + + (Contributed by Antoine Pitrou in :issue:`7451` and by Raymond Hettinger and + Antoine Pitrou in :issue:`10314`.) + +* Recursive locks (created with the :func:`threading.RLock` API) now benefit + from a C implementation which makes them as fast as regular locks, and between + 10x and 15x faster than their previous pure Python implementation. + + (Contributed by Antoine Pitrou; :issue:`3001`.) + +* The fast-search algorithm in stringlib is now used by the :meth:`split`, + :meth:`splitlines` and :meth:`replace` methods on + :class:`bytes`, :class:`bytearray` and :class:`str` objects. Likewise, the + algorithm is also used by :meth:`rfind`, :meth:`rindex`, :meth:`rsplit` and + :meth:`rpartition`. + + (Patch by Florent Xicluna in :issue:`7622` and :issue:`7462`.) + + +* Integer to string conversions now work two "digits" at a time, reducing the + number of division and modulo operations. + + (:issue:`6713` by Gawain Bolton, Mark Dickinson, and Victor Stinner.) + +There were several other minor optimizations. Set differencing now runs faster +when one operand is much larger than the other (patch by Andress Bennetts in +:issue:`8685`). The :meth:`array.repeat` method has a faster implementation +(:issue:`1569291` by Alexander Belopolsky). The :class:`BaseHTTPRequestHandler` +has more efficient buffering (:issue:`3709` by Andrew Schaaf). The +:func:`operator.attrgetter` function has been sped-up (:issue:`10160` by +Christos Georgiou). And :class:`ConfigParser` loads multi-line arguments a bit +faster (:issue:`7113` by Łukasz Langa). + + +Unicode +======= + +Python has been updated to `Unicode 6.0.0 +<http://unicode.org/versions/Unicode6.0.0/>`_. The update to the standard adds +over 2,000 new characters including `emoji <http://en.wikipedia.org/wiki/Emoji>`_ +symbols which are important for mobile phones. + +In addition, the updated standard has altered the character properties for two +Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric character +(U+19DA), making the former eligible for use in identifiers while disqualifying +the latter. For more information, see `Unicode Character Database Changes +<http://www.unicode.org/versions/Unicode6.0.0/#Database_Changes>`_. + + +Codecs +====== + +Support was added for *cp720* Arabic DOS encoding (:issue:`1616979`). + +MBCS encoding no longer ignores the error handler argument. In the default +strict mode, it raises an :exc:`UnicodeDecodeError` when it encounters an +undecodable byte sequence and an :exc:`UnicodeEncodeError` for an unencodable +character. + +The MBCS codec supports ``'strict'`` and ``'ignore'`` error handlers for +decoding, and ``'strict'`` and ``'replace'`` for encoding. + +To emulate Python3.1 MBCS encoding, select the ``'ignore'`` handler for decoding +and the ``'replace'`` handler for encoding. + +On Mac OS X, Python decodes command line arguments with ``'utf-8'`` rather than +the locale encoding. + +By default, :mod:`tarfile` uses ``'utf-8'`` encoding on Windows (instead of +``'mbcs'``) and the ``'surrogateescape'`` error handler on all operating +systems. + + +Documentation +============= + +The documentation continues to be improved. + +* A table of quick links has been added to the top of lengthy sections such as + :ref:`built-in-funcs`. In the case of :mod:`itertools`, the links are + accompanied by tables of cheatsheet-style summaries to provide an overview and + memory jog without having to read all of the docs. + +* In some cases, the pure Python source code can be a helpful adjunct to the + documentation, so now many modules now feature quick links to the latest + version of the source code. For example, the :mod:`functools` module + documentation has a quick link at the top labeled: + + **Source code** :source:`Lib/functools.py`. + + (Contributed by Raymond Hettinger; see + `rationale <http://rhettinger.wordpress.com/2011/01/28/open-your-source-more/>`_.) + +* The docs now contain more examples and recipes. In particular, :mod:`re` + module has an extensive section, :ref:`re-examples`. Likewise, the + :mod:`itertools` module continues to be updated with new + :ref:`itertools-recipes`. + +* The :mod:`datetime` module now has an auxiliary implementation in pure Python. + No functionality was changed. This just provides an easier-to-read alternate + implementation. + + (Contributed by Alexander Belopolsky in :issue:`9528`.) + +* The unmaintained :file:`Demo` directory has been removed. Some demos were + integrated into the documentation, some were moved to the :file:`Tools/demo` + directory, and others were removed altogether. + + (Contributed by Georg Brandl in :issue:`7962`.) + + +IDLE +==== + +* The format menu now has an option to clean source files by stripping + trailing whitespace. + + (Contributed by Raymond Hettinger; :issue:`5150`.) + +* IDLE on Mac OS X now works with both Carbon AquaTk and Cocoa AquaTk. + + (Contributed by Kevin Walzer, Ned Deily, and Ronald Oussoren; :issue:`6075`.) + +Code Repository +=============== + +In addition to the existing Subversion code repository at http://svn.python.org +there is now a `Mercurial <http://mercurial.selenic.com/>`_ repository at +http://hg.python.org/\. + +After the 3.2 release, there are plans to switch to Mercurial as the primary +repository. This distributed version control system should make it easier for +members of the community to create and share external changesets. See +:pep:`385` for details. + +To learn the new version control system, see the `tutorial by Joel +Spolsky <http://hginit.com>`_ or the `Guide to Mercurial Workflows +<http://mercurial.selenic.com/guide/>`_. + + +Build and C API Changes +======================= + +Changes to Python's build process and to the C API include: + +* The *idle*, *pydoc* and *2to3* scripts are now installed with a + version-specific suffix on ``make altinstall`` (:issue:`10679`). + +* The C functions that access the Unicode Database now accept and return + characters from the full Unicode range, even on narrow unicode builds + (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others). A visible difference + in Python is that :func:`unicodedata.numeric` now returns the correct value + for large code points, and :func:`repr` may consider more characters as + printable. + + (Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; :issue:`5127`.) + +* Computed gotos are now enabled by default on supported compilers (which are + detected by the configure script). They can still be disabled selectively by + specifying ``--without-computed-gotos``. + + (Contributed by Antoine Pitrou; :issue:`9203`.) + +* The option ``--with-wctype-functions`` was removed. The built-in unicode + database is now used for all functions. + + (Contributed by Amaury Forgeot D'Arc; :issue:`9210`.) + +* Hash values are now values of a new type, :c:type:`Py_hash_t`, which is + defined to be the same size as a pointer. Previously they were of type long, + which on some 64-bit operating systems is still only 32 bits long. As a + result of this fix, :class:`set` and :class:`dict` can now hold more than + ``2**32`` entries on builds with 64-bit pointers (previously, they could grow + to that size but their performance degraded catastrophically). + + (Suggested by Raymond Hettinger and implemented by Benjamin Peterson; + :issue:`9778`.) + +* A new macro :c:macro:`Py_VA_COPY` copies the state of the variable argument + list. It is equivalent to C99 *va_copy* but available on all Python platforms + (:issue:`2443`). + +* A new C API function :c:func:`PySys_SetArgvEx` allows an embedded interpreter + to set :attr:`sys.argv` without also modifying :attr:`sys.path` + (:issue:`5753`). + +* :c:macro:`PyEval_CallObject` is now only available in macro form. The + function declaration, which was kept for backwards compatibility reasons, is + now removed -- the macro was introduced in 1997 (:issue:`8276`). + +* There is a new function :c:func:`PyLong_AsLongLongAndOverflow` which + is analogous to :c:func:`PyLong_AsLongAndOverflow`. They both serve to + convert Python :class:`int` into a native fixed-width type while providing + detection of cases where the conversion won't fit (:issue:`7767`). + +* The :c:func:`PyUnicode_CompareWithASCIIString` function now returns *not + equal* if the Python string is *NUL* terminated. + +* There is a new function :c:func:`PyErr_NewExceptionWithDoc` that is + like :c:func:`PyErr_NewException` but allows a docstring to be specified. + This lets C exceptions have the same self-documenting capabilities as + their pure Python counterparts (:issue:`7033`). + +* When compiled with the ``--with-valgrind`` option, the pymalloc + allocator will be automatically disabled when running under Valgrind. This + gives improved memory leak detection when running under Valgrind, while taking + advantage of pymalloc at other times (:issue:`2422`). + +* Removed the ``O?`` format from the *PyArg_Parse* functions. The format is no + longer used and it had never been documented (:issue:`8837`). + +There were a number of other small changes to the C-API. See the +:source:`Misc/NEWS` file for a complete list. + +Also, there were a number of updates to the Mac OS X build, see +:source:`Mac/BuildScript/README.txt` for details. For users running a 32/64-bit +build, there is a known problem with the default Tcl/Tk on Mac OS X 10.6. +Accordingly, we recommend installing an updated alternative such as +`ActiveState Tcl/Tk 8.5.9 <http://www.activestate.com/activetcl/downloads>`_\. +See http://www.python.org/download/mac/tcltk/ for additional details. + +Porting to Python 3.2 +===================== + +This section lists previously described changes and other bugfixes that may +require changes to your code: + +* The :mod:`configparser` module has a number of clean-ups. The major change is + to replace the old :class:`ConfigParser` class with long-standing preferred + alternative :class:`SafeConfigParser`. In addition there are a number of + smaller incompatibilities: + + * The interpolation syntax is now validated on + :meth:`~configparser.ConfigParser.get` and + :meth:`~configparser.ConfigParser.set` operations. In the default + interpolation scheme, only two tokens with percent signs are valid: ``%(name)s`` + and ``%%``, the latter being an escaped percent sign. + + * The :meth:`~configparser.ConfigParser.set` and + :meth:`~configparser.ConfigParser.add_section` methods now verify that + values are actual strings. Formerly, unsupported types could be introduced + unintentionally. + + * Duplicate sections or options from a single source now raise either + :exc:`~configparser.DuplicateSectionError` or + :exc:`~configparser.DuplicateOptionError`. Formerly, duplicates would + silently overwrite a previous entry. + + * Inline comments are now disabled by default so now the **;** character + can be safely used in values. + + * Comments now can be indented. Consequently, for **;** or **#** to appear at + the start of a line in multiline values, it has to be interpolated. This + keeps comment prefix characters in values from being mistaken as comments. + + * ``""`` is now a valid value and is no longer automatically converted to an + empty string. For empty strings, use ``"option ="`` in a line. + +* The :mod:`nntplib` module was reworked extensively, meaning that its APIs + are often incompatible with the 3.1 APIs. + +* :class:`bytearray` objects can no longer be used as filenames; instead, + they should be converted to :class:`bytes`. + +* The :meth:`array.tostring` and :meth:`array.fromstring` have been renamed to + :meth:`array.tobytes` and :meth:`array.frombytes` for clarity. The old names + have been deprecated. (See :issue:`8990`.) + +* ``PyArg_Parse*()`` functions: + + * "t#" format has been removed: use "s#" or "s*" instead + * "w" and "w#" formats has been removed: use "w*" instead + +* The :c:type:`PyCObject` type, deprecated in 3.1, has been removed. To wrap + opaque C pointers in Python objects, the :c:type:`PyCapsule` API should be used + instead; the new type has a well-defined interface for passing typing safety + information and a less complicated signature for calling a destructor. + +* The :func:`sys.setfilesystemencoding` function was removed because + it had a flawed design. + +* The :func:`random.seed` function and method now salt string seeds with an + sha512 hash function. To access the previous version of *seed* in order to + reproduce Python 3.1 sequences, set the *version* argument to *1*, + ``random.seed(s, version=1)``. + +* The previously deprecated :func:`string.maketrans` function has been removed + in favor of the static methods :meth:`bytes.maketrans` and + :meth:`bytearray.maketrans`. This change solves the confusion around which + types were supported by the :mod:`string` module. Now, :class:`str`, + :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and + **translate** methods with intermediate translation tables of the appropriate + type. + + (Contributed by Georg Brandl; :issue:`5675`.) + +* The previously deprecated :func:`contextlib.nested` function has been removed + in favor of a plain :keyword:`with` statement which can accept multiple + context managers. The latter technique is faster (because it is built-in), + and it does a better job finalizing multiple context managers when one of them + raises an exception:: + + with open('mylog.txt') as infile, open('a.out', 'w') as outfile: + for line in infile: + if '<critical>' in line: + outfile.write(line) + + (Contributed by Georg Brandl and Mattias Brändström; + `appspot issue 53094 <http://codereview.appspot.com/53094>`_.) + +* :func:`struct.pack` now only allows bytes for the ``s`` string pack code. + Formerly, it would accept text arguments and implicitly encode them to bytes + using UTF-8. This was problematic because it made assumptions about the + correct encoding and because a variable-length encoding can fail when writing + to fixed length segment of a structure. + + Code such as ``struct.pack('<6sHHBBB', 'GIF87a', x, y)`` should be rewritten + with to use bytes instead of text, ``struct.pack('<6sHHBBB', b'GIF87a', x, y)``. + + (Discovered by David Beazley and fixed by Victor Stinner; :issue:`10783`.) + +* The :class:`xml.etree.ElementTree` class now raises an + :exc:`xml.etree.ElementTree.ParseError` when a parse fails. Previously it + raised a :exc:`xml.parsers.expat.ExpatError`. + +* The new, longer :func:`str` value on floats may break doctests which rely on + the old output format. + +* In :class:`subprocess.Popen`, the default value for *close_fds* is now + ``True`` under Unix; under Windows, it is ``True`` if the three standard + streams are set to ``None``, ``False`` otherwise. Previously, *close_fds* + was always ``False`` by default, which produced difficult to solve bugs + or race conditions when open file descriptors would leak into the child + process. + +* Support for legacy HTTP 0.9 has been removed from :mod:`urllib.request` + and :mod:`http.client`. Such support is still present on the server side + (in :mod:`http.server`). + + (Contributed by Antoine Pitrou, :issue:`10711`.) + +* SSL sockets in timeout mode now raise :exc:`socket.timeout` when a timeout + occurs, rather than a generic :exc:`~ssl.SSLError`. + + (Contributed by Antoine Pitrou, :issue:`10272`.) + +* The misleading functions :c:func:`PyEval_AcquireLock()` and + :c:func:`PyEval_ReleaseLock()` have been officially deprecated. The + thread-state aware APIs (such as :c:func:`PyEval_SaveThread()` + and :c:func:`PyEval_RestoreThread()`) should be used instead. + +* Due to security risks, :func:`asyncore.handle_accept` has been deprecated, and + a new function, :func:`asyncore.handle_accepted`, was added to replace it. + + (Contributed by Giampaolo Rodola in :issue:`6706`.) + +* Due to the new :term:`GIL` implementation, :c:func:`PyEval_InitThreads()` + cannot be called before :c:func:`Py_Initialize()` anymore. + diff --git a/Doc/whatsnew/index.rst b/Doc/whatsnew/index.rst index a1efe92993..8220bd28dd 100644 --- a/Doc/whatsnew/index.rst +++ b/Doc/whatsnew/index.rst @@ -11,8 +11,10 @@ anyone wishing to stay up-to-date after a new release. .. toctree:: :maxdepth: 2 + 3.2.rst 3.1.rst 3.0.rst + 2.7.rst 2.6.rst 2.5.rst 2.4.rst |
