summaryrefslogtreecommitdiff
path: root/Lib/tokenize.py
Commit message (Collapse)AuthorAgeFilesLines
* bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686)Anthony Sottile2019-05-301-1/+1
| | | https://bugs.python.org/issue5028
* bpo-5028: fix doc bug for tokenize (GH-11683)Andrew Carr2019-05-301-1/+1
| | | https://bugs.python.org/issue5028
* bpo-36766: Typos in docs and code comments (GH-13116)penguindustin2019-05-061-1/+1
|
* bpo-30455: Generate all token related code and docs from Grammar/Tokens. ↵Serhiy Storchaka2018-12-221-60/+6
| | | | | | | | | | | | | | | | | | | (GH-10370) "Include/token.h", "Lib/token.py" (containing now some data moved from "Lib/tokenize.py") and new files "Parser/token.c" (containing the code moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by "Tools/scripts/generate_token.py". The script overwrites files only if needed and can be used on the read-only sources tree. "Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py" instead of been executable itself. Added new make targets "regen-token" and "regen-symbol" which are now dependencies of "regen-all". The documentation contains now strings for operators and punctuation tokens.
* bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior ↵Ammar Askar2018-07-061-0/+10
| | | | | | | | | (GH-7891) Most of the change involves fixing up the test suite, which previously made the assumption that there wouldn't be a new line if the input didn't end in one. Contributed by Ammar Askar.
* bpo-12486: Document tokenize.generate_tokens() as public API (#6957)Thomas Kluyver2018-06-051-3/+6
| | | | | | | | | | | | * Document tokenize.generate_tokens() * Add news file * Add test for generate_tokens * Document behaviour around ENCODING token * Add generate_tokens to __all__
* bpo-33338: [tokenize] Minor code cleanup (#6573)Łukasz Langa2018-04-231-11/+8
| | | | This change contains minor things that make diffing between Lib/tokenize.py and Lib/lib2to3/pgen2/tokenize.py cleaner.
* bpo-33260: Regenerate token.py after removing ASYNC and AWAIT. (GH-6447)Serhiy Storchaka2018-04-111-1/+1
|
* bpo-30406: Make async and await proper keywords (#1669)Jelle Zijlstra2017-10-051-61/+1
| | | Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
* bpo-25324: copy tok_name before changing it (#1608)Albert-Jan Nijburg2017-05-311-9/+2
| | | | | | | | | | | | | | | | | | | | | | | | * add test to check if were modifying token * copy list so import tokenize doesnt have side effects on token * shorten line * add tokenize tokens to token.h to get them to show up in token * move ERRORTOKEN back to its previous location, and fix nitpick * copy comments from token.h automatically * fix whitespace and make more pythonic * change to fix comments from @haypo * update token.rst and Misc/NEWS * change wording * some more wording changes
* bpo-30377: Simplify handling of COMMENT and NL in tokenize.py (#1607)Albert-Jan Nijburg2017-05-241-5/+3
|
* bpo-30296 Remove unnecessary tuples, lists, sets, and dicts (#1489)Jon Dufresne2017-05-181-1/+1
| | | | | | | | * Replaced list(<generator expression>) with list comprehension * Replaced dict(<generator expression>) with dict comprehension * Replaced set(<list literal>) with set literal * Replaced builtin func(<list comprehension>) with func(<generator expression>) when supported (e.g. any(), all(), tuple(), min(), & max())
* Add ELLIPSIS and RARROW. Add tests (#666)Jim Fasarakis-Hilliard2017-03-141-1/+3
|
* Issue #26331: Implement the parsing part of PEP 515.Brett Cannon2016-09-091-8/+9
| | | | Thanks to Georg Brandl for the patch.
* Issue #26581: Use the first coding cookie on a line, not the last one.Serhiy Storchaka2016-03-201-1/+1
|\
| * Issue #26581: Use the first coding cookie on a line, not the last one.Serhiy Storchaka2016-03-201-1/+1
| |
* | Issue #25977: Fix typos in Lib/tokenize.pyBerker Peksag2015-12-301-5/+5
|\ \ | |/ | | | | Patch by John Walker.
| * Issue #25977: Fix typos in Lib/tokenize.pyBerker Peksag2015-12-301-4/+4
| | | | | | | | Patch by John Walker.
* | Issue 25311: Add support for f-strings to tokenize.py. Also added some ↵Eric V. Smith2015-10-261-51/+67
|/ | | | comments to explain what's happening, since it's not so obvious.
* Issue #24619: Simplify async/await tokenization.Yury Selivanov2015-07-231-16/+23
| | | | | | | | | | This commit simplifies async/await tokenization in tokenizer.c, tokenize.py & lib2to3/tokenize.py. Previous solution was to keep a stack of async-def & def blocks, whereas the new approach is just to remember position of the outermost async-def block. This change won't bring any parsing performance improvements, but it makes the code much easier to read and validate.
* Issue #24619: New approach for tokenizing async/await.Yury Selivanov2015-07-221-1/+6
| | | | | | | | | | | | | | | | | | | | | This commit fixes how one-line async-defs and defs are tracked by tokenizer. It allows to correctly parse invalid code such as: >>> async def f(): ... def g(): pass ... async = 10 and valid code such as: >>> async def f(): ... async def g(): pass ... await z As a consequence, is is now possible to have one-line 'async def foo(): await ..' functions: >>> async def foo(): return await bar()
* Issue #20387: Merge test and patch from 3.4.4Jason R. Coombs2015-06-281-0/+17
|\
| * Issue #20387: Restore retention of indentation during untokenize.Dingyuan Wang2015-06-221-0/+17
| |
* | (Merge 3.5) Issue #23840: tokenize.open() now closes the temporary binary fileVictor Stinner2015-05-261-5/+9
|\ \ | |/ | | | | on error to fix a resource warning.
| * Issue #23840: tokenize.open() now closes the temporary binary file on error toVictor Stinner2015-05-261-5/+9
| | | | | | | | fix a resource warning.
* | PEP 0492 -- Coroutines with async and await syntax. Issue #24017.Yury Selivanov2015-05-111-2/+54
| |
* | Issue #23615: Modules bz2, tarfile and tokenize now can be reloaded withSerhiy Storchaka2015-03-111-2/+1
|\ \ | |/ | | | | imp.reload(). Patch by Thomas Kluyver.
| * Issue #23615: Modules bz2, tarfile and tokenize now can be reloaded withSerhiy Storchaka2015-03-111-2/+1
| | | | | | | | imp.reload(). Patch by Thomas Kluyver.
* | Removed duplicated dict entries.Serhiy Storchaka2015-01-111-1/+0
| |
* | (Merge 3.4) Issue #22599: Enhance tokenize.open() to be able to call it duringVictor Stinner2014-12-051-3/+4
|\ \ | |/ | | | | | | | | | | | | | | | | | | | | | | Python finalization. Before the module kept a reference to the builtins module, but the module attributes are cleared during Python finalization. Instead, keep directly a reference to the open() function. This enhancement is not perfect, calling tokenize.open() can still fail if called very late during Python finalization. Usually, the function is called by the linecache module which is called to display a traceback or emit a warning.
| * Issue #22599: Enhance tokenize.open() to be able to call it during PythonVictor Stinner2014-12-051-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | finalization. Before the module kept a reference to the builtins module, but the module attributes are cleared during Python finalization. Instead, keep directly a reference to the open() function. This enhancement is not perfect, calling tokenize.open() can still fail if called very late during Python finalization. Usually, the function is called by the linecache module which is called to display a traceback or emit a warning.
* | PEP 465: a dedicated infix operator for matrix multiplication (closes #21176)Benjamin Peterson2014-04-091-2/+3
|/
* Merge with 3.3Terry Jan Reedy2014-02-231-1/+1
|\
| * whitespaceTerry Jan Reedy2014-02-231-1/+1
| |
* | Merge with 3.3Terry Jan Reedy2014-02-231-0/+6
|\ \ | |/
| * Issue #9974: When untokenizing, use row info to insert backslash+newline.Terry Jan Reedy2014-02-231-0/+6
| | | | | | | | Original patches by A. Kuchling and G. Rees (#12691).
* | Merge with 3.3Terry Jan Reedy2014-02-171-13/+11
|\ \ | |/
| * Issue #8478: Untokenizer.compat now processes first token from iterator input.Terry Jan Reedy2014-02-171-13/+11
| | | | | | | | Patch based on lines from Georg Brandl, Eric Snow, and Gareth Rees.
* | Untokenize, bad assert: Merge with 3.3Terry Jan Reedy2014-02-171-1/+3
|\ \ | |/
| * Untokenize: An logically incorrect assert tested user input validity.Terry Jan Reedy2014-02-171-1/+3
| | | | | | | | | | | | Replace it with correct logic that raises ValueError for bad input. Issues #8478 and #12691 reported the incorrect logic. Add an Untokenize test case and an initial test method.
* | Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-0/+3
|\ \ | |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
| * Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
* | #19620: merge with 3.3.Ezio Melotti2013-11-251-1/+1
|\ \ | |/
| * #19620: Fix typo in docstring (noticed by Christopher Welborn).Ezio Melotti2013-11-251-1/+1
| |
* | Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py scriptSerhiy Storchaka2013-09-161-4/+4
|\ \ | |/ | | | | now detect Python source code encoding only in comment lines.
| * Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py scriptSerhiy Storchaka2013-09-161-4/+4
| | | | | | | | now detect Python source code encoding only in comment lines.
* | Replace IOError with OSError (#16715)Andrew Svetlov2012-12-251-1/+1
|/
* #16152: merge with 3.2.Ezio Melotti2012-11-031-1/+3
|\
| * #16152: fix tokenize to ignore whitespace at the end of the code when no ↵Ezio Melotti2012-11-031-1/+3
| | | | | | | | newline is found. Patch by Ned Batchelder.
* | Merge branchFlorent Xicluna2012-07-071-1/+1
|\ \ | |/
View Lorry Controller Status