summaryrefslogtreecommitdiff
path: root/Parser/tokenizer.c
Commit message (Collapse)AuthorAgeFilesLines
* bpo-36459: Fix a possible double PyMem_FREE() due to tokenizer.c's ↵Miss Islington (bot)2019-03-281-1/+0
| | | | | | | | | tok_nextc() (12601) Remove the PyMem_FREE() call added in cb90c89. The buffer will be freed when PyTokenizer_Free() is called on the tokenizer state. (cherry picked from commit cda139d1ded6708665b53e4ed32ccc1d2627e1da) Co-authored-by: Zackery Spytz <zspytz@gmail.com>
* bpo-36367: Free buffer if realloc fails in tokenize.c (GH-12442) (GH-12471)Victor Stinner2019-03-201-2/+8
|
* bpo-35436: Add missing PyErr_NoMemory() calls and other minor bug fixes. ↵Zackery Spytz2018-12-071-0/+5
| | | | | (GH-11015) (GH-11020) (cherry picked from commit 4c49da0cb7434c676d70b9ccf38aca82ac0d64a9)
* tokenizer: Remove unused tabs options (#4422)Victor Stinner2017-11-171-31/+11
| | | | | | | | | | Remove the following fields from tok_state structure which are now used unused: * altwarning: "Issue warning if alternate tabs don't match" * alterror: "Issue error if alternate tabs don't match" * alttabsize: "Alternate tab spacing" Replace alttabsize variable with ALTTABSIZE define.
* bpo-30406: Make async and await proper keywords (#1669)Jelle Zijlstra2017-10-051-63/+0
| | | Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
* bpo-25324: add missing comma in Parser/tokenizer.c (GH-1910)Albert-Jan Nijburg2017-06-011-1/+1
|
* bpo-25324: copy tok_name before changing it (#1608)Albert-Jan Nijburg2017-05-311-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | * add test to check if were modifying token * copy list so import tokenize doesnt have side effects on token * shorten line * add tokenize tokens to token.h to get them to show up in token * move ERRORTOKEN back to its previous location, and fix nitpick * copy comments from token.h automatically * fix whitespace and make more pythonic * change to fix comments from @haypo * update token.rst and Misc/NEWS * change wording * some more wording changes
* Issue #28489: Merge from 3.6Berker Peksag2017-02-051-1/+1
|\
| * Issue #28489: Fix comment in tokenizer.cBerker Peksag2017-02-051-1/+1
| | | | | | | | Patch by Ryan Gonzalez.
* | Use _PyObject_CallNoArg()Victor Stinner2016-12-061-3/+3
|/ | | | | | | Replace: PyObject_CallObject(callable, NULL) with: _PyObject_CallNoArg(callable)
* Replaced outdated macros _PyUnicode_AsString and _PyUnicode_AsStringAndSizeSerhiy Storchaka2016-11-201-1/+1
| | | | with PyUnicode_AsUTF8 and PyUnicode_AsUTF8AndSize.
* merge 3.5 (#24022)Benjamin Peterson2016-09-181-1/+1
|\
| * merge 3.4Benjamin Peterson2016-09-181-1/+1
| |\
| | * properly handle the single null-byte file (closes #24022)Benjamin Peterson2016-09-181-1/+1
| | |
* | | merge 3.5 (#27981)Benjamin Peterson2016-09-121-15/+17
|\ \ \ | |/ /
| * | restructure fp_setreadl so as to avoid refleaks (closes #27981)Benjamin Peterson2016-09-121-15/+17
| | |
* | | Issue #26331: Implement the parsing part of PEP 515.Brett Cannon2016-09-091-68/+162
| | | | | | | | | | | | Thanks to Georg Brandl for the patch.
* | | Skip unused value in tokenizer codeChristian Heimes2016-09-091-1/+1
| | | | | | | | | | | | | | | | | | | | | In the case of an escape character, c is never read. tok_next() is used to advance the pointer. CID 1225097
* | | Issue #22570: Renamed Py_SETREF to Py_XSETREF.Serhiy Storchaka2016-04-061-1/+1
|\ \ \ | |/ /
| * | Issue #22570: Renamed Py_SETREF to Py_XSETREF.Serhiy Storchaka2016-04-061-1/+1
| | |
* | | remove duplicated check for fractions and complex numbers (closes #26076)Benjamin Peterson2016-03-241-4/+0
| | | | | | | | | | | | Patch by Oren Milman.
* | | Issue #26581: Use the first coding cookie on a line, not the last one.Serhiy Storchaka2016-03-201-0/+1
|\ \ \ | |/ /
| * | Issue #26581: Use the first coding cookie on a line, not the last one.Serhiy Storchaka2016-03-201-0/+1
| | |
* | | Issue #25923: Added more const qualifiers to signatures of static and ↵Serhiy Storchaka2015-12-251-3/+3
| | | | | | | | | | | | private functions.
* | | Issue #20440: Massive replacing unsafe attribute setting code with specialSerhiy Storchaka2015-12-241-2/+1
|\ \ \ | |/ / | | | | | | macro Py_SETREF.
| * | Issue #20440: Massive replacing unsafe attribute setting code with specialSerhiy Storchaka2015-12-241-2/+1
| | | | | | | | | | | | macro Py_SETREF.
* | | Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
|\ \ \ | |/ / | | | | | | with a null byte.
| * | Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
| |\ \ | | |/ | | | | | | with a null byte.
| | * Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
| | | | | | | | | | | | with a null byte.
* | | Issue #24965: Implement PEP 498 "Literal String Interpolation". ↵Eric V. Smith2015-09-191-3/+5
| | | | | | | | | | | | Documentation is still needed, I'll open an issue for that.
* | | Fixed indentation.Eric V. Smith2015-09-121-1/+1
|/ /
* | Issue #24619: Simplify async/await tokenization.Yury Selivanov2015-07-231-87/+39
| | | | | | | | | | | | | | | | | | | | This commit simplifies async/await tokenization in tokenizer.c, tokenize.py & lib2to3/tokenize.py. Previous solution was to keep a stack of async-def & def blocks, whereas the new approach is just to remember position of the outermost async-def block. This change won't bring any parsing performance improvements, but it makes the code much easier to read and validate.
* | Issue #24619: New approach for tokenizing async/await.Yury Selivanov2015-07-221-35/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit fixes how one-line async-defs and defs are tracked by tokenizer. It allows to correctly parse invalid code such as: >>> async def f(): ... def g(): pass ... async = 10 and valid code such as: >>> async def f(): ... async def g(): pass ... await z As a consequence, is is now possible to have one-line 'async def foo(): await ..' functions: >>> async def foo(): return await bar()
* | Issue 24226: Fix parsing of many sequential one-line 'def' statements.Yury Selivanov2015-05-181-8/+17
| |
* | PEP 0492 -- Coroutines with async and await syntax. Issue #24017.Yury Selivanov2015-05-111-0/+68
| |
* | merge 3.4 (#24022)Benjamin Peterson2015-04-211-4/+3
|\ \ | |/
| * do not call into python api if an exception is set (#24022)Benjamin Peterson2015-04-211-4/+3
| |
* | merge 3.4 (#21642)Benjamin Peterson2014-06-071-5/+14
|\ \ | |/
| * allow the keyword else immediately after (no space) an integer (closes #21642)Benjamin Peterson2014-06-071-5/+14
| |
* | PEP 465: a dedicated infix operator for matrix multiplication (closes #21176)Benjamin Peterson2014-04-091-1/+7
|/
* Merge with 3.3Martin v. Löwis2014-02-281-2/+12
|\
| * Issue #20731: Properly position in source code files even if theyMartin v. Löwis2014-02-281-2/+12
| | | | | | | | are opened in text mode. Patch by Serhiy Storchaka.
* | Do not reset the line number because we already set file position to correctSerhiy Storchaka2014-01-091-8/+0
|\ \ | |/ | | | | | | | | value. (fixes error in patch for issue #18960)
| * Do not reset the line number because we already set file position to correctSerhiy Storchaka2014-01-091-8/+0
| | | | | | | | | | | | value. (fixes error in patch for issue #18960)
* | Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-4/+22
|\ \ | |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
| * Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-4/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
* | Issue #1772673: The type of `char*` arguments now changed to `const char*`.Serhiy Storchaka2013-10-191-3/+4
| |
* | Issue #18571: Implementation of the PEP 446: file descriptors and file handlesVictor Stinner2013-08-281-0/+5
| | | | | | | | | | are now created non-inheritable; add functions os.get/set_inheritable(), os.get/set_handle_inheritable() and socket.socket.get/set_inheritable().
* | Issue #18722: Remove uses of the "register" keyword in C code.Antoine Pitrou2013-08-131-6/+6
| |
* | merge 3.3Benjamin Peterson2013-07-151-1/+2
|\ \ | |/
View Lorry Controller Status