delta/cpython-git.git/Parser/tokenizer.c, branch fix-misc-acks

bpo-38673: dont switch to ps2 if the line starts with comment or whitespace (GH-17421) (GH-17522)

2019-12-09T11:38:36+00:00

https://bugs.python.org/issue38673
(cherry picked from commit 109fc2792a490ee5cd8a423e17d415fbdedec5c8)

Co-authored-by: Batuhan Taşkaya <47358913+isidentical@users.noreply.github.com>

Indent code inside if block. (GH-15284)

2019-08-15T16:46:47+00:00

Without indendation, seems like strcpy line is parallel to `if` condition.
(cherry picked from commit 69f37bcb28d7cd78255828029f895958b5baf6ff)

Co-authored-by: Hansraj Das

bpo-36459: Fix a possible double PyMem_FREE() due to tokenizer.c's tok_nextc() (12601)

2019-03-28T15:08:35+00:00

Remove the PyMem_FREE() call added in cb90c89.  The buffer will be
freed when PyTokenizer_Free() is called on the tokenizer state.
(cherry picked from commit cda139d1ded6708665b53e4ed32ccc1d2627e1da)

Co-authored-by: Zackery Spytz

bpo-36367: Free buffer if realloc fails in tokenize.c (GH-12442) (GH-12471)

2019-03-20T12:03:11+00:00

bpo-35436: Add missing PyErr_NoMemory() calls and other minor bug fixes. (GH-11015) (GH-11020)

2018-12-07T12:17:43+00:00

(cherry picked from commit 4c49da0cb7434c676d70b9ccf38aca82ac0d64a9)

tokenizer: Remove unused tabs options (#4422)

2017-11-17T09:25:47+00:00

Remove the following fields from tok_state structure which are now
used unused:

* altwarning: "Issue warning if alternate tabs don't match"
* alterror: "Issue error if alternate tabs don't match"
* alttabsize: "Alternate tab spacing"

Replace alttabsize variable with ALTTABSIZE define.

bpo-30406: Make async and await proper keywords (#1669)

2017-10-06T03:24:46+00:00

Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.

bpo-25324: add missing comma in Parser/tokenizer.c (GH-1910)

2017-06-01T20:51:27+00:00

bpo-25324: copy tok_name before changing it (#1608)

2017-05-31T14:00:21+00:00

* add test to check if were modifying token

* copy list so import tokenize doesnt have side effects on token

* shorten line

* add tokenize tokens to token.h to get them to show up in token

* move ERRORTOKEN back to its previous location, and fix nitpick

* copy comments from token.h automatically

* fix whitespace and make more pythonic

* change to fix comments from @haypo

* update token.rst and Misc/NEWS

* change wording

* some more wording changes

Issue #28489: Merge from 3.6

2017-02-05T01:33:11+00:00