delta/cpython-git.git/Parser/tokenizer.c, branch fix-namedexpr-comment

bpo-38673: dont switch to ps2 if the line starts with comment or whitespace (GH-17421)

2019-12-09T04:36:27+00:00

https://bugs.python.org/issue38673

Indent code inside if block. (GH-15284)

2019-08-15T16:19:07+00:00

Without indendation, seems like strcpy line is parallel to `if` condition.

Fix `SyntaxError` indicator printing too many spaces for multi-line strings (GH-14433)

2019-07-29T13:59:13+00:00

bpo-36878: Only allow text after `# type: ignore` if first character ASCII (GH-13504)

2019-05-22T20:43:36+00:00

This disallows things like `# type: ignoreé`, which seems wrong.

Also switch to using Py_ISALNUM for the alnum check, for consistency
with other code (and maybe correctness re: locale issues?).


https://bugs.python.org/issue36878

bpo-36878: Track extra text added to 'type: ignore' in the AST (GH-13479)

2019-05-22T14:54:20+00:00

GH-13238 made extra text after a # type: ignore accepted by the parser.
This finishes the job and actually plumbs the extra text through the
parser and makes it available in the AST.

bpo-2180: Treat line continuation at EOF as a `SyntaxError` (GH-13401)

2019-05-18T18:27:16+00:00

This makes the parser consistent with the tokenize module (already the case
in `pypy`).

sample
------

```python
x = 5\
```

before
------

```console
$ python3 t.py
$ python3 -mtokenize t.py
t.py:2:0: error: EOF in multi-line statement
```

after
-----

```console
$ ./python t.py
  File "t.py", line 3
    x = 5\

         ^
SyntaxError: unexpected EOF while parsing
$ ./python -m tokenize t.py
t.py:2:0: error: EOF in multi-line statement
```



https://bugs.python.org/issue2180

bpo-36878: Allow extra text after `# type: ignore` comments (GH-13238)

2019-05-11T18:17:24+00:00

In the parser, when using the type_comments=True option, recognize
a TYPE_IGNORE as anything containing `# type: ignore` followed by
a non-alphanumeric character. This is to allow ignores such as
`# type: ignore[E1000]`.

bpo-36623: Clean parser headers and include files (GH-12253)

2019-04-13T16:05:14+00:00

After the removal of pgen, multiple header and function prototypes that lack implementation or are unused are still lying around.

bpo-36459: Fix a possible double PyMem_FREE() due to tokenizer.c's tok_nextc() (12601)

2019-03-28T13:53:00+00:00

Remove the PyMem_FREE() call added in cb90c89.  The buffer will be
freed when PyTokenizer_Free() is called on the tokenizer state.

bpo-36367: Free buffer if realloc fails in tokenize.c (GH-12442)

2019-03-19T17:17:58+00:00