delta/cpython-git.git/Lib/tokenize.py, branch alex-patch-1

bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686)

2019-05-30T22:06:32+00:00

https://bugs.python.org/issue5028

bpo-5028: fix doc bug for tokenize (GH-11683)

2019-05-30T19:31:51+00:00

https://bugs.python.org/issue5028

bpo-36766: Typos in docs and code comments (GH-13116)

2019-05-06T18:57:17+00:00

bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)

2018-12-22T09:18:40+00:00

"Include/token.h", "Lib/token.py" (containing now some data moved from
"Lib/tokenize.py") and new files "Parser/token.c" (containing the code
moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
"Tools/scripts/generate_token.py". The script overwrites files only if
needed and can be used on the read-only sources tree.

"Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
instead of been executable itself.

Added new make targets "regen-token" and "regen-symbol" which are now
dependencies of "regen-all".

The documentation contains now strings for operators and punctuation tokens.

bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891)

2018-07-06T07:19:08+00:00

Most of the change involves fixing up the test suite, which previously made
the assumption that there wouldn't be a new line if the input didn't end in
one.

Contributed by Ammar Askar.

bpo-12486: Document tokenize.generate_tokens() as public API (#6957)

2018-06-05T17:26:39+00:00

* Document tokenize.generate_tokens()

* Add news file

* Add test for generate_tokens

* Document behaviour around ENCODING token

* Add generate_tokens to __all__

bpo-33338: [tokenize] Minor code cleanup (#6573)

2018-04-23T08:07:11+00:00

This change contains minor things that make diffing between Lib/tokenize.py and
Lib/lib2to3/pgen2/tokenize.py cleaner.

bpo-33260: Regenerate token.py after removing ASYNC and AWAIT. (GH-6447)

2018-04-11T16:15:51+00:00

bpo-30406: Make async and await proper keywords (#1669)

2017-10-06T03:24:46+00:00

Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.

bpo-25324: copy tok_name before changing it (#1608)

2017-05-31T14:00:21+00:00

* add test to check if were modifying token

* copy list so import tokenize doesnt have side effects on token

* shorten line

* add tokenize tokens to token.h to get them to show up in token

* move ERRORTOKEN back to its previous location, and fix nitpick

* copy comments from token.h automatically

* fix whitespace and make more pythonic

* change to fix comments from @haypo

* update token.rst and Misc/NEWS

* change wording

* some more wording changes