From fc354f07855a9197e71f851ad930cbf5652f9160 Mon Sep 17 00:00:00 2001 From: Albert-Jan Nijburg Date: Wed, 31 May 2017 15:00:21 +0100 Subject: bpo-25324: copy tok_name before changing it (#1608) * add test to check if were modifying token * copy list so import tokenize doesnt have side effects on token * shorten line * add tokenize tokens to token.h to get them to show up in token * move ERRORTOKEN back to its previous location, and fix nitpick * copy comments from token.h automatically * fix whitespace and make more pythonic * change to fix comments from @haypo * update token.rst and Misc/NEWS * change wording * some more wording changes --- Parser/tokenizer.c | 3 +++ 1 file changed, 3 insertions(+) (limited to 'Parser/tokenizer.c') diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 5cc9533c3c..7f2f3e6028 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -106,6 +106,9 @@ const char *_PyParser_TokenNames[] = { "AWAIT", "ASYNC", "", + "COMMENT", + "NL", + "ENCODING" "" }; -- cgit v1.2.1