bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)

"Include/token.h", "Lib/token.py" (containing now some data moved from
"Lib/tokenize.py") and new files "Parser/token.c" (containing the code
moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
"Tools/scripts/generate_token.py". The script overwrites files only if
needed and can be used on the read-only sources tree.

"Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
instead of been executable itself.

Added new make targets "regen-token" and "regen-symbol" which are now
dependencies of "regen-all".

The documentation contains now strings for operators and punctuation tokens.

1 files changed, 206 insertions, 0 deletions

diff --git a/Doc/library/token-list.inc b/Doc/library/token-list.inc
new file mode 100644
index 0000000000..cd6e0f2696
--- /dev/null
+++ b/Doc/library/token-list.inc
@@ -0,0 +1,206 @@
+.. Auto-generated by Tools/scripts/generate_token.py
+.. data:: ENDMARKER
+
+.. data:: NAME
+
+.. data:: NUMBER
+
+.. data:: STRING
+
+.. data:: NEWLINE
+
+.. data:: INDENT
+
+.. data:: DEDENT
+
+.. data:: LPAR
+
+   Token value for ``"("``.
+
+.. data:: RPAR
+
+   Token value for ``")"``.
+
+.. data:: LSQB
+
+   Token value for ``"["``.
+
+.. data:: RSQB
+
+   Token value for ``"]"``.
+
+.. data:: COLON
+
+   Token value for ``":"``.
+
+.. data:: COMMA
+
+   Token value for ``","``.
+
+.. data:: SEMI
+
+   Token value for ``";"``.
+
+.. data:: PLUS
+
+   Token value for ``"+"``.
+
+.. data:: MINUS
+
+   Token value for ``"-"``.
+
+.. data:: STAR
+
+   Token value for ``"*"``.
+
+.. data:: SLASH
+
+   Token value for ``"/"``.
+
+.. data:: VBAR
+
+   Token value for ``"|"``.
+
+.. data:: AMPER
+
+   Token value for ``"&"``.
+
+.. data:: LESS
+
+   Token value for ``"<"``.
+
+.. data:: GREATER
+
+   Token value for ``">"``.
+
+.. data:: EQUAL
+
+   Token value for ``"="``.
+
+.. data:: DOT
+
+   Token value for ``"."``.
+
+.. data:: PERCENT
+
+   Token value for ``"%"``.
+
+.. data:: LBRACE
+
+   Token value for ``"{"``.
+
+.. data:: RBRACE
+
+   Token value for ``"}"``.
+
+.. data:: EQEQUAL
+
+   Token value for ``"=="``.
+
+.. data:: NOTEQUAL
+
+   Token value for ``"!="``.
+
+.. data:: LESSEQUAL
+
+   Token value for ``"<="``.
+
+.. data:: GREATEREQUAL
+
+   Token value for ``">="``.
+
+.. data:: TILDE
+
+   Token value for ``"~"``.
+
+.. data:: CIRCUMFLEX
+
+   Token value for ``"^"``.
+
+.. data:: LEFTSHIFT
+
+   Token value for ``"<<"``.
+
+.. data:: RIGHTSHIFT
+
+   Token value for ``">>"``.
+
+.. data:: DOUBLESTAR
+
+   Token value for ``"**"``.
+
+.. data:: PLUSEQUAL
+
+   Token value for ``"+="``.
+
+.. data:: MINEQUAL
+
+   Token value for ``"-="``.
+
+.. data:: STAREQUAL
+
+   Token value for ``"*="``.
+
+.. data:: SLASHEQUAL
+
+   Token value for ``"/="``.
+
+.. data:: PERCENTEQUAL
+
+   Token value for ``"%="``.
+
+.. data:: AMPEREQUAL
+
+   Token value for ``"&="``.
+
+.. data:: VBAREQUAL
+
+   Token value for ``"|="``.
+
+.. data:: CIRCUMFLEXEQUAL
+
+   Token value for ``"^="``.
+
+.. data:: LEFTSHIFTEQUAL
+
+   Token value for ``"<<="``.
+
+.. data:: RIGHTSHIFTEQUAL
+
+   Token value for ``">>="``.
+
+.. data:: DOUBLESTAREQUAL
+
+   Token value for ``"**="``.
+
+.. data:: DOUBLESLASH
+
+   Token value for ``"//"``.
+
+.. data:: DOUBLESLASHEQUAL
+
+   Token value for ``"//="``.
+
+.. data:: AT
+
+   Token value for ``"@"``.
+
+.. data:: ATEQUAL
+
+   Token value for ``"@="``.
+
+.. data:: RARROW
+
+   Token value for ``"->"``.
+
+.. data:: ELLIPSIS
+
+   Token value for ``"..."``.
+
+.. data:: OP
+
+.. data:: ERRORTOKEN
+
+.. data:: N_TOKENS
+
+.. data:: NT_OFFSET