bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)

"Include/token.h", "Lib/token.py" (containing now some data moved from
"Lib/tokenize.py") and new files "Parser/token.c" (containing the code
moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
"Tools/scripts/generate_token.py". The script overwrites files only if
needed and can be used on the read-only sources tree.

"Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
instead of been executable itself.

Added new make targets "regen-token" and "regen-symbol" which are now
dependencies of "regen-all".

The documentation contains now strings for operators and punctuation tokens.

2 files changed, 207 insertions, 58 deletions

diff --git a/Doc/library/token-list.inc b/Doc/library/token-list.inc
new file mode 100644
index 0000000000..cd6e0f2696
--- /dev/null
+++ b/Doc/library/token-list.inc
@@ -0,0 +1,206 @@
+.. Auto-generated by Tools/scripts/generate_token.py
+.. data:: ENDMARKER
+
+.. data:: NAME
+
+.. data:: NUMBER
+
+.. data:: STRING
+
+.. data:: NEWLINE
+
+.. data:: INDENT
+
+.. data:: DEDENT
+
+.. data:: LPAR
+
+   Token value for ``"("``.
+
+.. data:: RPAR
+
+   Token value for ``")"``.
+
+.. data:: LSQB
+
+   Token value for ``"["``.
+
+.. data:: RSQB
+
+   Token value for ``"]"``.
+
+.. data:: COLON
+
+   Token value for ``":"``.
+
+.. data:: COMMA
+
+   Token value for ``","``.
+
+.. data:: SEMI
+
+   Token value for ``";"``.
+
+.. data:: PLUS
+
+   Token value for ``"+"``.
+
+.. data:: MINUS
+
+   Token value for ``"-"``.
+
+.. data:: STAR
+
+   Token value for ``"*"``.
+
+.. data:: SLASH
+
+   Token value for ``"/"``.
+
+.. data:: VBAR
+
+   Token value for ``"|"``.
+
+.. data:: AMPER
+
+   Token value for ``"&"``.
+
+.. data:: LESS
+
+   Token value for ``"<"``.
+
+.. data:: GREATER
+
+   Token value for ``">"``.
+
+.. data:: EQUAL
+
+   Token value for ``"="``.
+
+.. data:: DOT
+
+   Token value for ``"."``.
+
+.. data:: PERCENT
+
+   Token value for ``"%"``.
+
+.. data:: LBRACE
+
+   Token value for ``"{"``.
+
+.. data:: RBRACE
+
+   Token value for ``"}"``.
+
+.. data:: EQEQUAL
+
+   Token value for ``"=="``.
+
+.. data:: NOTEQUAL
+
+   Token value for ``"!="``.
+
+.. data:: LESSEQUAL
+
+   Token value for ``"<="``.
+
+.. data:: GREATEREQUAL
+
+   Token value for ``">="``.
+
+.. data:: TILDE
+
+   Token value for ``"~"``.
+
+.. data:: CIRCUMFLEX
+
+   Token value for ``"^"``.
+
+.. data:: LEFTSHIFT
+
+   Token value for ``"<<"``.
+
+.. data:: RIGHTSHIFT
+
+   Token value for ``">>"``.
+
+.. data:: DOUBLESTAR
+
+   Token value for ``"**"``.
+
+.. data:: PLUSEQUAL
+
+   Token value for ``"+="``.
+
+.. data:: MINEQUAL
+
+   Token value for ``"-="``.
+
+.. data:: STAREQUAL
+
+   Token value for ``"*="``.
+
+.. data:: SLASHEQUAL
+
+   Token value for ``"/="``.
+
+.. data:: PERCENTEQUAL
+
+   Token value for ``"%="``.
+
+.. data:: AMPEREQUAL
+
+   Token value for ``"&="``.
+
+.. data:: VBAREQUAL
+
+   Token value for ``"|="``.
+
+.. data:: CIRCUMFLEXEQUAL
+
+   Token value for ``"^="``.
+
+.. data:: LEFTSHIFTEQUAL
+
+   Token value for ``"<<="``.
+
+.. data:: RIGHTSHIFTEQUAL
+
+   Token value for ``">>="``.
+
+.. data:: DOUBLESTAREQUAL
+
+   Token value for ``"**="``.
+
+.. data:: DOUBLESLASH
+
+   Token value for ``"//"``.
+
+.. data:: DOUBLESLASHEQUAL
+
+   Token value for ``"//="``.
+
+.. data:: AT
+
+   Token value for ``"@"``.
+
+.. data:: ATEQUAL
+
+   Token value for ``"@="``.
+
+.. data:: RARROW
+
+   Token value for ``"->"``.
+
+.. data:: ELLIPSIS
+
+   Token value for ``"..."``.
+
+.. data:: OP
+
+.. data:: ERRORTOKEN
+
+.. data:: N_TOKENS
+
+.. data:: NT_OFFSET
diff --git a/Doc/library/token.rst b/Doc/library/token.rst
index 373991027e..5358eb5a29 100644
--- a/Doc/library/token.rst
+++ b/Doc/library/token.rst
@@ -44,64 +44,7 @@ functions.  The functions mirror definitions in the Python C header files.
 
 The token constants are:
 
-.. data:: ENDMARKER
-          NAME
-          NUMBER
-          STRING
-          NEWLINE
-          INDENT
-          DEDENT
-          LPAR
-          RPAR
-          LSQB
-          RSQB
-          COLON
-          COMMA
-          SEMI
-          PLUS
-          MINUS
-          STAR
-          SLASH
-          VBAR
-          AMPER
-          LESS
-          GREATER
-          EQUAL
-          DOT
-          PERCENT
-          LBRACE
-          RBRACE
-          EQEQUAL
-          NOTEQUAL
-          LESSEQUAL
-          GREATEREQUAL
-          TILDE
-          CIRCUMFLEX
-          LEFTSHIFT
-          RIGHTSHIFT
-          DOUBLESTAR
-          PLUSEQUAL
-          MINEQUAL
-          STAREQUAL
-          SLASHEQUAL
-          PERCENTEQUAL
-          AMPEREQUAL
-          VBAREQUAL
-          CIRCUMFLEXEQUAL
-          LEFTSHIFTEQUAL
-          RIGHTSHIFTEQUAL
-          DOUBLESTAREQUAL
-          DOUBLESLASH
-          DOUBLESLASHEQUAL
-          AT
-          ATEQUAL
-          RARROW
-          ELLIPSIS
-          OP
-          ERRORTOKEN
-          N_TOKENS
-          NT_OFFSET
-
+.. include:: token-list.inc
 
 The following token type values aren't used by the C tokenizer but are needed for
 the :mod:`tokenize` module.