delta/python-packages/pygments-git.git/tests/test_mysql.py, branch fix/1579 github.com: pygments/pygments.git Overhaul the MySQL lexer (#1527) 2020-09-06T12:40:17+00:00 Kurt McKee contactme@kurtmckee.org 2020-09-06T12:40:17+00:00 b3f1691125bf611cc073fa0ad3303518ae04c094 * Overhaul the MySQL lexer Fixes #975, #1063, #1453 Changes include: Documentation ------------- * Note in the lexer docstring that Oracle MySQL is the target syntax. MariaDB syntax is not a target (though there is significant overlap). Unit tests ---------- * Add 140 unit tests for MySQL. Literals -------- * Hexadecimal/binary/date/time/timestamp literals are supported. * Integer mantissas are supported for scientific notation. * In-string escapes are now tokenized properly. * Support the "unknown" constant. Comments -------- * Optimizer hints are now supported, and keywords are recognized and tokenized as preprocessor instructions. * Remove nested multi-line comment support, which is no longer supported in MySQL. Variables --------- * Support the '@' prefix for variable names. * Lift restrictions on characters in unquoted variable names. (MySQL does not impose a restriction on lead characters.) * Support single/double/backtick-quoted variable names, including escapes. * Support the '@@' prefix for system variable names. * Support '?' as a variable so people can demonstrate prepared statements. Keywords -------- * Keyword / data type / function are now in a separate, auto-updating file. * Support 25 additional data types (including spatial and JSON types). * Support 460 additional MySQL keywords. * Support 372 MySQL functions. Explicit function support resolves a bug that causes non-function items to be treated as functions simply because they have a trailing opening parenthesis. * Support exceptions for the 'SET' keyword, which is both a datatype and a keyword depending on context. Schema object names ------------------- * Support Unicode in MySQL schema object names. * Support parsing of backtick-quoted schema object name escapes. (Escapes do not produce a distinct token type at this time.) Operators --------- * Remove non-operator characters from the list of operators. * Remove non-punctuation characters from the list of punctuation. * Cleanup items based on feedback * Remove an unnecessary optional newline lookahead for single-line comments 
* Overhaul the MySQL lexer

Fixes #975, #1063, #1453

Changes include:

Documentation
-------------

* Note in the lexer docstring that Oracle MySQL is the target syntax.
  MariaDB syntax is not a target (though there is significant overlap).

Unit tests
----------

* Add 140 unit tests for MySQL.

Literals
--------

* Hexadecimal/binary/date/time/timestamp literals are supported.
* Integer mantissas are supported for scientific notation.
* In-string escapes are now tokenized properly.
* Support the "unknown" constant.

Comments
--------

* Optimizer hints are now supported, and keywords are
  recognized and tokenized as preprocessor instructions.
* Remove nested multi-line comment support, which is no
  longer supported in MySQL.

Variables
---------

* Support the '@' prefix for variable names.
* Lift restrictions on characters in unquoted variable names.
  (MySQL does not impose a restriction on lead characters.)
* Support single/double/backtick-quoted variable names, including escapes.
* Support the '@@' prefix for system variable names.
* Support '?' as a variable so people can demonstrate prepared statements.

Keywords
--------

* Keyword / data type / function are now in a separate, auto-updating file.
* Support 25 additional data types (including spatial and JSON types).
* Support 460 additional MySQL keywords.
* Support 372 MySQL functions.
  Explicit function support resolves a bug that causes non-function
  items to be treated as functions simply because they have a trailing
  opening parenthesis.
* Support exceptions for the 'SET' keyword, which is both a datatype and
  a keyword depending on context.

Schema object names
-------------------

* Support Unicode in MySQL schema object names.
* Support parsing of backtick-quoted schema object name escapes.
  (Escapes do not produce a distinct token type at this time.)

Operators
---------

* Remove non-operator characters from the list of operators.
* Remove non-punctuation characters from the list of punctuation.

* Cleanup items based on feedback

* Remove an unnecessary optional newline lookahead for single-line comments