| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Lex identifiers after `case` as constants
Add a state for marking identifiers preceded by a `case` keyword as
constants.
Additionally, refactor the `label` rule to no longer permit a `case`
keyword before a label.
Consequentially, identifiers after a `case` keyword (like `foo` in
`case foo:`) are no longer wrongly lexed as `Name.Label`, but as
`Name.Constant`.
In addition, this fixes #2076, as multiple `case` keywords in one
line are lexed the same.
* Add test for multiple `case` keywords in one line
* Fix existing tests
* Lex `::` as Operator and not Name.Constant
After a `case`, when lexing a namespaced name, like `foo::bar`, lex the
namespace operator `::` as Operator, and not Name.Constant.
* Regenerate tokens
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Remove unused variable
This variable is unused since the first commit to this file
in the commit history, so it's probably safe to remove it.
* Lex identifier as label only if it's at line start
* Stop matching identifiers that begin with a digit
This is so we don't match numbers as labels with the new labels rule.
* Add label tests
* Fix existing tests
|
| |
|
|
|
| |
* Fix #1237 cpp whitespace token usage expanded
* Adapt tests change to 3eff56f5
|
| |
|
|
| |
The CFamilyLexer was matching whitespace as Text instead of Whitespace.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
CFamilyLexer failed to tokenize preprocessor macros when they were
preceded by line break surrounded by spaces. This was the case because
prerpocessor regex rule expected to start at the beginning of the line,
but the space regex rule matched also the whitespace after the line
break. Now the space rule has been refined not to match the line break.
Because of this, the preprocessor regex rule correctly matches
prerpocessor tokens even when they are preceded by white spaces, at the
cost of adding some more tokens in the token stream in some cases. This
change preserves the behavior of invalid preprocessor usage failing to
tokenize.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Fix lexing of function names
This fixes #1561.
Add a keywords state that matches inside and outside functions for
keywords.
Before this, when a keyword would appear the lexer would go to the
statements state, in which functions were not matched.
* Add tests for lexing of function names
* Unbreak previous tests
* Allow namespaced names in function statements
Add a second identifiers regex that matces all the previous identifiers
and also '::'.
I took the decision to create a second identifiers regex with '::'
inside, simply because using the old identifiers regex would hurt
performance massively on every solution I tried to craft.
* Add tests for namespaced names in functions
* Unbreak previous tests
* Add support for namespaces in namespace declarations
Add a namespace state that is entered each time the namespace keyword
is matched and lexes all name matches as namespaces.
Cases this approach doesn't cover:
* Namespaces in using declarations.
* Namespaces that prefix names in random code.
Unfortunately, in both of these cases the names before and after '::'
are not always namespaces.
* Add tests for namespace declartions
* Unbreak previous tests
* Tidy functions regex
Remove group nesting that became unneeded after fc56ab8 (the last big
refactor).
* Remove f string usage I introduced by mistake
|
|
|
Co-authored-by: Georg Brandl <georg@python.org>
|
