TextEditor: Highlight punctuators as Text

This change limits the set of tokens that fall under
Token::isOperator(). That allows cpphighlighter.cpp to
distinguish operator tokens from punctuator tokens
(without changing any logic in cpphighlighter.cpp).

This change moves punctuators from "Operator"
to the "Text" style category where they belong.
Punctuators are not operators. Punctuators are
dumb text tokens.

Why don't we let the clang backend alone separate
these tokens for us?

1. Clang is slow on big files. Sometimes the
   highlighting dictated by clang is painted _seconds_
   after cpphighlighter.cpp runs. CppHighlighter is way
   faster so we use it to "prepaint" code while clang is
   busy in the background.

2. Secondly, clang cannot yet handle all operator types.
   In particular, none if its "operator cursors"
     CXCursor_UnaryOperator:
     CXCursor_BinaryOperator:
     CXCursor_CompoundAssignOperator:
     CXCursor_ConditionalOperator:
   includes the -> and . operators.
   We still need CppHighlighter to paint those tokens.

However, once clang has finished processing the file some
operator tokens will be repainted. We need clang to get
all operators' semantics. In particular, we need clang to
tell us if < is a "smaller than"-operator or part of a
template parameter like set<int>.

Task-number: QTCREATORBUG-19659
Change-Id: I952cb58f7c79134b3281e2a8221425cc1d0ad263
Reviewed-by: Ivan Donchevskii <ivan.donchevskii@qt.io>

2 files changed, 2 insertions, 2 deletions

diff --git a/src/libs/cplusplus/ExpressionUnderCursor.cpp b/src/libs/cplusplus/ExpressionUnderCursor.cpp
index 1d768b2a09..0b4405412b 100644
--- a/src/libs/cplusplus/ExpressionUnderCursor.cpp
+++ b/src/libs/cplusplus/ExpressionUnderCursor.cpp
@@ -65,7 +65,7 @@ int ExpressionUnderCursor::startOfExpression(BackwardsScanner &tk, int index)
             break;
 
         default:
-            if (tok.isOperator())
+            if (tok.isPunctuationOrOperator())
                 return startOfExpression(tk, index - 1);
 
             break;
diff --git a/src/libs/cplusplus/MatchingText.cpp b/src/libs/cplusplus/MatchingText.cpp
index ade81c9b9f..c22d3fe89c 100644
--- a/src/libs/cplusplus/MatchingText.cpp
+++ b/src/libs/cplusplus/MatchingText.cpp
@@ -98,7 +98,7 @@ static bool insertQuote(const QChar ch, const BackwardsScanner &tk)
         return true;
 
     // Insert a matching quote after an operator.
-    if (token.isOperator())
+    if (token.isPunctuationOrOperator())
         return true;
 
     if (token.isKeyword())