C++: Core changes in preprocessing

Summary of most relevant items:

- Preprocessor output format change. No more gen true/false. Instead
  a more intuitive and natural expansion (like from a real compiler) is
  performed directly corresponding to the macro invocation. Notice that
  information about the generated tokens is not lost, because it's now
  embedded in the expansion section header (in terms of lines and columns
  as explained in the code). In addition the location on where the macro
  expansion happens is also documented for future use.

- Fix line control directives and associated token line numbers.
  This was not detected in tests cases because some of them were
  actually wrong: Within expansions the line information was being
  considered as originally computed in the macro definition, while
  the desired and expected for Creator's reporting mechanism (just
  like regular compilers) is the line from the expanded version
  of the tokens.

- Do not allow for eager expansion. This was previously being done
  inside define directives. However, it's not allowed and might
  lead to incorrect results, since the argument substitution should
  only happen upon the macro invocation (and following nested ones).
  At least GCC and clang are consistent with that. See test case
  tst_Preprocessor:dont_eagerly_expand for a detailed explanation.

- Revive the 'expanded' token flag. This is used to mark every token
  that originates from a macro expansion. Notice, however, that
  expanded tokens are not necessarily generated tokens (although
  every generated token is a expanded token). Expanded tokens that
  are not generated are those which are still considered by our
  code model features, since they are visible on the editor. The
  translation unit is smart enough to calculate line/column position
  for such tokens based on the information from the expansion section
  header.

- How expansions are tracked has also changed. Now, we simply add
  two surrounding marker tokens to each "top-level" expansion
  sequence. There is an enumeration that control expansion states.
  Also, no "previous" token is kept around.

- Preprocessor client methods suffered a change in signature so
  they now receive the line number of the action in question as
  a paramater. Previously such line could be retrieved by the client
  implementation by accessing the environment line. However, this
  is not reliable because we try to avoid synchronization of the
  output/environment lines in order to avoid unnecessary output,
  while expanding macros or handling preprocessor directives.

- Although macros are not expanded during define directives (as
  mentioned above) the preprocessor client is now "notified"
  when it sees a macro. This is to allow usage tracking.

- Other small stuff.

This is all in one patch because the fixes are a consequence
of the change in preprocessing control.

Change-Id: I8f4c6e6366f37756ec65d0a93b79f72a3ac4ed50
Reviewed-by: Roberto Raggi <roberto.raggi@nokia.com>

19 files changed, 429 insertions, 154 deletions

diff --git a/tests/auto/cplusplus/preprocessor/data/empty-macro.2.out.cpp b/tests/auto/cplusplus/preprocessor/data/empty-macro.2.out.cpp
index fced1fa8fd..03703e9bbd 100644
--- a/tests/auto/cplusplus/preprocessor/data/empty-macro.2.out.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/empty-macro.2.out.cpp
@@ -1,32 +1,15 @@
 # 1 "data/empty-macro.2.cpp"
-# 6 "data/empty-macro.2.cpp"
+
+
+
+
+
 class Test {
 private:
-                   Test
-#gen true
-# 3 "data/empty-macro.2.cpp"
-(const
-#gen false
-# 8 "data/empty-macro.2.cpp"
-                   Test
-#gen true
-# 3 "data/empty-macro.2.cpp"
-&);
-#gen false
-# 8 "data/empty-macro.2.cpp"
-                   Test
-#gen true
-# 4 "data/empty-macro.2.cpp"
-&operator=(const
-#gen false
-# 8 "data/empty-macro.2.cpp"
-                   Test
-#gen true
-# 4 "data/empty-macro.2.cpp"
-&);
-#gen false
+# expansion begin 182,14 8:19 ~2 8:19 ~3 8:19 ~5 8:19 ~3
+Test(const Test &); Test &operator=(const Test &);
+# expansion end
 # 10 "data/empty-macro.2.cpp"
 public:
     Test();
 };
-
diff --git a/tests/auto/cplusplus/preprocessor/data/empty-macro.cpp b/tests/auto/cplusplus/preprocessor/data/empty-macro.cpp
index d6ef56c60e..b86fccb7b9 100644
--- a/tests/auto/cplusplus/preprocessor/data/empty-macro.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/empty-macro.cpp
@@ -3,3 +3,11 @@
 class EMPTY_MACRO Foo {
 };
 
+class EMPTY_MACRO    Foo2 {
+};
+
+class       EMPTY_MACRO Foo3 {
+};
+
+class       EMPTY_MACRO    Foo3 {
+};
diff --git a/tests/auto/cplusplus/preprocessor/data/empty-macro.out.cpp b/tests/auto/cplusplus/preprocessor/data/empty-macro.out.cpp
index 4e3be9cb22..405cbbacf9 100644
--- a/tests/auto/cplusplus/preprocessor/data/empty-macro.out.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/empty-macro.out.cpp
@@ -4,3 +4,11 @@
 class             Foo {
 };
 
+class                Foo2 {
+};
+
+class                   Foo3 {
+};
+
+class                      Foo3 {
+};
diff --git a/tests/auto/cplusplus/preprocessor/data/identifier-expansion.1.out.cpp b/tests/auto/cplusplus/preprocessor/data/identifier-expansion.1.out.cpp
index b8e83bb077..a401d55dbf 100644
--- a/tests/auto/cplusplus/preprocessor/data/identifier-expansion.1.out.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/identifier-expansion.1.out.cpp
@@ -1,15 +1,18 @@
 # 1 "data/identifier-expansion.1.cpp"
-#gen true
-# 1 "data/identifier-expansion.1.cpp"
-test test
-#gen false
+
+
+# expansion begin 19,4 ~1
+test
+# expansion end
+# expansion begin 24,4 ~1
+test
+# expansion end
 # 3 "data/identifier-expansion.1.cpp"
          ;
 
 void
-#gen true
-# 1 "data/identifier-expansion.1.cpp"
+# expansion begin 36,4 ~1
 test
-#gen false
+# expansion end
 # 5 "data/identifier-expansion.1.cpp"
          ();
diff --git a/tests/auto/cplusplus/preprocessor/data/identifier-expansion.2.out.cpp b/tests/auto/cplusplus/preprocessor/data/identifier-expansion.2.out.cpp
index d6e37e4ac5..8d4c833a0d 100644
--- a/tests/auto/cplusplus/preprocessor/data/identifier-expansion.2.out.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/identifier-expansion.2.out.cpp
@@ -1,15 +1,19 @@
 # 1 "data/identifier-expansion.2.cpp"
-#gen true
-# 1 "data/identifier-expansion.2.cpp"
-test test
-#gen false
+
+
+
+# expansion begin 45,12 ~1
+test
+# expansion end
+# expansion begin 58,4 ~1
+test
+# expansion end
 # 4 "data/identifier-expansion.2.cpp"
                  ;
 
 void
-#gen true
-# 1 "data/identifier-expansion.2.cpp"
+# expansion begin 70,12 ~1
 test
-#gen false
+# expansion end
 # 6 "data/identifier-expansion.2.cpp"
                  ();
diff --git a/tests/auto/cplusplus/preprocessor/data/identifier-expansion.3.cpp b/tests/auto/cplusplus/preprocessor/data/identifier-expansion.3.cpp
index cf85dc953f..4f0e0bd81c 100644
--- a/tests/auto/cplusplus/preprocessor/data/identifier-expansion.3.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/identifier-expansion.3.cpp
@@ -2,13 +2,20 @@
     V(ADD) \
     V(SUB)
 
+#define OTHER_FOR_EACH(V) \
+    V(DIV) \
+    V(MUL)
+
 #define DECLARE_INSTR(op) #op,
 #define DECLARE_OP_INSTR(op) op_##op,
 
 enum op_code {
     FOR_EACH_INSTR(DECLARE_OP_INSTR)
+    OTHER_FOR_EACH(DECLARE_OP_INSTR)
 };
 
+
 static const char *names[] = {
 FOR_EACH_INSTR(DECLARE_INSTR)
+OTHER_FOR_EACH(DECLARE_INSTR)
 };
diff --git a/tests/auto/cplusplus/preprocessor/data/identifier-expansion.3.out.cpp b/tests/auto/cplusplus/preprocessor/data/identifier-expansion.3.out.cpp
index 689220f32e..bd8b2bdbf2 100644
--- a/tests/auto/cplusplus/preprocessor/data/identifier-expansion.3.out.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/identifier-expansion.3.out.cpp
@@ -1,24 +1,22 @@
 # 1 "data/identifier-expansion.3.cpp"
-# 8 "data/identifier-expansion.3.cpp"
+# 12 "data/identifier-expansion.3.cpp"
 enum op_code {
-#gen true
-# 6 "data/identifier-expansion.3.cpp"
+# expansion begin 195,14 ~4
 op_ADD, op_SUB,
-#gen false
-# 10 "data/identifier-expansion.3.cpp"
+# expansion end
+# expansion begin 232,14 ~4
+op_DIV, op_MUL,
+# expansion end
+# 15 "data/identifier-expansion.3.cpp"
 };
 
-static const char *names[] = {
-#gen true
-# 2 "data/identifier-expansion.3.cpp"
-"ADD"
-
 
-,
-# 3 "data/identifier-expansion.3.cpp"
- "SUB"
-
-,
-#gen false
-# 14 "data/identifier-expansion.3.cpp"
+static const char *names[] = {
+# expansion begin 301,14 ~4
+"ADD", "SUB",
+# expansion end
+# expansion begin 331,14 ~4
+"DIV", "MUL",
+# expansion end
+# 21 "data/identifier-expansion.3.cpp"
 };
diff --git a/tests/auto/cplusplus/preprocessor/data/identifier-expansion.4.out.cpp b/tests/auto/cplusplus/preprocessor/data/identifier-expansion.4.out.cpp
index e346f408f6..759eae4418 100644
--- a/tests/auto/cplusplus/preprocessor/data/identifier-expansion.4.out.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/identifier-expansion.4.out.cpp
@@ -5,5 +5,9 @@
 void baz()
 {
     int aaa;
-         aaa;
+# expansion begin 88,4 7:9
+aaa
+# expansion end
+# 7 "data/identifier-expansion.4.cpp"
+             ;
 }
diff --git a/tests/auto/cplusplus/preprocessor/data/identifier-expansion.5.out.cpp b/tests/auto/cplusplus/preprocessor/data/identifier-expansion.5.out.cpp
index 0bab742e80..3d328fe251 100644
--- a/tests/auto/cplusplus/preprocessor/data/identifier-expansion.5.out.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/identifier-expansion.5.out.cpp
@@ -1,2 +1 @@
 # 1 "data/identifier-expansion.5.cpp"
-# 9 "data/identifier-expansion.5.cpp"
diff --git a/tests/auto/cplusplus/preprocessor/data/macro-test.out.cpp b/tests/auto/cplusplus/preprocessor/data/macro-test.out.cpp
index cc8737a741..d13e2c5b0b 100644
--- a/tests/auto/cplusplus/preprocessor/data/macro-test.out.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/macro-test.out.cpp
@@ -1,8 +1,12 @@
 # 1 "data/macro-test.cpp"
-# 7 "data/macro-test.cpp"
+
+
+
+
+
+
 void thisFunctionIsEnabled();
 # 21 "data/macro-test.cpp"
 void thisFunctionIsEnabled2();
 # 31 "data/macro-test.cpp"
 void thisFunctionIsEnabled3();
-# 37 "data/macro-test.cpp"
diff --git a/tests/auto/cplusplus/preprocessor/data/macro_expand.out.c b/tests/auto/cplusplus/preprocessor/data/macro_expand.out.c
index 918ad379ff..24442a7567 100644
--- a/tests/auto/cplusplus/preprocessor/data/macro_expand.out.c
+++ b/tests/auto/cplusplus/preprocessor/data/macro_expand.out.c
@@ -3,8 +3,7 @@
 
 
 A:
-#gen true
-# 1 "data/macro_expand.c"
+# expansion begin 32,1 ~1
 Y
-#gen false
+# expansion end
 # 5 "data/macro_expand.c"
diff --git a/tests/auto/cplusplus/preprocessor/data/macro_expand_1.out.cpp b/tests/auto/cplusplus/preprocessor/data/macro_expand_1.out.cpp
index 6212601f71..e281d8d60f 100644
--- a/tests/auto/cplusplus/preprocessor/data/macro_expand_1.out.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/macro_expand_1.out.cpp
@@ -1,7 +1,7 @@
 # 1 "data/macro_expand_1.cpp"
-#gen true
-# 1 "data/macro_expand_1.cpp"
-class
-#gen false
+
+# expansion begin 33,13 ~1 2:14
+class QString
+# expansion end
 # 2 "data/macro_expand_1.cpp"
-              QString;
+                      ;
diff --git a/tests/auto/cplusplus/preprocessor/data/noPP.1.cpp b/tests/auto/cplusplus/preprocessor/data/noPP.1.cpp
index 9df1cd12bc..32b2797cee 100644
--- a/tests/auto/cplusplus/preprocessor/data/noPP.1.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/noPP.1.cpp
@@ -63,4 +63,3 @@ Write in C, write in C,
 Write in C, yeah, write in C.
 The government loves ADA,
 Write in C.
-
diff --git a/tests/auto/cplusplus/preprocessor/data/noPP.2.cpp b/tests/auto/cplusplus/preprocessor/data/noPP.2.cpp
new file mode 100644
index 0000000000..60f2cd9596
--- /dev/null
+++ b/tests/auto/cplusplus/preprocessor/data/noPP.2.cpp
@@ -0,0 +1,26 @@
+
+void hey(int a) {
+    int b = a + 10;
+    b;
+}
+
+class hello
+{
+public:
+    bool doit() { return true; }
+    bool dothat();
+    void run();
+};
+
+bool hello::dothat()
+{
+    bool should = true;
+    if (should) {
+        int i = 10;
+        while (i > 0) {
+            run();
+            --i;
+        }
+    }
+    return false;
+}
diff --git a/tests/auto/cplusplus/preprocessor/data/poundpound.1.out.cpp b/tests/auto/cplusplus/preprocessor/data/poundpound.1.out.cpp
index 40d92e888a..96b0706210 100644
--- a/tests/auto/cplusplus/preprocessor/data/poundpound.1.out.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/poundpound.1.out.cpp
@@ -1,8 +1,10 @@
 # 1 "data/poundpound.1.cpp"
 struct QQ {};
-#gen true
-# 3 "data/poundpound.1.cpp"
+
+
+
+# expansion begin 50,2 ~4
 typedef QQ PPCC;
-#gen false
+# expansion end
 # 7 "data/poundpound.1.cpp"
 typedef PPCC RR;
diff --git a/tests/auto/cplusplus/preprocessor/data/recursive.1.out.cpp b/tests/auto/cplusplus/preprocessor/data/recursive.1.out.cpp
index 2beaba7c5b..3f702f011e 100644
--- a/tests/auto/cplusplus/preprocessor/data/recursive.1.out.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/recursive.1.out.cpp
@@ -1,7 +1,11 @@
 # 1 "data/recursive.1.cpp"
-#gen true
-# 1 "data/recursive.1.cpp"
+
+
+
+# expansion begin 25,1 ~1
 b
- a
-#gen false
+# expansion end
+# expansion begin 27,1 ~1
+a
+# expansion end
 # 6 "data/recursive.1.cpp"
diff --git a/tests/auto/cplusplus/preprocessor/data/reserved.1.out.cpp b/tests/auto/cplusplus/preprocessor/data/reserved.1.out.cpp
index e4339882b4..8eb99b63fb 100644
--- a/tests/auto/cplusplus/preprocessor/data/reserved.1.out.cpp
+++ b/tests/auto/cplusplus/preprocessor/data/reserved.1.out.cpp
@@ -1,5 +1,8 @@
 # 1 "data/reserved.1.cpp"
-# 5 "data/reserved.1.cpp"
+
+
+
+
 int f() {
     foreach (QString &s, QStringList()) {
         doSomething();
diff --git a/tests/auto/cplusplus/preprocessor/preprocessor.pro b/tests/auto/cplusplus/preprocessor/preprocessor.pro
index 9caff07d06..05c463833a 100644
--- a/tests/auto/cplusplus/preprocessor/preprocessor.pro
+++ b/tests/auto/cplusplus/preprocessor/preprocessor.pro
@@ -3,11 +3,19 @@ include(../shared/shared.pri)
 SOURCES += tst_preprocessor.cpp
 
 OTHER_FILES = \
-    data/noPP.1.cpp data/noPP.1.errors.txt \
-    data/identifier-expansion.1.cpp data/identifier-expansion.1.out.cpp data/identifier-expansion.1.errors.txt \
-    data/identifier-expansion.2.cpp data/identifier-expansion.2.out.cpp data/identifier-expansion.2.errors.txt \
-    data/identifier-expansion.3.cpp data/identifier-expansion.3.out.cpp data/identifier-expansion.3.errors.txt \
-    data/identifier-expansion.4.cpp data/identifier-expansion.4.out.cpp data/identifier-expansion.4.errors.txt \
-    data/reserved.1.cpp data/reserved.1.out.cpp data/reserved.1.errors.txt \
-    data/macro_expand.c data/macro_expand.out.c data/macro_expand.errors.txt \
-    data/empty-macro.cpp data/empty-macro.out.cpp
+    data/noPP.1.cpp \
+    data/noPP.2.cpp \
+    data/identifier-expansion.1.cpp data/identifier-expansion.1.out.cpp \
+    data/identifier-expansion.2.cpp data/identifier-expansion.2.out.cpp \
+    data/identifier-expansion.3.cpp data/identifier-expansion.3.out.cpp \
+    data/identifier-expansion.4.cpp data/identifier-expansion.4.out.cpp \
+    data/identifier-expansion.5.cpp data/identifier-expansion.5.out.cpp \
+    data/reserved.1.cpp data/reserved.1.out.cpp \
+    data/recursive.1.cpp data/recursive.1.out.cpp \
+    data/macro_expand.c data/macro_expand.out.c \
+    data/macro_expand_1.cpp data/macro_expand_1.out.cpp \
+    data/macro-test.cpp data/macro-test.out.cpp \
+    data/poundpound.1.cpp data/poundpound.1.out.cpp \
+    data/empty-macro.cpp data/empty-macro.out.cpp \
+    data/empty-macro.2.cpp data/empty-macro.2.out.cpp \
+    data/macro_pounder_fn.c
diff --git a/tests/auto/cplusplus/preprocessor/tst_preprocessor.cpp b/tests/auto/cplusplus/preprocessor/tst_preprocessor.cpp
index bc33d43457..f528c52a81 100644
--- a/tests/auto/cplusplus/preprocessor/tst_preprocessor.cpp
+++ b/tests/auto/cplusplus/preprocessor/tst_preprocessor.cpp
@@ -114,18 +114,26 @@ public:
         m_definedMacrosLine.append(macro.line());
     }
 
-    virtual void passedMacroDefinitionCheck(unsigned /*offset*/, const Macro &/*macro*/) {}
+    virtual void passedMacroDefinitionCheck(unsigned /*offset*/,
+                                            unsigned /*line*/,
+                                            const Macro &/*macro*/) {}
     virtual void failedMacroDefinitionCheck(unsigned /*offset*/, const ByteArrayRef &/*name*/) {}
 
+    virtual void notifyMacroReference(unsigned offset, unsigned line, const Macro &macro)
+    {
+        m_macroUsesLine[macro.name()].append(line);
+        m_expandedMacrosOffset.append(offset);
+    }
+
     virtual void startExpandingMacro(unsigned offset,
+                                     unsigned line,
                                      const Macro &macro,
-                                     const ByteArrayRef &originalText,
                                      const QVector<MacroArgumentReference> &actuals
                                             = QVector<MacroArgumentReference>())
     {
-        m_expandedMacros.append(QByteArray(originalText.start(), originalText.length()));
+        m_expandedMacros.append(macro.name());
         m_expandedMacrosOffset.append(offset);
-        m_macroUsesLine[macro.name()].append(m_env->currentLine);
+        m_macroUsesLine[macro.name()].append(line);
         m_macroArgsCount.append(actuals.size());
     }
 
@@ -137,8 +145,7 @@ public:
     virtual void stopSkippingBlocks(unsigned offset)
     { m_skippedBlocks.last().end = offset; }
 
-    virtual void sourceNeeded(QString &includedFileName, IncludeType mode,
-                              unsigned line)
+    virtual void sourceNeeded(unsigned line, QString &includedFileName, IncludeType mode)
     {
 #if 1
         m_recordedIncludes.append(Include(includedFileName, mode, line));
@@ -300,15 +307,14 @@ protected:
     }
     static QString simplified(QByteArray buf);
 
-private /* not corrected yet */:
-    void macro_definition_lineno();
+private:
+    void compare_input_output();
 
 private slots:
-    void defined();
-    void defined_data();
-
     void va_args();
     void named_va_args();
+    void defined();
+    void defined_data();
     void empty_macro_args();
     void macro_args_count();
     void invalid_param_count();
@@ -318,14 +324,18 @@ private slots:
     void macro_arguments_notificatin();
     void unfinished_function_like_macro_call();
     void nasty_macro_expansion();
-    void tstst();
-    void test_file_builtin();
-
+    void glib_attribute();
+    void builtin__FILE__();
     void blockSkipping();
     void includes_1();
-
+    void dont_eagerly_expand();
+    void dont_eagerly_expand_data();
     void comparisons_data();
     void comparisons();
+    void comments_within();
+    void comments_within_data();
+    void multitokens_argument();
+    void multitokens_argument_data();
 };
 
 // Remove all #... lines, and 'simplify' string, to allow easily comparing the result
@@ -534,38 +544,72 @@ void tst_Preprocessor::macro_uses_lines()
              << buffer.lastIndexOf("ENABLE(LESS)"));
 }
 
-void tst_Preprocessor::macro_definition_lineno()
+void tst_Preprocessor::multitokens_argument_data()
 {
-    Client *client = 0; // no client.
-    Environment env;
-    Preprocessor preprocess(client, &env);
-    QByteArray preprocessed = preprocess.run(QLatin1String("<stdin>"),
-                                         QByteArray("#define foo(ARGS) int f(ARGS)\n"
-                                                    "foo(int a);\n"));
-    QVERIFY(preprocessed.contains("#gen true\n# 2 \"<stdin>\"\nint f"));
-
-    preprocessed = preprocess.run(QLatin1String("<stdin>"),
-                              QByteArray("#define foo(ARGS) int f(ARGS)\n"
-                                         "foo(int a)\n"
-                                         ";\n"));
-    QVERIFY(preprocessed.contains("#gen true\n# 2 \"<stdin>\"\nint f"));
-
-    preprocessed = preprocess.run(QLatin1String("<stdin>"),
-                              QByteArray("#define foo(ARGS) int f(ARGS)\n"
-                                         "foo(int  \n"
-                                         "    a);\n"));
-    QVERIFY(preprocessed.contains("#gen true\n# 2 \"<stdin>\"\nint f"));
-
-    preprocessed = preprocess.run(QLatin1String("<stdin>"),
-                              QByteArray("#define foo int f\n"
-                                         "foo;\n"));
-    QVERIFY(preprocessed.contains("#gen true\n# 2 \"<stdin>\"\nint f"));
-
-    preprocessed = preprocess.run(QLatin1String("<stdin>"),
-                              QByteArray("#define foo int f\n"
-                                         "foo\n"
-                                         ";\n"));
-    QVERIFY(preprocessed.contains("#gen true\n# 2 \"<stdin>\"\nint f"));
+    QTest::addColumn<QByteArray>("input");
+    QTest::addColumn<QByteArray>("output");
+
+    QByteArray original;
+    QByteArray expected;
+
+    original =
+            "#define foo(ARGS) int f(ARGS)\n"
+            "foo(int a);\n";
+    expected =
+            "# 1 \"<stdin>\"\n"
+            "\n"
+            "# expansion begin 30,3 ~3 2:4 2:8 ~1\n"
+            "int f(int a)\n"
+            "# expansion end\n"
+            "# 2 \"<stdin>\"\n"
+            "          ;\n";
+    QTest::newRow("case 1") << original << expected;
+
+    original =
+            "#define foo(ARGS) int f(ARGS)\n"
+            "foo(int   \n"
+            "    a);\n";
+    expected =
+            "# 1 \"<stdin>\"\n"
+            "\n"
+            "# expansion begin 30,3 ~3 2:4 3:4 ~1\n"
+            "int f(int a)\n"
+            "# expansion end\n"
+            "# 3 \"<stdin>\"\n"
+            "      ;\n";
+    QTest::newRow("case 2") << original << expected;
+
+    original =
+            "#define foo(ARGS) int f(ARGS)\n"
+            "foo(int a = 0);\n";
+    expected =
+            "# 1 \"<stdin>\"\n"
+            "\n"
+            "# expansion begin 30,3 ~3 2:4 2:8 2:10 2:12 ~1\n"
+            "int f(int a = 0)\n"
+            "# expansion end\n"
+            "# 2 \"<stdin>\"\n"
+            "              ;\n";
+    QTest::newRow("case 3") << original << expected;
+
+    original =
+            "#define foo(X) int f(X = 0)\n"
+            "foo(int \n"
+            "    a);\n";
+    expected =
+            "# 1 \"<stdin>\"\n"
+            "\n"
+            "# expansion begin 28,3 ~3 2:4 3:4 ~3\n"
+            "int f(int a = 0)\n"
+            "# expansion end\n"
+            "# 3 \"<stdin>\"\n"
+            "      ;\n";
+    QTest::newRow("case 4") << original << expected;
+}
+
+void tst_Preprocessor::multitokens_argument()
+{
+    compare_input_output();
 }
 
 void tst_Preprocessor::objmacro_expanding_as_fnmacro_notification()
@@ -608,9 +652,17 @@ void tst_Preprocessor::unfinished_function_like_macro_call()
 
     Preprocessor preprocess(client, &env);
     QByteArray preprocessed = preprocess.run(QLatin1String("<stdin>"),
-                                         QByteArray("\n#define foo(a,b) a + b"
-                                         "\nfoo(1, 2\n"));
-    QByteArray expected__("# 1 \"<stdin>\"\n\n\n    1\n#gen true\n# 2 \"<stdin>\"\n+\n#gen false\n# 3 \"<stdin>\"\n       2\n");
+                                             QByteArray("\n"
+                                                        "#define foo(a,b) a + b\n"
+                                                        "foo(1, 2\n"));
+    QByteArray expected__("# 1 \"<stdin>\"\n"
+                          "\n"
+                          "\n"
+                          "# expansion begin 24,3 3:4 ~1 3:7\n"
+                          "1 + 2\n"
+                          "# expansion end\n"
+                          "# 4 \"<stdin>\"\n");
+
 //    DUMP_OUTPUT(preprocessed);
     QCOMPARE(preprocessed, expected__);
 }
@@ -668,12 +720,10 @@ void tst_Preprocessor::nasty_macro_expansion()
     QVERIFY(!preprocessed.contains("FIELD32"));
 }
 
-void tst_Preprocessor::tstst()
+void tst_Preprocessor::glib_attribute()
 {
-    Client *client = 0; // no client.
     Environment env;
-
-    Preprocessor preprocess(client, &env);
+    Preprocessor preprocess(0, &env);
     QByteArray preprocessed = preprocess.run(
                 QLatin1String("<stdin>"),
                 QByteArray("\n"
@@ -681,15 +731,14 @@ void tst_Preprocessor::tstst()
                            "namespace std _GLIBCXX_VISIBILITY(default) {\n"
                            "}\n"
                            ));
-    const QByteArray result____ ="# 1 \"<stdin>\"\n\n\n"
+    const QByteArray result____ =
+            "# 1 \"<stdin>\"\n"
+            "\n"
+            "\n"
             "namespace std\n"
-            "#gen true\n"
-            "# 2 \"<stdin>\"\n"
-            "__attribute__ ((__visibility__ (\n"
-            "\"default\"\n"
-            "# 2 \"<stdin>\"\n"
-            ")))\n"
-            "#gen false\n"
+            "# expansion begin 85,19 ~9\n"
+            "__attribute__ ((__visibility__ (\"default\")))\n"
+            "# expansion end\n"
             "# 3 \"<stdin>\"\n"
             "                                           {\n"
             "}\n";
@@ -698,7 +747,7 @@ void tst_Preprocessor::tstst()
     QCOMPARE(preprocessed, result____);
 }
 
-void tst_Preprocessor::test_file_builtin()
+void tst_Preprocessor::builtin__FILE__()
 {
     Client *client = 0; // no client.
     Environment env;
@@ -710,13 +759,8 @@ void tst_Preprocessor::test_file_builtin()
                            ));
     const QByteArray result____ =
             "# 1 \"some-file.c\"\n"
-            "const char *f =\n"
-            "#gen true\n"
-            "# 1 \"some-file.c\"\n"
-            "\"some-file.c\"\n"
-            "#gen false\n"
-            "# 2 \"some-file.c\"\n"
-            ;
+            "const char *f = \"some-file.c\"\n";
+
     QCOMPARE(preprocessed, result____);
 }
 
@@ -727,6 +771,7 @@ void tst_Preprocessor::comparisons_data()
     QTest::addColumn<QString>("errorfile");
 
     QTest::newRow("do nothing") << "noPP.1.cpp" << "noPP.1.cpp" << "";
+    QTest::newRow("no PP 2") << "noPP.2.cpp" << "noPP.2.cpp" << "";
     QTest::newRow("identifier-expansion 1")
         << "identifier-expansion.1.cpp" << "identifier-expansion.1.out.cpp" << "";
     QTest::newRow("identifier-expansion 2")
@@ -766,6 +811,7 @@ void tst_Preprocessor::comparisons()
     QByteArray errors;
     QByteArray preprocessed = preprocess(infile, &errors, infile == outfile);
 
+
     // DUMP_OUTPUT(preprocessed);
 
     if (!outfile.isEmpty()) {
@@ -976,6 +1022,176 @@ void tst_Preprocessor::defined_data()
         "#endif\n";
 }
 
+void tst_Preprocessor::dont_eagerly_expand_data()
+{
+    QTest::addColumn<QByteArray>("input");
+    QTest::addColumn<QByteArray>("output");
+
+    QByteArray original;
+    QByteArray expected;
+
+    // Expansion must be processed upon invocation of the macro. Therefore a particular
+    // identifier within a define must not be expanded (in the case it matches an
+    // already known macro) during the processor directive handling, but only when
+    // it's actually "used". Naturally, if it's still not replaced after an invocation
+    // it should then be expanded. This is consistent with clang and gcc for example.
+
+    original = "#define T int\n"
+               "#define FOO(T) T\n"
+               "FOO(double)\n";
+    expected =
+            "# 1 \"<stdin>\"\n"
+            "\n"
+            "\n"
+            "# expansion begin 31,3 3:4\n"
+            "double\n"
+            "# expansion end\n"
+            "# 4 \"<stdin>\"\n";
+    QTest::newRow("case 1") << original << expected;
+
+    original = "#define T int\n"
+               "#define FOO(X) T\n"
+               "FOO(double)\n";
+    expected =
+            "# 1 \"<stdin>\"\n"
+            "\n"
+            "\n"
+            "# expansion begin 31,3 ~1\n"
+            "int\n"
+            "# expansion end\n"
+            "# 4 \"<stdin>\"\n";
+    QTest::newRow("case 2") << original << expected;
+}
+
+void tst_Preprocessor::dont_eagerly_expand()
+{
+    compare_input_output();
+}
+
+void tst_Preprocessor::comments_within()
+{
+    compare_input_output();
+}
+
+void tst_Preprocessor::comments_within_data()
+{
+    QTest::addColumn<QByteArray>("input");
+    QTest::addColumn<QByteArray>("output");
+
+    QByteArray original;
+    QByteArray expected;
+
+    original = "#define FOO int x;\n"
+               "\n"
+               "   // comment\n"
+               "   // comment\n"
+               "   // comment\n"
+               "   // comment\n"
+               "FOO\n"
+               "x = 10\n";
+    expected =
+            "# 1 \"<stdin>\"\n"
+            "\n"
+            "\n"
+            "\n"
+            "\n"
+            "\n"
+            "\n"
+            "# expansion begin 76,3 ~3\n"
+            "int x;\n"
+            "# expansion end\n"
+            "# 8 \"<stdin>\"\n"
+            "x = 10\n";
+    QTest::newRow("case 1") << original << expected;
+
+
+    original = "#define FOO int x;\n"
+               "\n"
+               "   /* comment\n"
+               "      comment\n"
+               "      comment\n"
+               "      comment */\n"
+               "FOO\n"
+               "x = 10\n";
+    expected =
+            "# 1 \"<stdin>\"\n"
+            "\n"
+            "\n"
+            "\n"
+            "\n"
+            "\n"
+            "\n"
+            "# expansion begin 79,3 ~3\n"
+            "int x;\n"
+            "# expansion end\n"
+            "# 8 \"<stdin>\"\n"
+            "x = 10\n";
+    QTest::newRow("case 2") << original << expected;
+
+
+    original = "#define FOO int x;\n"
+               "\n"
+               "   // comment\n"
+               "   // comment\n"
+               "   // comment\n"
+               "   // comment\n"
+               "FOO\n"
+               "// test\n"
+               "// test again\n"
+               "x = 10\n";
+    expected =
+            "# 1 \"<stdin>\"\n"
+            "\n"
+            "\n"
+            "\n"
+            "\n"
+            "\n"
+            "\n"
+            "# expansion begin 76,3 ~3\n"
+            "int x;\n"
+            "# expansion end\n"
+            "# 10 \"<stdin>\"\n"
+            "x = 10\n";
+    QTest::newRow("case 3") << original << expected;
+
+
+    original = "#define FOO int x;\n"
+               "\n"
+               "   /* comment\n"
+               "      comment\n"
+               "      comment\n"
+               "      comment */\n"
+               "FOO\n"
+               "/*  \n"
+               "*/\n"
+               "x = 10\n";
+    expected =
+            "# 1 \"<stdin>\"\n"
+            "\n"
+            "\n"
+            "\n"
+            "\n"
+            "\n"
+            "\n"
+            "# expansion begin 79,3 ~3\n"
+            "int x;\n"
+            "# expansion end\n"
+            "# 10 \"<stdin>\"\n"
+            "x = 10\n";
+    QTest::newRow("case 4") << original << expected;
+}
+
+void tst_Preprocessor::compare_input_output()
+{
+    QFETCH(QByteArray, input);
+    QFETCH(QByteArray, output);
+
+    Environment env;
+    Preprocessor preprocess(0, &env);
+    QByteArray prep = preprocess.run(QLatin1String("<stdin>"), input);
+    QCOMPARE(output, prep);
+}
+
 QTEST_APPLESS_MAIN(tst_Preprocessor)
 
 #include "tst_preprocessor.moc"