BR3066383: Restore backward compatibility with token pasting

It seems to be a bit long story for the reason if this bug. But
lets be verbose and describe all byte-to-byte. And it is all about
preprocessor code, in particular paste_tokens and expand_mmac_params.

Initially the problem (not the same but similar) was noticed and
fixed in commit ec88c1be. The problem reveals itself with code snippets
like

 | %macro m 1
 |  %push
 |      %define %$arg %1
 | %%top_%$arg:
 |      resb ($ - %%top_%$arg)
 |  %pop
 | %endmacro

So with commits ec88c1be, 51fd86e0, 1f6741fc, 985d880c we did expand
local single macro before processing tokens pasting unconditionally.

But then it being found that such approach breaks %assign directive.
The snippets like below didn't work

 | %macro m 1
 |  %push
 |      %assign %$arg %1
 |      %assign %$arg %1+%$arg
 |  %pop
 | %endmacro

So all these commits were reverted and we just stop pasting tokens
in paste_tokens() after TOK_PREPROC_ID (commit 20a94ad7). Unfortunately
this breaks %assign with compound preproc id

 | %macro m3 1
 |    %push
 |        %assign %$_uses 0
 |        %rep 4
 |            %assign %$_ur%$_uses %$_uses
 |            mov ecx, %$_ur%$_uses
 |            %assign %$_uses %$_uses+1
 |        %endrep
 |    %pop
 | %endmacro

To fix this bug we have to combine two approaches at once,
we should continue pasting after TOK_PREPROC_ID and expand
sequential TOK_PREPROC_IDs except first one.

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

1 files changed, 30 insertions, 0 deletions

diff --git a/preproc.c b/preproc.c
index 9a8085d8..1a1905a4 100644
--- a/preproc.c
+++ b/preproc.c
@@ -3660,6 +3660,7 @@ static bool paste_tokens(Token **head, bool handle_paste_tokens)
             }
             break;
         case TOK_ID:
+        case TOK_PREPROC_ID:
         case TOK_NUMBER:
         case TOK_FLOAT:
         {
@@ -3972,6 +3973,35 @@ static Token *expand_mmac_params(Token * tline)
             }
             delete_Token(t);
             changed = true;
+        } else if (tline->type == TOK_PREPROC_ID                &&
+                   tline->text[0] == '%'                        &&
+                   tline->text[1] == '$'                        &&
+                   !tok_type_(tline->next, TOK_WHITESPACE)      &&
+                   (tok_type_(tline->next, TOK_ID)              ||
+                    tok_type_(tline->next, TOK_PREPROC_ID)      ||
+                    tok_type_(tline->next, TOK_NUMBER)          ||
+                    tok_type_(tline->next, TOK_OTHER)           ||
+                    tok_type_(tline->next, TOK_FLOAT))) {
+            /*
+             * In a sake of backward compatibility we allow
+             * to expand local single macro that early before
+             * pasting token code have place
+             *
+             * NOTE: that new code MUST use %+ macro to obtain
+             * same result
+             */
+            t = tline;
+            tline = tline->next;
+            tt = tokenize(t->text);
+            tt = expand_smacro(tt);
+            *tail = tt;
+            while (tt) {
+                tt->a.mac = NULL;
+                tail = &tt->next;
+                tt = tt->next;
+            }
+            delete_Token(t);
+            changed = true;
         } else {
             t = *tail = tline;
             tline = tline->next;