_yaml.py: Support merging of conditional lists

During composition of project.conf, it can happen that we
are including and composing node hierarchies before options
are resolved and before we are ready to evaluate conditional
statements.

Without this patch, conditional statements are treated like
regular lists, causing existing conditional statements to
be overwritten by subsequently composited conditional statements
without ever having been resolved.

This patch introduces a special case for composing lists,
when composing a sequence that is a conditional statement, the
sequence is appended instead of overwriting the underlying
list.

Note that precedence of declarations in including vs included
yaml fragments did raise some concern, and a test in the following
commit is added to cover this case. Composition of conditional lists
on top of other conditional lists are unconditionally appended,
however the semantics of includes ensures that the including fragment's
conditionals is always composited *on top* over the included fragment,
ensuring that we have the correct pecedence even when compositing
yet to be resolved conditional directives.

1 files changed, 14 insertions, 1 deletions

diff --git a/buildstream/_yaml.py b/buildstream/_yaml.py
index 4f2f88329..405a10ec8 100644
--- a/buildstream/_yaml.py
+++ b/buildstream/_yaml.py
@@ -738,7 +738,20 @@ def composite_list(target_node, source_node, key):
                             "{}: List cannot overwrite value at: {}"
                             .format(source_key_provenance, target_key_provenance))
 
-        composite_list_overwrite(target_node, key, source_node, key)
+        # Special case: The `project.conf` in some cases needs to composite
+        # include files before having resolved options, so there can be
+        # conditionals that need to be merged at this point.
+        #
+        # This unconditionally appends conditional statements to a matching
+        # conditional in the target so as to preserve them. The precedence
+        # of include files is preserved regardless due to the order in which
+        # included dictionaries are composited.
+        #
+        if key == "(?)":
+            composite_list_append(target_node, key, source_node, key)
+        else:
+            composite_list_overwrite(target_node, key, source_node, key)
+
         return True
 
     # When a composite list is encountered in the source, then