nir: Combine if_uses with instruction uses

Every nir_ssa_def is part of a chain of uses, implemented with doubly linked
lists.  That means each requires 2 * 64-bit = 16 bytes per def, which is
memory intensive. Together they require 32 bytes per def. Not cool.

To cut that memory use in half, we can combine the two linked lists into a
single use list that contains both regular instruction uses and if-uses. To do
this, we augment the nir_src with a boolean "is_if", and reimplement the
abstract if-uses operations on top of that list. That boolean should fit into
the padding already in nir_src so should not actually affect memory use, and in
the future we sneak it into the bottom bit of a pointer.

However, this creates a new inefficiency: now iterating over regular uses
separate from if-uses is (nominally) more expensive. It turns out virtually
every caller of nir_foreach_if_use(_safe) also calls nir_foreach_use(_safe)
immediately before, so we rewrite most of the callers to instead call a new
single `nir_foreach_use_including_if(_safe)` which predicates the logic based on
`src->is_if`. This should mitigate the performance difference.

There's a bit of churn, but this is largely a mechanical set of changes.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22343>

3 files changed, 7 insertions, 8 deletions

diff --git a/src/panfrost/.clang-format b/src/panfrost/.clang-format
index 2d75ff1089f..e6829560e59 100644
--- a/src/panfrost/.clang-format
+++ b/src/panfrost/.clang-format
@@ -46,6 +46,8 @@ ForEachMacros:
   - nir_foreach_register_safe
   - nir_foreach_use
   - nir_foreach_use_safe
+  - nir_foreach_use_including_if
+  - nir_foreach_use_including_if_safe
   - nir_foreach_if_use
   - nir_foreach_if_use_safe
   - nir_foreach_def
diff --git a/src/panfrost/midgard/nir_fuse_io_16.c b/src/panfrost/midgard/nir_fuse_io_16.c
index f4b052ea438..5fe4761f969 100644
--- a/src/panfrost/midgard/nir_fuse_io_16.c
+++ b/src/panfrost/midgard/nir_fuse_io_16.c
@@ -77,13 +77,10 @@ nir_fuse_io_16(nir_shader *shader)
             if (!intr->dest.is_ssa)
                continue;
 
-            if (!list_is_empty(&intr->dest.ssa.if_uses))
-               return false;
-
             bool valid = true;
 
-            nir_foreach_use(src, &intr->dest.ssa)
-               valid &= nir_src_is_f2fmp(src);
+            nir_foreach_use_including_if(src, &intr->dest.ssa)
+               valid &= !src->is_if && nir_src_is_f2fmp(src);
 
             if (!valid)
                continue;
diff --git a/src/panfrost/util/nir_mod_helpers.c b/src/panfrost/util/nir_mod_helpers.c
index 7a1fef990a7..466a4d4252a 100644
--- a/src/panfrost/util/nir_mod_helpers.c
+++ b/src/panfrost/util/nir_mod_helpers.c
@@ -86,13 +86,13 @@ pan_has_dest_mod(nir_dest **odest, nir_op op)
       return false;
 
    /* Check the uses. We want a single use, with the op `op` */
-   if (!list_is_empty(&dest->ssa.if_uses))
-      return false;
-
    if (!list_is_singular(&dest->ssa.uses))
       return false;
 
    nir_src *use = list_first_entry(&dest->ssa.uses, nir_src, use_link);
+   if (use->is_if)
+      return false;
+
    nir_instr *parent = use->parent_instr;
 
    /* Check if the op is `op` */