GVSoC: There's currently only one icache for all tiles

[jpf-h]

Hey @Aquaticfuller @DiyouS,

found when running some kernels on the 64 tile variant: Unlike RTL, GVSoC currently only has one Icache that was shared for all tiles. At very large tile counts, this seems to be creating issues.

Note: Found by AI, I have not verified the diagnosis and patch below completely and unfortunately don't have time to clean it up now. But I wanted to report before you run into this issue and have to debug it again :-)

Is it correct that refill is connected to the same port for all ICACHES?

From 24c2f8ab4ba1080dd73d9f8e6e75d0ebd6509767 Mon Sep 17 00:00:00 2001
From: Johannes Pfau <johannes.pfau@h-partners.com>
Date: Wed, 26 Aug 2026 01:18:19 +0200
Subject: [PATCH] snitch_cluster: one instruction cache per tile, not one per
 cluster
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The cluster model gave the whole cluster a single Hierarchical_cache with a
single L1 bank shared by every core. The RTL instantiates a 16 kB icache
inside each tile (cachepool_tile.sv), over its own NrCores. Fine to 128
cores; at 256 it fell off a cliff — after every barrier release all cores
miss their one-line L0s at once into that lone L1 bank, whose refill queue
then grows faster than it drains and charges single instruction fetches
millions of cycles. Now one icache per tile, cores bound to their own,
refill and flush fan-out per tile; clusters without a tile geometry keep
the single cache. Contributes to the full 256-core rlc_am selftest passing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 pulp/snitch/snitch_cluster/snitch_cluster.py | 28 ++++++++++++++------
 1 file changed, 20 insertions(+), 8 deletions(-)

diff --git a/pulp/snitch/snitch_cluster/snitch_cluster.py b/pulp/snitch/snitch_cluster/snitch_cluster.py
index fc5c1fa..de11758 100644
--- a/pulp/snitch/snitch_cluster/snitch_cluster.py
+++ b/pulp/snitch/snitch_cluster/snitch_cluster.py
@@ -190,9 +190,19 @@ class SnitchCluster(gvsoc.systree.Component):
         # Zero memory
         zero_mem = ZeroMem(self, 'zero_mem', size=arch.zero_mem.size)
 
-        # Shared icache
-        icache = Hierarchical_cache(self, 'icache', nb_cores=arch.nb_core, has_cc=0,
-            l1_line_size_bits=7)
+        # Instruction caches: one per TILE, as the RTL has it (cachepool_tile.sv instantiates
+        # the 16 kB icache inside the tile, over its own NrCores). One cluster-wide icache with a
+        # single L1 bank shared by every core was fine to 128 cores and collapsed at 256: after a
+        # barrier release all cores miss their one-line L0s at once, the lone L1 bank's refill queue
+        # grows faster than it drains, and a single instruction fetch ends up charged millions of
+        # cycles (the 64-tile "hang" of 2026-08-25). Clusters without a tile geometry keep one.
+        _cpt = getattr(arch, 'cachepool_cores_per_tile', 0) or arch.nb_core
+        _nb_icache = (arch.nb_core + _cpt - 1) // _cpt
+        icaches = [Hierarchical_cache(self, ('icache_%d' % t) if _nb_icache > 1 else 'icache',
+                                      nb_cores=_cpt, has_cc=0, l1_line_size_bits=7)
+                   for t in range(_nb_icache)]
+        def _icache_of(core_id):
+            return icaches[core_id // _cpt], core_id % _cpt
 
         # Cores
         cores = []
@@ -289,7 +299,8 @@ class SnitchCluster(gvsoc.systree.Component):
         wide_axi.o_MAP(self.i_WIDE_SOC())
 
         # Icache
-        icache.o_REFILL( wide_axi.i_INPUT() )
+        for _ic in icaches:
+            _ic.o_REFILL( wide_axi.i_INPUT() )
 
         # Remote Access to TCDM
         wide_axi.o_MAP(tcdm.i_DMA_INPUT(), base=arch.tcdm.area.base, size=arch.tcdm.area.size, rm_base=True)
@@ -509,11 +520,12 @@ class SnitchCluster(gvsoc.systree.Component):
                     tcdm_port += 1
 
             cores_ico[core_id].o_MAP(narrow_axi.i_INPUT())
-            cores[core_id].o_FETCH(icache.i_INPUT(core_id))
+            _ic, _local = _icache_of(core_id)
+            cores[core_id].o_FETCH(_ic.i_INPUT(_local))
 
-            # Icache
-            cores[core_id].o_FLUSH_CACHE(icache.i_FLUSH())
-            icache.o_FLUSH_ACK(cores[core_id].i_FLUSH_CACHE_ACK())
+            # Icache (this core's tile's)
+            cores[core_id].o_FLUSH_CACHE(_ic.i_FLUSH())
+            _ic.o_FLUSH_ACK(cores[core_id].i_FLUSH_CACHE_ACK())
 
         # Cache's L2 output → wide_axi. Existing wide_axi mapping already routes
         # TCDM-range addresses to tcdm.i_DMA_INPUT, so misses land in the SPM as expected.
-- 
2.50.1