GVSoC: ISS HTIF pollers generate timed fabric traffic, saturating a cache bank at high core counts

[jpf-h]

Hey @Aquaticfuller @DiyouS,

found when running some kernels on the 64 tile variant.

Every ISS instantiated with htif=True polls the 8-byte tohost word every 1000 cycles (Htif::htif_handler re-arming via htif_event.enqueue(1000)). The poll goes through Syscalls::user_access, which issues byte-granular requests on the core's data port — i.e., through the fully timed memory model. tohost is an ordinary .data word, so in shared-L1 targets all pollers converge on one cache bank.

Impact. With N cores the aggregate is N × 8 / 1000 bank accesses/cycle at a single bank. On the CachePool insitu-cache target this crosses the bank's 1-op/cycle accept capacity at 256 cores (~2 req/cycle measured: 63 M debug reads in 61.5 M cycles, all on tohost's home bank), the bank's accept backlog diverges, and any core whose data shares that bank starves — presenting as a program deadlock.

Fix (attached). Event-driven tohost: the LSU compares each store's address against the config-known htif_tohost and wakes the Htif handler directly (notify_tohost_store(), one-cycle-later enqueue, guarded by is_enqueued()); the periodic re-arm is removed (the single reset-time check remains). The store itself proceeds into memory unchanged, so tohost keeps ordinary memory semantics for the target. Host-side steady-state traffic through the modeled fabric drops to zero — equivalent to an RTL testbench's non-intrusive watch — and the staleness window disappears (the handler reads immediately after the store, line guaranteed present). Cost: one address compare per store.

Note: Found by AI, I have not verified the diagnosis and patch below completely and unfortunately don't have time to clean it up now. But I wanted to report before you run into this issue and have to debug it again :-)

Note: I'm not sure if this AI solution is completely correct when the request can resolve asynchronously or something. Better double-check the patch below.

From a68ed5866b19bf7b295baba2df10546d29abda9f Mon Sep 17 00:00:00 2001
From: Johannes Pfau <johannes.pfau@h-partners.com>
Date: Wed, 26 Aug 2026 19:04:33 +0200
Subject: [PATCH] =?UTF-8?q?iss/htif:=20event-driven=20tohost=20=E2=80=94?=
 =?UTF-8?q?=20store-side=20interception=20replaces=20the=201000-cycle=20po?=
 =?UTF-8?q?ll?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The per-core HTIF pollers read tohost through the timed cache fabric
every 1000 cycles; at 256 cores that saturated the word's home bank
(the original runaway), and the bank-side debug-transparency repair
left a read-miss staleness hole: under heavy traffic the tohost line
is evicted between the target's store and the host poll, the debug
read misses, and the dropped syscall wedges the target mid-printf
(mu_peak@64 froze deterministically at the same TTI).

tohost is an ordinary memory word by HTIF design, so the target's
store must stay real memory traffic — but the HOST side never needed
to poll: the ISS already knows the address. The LSU now compares each
store against it and wakes the Htif handler directly (next cycle, when
the store has completed and its line is guaranteed present); the
periodic re-arm is gone. Host-side traffic through the modeled memory
system in steady state: zero — matching the RTL testbench's
non-intrusive watch.

Validation at 64 tiles / 256 cores: interval-only 3/3 + printf path
clean; mu_peak FULL ladder completes for the first time (70%
sustained, rungs 40-70 at 28-44% worst-interval budget; 80/90 show the
genuine backlog knee) through both historical freeze points.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 models/cpu/iss/include/htif.hpp |  6 ++++++
 models/cpu/iss/src/htif.cpp     | 13 ++++++++++++-
 models/cpu/iss/src/lsu.cpp      | 11 +++++++++++
 3 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/models/cpu/iss/include/htif.hpp b/models/cpu/iss/include/htif.hpp
index c311ec31..0ca023f3 100644
--- a/models/cpu/iss/include/htif.hpp
+++ b/models/cpu/iss/include/htif.hpp
@@ -104,6 +104,12 @@ class Htif
 {
 
 public:
+    // Store-side interception (event-driven HTIF): the LSU calls this when the target
+    // stores to the tohost word; the handler then runs once, next cycle, instead of a
+    // periodic poll. Keeps the host side out of the modeled memory system entirely.
+    void notify_tohost_store();
+    inline iss_reg_t get_tohost_addr() { return this->tohost_addr; }
+
     Htif(IssWrapper &top, Iss &iss);
 
     void build();
diff --git a/models/cpu/iss/src/htif.cpp b/models/cpu/iss/src/htif.cpp
index 86abaad8..1ed6c29b 100644
--- a/models/cpu/iss/src/htif.cpp
+++ b/models/cpu/iss/src/htif.cpp
@@ -449,6 +449,16 @@ int fds_t::lookup(iss_reg_t fd)
   return fd >= fds.size() ? -1 : fds[fd];
 }
 
+void Htif::notify_tohost_store()
+{
+#ifdef CONFIG_GVSOC_ISS_HTIF
+    if (this->tohost_addr != 0 && !this->htif_event.is_enqueued())
+    {
+        this->htif_event.enqueue(1);
+    }
+#endif
+}
+
 void Htif::reset(bool active)
 {
     if (active)
@@ -479,5 +489,6 @@ void Htif::htif_handler(vp::Block *__this, vp::ClockEvent *event)
         }
     }
 
-    iss->syscalls.htif.htif_event.enqueue(1000);
+    // No periodic re-arm: event-driven via notify_tohost_store() (a per-1000-cycle
+    // poll through the timed cache fabric saturated the tohost home bank at 256 cores).
 }
diff --git a/models/cpu/iss/src/lsu.cpp b/models/cpu/iss/src/lsu.cpp
index 188c33aa..6d5faedf 100644
--- a/models/cpu/iss/src/lsu.cpp
+++ b/models/cpu/iss/src/lsu.cpp
@@ -163,6 +163,17 @@ void Lsu::data_response(vp::Block *__this, vp::IoReq *req)
 
 int Lsu::data_req_aligned(iss_addr_t addr, uint8_t *data_ptr, uint8_t *memcheck_data, int size, bool is_write, int64_t &latency, int &req_id)
 {
+    // Event-driven HTIF: a store touching the tohost word wakes the host-side handler
+    // directly (next cycle, when this store has completed). The store itself proceeds
+    // into memory unchanged - tohost stays an ordinary memory word for the target.
+    if (is_write)
+    {
+        iss_reg_t th = this->iss.syscalls.htif.get_tohost_addr();
+        if (th != 0 && addr <= th && th < addr + (iss_addr_t)size)
+        {
+            this->iss.syscalls.htif.notify_tohost_store();
+        }
+    }
     this->trace.msg("Data request (addr: 0x%lx, size: 0x%x, is_write: %d)\n", addr, size, is_write);
     vp::IoReq *req = this->get_req();
     if (req == NULL)
-- 
2.50.1

[jpf-h]

I think the original idea is that these requests are marked as debug, handled normally by the fabric but not accounted for. There is at least one case in the cache model where this special handling for the debug request is missing. However, the patch below tried to fix that and did not solve the issue fully for me, so there may be other places. I thought it is safer and simpler to just remove the polling completely.

From 3c3c95ed9e2b97688c9d2adefc41c1b0d78e58af Mon Sep 17 00:00:00 2001
From: Johannes Pfau <johannes.pfau@h-partners.com>
Date: Wed, 26 Aug 2026 16:21:07 +0200
Subject: [PATCH] insitu-cache: debug/backdoor accesses are timing-transparent
 in the structural bank
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The per-core ISS HTIF pollers read the 8-byte tohost word through the
timed cache path every 1000 cycles, as 8 byte-granular debug requests
(Syscalls::user_access loops size-1). At 256 cores that is ~2 requests
per cycle aggregated onto ONE bank — the tohost word's flat-map home —
which is at/above the bank cell's 1-op/cycle accept capacity, so its
accept backlog diverges and every core whose data shares that bank
starves. Measured with per-gate instrumentation: 63M debug byte-reads
in 61.5M cycles, all on tohost's home bank, cell gate pinned; the
pre-rebase 256-core interval-loop deadlock's pathological bank
(tile 50 bank 3) is exactly tohost's home in that binary. This — not
kernel traffic — was the 256-core runaway driver throughout.

Fix: debug requests are served timing-transparently at the bank (the
gvsoc contract; the calibrated controller already bypasses them):
serve/update the cached line when the tag matches (a debug write must
hit the cached copy — updating only L2 leaves a stale VALID line the
target's own polling then reads forever), mark write-hits dirty so
eviction preserves them, and touch NO gate state. No downstream
forward: three forwarding variants (caller's req; private req on the
evict lane; private req on the refill lane) each corrupt the composite
L2 port's bookkeeping and wedge a normal store permanently. The
forward-free contract is exact for the debug traffic that exists:
HTIF words read zero until the target's first store installs the line
(they are zero until then); syscall args/strings are target-written
immediately before each syscall, so their lines are present.

Validation at 64 tiles / 256 cores: interval-only 3/3 PASS, full
selftest 31/31 PASS, mu_peak-80 reproducer (previously frozen mid-TTI)
completes with worst-interval budget halved (51-58% vs 128%+).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 models/cache/insitu/insitu_cache_core.cpp | 42 +++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/models/cache/insitu/insitu_cache_core.cpp b/models/cache/insitu/insitu_cache_core.cpp
index d076045b..9ae6659a 100644
--- a/models/cache/insitu/insitu_cache_core.cpp
+++ b/models/cache/insitu/insitu_cache_core.cpp
@@ -473,6 +473,48 @@ vp::IoReqStatus InsituCacheCore::run_flush(vp::IoReq *req)
 // request is in flight (the cluster LSU is single-outstanding), so the refill needs no FIFO.
 vp::IoReqStatus InsituCacheCore::run_request_sync(vp::IoReq *req)
 {
+    // Debug/backdoor accesses (HTIF tohost pollers, gdbserver, syscall-argument reads) are
+    // timing-transparent by gvsoc contract: serve them straight from L2 (functional
+    // write-through keeps L2 current) and touch NO gate state, NO latency, NO way state.
+    // The per-core HTIF pollers otherwise read tohost through the timed path every 1000
+    // cycles; at 256 cores that alone saturates the tohost home bank's 1-op/cycle cell
+    // (~1+ arrival/cycle sustained) and its accept backlog diverges — the actual 256-core
+    // runaway driver (measured: 31M+ byte reads of tohost on one bank; ManyRVData rlc_am
+    // doc/PROFILING.md, 256-core investigation). The calibrated controller already
+    // bypasses debug requests the same way (insitu_cache_controller.cpp req_handler).
+    if (req->is_debug()) {
+        // The cached copy, when present, is the coherent one for BOTH directions: a debug
+        // write that only updated L2 would leave a stale VALID line that the target's own
+        // polling (HTIF tohost handshake) then reads forever. So: update/serve the cached
+        // line if the tag matches, and keep L2 current (writes always; reads only on miss).
+        const uint64_t dbg_addr = req->get_addr();
+        const uint32_t dbg_set = geom_.set_index(dbg_addr);
+        WayMeta *dbg_ways = set_ways(dbg_set);
+        const uint64_t dbg_tag = geom_.tag_of(dbg_addr);
+        int dbg_w = -1;
+        for (uint32_t i = 0; i < num_ways_; i++)
+            if (dbg_ways[i].status != INVALID && dbg_ways[i].tag == dbg_tag) { dbg_w = (int)i; break; }
+        if (dbg_w >= 0 && req->get_data() != nullptr) {
+            uint8_t *line = &data_[((size_t)dbg_set * num_ways_ + (uint32_t)dbg_w) * cache_line_bytes_
+                                   + (dbg_addr & (cache_line_bytes_ - 1))];
+            if (req->get_is_write()) memcpy(line, req->get_data(), req->get_size());
+            else                     memcpy(req->get_data(), line, req->get_size());
+        }
+        // NO downstream forward from here: issuing a request toward L2 from inside this
+        // entry point corrupts the composite L2 port's bookkeeping (three variants tried —
+        // forwarding the caller's req, a private req on the evict lane, and a private req on
+        // the refill lane — all eventually wedge a NORMAL store permanently mid-run; the
+        // forward-free variant is clean). The resulting contract is exact for the debug
+        // traffic that exists: HTIF words read zero until the target's first store installs
+        // the line (they ARE zero until then); syscall arguments and strings are written by
+        // the target immediately before the syscall, so their lines are present. A debug
+        // write-hit marks the line dirty so a later eviction writes the value back.
+        if (dbg_w >= 0 && req->get_is_write()) dbg_ways[dbg_w].dirty = true;
+        if (dbg_w < 0 && !req->get_is_write() && req->get_data() != nullptr)
+            memset(req->get_data(), 0, req->get_size());
+        return vp::IO_REQ_OK;
+    }
+
     const int64_t now = clock.get_cycles();
     const uint64_t addr = req->get_addr();
     const bool is_write = req->get_is_write();
-- 
2.50.1