EPIC: Shutter API — Production Readiness for Continuous Time-Based Triggers

[ylembachar]

Driver: game customer, ~1-min rounds, time-based triggers on the internal set (1003).
Load: ~1,440 registrations/day and ≥1,440 key reads/day, sustained.

• Chiado rotated onto a new signer, verified end to end (register → key → decrypt), event registry ownership transferred, 8 consecutive rounds at 1/min passing
• Handed to the customer with an integration note
• Gas measured: 46,860 per registration
• Chiado alerting coverage verified (API and keypers both on InstanceDown, KeyperSyncStalled covers the crash-loop case)
• Issued mainnet key with higher limits and added to the config (see API config)
• Opened port to fallback Keyper on 1003 API
• Mainnet soak on 1003: 10 consecutive rounds at 1/min, 10/10 passing, key released within ~2s of the decryption timestamp (eon 1 live)
• Persist caddy limits in env variables
• One API PR: nonce serialisation + receipt confirmation + gas bump in a shared sender (crypto.go:520/:536, same bugs in eventtrigger.go:406/:422), gauges→counters, add keys-served / key-miss / signer balance.
• Soak + overlapping-registration stress on 1003, and pass it.

Done when

• 60 rounds at 1/min on 1003, a key every round
• No silently dropped registrations under overlapping load
• Their key on the raised limits
• Alerts fire on keys-stopped and low signer balance
• Signer funded to a documented runway with a named owner

Notes

1. 200 OK from register_identity doesn't mean registered — reproduced live: two concurrent calls both returned 200, second tx reverted AlreadyRegistered(), success metric counted both.
2. Caddy rate-limit counters are in-memory and the snippet is read only at startup, so any restart resets everyone's window.
3. The Go suite never touches Caddy, so it cannot test rate limits. Use curl against the deployed host.
4. tests/integration spends real gas and runs migrations on whatever DB_URL you point it at.