Feature: Return 503 when a goo SPARQL circuit breaker is open

[mdorf]

What this does

goo Ship 2 (ncbo/goo#196) can shed load when the Redis cache or the triple store is failing, but as a library it can only raise Goo::SPARQL::Resilience::CircuitOpenError. Nothing here translated that, so an open breaker surfaced as a 500 with a stack trace: indistinguishable from an application bug, and telling the client nothing about retrying.

This adds config/resilience.rb, the web-layer half of that design. It is the last piece needed before the breaker can actually be switched on.

Inert unless OP_SPARQL_CIRCUIT_BREAKER is set, so merging changes no behavior.

What's in it

• CircuitOpenError → 503, with Retry-After taken from the breaker's own cool-off so clients back off for roughly as long as the breaker stays open, and Cache-Control: no-store. The no-store matters: Rack::Cache fronts this app, and a cached 503 would keep serving the outage after the dependency recovered. Body follows the usual {errors: [...], status: 503} shape.
• on_state_change wired to the app logger and a New Relic custom event. goo only warns to stderr, which nothing watches, and a load-bearing dependency tripping is page-worthy. goo calls this per transition rather than per rejected request, so an outage produces one notice, not one per request.
• on_invalidation_failure wired to a log line and a counter, since a dropped invalidation leaves stale cache entries for that graph until its next successful write.

Both hooks swallow their own exceptions: reporting must never be the thing that breaks a request.

Gemfile.lock moves goo to 8a80f1c (development), the first revision containing Goo::SPARQL::Resilience. That also brings in Ship 3 (query logging), likewise off by default.

Why 503 rather than 500

The request failed because a dependency is unavailable, not because the request was wrong. 503 is what clients and load balancers already treat as "retry later", and it is the only status that carries Retry-After. Returning 500 also pollutes error dashboards with what is really deliberate load shedding.

Testing

test/controllers/test_resilience_503.rb asserts an open circuit returns 503 and not 500, that Retry-After matches the breaker cool-off and the response is no-store, that normal requests are untouched, and that both hooks are wired and cannot raise. The breaker itself is goo's to test (test/test_resilience.rb); this covers only the translation, by making a goo query raise the way an open breaker would.

Verified the two breaker tests error out with the handler removed, so they are testing the mapping rather than passing incidentally. Locally: the new file plus test_ontologies_controller and test_metrics_controller against 4store, 25 tests, 0 failures.

Enabling it

Set OP_SPARQL_CIRCUIT_BREAKER=true in a deployment's environment once this and goo Ship 2 are released. OP_SPARQL_BREAKER_THRESHOLD (default 5) and OP_SPARQL_BREAKER_COOL_OFF (default 30s) tune it. Worth doing on staging first and watching for the state-change log line, since breaker state is per worker (goo D14), so a real outage produces one notice per unicorn worker.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.94%. Comparing base (6f1bbad) to head (41c5cda).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop     #254   +/-   ##
========================================
  Coverage    78.94%   78.94%           
========================================
  Files           69       69           
  Lines         3767     3767           
========================================
  Hits          2974     2974           
  Misses         793      793           

Flag	Coverage Δ
ag	78.94% <ø> (ø)
fs	78.94% <ø> (ø)
gd	78.94% <ø> (ø)
unittests	78.94% <ø> (ø)
vo	78.94% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.