fix: NoTicket infinite loop when draining non-query response streams (HTTP/2 stream reset) - #758
fix: NoTicket infinite loop when draining non-query response streams (HTTP/2 stream reset)#758Koushik-Mummina wants to merge 3 commits into
Conversation
Propagate IOException from InputStreamUtil.readAllBytes instead of catching forever so HTTP/2 stream resets fail executeUpdate with a SQLException rather than hanging and WARN-spamming.
Per engineering review: FireboltException from a failed non-query response drain now carries SQLState 08007 (transaction resolution unknown) so applications can detect the in-doubt outcome programmatically, and the OkHttp connection pool is evicted so the next statement gets a fresh connection instead of reusing the one that was reset. Recovery test asserts the second statement lands on a distinct connection.
678fbf2 to
b6ca6f4
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b6ca6f4. Configure here.
| @Override | ||
| public void evictConnectionPool() { | ||
| getHttpClient().connectionPool().evictAll(); | ||
| } |
There was a problem hiding this comment.
Global pool eviction blast radius
Medium Severity
evictConnectionPool() calls evictAll() on the OkHttp client from HttpClientConfig, which is a JVM-wide singleton shared by every Firebolt JDBC connection (and auth/gateway traffic). One non-query drain failure therefore drops idle pooled connections for unrelated connections, forcing reconnects process-wide instead of only retiring the stale route for that statement.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit b6ca6f4. Configure here.
There was a problem hiding this comment.
That's accurate. The client and its pool are JVM-wide. We considered this and kept evictAll deliberately. It only closes idle connections, so nothing in flight is affected, and the cost to other connections is one extra TLS handshake on their next request. This path also only fires on a mid-body drain failure, which is rare. The other side of it is that the usual cause of the reset (LB idle timeout or GOAWAY on the shared route) likely also stalled the other idle connections in the same pool, so evicting them is more preemptive than collateral. A per-connection eviction would be nicer, but okhttp doesn't expose one publicly, and detecting connection-level causes in the service layer would mean depending on okhttp internals there.
Delete the env-gated live smoke test and the hang-capture paths from the repro tests; they demonstrated the bug on the pre-fix code and are unreachable on the fixed driver. Restructure the loop-repro assertion to a single-invocation lambda. Regression coverage (fail-fast with SQLState 08007, pool eviction, recovery on a fresh connection) is unchanged.
|





A customer on 3.7.1 hit this in their integration environment. When the HTTP/2 stream is reset mid-response (StreamResetException: stream was reset: CANCEL, usually due to a stale pooled connection), executeUpdate() never returns. The drain loop in InputStreamUtil.readAllBytes() catches IOException and logs it, so once the stream is dead, every read throws an exception, and there is no way out. The stuck thread also logs the same WARN line nonstop. I measured around 190k lines per second. This has been in the driver since #216, and I reproduced it on both v3.7.1 and master.
There is nothing on the client side that can stop it once it starts. The reads fail instantly instead of blocking, so socket timeouts, setQueryTimeout, and Statement.cancel() never get a chance.
What this PR changes:
Tests: a unit test for the loop itself. A MockWebServer test that resets the stream with CANCEL mid-body, which used to hang before the fix. A recovery test showing the follow-up statement lands on a different connection and unit coverage for SQLState and eviction.
The customer is waiting on a version to upgrade to, so please include this in a patch release once it's merged.
Note
Medium Risk
Changes how DML/DDL response drain failures are handled and evicts the entire OkHttp connection pool on those errors, which can affect subsequent statements and in-doubt outcomes.
Overview
Fixes a hang in non-query execution:
InputStreamUtil.readAllBytesno longer swallowsIOExceptionin an infinite loop (HTTP/2CANCEL/ stream reset). Failures now propagate immediately instead of spinning and flooding WARN logs.FireboltStatementServicewraps drain failures asFireboltExceptionwith SQLState08007(transaction resolution unknown) and an in-doubt message, then always closes the stream. It also calls newStatementClient.evictConnectionPool()so the next statement does not reuse a reset HTTP/2 connection.Adds unit and MockWebServer HTTP/2 tests for fail-fast behavior, SQLState, pool eviction, and recovery on a fresh connection. Includes
REPRO_RESULTS.mddocumenting the hang and fix.Reviewed by Cursor Bugbot for commit bc07cdb. Bugbot is set up for automated code reviews on this repo. Configure here.