FE-1569: Add durable Brunch storage and telemetry - #9487
Open
lunelson wants to merge 8 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
github-actions
Bot
added
area/deps
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
3 tasks
Contributor
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
4 tasks
lunelson
force-pushed
the
ln/fe-1569-brunch-agent-deployment
branch
from
September 1, 2026 15:55
089310c to
157730c
Compare
TimDiekmann
reviewed
Sep 2, 2026
TimDiekmann
left a comment
TimDiekmann
left a comment
Member
There was a problem hiding this comment.
It would be good if we could isolate the Dockerfile changes into an own PR. The changes to .github/workflows/deploy.yml will make sure that the build is covered in CI and will upload it on merge to main. I will make sure that a ECR is set up where we can upload the image to and extend the definition with that.
lunelson
changed the base branch from
ln/fe-1563-redesign-runbook-workpiece
to
graphite-base/9487
September 2, 2026 11:12
lunelson
force-pushed
the
ln/fe-1569-brunch-agent-deployment
branch
from
September 2, 2026 11:12
157730c to
a118473
Compare
lunelson
changed the base branch from
graphite-base/9487
to
ln/fe-1569-brunch-agent-container
September 2, 2026 11:12
3 tasks
lunelson
changed the title
lunelson
had a problem deploying
to
pull-request
September 2, 2026 11:13 — with 
GitHub Actions
Error
GitHub Actions
Error
lunelson
had a problem deploying
to
pull-request
September 2, 2026 11:13 — with
GitHub Actions
Error
GitHub Actions
Error
lunelson
force-pushed
the
ln/fe-1569-brunch-agent-deployment
branch
from
September 4, 2026 09:58
6b78c5c to
b3ae475
Compare
lunelson
force-pushed
the
ln/fe-1569-brunch-agent-container
branch
from
September 4, 2026 09:58
bda25ab to
9c3440d
Compare
lunelson
marked this pull request as ready for review
September 4, 2026 09:59
PR SummaryHigh Risk Overview Persistence: In Telemetry: OpenTelemetry moves from inline Ops & proof: New scripts Tests: Unit coverage for config, Postgres/IAM, telemetry lifecycle, deployment smoke validation; build-artifact asserts production store wiring in the bundle. Reviewed by Cursor Bugbot for commit 4dc89cd. Bugbot is set up for automated code reviews on this repo. Configure here. |
lunelson
had a problem deploying
to
pull-request
September 4, 2026 09:59 — with
GitHub Actions
Error
GitHub Actions
Error
lunelson
had a problem deploying
to
pull-request
September 4, 2026 09:59 — with
GitHub Actions
Error
GitHub Actions
Error
lunelson
force-pushed
the
ln/fe-1569-brunch-agent-deployment
branch
from
September 4, 2026 10:02
b3ae475 to
e16e587
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Moderate issues remain in recovery validation, database timeouts, and Flue telemetry export.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds durable PostgreSQL persistence, RDS IAM support, operational telemetry, and deployment diagnostics for the Brunch service.
Changes:
- Adds fail-closed PostgreSQL configuration with TLS and IAM/password authentication.
- Adds OTLP telemetry lifecycle and database-failure reporting.
- Adds deployment probes, integration tests, and infrastructure handoff documentation.
File summaries
| File | Review |
|---|---|
yarn.lock |
Locks new database, AWS, and telemetry dependencies. |
libs/@hashintel/brunch-agent/MISSION.next.md |
Updates deployment backlog and planning context. |
libs/@hashintel/brunch-agent/MISSION.md |
Defines the restricted-deployment mission. |
libs/@hashintel/brunch-agent/docs/evidence/implementations/mission-8-deployment-handoff.md |
Records proof status and infrastructure handoff. |
apps/brunch-agent/turbo.json |
Adds telemetry environment and integration-test wiring. |
apps/brunch-agent/test/telemetry.test.ts |
Tests telemetry configuration and disposal. |
apps/brunch-agent/test/postgres.test.ts |
Tests pooling, IAM tokens, transactions, and shutdown. |
apps/brunch-agent/test/database-config.test.ts |
Tests production database validation. |
apps/brunch-agent/test/container-smoke.integration.ts |
Exercises the production container with PostgreSQL and OTel. |
apps/brunch-agent/test/build-artifact.test.ts |
Verifies persistence and telemetry bundle wiring. |
apps/brunch-agent/test/architecture/boundaries.integration.ts |
Registers the telemetry architecture boundary. |
apps/brunch-agent/src/telemetry.ts |
Configures OTLP providers and failure spans. Moderate (1 vote): Flue metrics use the unregistered global meter and are not exported. Moderate (1 vote): Flue also receives no logger, dropping its logs as well as metrics. |
apps/brunch-agent/src/telemetry-bootstrap.ts |
Installs telemetry before application initialization. |
apps/brunch-agent/src/rds-iam-probe.ts |
Adds the two-connection IAM diagnostic. |
apps/brunch-agent/src/postgres.ts |
Implements the Flue PostgreSQL runner. Moderate (1 vote): No connection timeout is configured, allowing database operations and diagnostics to wait indefinitely. |
apps/brunch-agent/src/deployment-smoke.ts |
Adds remote turn and history diagnostics. Moderate (2 votes): Recovery can report success for missing/malformed history, and streamed turns do not require a successful finish. |
apps/brunch-agent/src/db.ts |
Selects PostgreSQL in production and SQLite locally. |
apps/brunch-agent/src/database-config.ts |
Defines fail-closed production database configuration. |
apps/brunch-agent/src/app.ts |
Bootstraps telemetry before route setup. |
apps/brunch-agent/README.md |
Documents runtime and deployment requirements. |
apps/brunch-agent/package.json |
Adds dependencies and operational scripts. |
apps/brunch-agent/docs/task-dependencies.json |
Documents the new task dependencies. |
Review details
Suppressed comments (4)
apps/brunch-agent/src/postgres.ts:102
- A
pgpool emits anerrorevent when an idle connection is broken; without a listener, Node treats that event as unhandled and terminates the process. A routine RDS restart or network partition can therefore crash Brunch instead of producing the operational database-failure telemetry added by this PR. Attach a pool error listener and report it throughreportDatabaseFailure.
export const createPostgresPool = (
config: PostgresDatabaseConfig,
options?: ConnectionOptions,
): Pool => new Pool(createPostgresPoolConfig(config, options));
apps/brunch-agent/src/postgres.ts:200
- If one
pool.connect()resolves and the other rejects, thePromise.allassignment never completes, soclientsremains empty and the successful checked-out client is omitted from thefinallycleanup. The failed IAM probe then leaks that client and can leavepool.end()waiting indefinitely. Acquire while appending each client (the first remains checked out, so the second is still guaranteed to be a distinct physical connection).
let clients: PoolClient[] = [];
try {
clients = await Promise.all([pool.connect(), pool.connect()]);
apps/brunch-agent/src/postgres.ts:128
- A transaction that fails while acquiring a pooled connection bypasses the
tryblock, so nodatabase_operationfailure span is recorded. This is the exact path taken when Postgres is unavailable during Flue startup/migration, contradicting the operational database-failure telemetry contract. Report and rethrow acquisition failures before entering the transaction body.
const client = await pool.connect();
apps/brunch-agent/src/postgres.ts:156
- Both expressions passed to
Promise.allSettledare invoked immediately, so telemetry shutdown begins beforepool.end()has completed. The unit test only observes synchronous mock side effects and therefore does not enforce the documented “close Postgres, then flush OTel” lifecycle; with a real pool, final database activity can race provider shutdown. Await the Postgres result first, then invoke the telemetry closer while still collecting both failures.
const results = await Promise.allSettled([pool.end(), afterClose?.()]);
- Files reviewed: 21/22 changed files
- Comments generated: 4
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+32
to
+36
| const history = (await response.json()) as { messages?: readonly unknown[] }; | ||
| process.stdout.write( | ||
| `${JSON.stringify({ | ||
| conversationId, | ||
| messages: history.messages?.length ?? 0, |
Contributor
Author
There was a problem hiding this comment.
Fixed in 4dc89cd. History mode now requires a stable conversation ID plus BRUNCH_SMOKE_EXPECTED_TEXT and verifies a text part contains it; turn mode parses the UI stream, rejects error/abort, and requires finish.
Comment on lines
+67
to
+71
| const common: PoolConfig = { | ||
| application_name: "brunch-agent", | ||
| database: config.database, | ||
| host: config.host, | ||
| port: config.port, |
Contributor
Author
There was a problem hiding this comment.
Fixed in 4dc89cd. Every pool now sets connectionTimeoutMillis to the documented 10-second bound, and the pool-config test pins that value.
Comment on lines
+85
to
+89
| const unregisterInstrumentations = registerInstrumentations({ | ||
| instrumentations: [new HttpInstrumentation(), new UndiciInstrumentation()], | ||
| meterProvider, | ||
| tracerProvider: traceProvider, | ||
| }); |
Contributor
Author
There was a problem hiding this comment.
Fixed in 4dc89cd. The HASH setup now supplies its application-owned tracer, meter, and structural logger explicitly when constructing Flue instrumentation.
Comment on lines
+148
to
+151
| const createFlue = | ||
| dependencies.createFlueInstrumentation ?? | ||
| createOpenTelemetryInstrumentation; | ||
| const flueInstrumentation = createFlue({ content: false }); |
Contributor
Author
There was a problem hiding this comment.
Fixed in 4dc89cd. Flue receives meterProvider.getMeter(...) explicitly, so its instruments bind to the exporting provider without relying on the global metrics API.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1002 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1527 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 108 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 27 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |
Co-authored-by: Cursor <cursoragent@cursor.com> Normalize deployment dependency lockfile Co-authored-by: Cursor <cursoragent@cursor.com> Publish the Brunch image to GHCR Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4dc89cd. Configure here.
| export const createPostgresPool = ( | ||
| config: PostgresDatabaseConfig, | ||
| options?: ConnectionOptions, | ||
| ): Pool => new Pool(createPostgresPoolConfig(config, options)); |
There was a problem hiding this comment.
Idle pool errors crash the process
High Severity
The new Pool never listens for error. When RDS drops an idle client, pg emits that event on the pool; Node then treats it as an uncaught exception and exits. A routine failover or idle timeout can take the whole Brunch task down after it has already become healthy.
Reviewed by Cursor Bugbot for commit 4dc89cd. Configure here.
| throw failure; | ||
| } finally { | ||
| client.release(); | ||
| } |
There was a problem hiding this comment.
Transaction connect failures skip telemetry
Medium Severity
transaction calls pool.connect() outside the try that reports database failures. Checkout failures, including IAM token errors raised from the password callback, never emit the operational database_operation span that query records, so hosted telemetry misses the main write-path outage.
Reviewed by Cursor Bugbot for commit 4dc89cd. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🌟 What is the purpose of this PR?
Add the runtime behavior needed to take the containerized Brunch service to the application-to-infrastructure deployment boundary: durable Flue Postgres state, content-free operational telemetry, and explicit diagnostics for restricted deployment and recovery.
This PR does not claim that Brunch is deployed. The downstack container PR publishes to ECR and GHCR, but no Brunch ECS service, RDS target, hosted collector, restricted ingress, or deployment owner is confirmed, so Mission 8 remains open at its required handoff boundary.
🔗 Related links
🚫 Blocked by
🔍 What does this change?
@flue/postgrespersistence using dedicated fields, verified TLS, RDS IAM tokens per physical connection, and a runtime-password fallback.🏗️ Agent notes
The branch authority is
libs/@hashintel/brunch-agent/MISSION.md. It records Mission 8 as live but stopped at the explicit application-to-infrastructure handoff boundary.Locally established:
Still open:
The downstack PR registers ECR and GHCR publication with no ECS target. Infrastructure provisioning and remote proof remain outside this local application artifact.
Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
turbo.jsonfiles have been updated to reflect this/healthproves process liveness only. Database connection and migration gate startup before the server listens.🐾 Next steps
🛡 What tests cover this?
❓ How to test this?
yarn workspace @apps/brunch-agent lint:tsc.yarn workspace @apps/brunch-agent test:unit.yarn workspace @apps/brunch-agent build.yarn workspace @apps/brunch-agent test:integrationwith Docker available.apps/brunch-agent/README.mdand the Mission 8 deployment handoff for the IAM and remote smoke commands.📹 Demo
No visual UI change. The proof artifact is the production container and its deployment diagnostics.