feat(subscriptions): ground AI reports in saved context by vdekrijger · Pull Request #91787 · PostHog/posthog

vdekrijger · 2026-08-31T16:08:05Z

Problem

AI reports cannot use saved dashboard or insight context yet, so the report harness still relies on prompt inference.

This execution layer depends on #91428. #91788 adds the UI.

Changes

The report harness resolves up to three saved contexts into authoritative query results before planning.
Dashboard contexts contribute the six most-viewed eligible tiles.
Insight contexts contribute their saved query result.
Contextful reports always replan so deliveries use current data.
Timeouts, character budgets, and partial-failure handling bound context resolution.
Deliveries retain compact context provenance and recheck access before execution.

Before:

flowchart LR
    Prompt[Prompt] --> Harness{{Report harness}}
    Harness --> Project[Project data]
    Project --> Delivery[Report delivery]

    class Prompt,Delivery phYellow
    class Harness phBlue
    class Project phGray

    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;

After:

flowchart LR
    Prompt[Prompt] --> Harness{{Report harness}}
    Context[Saved context] --> Resolve[Bounded query execution]
    Resolve --> Harness
    Harness --> Delivery[Report delivery]

    class Prompt,Delivery phYellow
    class Harness phBlue
    class Context,Resolve phGray

    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;

How did you test this code?

Resolver tests cover limits, popularity ordering, timeouts, access revocation, partial failures, and evidence bounds.
Pipeline tests cover authoritative context, contextful replanning, and unchanged contextless frozen plans.
Sanitization and delivery tests cover marker handling and compact provenance.
Repository-wide mypy and strict preflight completed locally.
Manual delivery testing was not run.

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Automatic notifications

Publish to changelog?

Docs update

No public docs update. The feature remains unavailable until the final stack layer enables its UI.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Codex retained query evidence and removed unrelated delivery hardening, recovery UI, and speculative indexing.
Skills: /writing-dataclasses, /writing-tests, /qa-team, /running-ci-preflight, /writing-pr-descriptions, and /stacking-prs.
Public-artifact review found no customer-derived or private material.

github-actions · 2026-08-31T16:11:01Z

🤖 CI report

⚠️

Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

✅ Django migration risk — migration analysis complete

We've analyzed your migrations for potential risks.

Summary: 0 Safe | 2 Needs Review | 0 Blocked

⚠️ Needs Review

May have performance impact

exports.0012_subscription_context
  └─ #1 ⚠️ AddField
     Adding NOT NULL field with callable default (list) - verify it's stable
     model: subscriptiondelivery, field: context_refs, default: list
  └─ #2 ✅ CreateModel
     Creating new table is safe
     model: SubscriptionContext
  └─ #3 ⚠️ AddForeignKeyNotValid: Unknown operation type: AddForeignKeyNotValid
  │
  └──> ℹ️  INFO:
       ℹ️  Skipped operations on newly created tables (empty tables
       don't cause lock contention).

exports.0013_validate_subscription_context_team_fk
  └─ #1 ⚠️ ValidateForeignKey: Unknown operation type: ValidateForeignKey

Last updated: 2026-09-08 20:02 UTC (d831fac)

✅ ClickHouse migration SQL — none

No ClickHouse migrations in the latest push.

trunk-io · 2026-08-31T16:50:07Z

_{View Full Report ↗︎ ⋅ Docs}

posthog · 2026-09-08T14:52:08Z

🦔 PostHog Review reviewed this pull request

Found 2 must fix, 6 should fix, 3 consider.

Published 11 findings (view the review).

Resolved comments: 2 already settled, 2 left for you

posthog · 2026-09-08T15:39:48Z

PostHog Review alpha 🦔 If you find any issues helpful - please reply "valid", "invalid", etc., for evaluation purposes 🙏

posthog

PostHog Review

Found 7 must fix, 14 should fix, 2 consider.

posthog

PostHog Review

Found 2 must fix, 6 should fix, 3 consider.

created_by is nullable, so select_related() joins it as a LEFT OUTER JOIN and PostgreSQL rejects an unscoped FOR UPDATE that reaches it. Every AI subscription delivery failed with NotSupportedError before report generation started. Lock only the subscription row with of=("self",), matching the existing pattern in the recording rasterizer. Add a real-database test for the resolver, which the rest of the delivery suite mocks out. Generated-By: PostHog Desktop Task-Id: 11780ac9-ccc8-44fc-91d2-32b7b582a523

The last-delivery cutoff lookup swallows database errors and degrades to the cadence window. It now runs inside the resolver's transaction, where an un-rolled-back statement error aborts every later query in the block — including the creator's access check — so the delivery failed instead of falling back. Read the row inside a savepoint so the error rolls back to it and the caller's transaction stays usable. Add a test that drives a real statement error through the lookup and asserts the resolver still completes. Generated-By: PostHog Desktop Task-Id: 11780ac9-ccc8-44fc-91d2-32b7b582a523

A dashboard whose tiles all failed reported "failed", and one truncated tile made the whole dashboard "truncated", but a mix of succeeded and failed tiles fell through to "success". The degradation rollup reads only the dashboard-level status, so losing most of a dashboard's evidence to query timeouts left the report tagged healthy. Treat any tile short of a clean success as incomplete evidence, which is what the dashboard-level "truncated" already reports for tiles dropped to fit the character budget. All-failed still reports "failed". Generated-By: PostHog Desktop Task-Id: 11780ac9-ccc8-44fc-91d2-32b7b582a523

The sanitizer cuts content to the character budget with no marker and returns a plain string, so the cut was invisible: the status check only looked for the formatter's per-cell marker, and nothing downstream could still detect it. A single oversized standalone insight was stored with status "success" while its table reached the model stopping mid-row. Sanitize uncapped, then bound the content here so an over-budget result carries the truncation marker and the truncated status. Generated-By: PostHog Desktop Task-Id: 11780ac9-ccc8-44fc-91d2-32b7b582a523

The planner may answer with zero queries only when computed context already covers the prompt. That gate tested the context block for emptiness, but a failed context still fills the block with unavailability markers, so a report whose every context failed could ship with no query data at all while its synthesis input said no supplemental queries were needed. Thread the existing has_successful_evidence signal down to the gate so an empty plan is refused unless at least one context produced a result. Generated-By: PostHog Desktop Task-Id: 11780ac9-ccc8-44fc-91d2-32b7b582a523

pr-assigner-resolver-posthog Bot assigned vdekrijger Aug 31, 2026

This was referenced Aug 31, 2026

feat(subscriptions): add AI report context picker #91788

Draft

feat(subscriptions): add AI report context contract #91428

Open

vdekrijger force-pushed the codex/ai-subscription-contexts-stack-v1 branch from eb7358a to 275ce18 Compare August 31, 2026 16:19

vdekrijger force-pushed the codex/ai-subscription-context-execution-v1 branch from 78016f1 to 9a6188d Compare August 31, 2026 16:19

vdekrijger force-pushed the codex/ai-subscription-context-execution-v1 branch 2 times, most recently from 97058bc to 8a32ee0 Compare September 1, 2026 13:01

vdekrijger added the reviewhog ($$$) Reviews pull requests before humans do label Sep 8, 2026

posthog Bot reviewed Sep 8, 2026

View reviewed changes

posthog Bot removed the reviewhog ($$$) Reviews pull requests before humans do label Sep 8, 2026

vdekrijger added 3 commits September 8, 2026 18:45

feat(subscriptions): ground AI reports in saved context

3a0e266

fix(subscriptions): persist generic report context refs

921cf3a

fix(subscriptions): harden AI report context execution

8730b54

vdekrijger force-pushed the codex/ai-subscription-context-execution-v1 branch from 8a32ee0 to 8730b54 Compare September 8, 2026 16:47

vdekrijger added the reviewhog ($$$) Reviews pull requests before humans do label Sep 8, 2026

fix(subscriptions): satisfy execution CI checks

03f17e0

posthog Bot reviewed Sep 8, 2026

View reviewed changes

posthog Bot removed the reviewhog ($$$) Reviews pull requests before humans do label Sep 8, 2026

vdekrijger and others added 8 commits September 8, 2026 20:24

chore(subscriptions): sync execution stack

2c49541

fix(subscriptions): address remaining context review findings

5596852

fix(subscriptions): support legacy insight filters

424de4d

test(subscriptions): scope query access mock

d831fac

Uh oh!

Conversation

vdekrijger commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

How did you test this code?

Automatic notifications

Docs update

🤖 Agent context

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 CI report

⚠️ Needs Review

Uh oh!

trunk-io Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

posthog Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦔 PostHog Review reviewed this pull request

Uh oh!

posthog Bot commented Sep 8, 2026

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

PostHog Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

PostHog Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vdekrijger commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

trunk-io Bot commented Aug 31, 2026 •

edited

Loading

posthog Bot commented Sep 8, 2026 •

edited

Loading