Add AI usage statistics with SQLite storage and Console UI - #96
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 90bb8ea3b1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return useInfiniteQuery<UsageCallPage>({ | ||
| queryKey: [QK_USAGE_CALLS, fingerprint], | ||
| queryFn: async ({ pageParam }) => { | ||
| const params = buildParams() |
There was a problem hiding this comment.
With preset ranges such as the default 30d, this recomputes from/to on every infinite-query page request. The backend cursor is bound to a hash of the exact UsageQuery, so a cursor returned by page 1 no longer matches page 2's slightly later range, causing the load-more request to be rejected as an invalid cursor. Keep a stable range for the lifetime of the calls query, or omit explicit dates and let the backend's stable defaults handle cursor paging.
Useful? React with 👍 / 👎.
| service.recordExecution(new UsageExecutionRecord(id, start.id(), start.epoch(), kind, | ||
| unitIndex, attemptIndex, startedAt, clock.instant(), status, UsageError.from(error), | ||
| start.requestModelId(), responseModelId, normalized)); |
There was a problem hiding this comment.
When the writer queue is temporarily full or unavailable for an execution event but later accepts the terminal event, this call has no way to tell the session that the execution row was dropped. The session has already added the usage locally, so finish() can persist the logical call as complete with zero missing executions even though the detailed execution evidence is absent; under load this hides per-call data loss in the call table/details. Have recordExecution report submission failure and mark the session incomplete just like a dropped start event.
Useful? React with 👍 / 👎.
e63aad5 to
62bf82e
Compare
Introduce end-to-end AI usage statistics for Halo admins. This adds runtime usage instrumentation across language, embedding, rerank, and image generation calls, persists logical calls and execution attempts in SQLite with retention and health tracking, exposes summary/trend/history/reset console APIs, regenerates the OpenAPI client, and adds a Chinese console view with filters, charts, call details, and reset flow.
62bf82e to
62e9002
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 62e9002a27
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| var callDays = rollupDays(writer, rollupBefore, | ||
| "call_rollup_frozen_watermark"); | ||
| var tokenDays = rollupDays(writer, rollupBefore, "rollup_frozen_watermark"); |
There was a problem hiding this comment.
When an execution crosses UTC midnight from its parent call into a day on which no logical call starts, tokenDays omits that day because rollupDays selects exclusively from ai_calls.started_at_ms. Consequently, rollupTokenDay is never invoked for the execution's day, so full-day summaries and trends stop reporting those tokens as soon as the rollup watermark advances, and execution retention later deletes the only remaining fact permanently. Select token rollup days from execution timestamps as well as fallback call facts.
Useful? React with 👍 / 👎.
| private static void succeedProjection(UsageCallSession session) { | ||
| if (session.hasExecutions()) { | ||
| session.succeed(NormalizedUsage.missing(), null, 0); |
There was a problem hiding this comment.
When streaming middleware returns a successful synthetic stream without invoking the provider, this guard prevents the logical session from reaching a terminal state. The repository's RagLanguageModelMiddleware.emptyContextStream does exactly this for skipModel(), and consumers commonly subscribe only to fullStream() or textStream() rather than result(), leaving these calls IN_PROGRESS until restart marks them ABANDONED. Successful completion of an execution-driving projection must terminate the session even when it legitimately has no physical executions.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc1e3a78d9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| () -> success.accept(session, result))) | ||
| .doOnError(error -> UsageTelemetry.safely( | ||
| () -> session.fail(error, NormalizedUsage.missing(), failureStepCount))) | ||
| .doOnCancel(() -> UsageTelemetry.safely(session::cancel)) |
There was a problem hiding this comment.
When a subscriber cancels an active non-streaming call, Reactor invokes this downstream doOnCancel before propagating cancellation to the delegate's UsageExecutionObserver. Consequently, session.cancel() persists the terminal call before the observer records its cancelled execution, so the call is saved with missingExecutionCount == 0 and complete == true even though that execution subsequently reports missing usage. Finalize cancellation after upstream cancellation callbacks have run so the terminal snapshot includes the execution evidence.
Useful? React with 👍 / 👎.
| : usageExecutionObserver.observe(UsageUnitKind.RERANK, 0, invocation, | ||
| response -> NormalizedUsage.from(response.getUsage()), | ||
| response -> null); |
There was a problem hiding this comment.
When a rerank provider returns RerankResponse.response.model, this mapper always supplies null to the execution observer. The logical call later snapshots the same model in AuditedRerankingModel, but the physical execution row permanently loses its available response-model identity, making execution details inconsistent with other model types and less useful for explaining provider behavior.
Useful? React with 👍 / 👎.
# Conflicts: # app/src/main/java/run/halo/aifoundation/service/language/LanguageModelImpl.java
1 participant
What this PR changes
Adds AI usage statistics for calls made through the audited model wrappers (language, embedding, reranking, image generation). Usage is recorded to a local SQLite database and surfaced through a new Console page.
Statistical model
usageQualityreports how complete the token data is (REPORTED_COMPONENTS,REPORTED_TOTAL,PARTIAL,ESTIMATED,MISSING);accountedTotalTokensis derived from the best available source.Console API
GET /apis/console.api.aifoundation.halo.run/v1alpha1/usage-statistics/summary— totals with filter dimensions and coverageGET .../usage-statistics/trends?resolution=DAY|HOUR— time-bucketed execution trendsGET .../usage-statistics/calls?page=...&size=...— paginated call list with filters (status, provider, model, caller, feature, operation, date range)GET .../usage-statistics/calls/{id}— call detail with per-execution recordsGET .../usage-statistics/health— persistence health (restore status, queue depth, dropped events)POST .../usage-statistics/reset— clear statistics dataResilience
%DATA_DIR%; daily rotation uses the same backup scheme as plugin-seo-tools.Test plan
./gradlew buildpasses (backend tests included).pnpm lint,pnpm type-check,pnpm test:unit(39 files / 245 tests) pass.