EAKS Reference Engine is an open-source reference implementation of the Enterprise Agentic Knowledge Service
(EAKS) API: a LangGraph hybrid (structured + unstructured) insight engine that turns a
natural-language question into a typed, lineage-traceable, cached answer. It is built directly on
LangGraph with no dependency on any external orchestration or data-modeling framework. The structured
path is plain validated Spark SQL with a module_key-keyed Delta result cache; the unstructured
path is embedding retrieval over policy docs.
The LLM is a semantic parser, never a query executor. It fills typed contracts; deterministic code compiles + validates the SQL, drives retrieval, and owns caching/versioning/lineage. That boundary is what makes the engine governable, auditable, and cheap to re-run.
Reference-implementation stance. It is meant to be forked and adapted, so it favors clarity and
generic building blocks over any proprietary dependency: it runs fully local and credential-free
(local Spark + Delta + Ollama), swaps to Databricks by env only, and holds its safety guarantees
(SELECT-only SQL, catalog allowlist, never-invent-a-metric, enforced LIMIT) regardless of the model.
Identity is consumed, not implemented — the engine reads whatever the platform SSO proxy forwards
(X-Forwarded-Email) and mints no tokens of its own; user_id is optional per the AKS spec (it
defaults to a single local user when no proxy is present). See Known limitations and dev-model
gaps for what the dev model (granite) does and doesn't do well.
- Blog — the design & lessons (start here): Lessons from building an Enterprise Agentic Knowledge Service — read the published article. This README is the engineering companion to that post — the section below maps its 10 lessons to code.
- Out-of-box deploy quickstart:
docs/DEPLOY.md.
This repository is self-contained — it vendors its own data/ and deploys to a fresh Databricks
workspace out of the box. Nothing outside it is required at runtime.
The companion blog distills this build into 10 lessons. This repo is those lessons in code; use the table to jump straight to the part you care about. (Same spine, same order as the post.)
| # | Lesson (from the blog) | Where it lives in this repo |
|---|---|---|
| 1 | Typed contract in, deterministic engine behind it | main_agent/contracts.py (QueryPlan · StructuredOperation); the LLM's only latitude is the single plan_query node (main_agent/planner.py) → Architecture |
| 2 | Govern in data, not prompts (catalog + taxonomy) | data/catalogs/ (metric · semantic · source) + structured_agent/sql_validator.py; taxonomy in main_agent/taxonomy.py → Data catalog and taxonomy |
| 3 | Structured · unstructured · hybrid as one engine | routes → structured_agent/ · unstructured_agent/ (RAG) · hybrid_agent/ (multi-step executor) → Taxonomy |
| 4 | Rules first, LLM only when needed (and it defers) | main_agent/rule_planner.py (rule tier, no LLM) → main_agent/planner.py (LLM fallback + follow-up rewrite) |
| 5 | "Can't answer" is a specific result — a 404, not a 500 | error classification in api/mapping.py; typed AksError in api/contracts.py |
| 6 | Caching + lineage from one content-addressed key | structured_agent/module_key.py + adapters/execution.py (res_<module_key> / Delta cache) |
| 7 | Observability: OTel spans → collector → pluggable target | observability.py (OTLP → Langfuse) + adapters/audit.py (always-on agent_audit) → Observability |
| 8 | Eval layers — test for "200-but-wrong" | tests/ (hermetic) + data/evals/ (golden + demo_expectations.json) + scripts/run_eval.py → Tests |
| 9 | Test the pixels too (if there's a UI) | Playwright harness in .claude/skills/ui-troubleshooting/ |
| 10 | The model is the lever — don't over-engineer a weak one | adapters/model.py (Ollama ⇄ Claude, an adapter swap) → Known limitations and dev-model gaps |
The blog is engine-neutral (the lessons apply on any warehouse/lakehouse); this repo is one concrete realization of them on Spark + Delta locally / Databricks in the cloud.
One question flows through a single graph. The LLM is confined to four bounded, open-ended steps (memory · follow-up rewrite · planning · grounded synthesis — shown in blue below); every decision that must be safe, reproducible, or testable — routing, SQL compile/validate/execute, retrieval, aggregation — is deterministic code. The model proposes (fills a typed contract or writes cited prose); code decides and executes. And each LLM call is skipped whenever deterministic code can do the job (single-turn → no summary; standalone question → no rewrite; rule-planner hit or plan-cache hit → no LLM plan; keyword retrieval → no embed model).
flowchart TD
Q["🗣️ NL question + conversation_id"] --> SH
SH["summarize_history<br/><small>fold older turns into a rolling summary</small>"] --> PQ
subgraph PQ["plan_query"]
direction TB
RW["① rewrite follow-up → self-contained question<br/><small>only when this turn depends on prior context</small>"]
RW --> RULE["② rule planner<br/><small>catalog aliases · group-by · region gazetteer · time window</small>"]
RULE -->|confident| POUT["typed QueryPlan"]
RULE -->|defers| LLMP["③ LLM structured plan<br/><small>fills the same typed QueryPlan when the rules can't</small>"]
LLMP --> POUT
end
PQ --> R{{"route by intent"}}
R -->|structured| SA["⚙️ structured agent<br/><small>build → compile → SELECT-only validate → execute</small>"]
R -->|unstructured| RET
R -->|hybrid / multi-step| MS["🔗 multi-step executor<br/><small>task DAG · partial results (legs reuse the nodes above)</small>"]
R -->|unclear| CL["❓ clarification<br/><small>returns the planner's question</small>"]
subgraph UA["unstructured (RAG)"]
direction LR
RET["retrieve<br/><small>embedding / keyword similarity — not the chat LLM</small>"] --> SY["grounded synthesis<br/><small>prose that cites the retrieved passages</small>"]
end
SA --> AG
SY --> AG
MS --> AG
AG["📦 aggregate<br/><small>locked facts + evidence + lineage</small>"] --> ANS["✅ typed answer + traceability"]
CL --> ANS
classDef llm fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e3a8a;
classDef det fill:#f1f5f9,stroke:#64748b,color:#0f172a;
class SH,RW,LLMP,SY llm;
class RULE,POUT,R,SA,MS,CL,RET,AG,ANS det;
Where the LLM runs, and why:
| Step | LLM? | Why |
|---|---|---|
summarize_history |
🧠 LLM — only with prior turns | Compressing free-form dialogue into a rolling summary is open-ended language work with no deterministic rule; it's a no-op on the first turn. |
| follow-up rewrite | 🧠 LLM — only for follow-ups | "what about EMEA?" only means something in context; rewriting it to a standalone question is a language task. Skipped entirely for fresh/standalone questions. |
| rule planner | ⚙️ deterministic | Most questions match known metric aliases / by <dim> / time-window patterns — resolved by code: faster, testable, and it sidesteps exactly where weak models fumble (metric extraction). |
| LLM structured plan | 🧠 LLM — only when the rules defer | The escape hatch for phrasings the rules don't cover. Fills the same typed QueryPlan, and sees schema + question only — never row data. |
| route | ⚙️ deterministic | "query vs retrieve" must be predictable and testable — never a runtime model decision. |
| structured (compile → validate → execute) | ⚙️ deterministic | SQL is compiled from the typed plan and passed through a SELECT-only + allowlist validator; a wrong query here is unsafe. The model never emits SQL and never touches rows. |
| retrieve | ⚙️ not the chat LLM | Vector similarity (embedding model) or keyword — a reproducible per-input operation; the keyword default needs no model at all. |
| grounded synthesize | 🧠 LLM | Writing prose that faithfully cites retrieved passages is generation; constrained to the evidence, with a model-free top-excerpt fallback. |
| aggregate | ⚙️ deterministic | Merges typed leg results + evidence + lineage into one answer shape — pure assembly. |
The hybrid / multi-step executor doesn't add new kinds of step — it runs the structured and unstructured legs above over a task DAG, so its LLM/deterministic split is exactly theirs.
Everything is an adapter swap (dev ⇄ deploy); the same engine code runs in all combinations, only env vars change:
| Adapter | Local (dev) | Databricks (deploy) | Code |
|---|---|---|---|
ModelAdapter |
Ollama (granite4.1:8b) |
Claude / Llama serving endpoint | adapters/model.py |
ExecutionAdapter |
warm local SparkSession + Delta |
serverless SQL warehouse + UC Delta | adapters/execution.py |
VectorStoreAdapter |
keyword / local embeddings | Databricks embeddings → UC Delta doc_embeddings |
adapters/vector_store.py |
AuditBackend |
SQLite (.cache/) |
agent_audit Delta via SQL warehouse |
api/sessions.py, adapters/audit.py |
One join key: conversation_id (== EAKS session_id == LangGraph thread_id) ties the
agent_audit table and Langfuse traces together across every step.
| Path | Role |
|---|---|
main_agent/ |
graph · planner · rule_planner · memory · taxonomy · contracts · aggregator · edges |
structured_agent/ |
operation_builder · sql_compiler · sql_validator (SELECT-only + allowlist) · module_key · boot (backbone views) |
unstructured_agent/ |
RAG worker + prompts |
hybrid_agent/ |
multi-step executor |
adapters/ |
model · execution · catalog · vector_store · embeddings · audit · dbx_auth |
api/ |
eaks.py (FastAPI /api/v1/query) · mapping.py (graph→typed response) · sessions.py (durable) · contracts.py |
ui/ |
gradio_app.py · app_ui.py (deploy entry) · css.py · eaks_client.py |
deploy/ |
api/ · ui/ · combined/ app.yaml + requirements |
scripts/ |
bootstrap · deploy · grants · setup · demo/eval runners |
data/ |
vendored fixtures: star-schema CSVs · 3 catalogs · RAG docs · golden evals |
docs/ |
DEPLOY.md (deploy quickstart) · demo-questions.md (the demo question set) |
tests/ |
286 fast tests + gated Spark/Databricks suites |
A synthetic B2B partner-marketing star schema under data/ — 5 fact + 11 dimension CSVs (~5,048
rows) + 3 catalogs + 5 RAG policy docs. Eval clock = 2026-07-19 (so "last month" = June
2026). Golden anchor: top APAC partner by claim amount, June 2026 = P001 = $183,001.25.
| Facts (measures) | Dimensions |
|---|---|
fact_sales (3,192) · fact_claims (663) · fact_campaign_performance (149) · fact_burn_rate (84) · fact_document_events (222) |
dim_date · dim_partner · dim_region (NA/EMEA/APAC/LATAM) · dim_customer · dim_campaign · dim_status · dim_document · dim_product · dim_product_category · dim_action · dim_event |
Full column-level schema is in data/catalogs/source_catalog.csv (142 columns across 16 tables, with
is_allowlisted + classification per column). The engine builds four conformed backbone views as
Spark temp views (structured_agent/boot.py): claims_enriched, sales_enriched, campaign_enriched,
burn_rate_enriched.
The catalog is the guardrail: the LLM may reference only metrics/tables that exist here; it cannot invent one. Each metric carries a deterministic SQL formula, allowed dimensions, and source tables.
| metric_id | display | formula | domain | backbone |
|---|---|---|---|---|
total_sales |
Total Sales | SUM(fact_sales.total_sales) |
Sales | sales_enriched |
claim_amount |
Claim Amount | SUM(fact_claims.claim_amount) |
Claims | claims_enriched |
claim_count |
Claim Count | COUNT(DISTINCT fact_claims.claim_id) |
Claims | claims_enriched |
burn_rate |
Burn Rate | SUM(fact_burn_rate.burn_rate) |
Finance | burn_rate_enriched |
campaign_revenue |
Campaign Revenue | SUM(fact_campaign_performance.revenue) |
Marketing | campaign_enriched |
campaign_roi |
Campaign ROI | (SUM(revenue) − SUM(spend)) / NULLIF(SUM(spend),0) |
Marketing | campaign_enriched |
proof_gap_count |
Proof-of-Performance Gap Count | SUM(fact_campaign_performance.proof_gap_count) |
Marketing Compliance | campaign_enriched |
Maps business terms + synonyms → canonical entity/dimension/document values (e.g. "reseller" → Partner,
"POP" → Proof of Performance, "low roi" → roi_tier). This is how the planner canonicalizes free text
into typed entities without hallucinating column names.
Defined in main_agent/taxonomy.py (IntentType, EntityType, deterministic default_route() —
every intent maps to exactly one route, enforced by tests). The typed chain is
QueryPlan → PlannedTask → StructuredOperation → validated SQL → ResultEnvelope.
| Route | Worker | Intents |
|---|---|---|
| structured | structured_agent |
GetMetricValue · TopNQuery · GroupByQuery · TrendQuery · CompareValues · StatusQuery · ListEntities · FilterQuery · YesNoQuery · DiscoverCatalog |
| unstructured | unstructured_agent |
DefinitionQuery · ProcessQuery · SemanticSearch |
| multi-step | multi_step_executor |
HybridQuery · MultiStepQuery |
| clarify | request_clarification |
any, when is_clear=false |
Intent definitions
| Intent | Meaning |
|---|---|
GetMetricValue |
A single scalar metric for a filter (e.g. "total sales in APAC last month"). |
TopNQuery |
Rank entities by a metric, limited to N ("top 10 partners by claim amount"). |
GroupByQuery |
A metric broken down "by <dimension>" (region, partner, campaign, …). |
TrendQuery |
A metric over time at a grain (monthly sales trend). |
CompareValues |
Compare a metric across two filters/periods (this month vs last). |
StatusQuery |
Look up a status/state field for an entity (claim approval state). |
ListEntities |
List rows of a dimension/entity ("which partners are Gold tier?"). |
FilterQuery |
Filtered row selection without a ranking/aggregate. |
YesNoQuery |
A boolean question from structured data (planner may re-route to docs). |
DiscoverCatalog |
"What metrics/datasets do we have?" — answered from the catalog, no metric value. |
DefinitionQuery |
"What is X?" — a definition grounded in the policy docs. |
ProcessQuery |
"How does X work?" — a process/procedure from the docs. |
SemanticSearch |
Free-text search over the document corpus. |
HybridQuery |
Needs BOTH a metric and document evidence in one answer. |
MultiStepQuery |
Dependent steps (a value feeds a follow-on lookup); a small task DAG. |
Entity types (10) — extracted into typed EntityMentions and bound to the catalogs:
| EntityType | Captures | Example → canonical |
|---|---|---|
Metric |
a catalog metric | "revenue from sales" → total_sales |
BusinessConcept |
a core business object/dimension | "reseller" → Partner |
TimePeriod |
absolute/relative time window | "last month" → 2026-06-01/2026-06-30 |
Region |
a geography value | "APAC" → APAC |
StatusField |
a status/state or derived band | "low ROI" → roi_tier |
Dimension |
a grouping attribute | "by category" → product_category |
EntityId |
a concrete identifier | "partner P001" → P001 |
Event |
a business event / policy topic | "funding window" → Funding Window |
Document |
a document type / policy topic | "proof of performance" → Proof of Performance |
ActionConcept |
a business action verb | "file" → Submit |
No Databricks account, no cloud, no secrets. Warm Spark + Delta over the vendored data/ fixtures,
Ollama for the model, keyword RAG, and durable sessions in a local SQLite (.cache/).
# 0. one-time setup
python -m venv .venv && .venv/bin/pip install -r requirements.txt
# Ollama running with the dev model: ollama pull granite4.1:8b (see .env.example for the name)
# 1. start the API (warm Spark + Ollama + keyword RAG + SQLite sessions) on :8000
./scripts/run_api.sh local
# smoke test:
curl -s localhost:8000/api/v1/query -H 'content-type: application/json' \
-d '{"session_id":"'"$(python -c 'import uuid;print(uuid.uuid4())')"'",
"query":"Give me the top 10 partners by claim amount in APAC last month."}' | jq .
# 2. start the UI (points at the local API) on :7860 — in a second terminal
EAKS_API_URL=http://localhost:8000 PYTHONPATH=. \
.venv/bin/python .claude/skills/ui-troubleshooting/scripts/run_local_ui.pysession_id must be a UUID — the API rejects non-UUIDs with 400 PARAMETER_INVALID (the UI mints
one per conversation and reuses it across turns).
.venv/bin/python -m pytest -q # 286 fast tests (no Spark/cloud)
# Spark-gated suite (local warm Spark path):
RUN_SPARK=1 JAVA_HOME=/opt/homebrew/opt/openjdk@17 SPARK_LOCAL_IP=127.0.0.1 \
.venv/bin/python -m pytest -qpytest.ini pins testpaths = tests so the vendored data/evals/ scripts aren't collected. Tests are
fixture-backed from data/evals/ (routing golden set, SQL-validator reject cases, expected results).
Follow TDD — a failing test first — for any change.
With Databricks credentials and an empty workspace, the scripts provision the infra and deploy both
apps. See docs/DEPLOY.md for the full quickstart; summary:
export DATABRICKS_CONFIG_PROFILE=$PROFILE # a `databricks auth login` profile
export OLAP_CATALOG=eaks OLAP_SCHEMA=runtime # optional — these are the defaults
# 1) Provision: SQL warehouse (discover/create) + catalog/schema/volume + 16 fixture tables +
# backbone views + agent_audit/agent_artifacts. Prints the OLAP_* export line — copy it.
./scripts/bootstrap_databricks.sh
# 2) Deploy the API + UI (env fills the app.yaml placeholders)
export OLAP_CATALOG=… OLAP_SCHEMA=… OLAP_WAREHOUSE_ID=… # from step 1
./scripts/deploy_databricks_app.sh api eaks-api "$PROFILE"
./scripts/grant_api_app_sp.sh eaks-api "$PROFILE" # least-privilege UC grants for its SP
API_URL=$(databricks apps get eaks-api --profile "$PROFILE" --output json \
| python3 -c 'import sys,json;print(json.load(sys.stdin)["url"])')
./scripts/deploy_databricks_app.sh ui eaks-ui "$PROFILE" "$API_URL"
UI_SP=$(databricks apps get eaks-ui --profile "$PROFILE" --output json \
| python3 -c 'import sys,json;print(json.load(sys.stdin)["service_principal_client_id"])')
databricks apps update-permissions eaks-api --profile "$PROFILE" \
--json "{\"access_control_list\":[{\"service_principal_name\":\"$UI_SP\",\"permission_level\":\"CAN_USE\"}]}"The API app runs the engine (uvicorn api.eaks:app) and needs least-privilege UC grants; the UI
app (uvicorn ui.app_ui:app) calls the remote API, mints its own SP bearer (EAKS_AUTH_MODE=databricks_sp),
ships no engine code, and needs no UC access. Runtime auth is the app's ambient service principal
(DATABRICKS_CONFIG_PROFILE=""). The first RAG query self-embeds the bundled data/docs into a
doc_embeddings table (one-time; no pre-embed step).
- A workspace + CLI profile (
databricks auth login). The principal must be able to CREATE CATALOG (or point at a writable one) and create a SQL warehouse. - Two foundation-model serving endpoints — an LLM (
DATABRICKS_LLM_ENDPOINT) and embeddings (OLAP_EMBED_ENDPOINT). These are the only things the scripts can't create. If the workspace isn't entitled to Claude, pointDATABRICKS_LLM_ENDPOINTat a model it does have (e.g.databricks-meta-llama-3-3-70b-instruct) — the planner falls back across structured-output methods, so Llama-class models work.
Langfuse is opt-in — a fresh deploy needs no secret scope. To turn it on:
- Create a Databricks secret scope holding your Langfuse keys (never commit them).
- Attach them to the app as resources named exactly
langfuse-public-keyandlangfuse-secret-key. - Deploy with the flag set:
The deploy script appends the
export LANGFUSE_ENABLED=true export LANGFUSE_BASE_URL=https://us.cloud.langfuse.com # US region (default) ./scripts/deploy_databricks_app.sh api eaks-api "$PROFILE"
LANGFUSE_BASE_URL+ the twovalueFromenv entries to the stagedapp.yamlautomatically whenLANGFUSE_ENABLED=true. Traces export via direct OTLP to${LANGFUSE_BASE_URL}/api/public/otel, keyed byconversation_id→langfuse.session.id(OpenInference LangChain instrumentation, backgroundBatchSpanProcessor— never on the request path).
| Env | Default | Purpose |
|---|---|---|
MODEL_BACKEND |
ollama |
ollama | claude | databricks |
EXECUTION_BACKEND |
local |
local (warm Spark) | databricks (serverless SQL) |
VECTOR_BACKEND |
keyword |
keyword | databricks (embeddings→Delta) |
AUDIT_BACKEND |
local |
local (SQLite) | databricks (agent_audit) | none |
DATA_DIR |
data (vendored) |
fixtures + catalogs (absolute default, cwd-independent) |
OLAP_CATALOG / OLAP_SCHEMA |
eaks / runtime |
where UC objects live |
OLAP_WAREHOUSE_ID |
auto-discover/create | SQL warehouse for execution + audit |
DATABRICKS_LLM_ENDPOINT |
databricks-claude-sonnet-4-6 |
planner serving endpoint (semantic parsing; no narrator) |
OLLAMA_MODEL |
granite4.1:8b |
local dev model name (Ollama); use a stronger model for closer-to-prod local planning |
OLAP_EMBED_ENDPOINT |
databricks-bge-large-en |
RAG embedding endpoint |
LANGFUSE_ENABLED |
false |
tracing opt-in (see above) |
LANGFUSE_BASE_URL |
https://us.cloud.langfuse.com |
Langfuse region for OTLP |
EAKS_UI_STATE_SECRET |
olap-v1-ui-chat-state |
stable gr.BrowserState secret (keep stable so refresh keeps history) |
EAKS_CLIENT_TIMEOUT |
180 |
UI→API request timeout (s); raise it for slow local-model (granite) queries |
Two sinks, both keyed by conversation_id (observability.py, adapters/audit.py):
agent_audit(Delta) — durable in-workspace trajectory, one row per plan/worker/aggregate step plus aturnrow carrying the full response body. Written via the SQL warehouse (works even from the egress-restricted App); locally it mirrors to.cache/agent_audit.jsonl(dev) / SQLite. Schema and ops queries below.- Langfuse Cloud via direct OTLP (opt-in; see above) — the rich external span tree. Join the two on
agent_audit.trace_id↔ the Langfuse trace id.
MLflow tracing was removed: from a Databricks App its trace-artifact-storage egress is blocked, which hung requests.
agent_audit(in-workspace) + Langfuse (external) cover the need without the egress dependency, and dropping it roughly halved warm latency. Non-LLM steps (SQL, retrieval, aggregate) get explicit spans; re-enter OTel context insideSend(...)fan-out or worker spans detach.
One Delta table, two kinds of row, all keyed by conversation_id (adapters/audit.py writes the
trajectory rows from the graph nodes; api/sessions.py writes the turn rows per request):
| Column | Type | Notes |
|---|---|---|
conversation_id |
STRING | The join key — == session_id == thread_id == the Langfuse sessionId. |
step_no |
BIGINT | 1 = plan · 2 = route/worker (or clarify) · 3 = aggregate. |
step |
STRING | plan · clarify · structured · unstructured · aggregate · turn. |
task_id |
STRING | Per-task id within a run (multiple on hybrid / multi-step). |
intent |
STRING | Resolved intent, e.g. TopNQuery, MetricAggregation, DocSearch. |
modality |
STRING | structured | unstructured (null on plan/aggregate rows). |
status |
STRING | ok · error · cache · unclear · clarification; turn rows carry the API status (success/…). |
detail |
STRING | Step-specific. plan → tasks+entities; worker → generated SQL or matched doc sources; turn → compact JSON {"u":user,"q":question,"a":answer,"s":status,"i":intent,"rid":request_id}. |
module_key |
STRING | Content-addressed cache/lineage key = SHA1(validated SQL + source-table versions). |
sql_hash |
STRING | Hash of the validated SQL (structured evidence ref). |
artifact_uri |
STRING | The result artifact (res_<hash> table) a cache hit re-reads. |
row_count |
BIGINT | Rows the step produced. |
cache_hit |
BOOLEAN | Whether the artifact was reused (no recompute). |
trace_id |
STRING | Links the run to its Langfuse/OTel trace. |
created_at |
TIMESTAMP | Server time (current_timestamp()). |
response_json |
STRING | turn rows only — the full API response body (bound param, stored verbatim). |
http_code |
INT | turn rows only — the HTTP status of the request. |
-- Reconstruct one full run (the trajectory), in order:
SELECT step_no, step, intent, modality, status, cache_hit, row_count, detail
FROM agent_audit
WHERE conversation_id = '…' AND step <> 'turn'
ORDER BY step_no, created_at;
-- Recent errors (last 24h), newest first:
SELECT created_at, conversation_id, step, intent, detail
FROM agent_audit
WHERE status = 'error' AND created_at > current_timestamp() - INTERVAL 24 HOURS
ORDER BY created_at DESC;
-- Cache-hit rate + volume by intent (structured steps):
SELECT intent,
count(*) AS steps,
round(100.0 * avg(CASE WHEN cache_hit THEN 1 ELSE 0 END), 1) AS cache_hit_pct,
sum(row_count) AS rows_out
FROM agent_audit
WHERE modality = 'structured'
GROUP BY intent ORDER BY steps DESC;
-- One user's sessions (what the UI left panel shows) — turn rows carry user in detail.u.
-- detail.u is the resolved user_id: the SSO email in a deployed workspace (where agent_audit lives),
-- 'analyst' for local dev with no SSO proxy, or 'anonymous' for a bare API call with no user_id.
SELECT conversation_id,
min_by(get_json_object(detail, '$.q'), created_at) AS first_question,
max(created_at) AS last_activity,
count(*) AS turns
FROM agent_audit
WHERE step = 'turn' AND get_json_object(detail, '$.u') = 'jane.doe@example.com'
GROUP BY conversation_id ORDER BY last_activity DESC;
-- Daily volume + latency-proxy (turns vs governed 404s):
SELECT date(created_at) AS day, count(*) AS turns,
sum(CASE WHEN http_code = 404 THEN 1 ELSE 0 END) AS not_found
FROM agent_audit WHERE step = 'turn'
GROUP BY date(created_at) ORDER BY day DESC;Ops helpers:
scripts/show_audit.pyreconstructs a trajectory from either backend, andscripts/dump_trajectories.pyexports rows for analysis. Locally,agent_auditis.cache/agent_audit.jsonl(trajectory) + a SQLiteturnstable (sessions) — same columns, so the queries above translate directly.
The local dev model is a convenience, not a production model. granite4.1:8b (via Ollama) exists so
the stack runs credential-free on a laptop — but it is materially weaker at structured extraction
than the production serving model (Claude Sonnet on Databricks). The deterministic guardrails hold on
either model; the gaps below are about the LLM planning quality on the small local model. All were
observed and cross-checked on 2026-07-29 (local granite vs deployed Databricks/Claude).
Why not "fix" these in the engine? They are inherent to a weak model, and the safe, deterministic layers already contain the blast radius (SELECT-only + allowlist + never-invent-a-metric + enforced
LIMIT; and the rule-planner-first path plans simple/standalone/rewritten queries with no LLM at all). Adding heuristics to paper over granite's decomposition could fight the strong model's already-correct behavior. The right lever is the model: production uses Claude; for closer-to-prod local behavior, pointOLLAMA_MODELat a stronger local model (e.g.llama3.1:8b,qwen2.5).
A single question with several parts is where the small model mis-plans. Example:
"Find the top 3 partners by claim amount in APAC last month, show those partners' total sales, and what does the claim approval process require?"
| local granite | Databricks Claude | |
|---|---|---|
| "top 3" partners leg | ❌ returns 10 — the TopN sub-task's rewritten_question drops the "3", so _limit_for falls back to the default 10 |
✅ 3 rows (limit preserved) |
| "their total sales" leg | ❌ mis-typed as a raw FilterQuery → 100 unaggregated rows, partner names repeated, no values (not grouped, not scoped to the top-3) |
✅ aggregated GetMetricValue (single total) |
| "claim approval process" leg | ✅ doc retrieved | ✅ doc retrieved |
The engine still answers all three parts and the final_result.summary does compose every executed
leg — but on granite the first two legs are low quality. A standalone "top 3 …" is honored (the
deterministic rule planner extracts the limit); the drop only happens inside the multi-part LLM split.
Claude decomposes it correctly, so this is treated as a dev-model-only limitation and left as-is.
Follow-ups ("what about EMEA?") are inherently context-dependent, so they route through the LLM. On
granite, a follow-up after a noisy/compound prior turn used to mis-extract the metric and return a
governed NO_DATA_PRODUCT 404 — even though the equivalent standalone query resolved fine. This is
mitigated (not by tuning granite):
- Rewrite → rule planner: the follow-up is first rewritten into a self-contained query, then
planned by the deterministic rule planner (which resolves metrics robustly). LLM structured
planning is only the fallback. Result: clean follow-ups (
APAC → "what about EMEA" → "and total sales there") now work end-to-end on granite. - Conservative fuzzy resolve: if the model still emits a noisy metric entity,
catalog.resolve_metric_fuzzyresolves it only when it unambiguously full-covers exactly one catalog metric's canonical name — otherwise it still refuses (never invents).
Residual: a follow-up after a genuinely ambiguous compound turn (three metrics in play) returns a
governed ENTITY_AMBIGUOUS clarification ("which metric for EMEA?") — correct behavior, not a bug.
Claude is robust across all of these.
granite on a laptop is slow, especially on compound/multi-turn questions, and can exceed the default
180 s client timeout → a 504 MODEL_TIMEOUT (framed for local: the local model is still
generating; try a simpler question or raise the timeout). Raise EAKS_CLIENT_TIMEOUT (seconds) to give
it headroom. On Databricks the model is fast; a 504 there instead means COMPUTE_WARMING_UP (cold
serverless compute on first use — retry shortly). The client picks the right framing from the
environment (EAKS_AUTH_MODE / MODEL_BACKEND).
When granite4.1:8b handles the planner prompt directly, it can return a schema-valid QueryPlan with
an empty entities list even when the question names recognizable business entities. The opt-in live
smoke test (RUN_OLLAMA=1 OLLAMA_MODEL=granite4.1:8b pytest tests/test_live_model.py -q) reproduced
this for "Top 10 partners by claim amount in APAC last month" in 5 of 5 runs on 2026-07-31. The
failure is model extraction quality, not structured-output parsing: the response validates as the
expected typed plan, but omits the entities required by the test.
This does not mean every request fails under the Ollama configuration. The production request path runs the deterministic rule planner first, so supported standalone questions—including the example above—can bypass LLM planning and work end-to-end. Queries that fall through to direct Granite planning remain exposed to this gap. Model output can vary by model build, Ollama version, hardware, and sampling, so the 5/5 result documents the tested environment rather than claiming universal failure.
The safety/quality invariants hold on any model: no invented metrics/tables/columns, SELECT-only
validated SQL, enforced LIMIT, deterministic caching/lineage, and correct results for simple,
single-metric, and standalone questions (which the rule planner handles deterministically, often with
no LLM call). Structured parsing quality on compound questions is the part that scales with model
strength.
- The LLM never writes SQL and never sees row data. It fills a typed
StructuredOperation; a deterministic compiler emits Spark SQL; a SELECT-only validator + table/column allowlist (= backbone views) gates it; an enforcedLIMITcaps rows. Free-form Python codegen is deferred behind the same gate. RBAC is Unity Catalog pass-through — the engine does not manage roles. - Caching/lineage =
module_key(SHA-1 of the validated SQL + source table versions) over plain Delta. A repeat question is a cache hit. The persisted module output is the Delta result artifact — don't double-write. - Identity & durable sessions. The UI resolves the caller's identity per request from the SSO
proxy's
X-Forwarded-*headers (_effective_user), threads it asuser_id, and lists/records sessions under it — so each user sees only their own history (no shared bucket). Locally, with no proxy, everything keys under the single defaultUSER_ID(analyst); a bare API call with nouser_idrecords underanonymous(the spec default). Session merge (_merge_server_sessions) drops chats deleted server-side but is conservative on an empty server list — it never wipes the browser's localStorage history on a plain refresh (only reconciles against a real list). - Multi-turn follow-ups are made robust on weak models by rewriting an elliptical follow-up
("what about EMEA?") into a self-contained query first, then planning it through the deterministic
rule planner; a conservative fuzzy metric-resolve (
catalog.resolve_metric_fuzzy, unique full-cover only) backstops the LLM without ever inventing a metric. See the known-gaps section. - Long-lived app auth: both the model adapter (
adapters/model.py) and the embedder (adapters/embeddings.py) mint an OAuth bearer with a 30-min TTL + retry-on-auth-error. Without this, a warm app reuses an expired token after ~1h idle and every call fails. Preserve this when touching either adapter. - A successful 0-row result is not "no answer":
api/mapping.py::_answer_summaryreturns an explicit "No matching data found…" message so the UI panel is never blank. Keep that branch. - Planner robustness:
plan_querysurfaces real platform errors (403/connection/timeout) instead of masking them as a bogus "rephrase"422;invoke_structuredfalls back across structured-output methods (default → json_schema → function_calling → json_mode) for models that don't support the default. - Self-contained: core modules default
DATA_DIRto an absolute path (Path(__file__).resolve().parents[1] / "data"), so the engine finds fixtures regardless of cwd. The deploy script bundlesdata/catalogs+data/docsinto the app.
The unstructured/RAG path started from Agentic RAG for Dummies by Giovanni Pasquariello — this engine was seeded from that baseline LangGraph RAG project and then wrapped as one modality of this governed, multi-modal engine. Grateful for that clear, well-structured starting point; it's also a great map of the territory if you're going deeper on the RAG leg.