One agent over everything your team ships and says — JIRA tickets, GitHub PRs, and Slack messages, cross-linked and queryable in plain English.
A ticket closes with no PR behind it. A PR merges but its ticket stays open. A
decision lands in Slack and never becomes a ticket. No single tool sees all of
it — yoku ingests all three into MongoDB, links them (ENG-123 in a branch
name ties the PR to its ticket to the thread that argued about it), indexes
them with OpenAI embeddings, and puts a planning agent (LangChain deepagents)
on top. Ask "what shipped for the payments epic and who reviewed it?" and get
an answer with citations, each one click from its source.
- FastAPI REST API with JWT auth + multi-tenant mongo isolation (SSE streaming chat)
- React (Vite + TypeScript) frontend
- Click CLI for ingest, embed, link, health checks, user management
- Schema-driven: adding a data source is registry entries, not tool or prompt edits
┌──────────────┐ JWT ┌────────────────────┐ invoke ┌──────────────────┐
│ React UI │ ──────────▶ │ FastAPI │ ───────────▶ │ deepagent │
│ (Vite :5173)│ │ (uvicorn :8000) │ │ + 8 tools │
└──────────────┘ │ + tenant ContextVar│ │ + source registry│
└──────┬─────────────┘ └──────┬───────────┘
│ │
▼ db-per-tenant ▼
┌────────────────────────────────────────────────┐
│ MongoDB (single cluster) │
│ yoku_<tenant_id> (one db per tenant) │
│ collections: dc-jira, dc-github, dc-slack, │
│ ds-work-item, ds-pull-request, │
│ ds-conversation, ds-entity-links, │
│ ds-unified-users, chat_sessions, │
│ chat_messages, auth_users … │
└────────────────────────────────────────────────┘
yoku/
├── pyproject.toml Makefile Dockerfile .dockerignore
├── README.md vision.md CLAUDE.md .env.example .gitignore .python-version
├── .pre-commit-config.yaml
├── docs/ # reference docs (see docs/README.md)
│ └── adding-a-connector.md # step-by-step connector recipe
├── yoku/ ← installable Python package
│ ├── config.py # Pydantic Settings (JWT secret, Mongo, OpenAI, etc.)
│ ├── constants.py · exceptions.py
│ ├── logging.py # session_id ContextVar + optional Sentry
│ ├── cli.py # Click entry: `yoku …`
│ ├── main.py # FastAPI app factory + lifespan
│ ├── auth.py # JWT, current_user, password hashing
│ ├── routers/ # FastAPI route handlers
│ │ ├── auth.py # login/signup + JWT
│ │ ├── sessions.py # session CRUD
│ │ ├── chat.py # SSE streaming chat
│ │ ├── connectors.py # on-demand sync + connector config
│ │ └── stats.py # analytics
│ ├── middleware/ # rate_limit, request_context,
│ │ # request_logger, exception_handler
│ ├── schemas/ # Pydantic models = collection schemas
│ │ ├── api.py # request/response models
│ │ ├── relationships.yaml # cross-collection joins
│ │ └── jira.py · github.py · slack.py · unified.py · …
│ ├── connectors/ # 🔌 drop a new source folder here
│ │ ├── base.py # Connector contract + auto-discovery
│ │ ├── jira/ (client, ingest, users_*)
│ │ ├── github/ (client, ingest, users_ingest)
│ │ └── slack/ (client, ingest, export_ingest, users_ingest)
│ ├── db/
│ │ ├── mongo.py # tenant-aware collection accessors
│ │ ├── tenancy.py # ContextVar + db name routing
│ │ ├── sessions.py # conversation persistence
│ │ ├── unified_users.py # JIRA ↔ GitHub user join
│ │ ├── connector_configs.py # per-tenant connector credentials
│ │ ├── freshness.py # data freshness tracking
│ │ └── backfill.py # author_email backfill
│ ├── mappers/ # dc-* → canonical ds-* projections
│ ├── pipeline/
│ │ ├── embed.py · pr_to_jira.py · unify.py · entity_links.py
│ │ ├── consistency.py # JIRA/GitHub data integrity checks
│ │ ├── scheduler.py # APScheduler background sync loop
│ │ └── sync_service.py # per-tenant full pipeline orchestration
│ ├── eval/retrieval.py # retrieval quality evaluation
│ ├── agent/
│ │ ├── agent.py # deepagent factory
│ │ ├── chat.py # CLI ask/final_answer helpers
│ │ ├── prompts.py · rerank.py · usage.py
│ │ ├── sources.py # source registry — one entry per connector
│ │ ├── schema_registry.py # collection name → Pydantic model
│ │ └── tools/ # 8 tools, auto-discovered
│ └── utils/ # cross-cutting helpers
│ ├── bson.py · jira_keys.py · http.py
├── web/ # React + Vite frontend
│ ├── package.json vite.config.ts tsconfig.json
│ └── src/
│ ├── main.tsx · App.tsx · styles.css
│ ├── pages/Login.tsx pages/Chat.tsx pages/Settings.tsx
│ ├── components/AppChrome.tsx
│ └── lib/api.ts # typed fetch client
├── scripts/
│ ├── dump_schemas.py # Pydantic → JSON Schema
│ ├── generate_models.py # regenerate yoku/schemas/_generated
│ ├── agent_smoke.py # agent regression suite
│ └── retrieval_eval.py # score retrieval quality on golden set
├── tests/ # 64 test files: unit + integration + pipeline
└── schemas/ # generated JSON Schemas (gitignored)
# 1. Mongo running locally
brew services start mongodb-community
# 2. Install
poetry install
poetry run pre-commit install # one-time
# 3. Configure — required: JIRA_EMAIL, JIRA_TOKEN, JIRA_BASE_URL, JIRA_PROJECT,
# GITHUB_TOKEN, GITHUB_ORG, OPENAI_API_KEY (JWT_SECRET before any real deploy).
# Optional extra env file: export YOKU_ENV_FILE=/path/to/.env
cp .env.example .env
# 4. Pull data + index + link
make refresh-all # ~10 min on first run, idempotent
# Data commands act on tenant 'default'; override with TENANT=<id>,
# `yoku --tenant <id> …`, or export YOKU_TENANT=<id>.
# 5. Bootstrap an admin user
poetry run yoku auth create-user --email you@example.com --tenant default --admin
# 6. Run
make api # http://localhost:8000 (FastAPI + Swagger /docs)
make web-install # one-time: npm install in web/
make web # http://localhost:5173 (React UI)
make web-lint # TypeScript static checks for the React appSign in at http://localhost:5173 with the exact tenant name you created.
The API runs a background scheduler that re-runs the full refresh pipeline
(ingest → embed → unify-users → link → backfill → unify → entity-links) for
every tenant with a configured connector, using each tenant's own stored
credentials. The final steps project every source into the canonical ds-*
collections and derive ds-entity-links edges. It starts with the API and is
controlled by env:
AUTO_SYNC_ENABLED(defaulttrue) — setfalseto turn it off.SYNC_INTERVAL_MINUTES(default60) — cadence between runs.
The first run fires one interval after boot (not at startup), and runs are
non-overlapping. Auto-sync is always off under ENV=test/ci. On-demand
POST /api/connectors/{name}/sync still works regardless.
$ yoku --help
Commands:
ingest Pull source data into mongo
jira … tickets from the configured JIRA project
jira-users … JIRA user directory
github … PRs from the configured org (last 365d, all non-archived repos)
github-users … GitHub org members
slack-export … Slack workspace export (messages + users)
embed Generate embeddings for any docs with embedding=null
link Reverse-link PRs onto their referenced JIRA tickets
unify-users Build unified_users by joining JIRA + GitHub users
unify Project all sources into the canonical ds-* collections
entity-links Derive cross-source ds-entity-links edges from canonical doc refs
backfill-pr-emails Backfill author_email from unified_users
refresh-all Full pipeline: ingest → embed → unify-users → link → backfill → unify → entity-links
list-connectors Enumerate registered data connectors
status Mongo collection counts (current tenant)
doctor Health checks: mongo, secrets, connectors, freshness
consistency Report JIRA/GitHub inconsistencies (done-no-PR, merged-no-ticket)
chat <Q…> One-shot agent query
api Launch FastAPI on :8000
auth User management
create-user … Create an account in a tenant
list-users … List users in a tenant
One mongo cluster, one db per tenant:
- every tenant gets its own db:
yoku_<tenant_id> - the db is auto-created on first signup for that tenant
Tenant flow:
- Signup at
POST /api/auth/signup?tenant=<id>→ first user becomes admin. - JWT carries
sub,email,tenant_id,is_admin,exp. - Every authenticated request sets
tenancy.current_tenant_varincurrent_userdependency → everydb/mongo.py::*_collection()call routes to that tenant's db. Cross-tenant access is impossible by construction.
Passwords are bcrypt-hashed (72-byte cap enforced).
JWT secret comes from settings.jwt_secret — rotate before any non-local deploy.
Full recipe with the registry contract,
ConnectorMetafields, and gotchas:docs/adding-a-connector.md. Working on this repo with a coding agent? Start fromCLAUDE.md.
- Copy
yoku/connectors/jira/→yoku/connectors/<source>/. - Replace
client.pywith your auth/REST helpers — use@make_retry("yourname", log)fromyoku.utils. - Adapt
ingest.pyto upsert into a new mongo collection. - In
__init__.pydeclareMETA: ConnectorMeta = {...}. - Optional
users_ingest.pyfor member directories. - Add a CLI subcommand in
yoku/cli.py. - Register the collection in
yoku/db/mongo.py— rawdc-*collections inDC_COLLECTIONS; canonicalds-*collections in theALLOWED_COLLECTIONSagent whitelist. - Add a Pydantic model in
yoku/schemas/(its docstring is the collection description; fields carrydescription+display/filterablemetadata) and map it inyoku/agent/schema_registry.py::COLLECTION_MODELS. - Add a
SourceSpectoyoku/agent/sources.py::SOURCES(key shape + example) and any links toyoku/schemas/relationships.yaml.semantic_search/list_collections/describe_collection/mongo_querythen cover the source automatically — no tool or prompt edits.
yoku list-connectors will auto-discover it.
- Mongo is the source of truth. No vector DB — embeddings live as a
embeddingfield on each doc; cosine runs in-memory via numpy. - Single planning agent. One deepagent plans with
write_todosand answers over a source-agnostic toolkit (no sub-agents). - Schema-driven, nothing hardcoded. Three registries are the single sources
of truth: the Pydantic models (
yoku/schemas/) define each collection's description + fields + what'sdisplay/filterable; the source registry (agent/sources.py) holds only key-routing facts; the relationship registry (schemas/relationships.yaml) declares cross-collection joins.list_collections/describe_collection/semantic_search/mongo_queryall read from them, so onboarding a connector touches no tool, prompt, or description. - Source-agnostic toolkit.
semantic_searchhybrid-searches every embeddable source;list_collections/describe_collection(s)expose the registries (schemas, relationships, freshness);resolve_user/who_knowshandle people.mongo_query/mongo_countcover filters and analytics the narrow tools don't. The mongo tools return errors as{"error": "..."}; the narrow tools raiseValueErrorwith a clear message — either way the agent can adapt rather than crash. - Cross-source link. PRs auto-extract JIRA keys (
ENG-123) from branch / title / body intojira_keys. Reverse pass writeslinked_prsonto each JIRA ticket. - Sessions persist in mongo (
chat_sessions,chat_messages); compact history (user + final-AI per past turn) replays to the agent on follow-ups. - Tenant routing via ContextVar. Set by
current_userin FastAPI; storage helpers read it transparently. No tenant string passed through call chains. - Pydantic Settings centralizes config;
SecretStrkeeps tokens out of logs. - Tenacity retries on every external HTTP call (shared
utils/http.make_retry).
- Rotating file log at
logs/yoku.log+ stderr. - Per-session correlation: every record carries
[sid=<8-char>]. LOG_JSON=1switches the file handler to JSON-line output.SENTRY_DSN=…auto-attaches the sentry SDK (no-op if package missing).
CI runs on every push to main and every pull request via GitHub Actions.
make fmt # autoflake + ruff --fix + black
make lint # ruff + black --check
make test # pytest
make pre-commit # all pre-commit hooks
make web-lint # TypeScript static checks
make web-build # React production build
make schemas # regenerate JSON Schema docs under ./schemas/
make agent-smoke # 5-query agent regression suite (hits OpenAI)
Pre-commit hooks: autoflake, ruff, black, standard hygiene.
docker build -t yoku .
docker run --rm -p 8000:8000 --env-file .env yoku
# API on http://localhost:8000 /docs for SwaggerFor batch ingest:
docker run --rm --env-file .env yoku python -m yoku.cli refresh-allENV defaults to local, which tolerates dev conveniences. Before exposing
yoku beyond your machine, set:
ENV=prod— startup then refuses the built-in devJWT_SECRETand a missingCONNECTOR_SECRET_KEY, and closes self-serve signup (first account per tenant becomes its admin, so open signup on a public host means anyone can create a tenant; re-open deliberately withSIGNUP_ENABLED=true).- A strong
JWT_SECRETand a separateCONNECTOR_SECRET_KEY(encrypts stored connector tokens at rest — separate so one leaked secret doesn't compromise both). - Bootstrap accounts with
yoku auth create-user. - Rate limiting and login lockout are per-process in-memory — put a reverse proxy with real limits in front if you run multiple workers.
| Collection | Holds |
|---|---|
dc-jira |
JIRA tickets + embedding + linked_prs |
dc-jira-users |
JIRA user directory |
dc-github |
GitHub PRs + embedding + jira_keys + author_email |
dc-github-users |
GitHub org members |
dc-slack |
Slack messages + embedding (what the team is discussing) |
dc-slack-users |
Slack workspace user directory |
ds-work-item |
Canonical work items projected from dc-jira |
ds-pull-request |
Canonical pull requests projected from dc-github |
ds-conversation |
Canonical conversations projected from dc-slack |
ds-entity-links |
Typed cross-source edges derived from canonical refs |
ds-unified-users |
JIRA ↔ GitHub cross-walk |
chat_sessions |
One row per New Session click |
chat_messages |
Every persisted agent message |
auth_users |
Login users (per tenant, bcrypt-hashed passwords) |
JSON Schemas live in ./schemas/ after make schemas.