A standalone FastAPI web app for browsing and editing the ARI autoimmune‑disease catalogue, stored as a single Protégé‑compatible OWL ontology. Editors sign in with their own GitHub account; every saved change is committed and opened as a pull request under their identity, so contributions are attributed on GitHub. Deployable on AWS Lightsail behind nginx.
Repository move: this app was originally developed inside the main
KrishnaTO/ARIrepository under themetadata-manager/area. It now lives in this standalone repository,KrishnaTO/ARI-metadata-manager, so paths and setup commands in this README are relative to this repository root.
This repository contains the metadata-manager app only: the FastAPI backend, vanilla-JS
frontend, ontology seed/import tooling, and deployment assets. It no longer includes the
full ARI monorepo history or sibling data/ and notebook/ areas from
KrishnaTO/ARI. The local ontology used by the app is
kept under ontologies/.
ARI-metadata-manager/
├── run.py # Local launcher: uvicorn on 127.0.0.1:8001, opens browser
├── requirements.txt # owlready2, fastapi, uvicorn, httpx, itsdangerous, openpyxl
├── requirements-dev.txt # Dev/CI tooling only: pytest, ruff
├── ruff.toml # Lint config
├── instructions.md # Original product brief for the app
├── changelog.md # Human-facing change log
├── .env.example # Config template (copy to .env; secrets stay server-side)
├── DEPLOY.md # AWS Lightsail + nginx + Cloudflare SSL deployment guide
│
├── app/ # ── Backend (FastAPI) ──
│ ├── main.py # App assembly: middleware, health, error handlers, routers
│ ├── config.py # Every setting read at startup (env vars, paths, versions)
│ ├── sessions.py # Server-side token store; who the caller is and may do
│ ├── workspace.py # Per-curator working copies: branch, edits, sweep, expiry
│ ├── stores.py # The assignment, id-provenance + removal ledger singletons
│ ├── routes/ # Endpoints, grouped by the page they serve:
│ │ ├── ontology.py # Disease records: trees, indexes, edits, releases
│ │ ├── review.py # /ref-edits matrix: xrefs, mappings, predictions, session
│ │ ├── feedback.py # Per-term curator commentary
│ │ ├── auth.py # GitHub OAuth round trip, identity, open PRs
│ │ ├── publish.py # Commit + pull request, replay guard, rollback
│ │ ├── settings.py # Source/PR branch selection, fetch, .xlsx export
│ │ ├── assignments.py # Review queue
│ │ └── pages.py # The two HTML pages (asset-token injection)
│ ├── ontology_service.py # owlready2 read/edit layer (one World per working copy)
│ ├── schema.py # Editable data-item field schema (drives forms + writes)
│ ├── xref_registry.py # Single source of truth for cross-reference databases
│ ├── github_service.py # Per-user OAuth, commit, fork, cross-repo pull request
│ ├── merge_service.py # Rebases a curator's diseases onto the source branch
│ ├── sssom_service.py # Confirmed cross-refs -> SSSOM + equivalencies TSV
│ ├── id_provenance.py # Who added each cross-reference id (separation of duties)
│ ├── xref_removals.py # Ids edited off a record, pending as negative judgments
│ ├── enrich_service.py # Confirmed cross-refs -> disease synonyms + clinical subtypes
│ ├── diff_service.py # Human-readable change summary for PR bodies
│ ├── export_service.py # Export ontology -> 1_Core_ARI_Diseases.xlsx (marks changes)
│ └── feedback_service.py # File-backed per-term feedback log
│
├── scripts/ # ── Data builders ──
│ ├── build_t1d_ontology.py # Generate the seed T1D ontology from scratch
│ ├── import_reports.py # Fold data/4-reports/ catalogue into the ontology
│ └── backfill_id_authors.py # Seed the id-authorship ledger from the curated mappings
│
├── data/2-databases/ # Reference-database indexes (built by fetch_databases.py)
│ ├── <db>.index.tsv # term -> label, synonyms, cross-referenced ids
│ ├── <db>.details.tsv # term -> definition, parent labels (on-demand lookup)
│ └── <db>.subtypes.tsv # direct is_a parent->child edge ids (OBO sources)
├── tests/ # pytest suite for the service layer
├── mappings/ # Accumulated cross-reference judgments (merged into PRs)
│ ├── ari.sssom.tsv # SSSOM exactMatch mappings
│ └── ari.equivalencies.tsv # biomappings-style equivalencies
├── .github/workflows/ci.yml # CI: pytest + advisory ruff lint
│
├── static/ # ── Frontend (vanilla JS, no build step) ──
│ ├── index.html # Main page skeleton (two-column record shell)
│ ├── css/styles.css # All main-page styles (light/dark); ref-edits has its own
│ ├── js/ # Classic scripts, loaded in order:
│ │ ├── core.js # state, constants, API helper, BASE_PATH detection
│ │ ├── trees.js # left nav: alphabetical / tissue trees, search
│ │ ├── detail.js # record view: header, reading column, story spine, sidebar
│ │ ├── panels.js # deep-dive read views (inline card / drawer)
│ │ ├── graph.js # D3 force-directed pathophysiology graph
│ │ ├── editor.js # edit mode, field/item editors, admin releases
│ │ ├── symptoms.js # "search by symptoms" multi-select board
│ │ ├── feedback.js # per-term feedback panel
│ │ ├── github.js # sign-in + publish control
│ │ ├── settings.js # fetch-from-branch, switch source, PR target
│ │ └── main.js # bootstrap
│ └── ref-edits/ # Cross-reference review subpage (matrix)
│ ├── index.html
│ └── ref-edits.js # diseases x databases grid, side-panel review, SSSOM publish
│
├── deploy/ # ── Hosting (systemd + nginx) ──
│ ├── ari-mm.service # uvicorn service (runs as ariapp on :8001)
│ ├── ari-mm-update.service # oneshot wrapper for update.sh
│ ├── ari-mm-update.timer # every 10 min: pull app branch, restart only if changed
│ ├── update.sh # git reset --hard origin/<branch>; restart on change
│ ├── ari-mm-ontology-update.service # oneshot wrapper for update-ontology.sh
│ ├── ari-mm-ontology-update.timer # every 10 min: refresh ontology from the ARI repo
│ ├── update-ontology.sh # fetch ontology file from GitHub; restart only if changed
│ └── nginx.conf # reverse proxy; strips the /ari-editor prefix
│
├── ontologies/ari_t1d.owl # The ontology data file (RDF/XML, Protégé-compatible)
├── releases/ # Versioned OWL snapshots (gitignored)
├── feedback/ # Runtime feedback log (gitignored)
├── .user-data/ # Per-user working copies (gitignored, auto-swept)
└── .sessions.json # Server-side session store (gitignored, chmod 600)
The main page is a two-column record view in the AurInt design system: a 288px index rail
and one record column, sized against a 1440px reference. The rail holds the A–Z and Tissue
trees and scrolls independently of the record. The record is a fixed header
(breadcrumb, title, Copy link · Edit record) over a body split into a
reading column — definition, citations, and the disease story spine — and a 340px sidebar
carrying synonyms, target tissue, the cross-reference ledger, clinical subtypes and the record
links — each a block whose section label collapses it (expanded by default; a collapsed block
is remembered under ari_side_collapsed).
The story spine is one column per numbered STORY_GROUPS step (01–05); the category lines
inside a column are the click targets. Picking one expands its deep dive as a card inline
under the spine (below 1024px, as a drawer over the record with a scrim). The card is the
#right-col element from the shell, re-parented into the reading column on every render, so
panels.js and editor.js keep rendering into #right-panel-content unchanged.
The record header's Edit record button (#edit-toggle) is the single way in and out of
edit mode; it reads Done editing while editing, and entering edit mode opens the
disease-field form in the deep-dive slot. It is enabled only once a record is loaded and
a curator is signed in, matching the server-side write gate in service_for() — on a
deployment with GitHub integration switched off there is no identity to demand, so a loaded
record is the only requirement.
Leaving the form — Cancel, Done editing, dismissing the drawer, or opening a category
from the story — routes through the same snapshot comparison (fieldsDirty()), so nothing
unsaved is discarded without a prompt. A category panel opened while curating carries a
Disease fields button back to the form, and its Close returns there rather than
dropping out of edit mode.
Design tokens (colour, type, spacing, radius, shadow) live in :root in
static/css/styles.css, with a dark set under [data-theme="dark"]. Type is Barlow Condensed
(wordmark), IBM Plex Sans (everything), IBM Plex Mono (ids, codes, counts) and IBM Plex Serif
(blockquotes). There are no emoji anywhere in the UI; icons are inline Lucide SVGs, because
the page pins its third-party scripts with SRI and cannot add an icon CDN.
Sign-in uses the GitHub OAuth Authorization-Code flow. The access token is held
server-side only (in the session store); the browser keeps just an opaque session id.
On Publish, the app commits the edited OWL on a branch named after the disease and opens
a PR authored by the signed-in user, so GitHub attributes the contribution to them.
Contributors without push access are handled by forking: the app creates a fork, commits
there, and opens a cross-repo PR with maintainer_can_modify. Re-publishing the same disease
appends commits to the existing PR. The only persistent secret is the OAuth client secret,
which never leaves the server. app/github_service.py owns this logic.
A working copy is made once and lives for days, so committing it whole reverted every record
merged into the branch since — the incident in issue #146. Publishing now fetches
GH_ONTOLOGY_PATH at the source branch, writes only the diseases in this curator's touched
set over it (app/merge_service.py), and commits that; the mapping files were always built
this way, which is why no mapping row was ever lost. The touched set lives at
.user-data/<login>.touched.json so a restart cannot empty it, and it is cleared on a
successful publish. A disease the curator edited that also changed on the branch fails the
publish with 409 naming it, rather than silently winning.
Each signed-in editor edits an isolated copy of the ontology at .user-data/<login>.owl
(its own owlready2 World), so one editor's unpublished changes never leak into another's view
or into the shared baseline. A startup background task sweeps idle copies older than
USER_DATA_TTL_DAYS (default 14) to bound disk use. Defined in app/config.py,
swept by app/workspace.py.
The ref-edits subpage lays out every disease against its database cross-references
(SNOMED, OMOP, DOID, UMLS, MONDO, ICD-10, MeSH, NCI). A curator reviews each id in a
resizable side panel and marks it correct or needs-change; empty cells link out to the target
database's search, and can be marked ∅ Not in DB when that database has no term for the
disease at all. Confirmed matches become skos:exactMatch rows in an SSSOM TSV plus a
simpler biomappings-style equivalencies TSV — both merged idempotently under mappings/
and included in the PR; an absence is recorded against SSSOM's NoTermFound object with the
database named in object_source. Built by app/sssom_service.py + static/ref-edits/.
The page's toolbar carries only what a review turn needs — the filter, the queue scope, review
progress, the pending and ⚗ enrichment drawers, and Publish (its ▾ opens the tracked PR or
starts a fresh one). Set-and-forget options live behind ⚙: theme, row density, and whether
the glyph legend shows. All three persist in localStorage per browser.
Review is two-person: whoever adds a cross-reference id may not also confirm the mapping
it stands for, so a second curator always vouches for every match. Every edit that introduces
an id credits its author in a ledger (app/id_provenance.py, served by GET /api/v2/id-authors); the page then withholds the ✓ from that curator — flagging the id or
declaring the database empty stays open to everyone — and POST /api/v2/publish re-checks
authorship, so the rule holds regardless of what the client sends.
Removing an id is a judgment too, and until recently only the review page made one. Editing a
cross-reference out of the disease record wrote the ontology and nothing else, so a curated id
could leave the registry with no decision behind it — which the data repo rejects on publish.
Both write paths now park what they take away (app/xref_removals.py), and the next publish
folds those removals into the same flagged list the review page fills, so one code path
produces the negative SSSOM row either way. An id put back before publishing is not a
judgment and leaves the ledger; a malformed one never enters it, because the published mapping
set rejects a row carrying an id of the wrong shape.
The ledger only sees edits made through the app, so ids curated before it existed are seeded
from the curator named on each accumulated mapping — SSSOM author_id, equivalencies
source — once per deployment:
python scripts/backfill_id_authors.pyIt reads the mappings and the ontology from the source ARI repo (KrishnaTO/ARI@main),
where publish accumulates them; this repo's tracked copies lag behind. --local reads the
working tree instead. Positive rows only — a negative names who flagged a mapping, not who
added the id — github: authors only, and a mapping is recorded only while its id is still on
file for that disease and database. Both files are read because neither is a superset of the
other; a mapping in both is recorded once. Re-running is safe: an id keeps its first author. The set of
databases (labels, CURIE prefixes, and link-out/search URL templates) lives in one place,
app/xref_registry.py, which both frontend pages fetch via GET /api/v2/xref-databases, so a
database is added or changed once instead of in four hand-synced spots.
For signed-in users, review progress is saved to the server as they work (verdicts,
edited-id markers and the open-PR pointer, via GET/PUT /api/v2/ref-session), so a page
reload resumes where they left off. The session is stored per user beside their working
ontology copy and is dropped when they switch source branch. Once a pull request exists,
Publish commits to that same PR, while a New PR button opens a fresh one instead.
A blank cell is not left for the curator to research from nothing: app/predict_service.py
proposes candidate ids from the data/2-databases/<db>.index.tsv files, surfaced as yellow
cells and written to mappings/ari.predicted.sssom.tsv with a semapv:LexicalMatching
justification. Nothing is ever written to the ontology by prediction — a curator confirms
every one. Four routes propose candidates, in descending order of how much they are worth:
| Route | How the candidate was found |
|---|---|
xref |
An id already on file for this disease cross-references it. One confirmed MONDO id carries SNOMED, DOID, NCI, ICD-10, Orphanet, UMLS and MeSH at once. |
label |
The disease's own label is exactly a name of the term. |
synonym |
Only one of the disease's synonyms matched; used only when the label matched nothing. |
fuzzy |
Nothing matched exactly, and the term's name merely shares most of the label's words ("Adult onset Still's disease" → "adult-onset Still disease"). |
The label is the disease's identity anchor: when it matches exactly, only those terms are
used, because ARI synonym lists sometimes name an associated condition rather than a
variant (filtered further by mappings/ari.synonym_blocklist.tsv). Each weaker route runs
only when every stronger one came up empty, so fuzzy never competes with a real match.
Every candidate carries a 0–100 score and a strong/fair/weak band, so the easy
confirmations sort to the top of a column: the route sets the floor, independent indexes
agreeing raises it, and the candidate's own label being the disease's label raises it most.
A fuzzy candidate scores below a synonym even at perfect overlap and always lands in
weak — it is inference from a string, not an assertion by anyone.
FUZZY_THRESHOLD (the minimum word overlap) is derived, not guessed. scripts/eval_fuzzy.py
replays every curator-confirmed mapping in mappings/ari.sssom.tsv at a range of thresholds
and reports what each recovers against the candidates it costs a curator to read; re-run it
when the confirmed corpus grows.
python scripts/eval_fuzzy.pyNames sharing no words at all ("Kawasaki disease" for "Acute febrile mucocutaneous lymph node syndrome") are out of reach of any lexical route and need a curator or ARI's own synonym list.
When it runs. There is no batch job. GET /api/v2/predictions computes predictions inside
the request that serves the review page, against the caller's own working copy. Two in-memory
caches keep that cheap: the parsed index files sit in a module-level cache shared by the whole
process (keyed on each file's mtime and size), and each OntologyService caches its cell list
against the ontology file's mtime. So a repeat page load is a cache hit (~0.1 ms); a load
after that curator edited, published or fetched recomputes (~40 ms); and the first load
after a restart pays the cold start (~2.2 s) to parse the indexes. Merging a PR deploys
code — the systemd timers in deploy/ pull app code, ontology and mappings every 10 minutes
— it does not run predictions.
What it costs. ~250 MB resident process-wide (index files, plus the lazily built reverse-id and word-token maps), and ~6.5 MB per concurrently active curator for their working copy and cached cells. A 1 GB instance is comfortable; 512 MB is not.
Confirming a cross-reference asserts that the external term is the disease, so two of that
term's facts can be folded back into the ARI record: its label and exact synonyms extend
ARI_Synonym, and its direct children extend ARI_ClinicalSubtype. app/enrich_service.py
computes both; OntologyService.apply_enrichment writes them. Only confirmed (positive)
mappings feed it — flagged ones never do — and additions are de-duplicated against existing
values and filtered by the predictor's synonym blocklist.
Many cross-references are coarser than a disease concept: one ICD-10 code is cross-referenced by up to 134 DOID terms. An id matching more than one term within a single source therefore identifies nothing there and is ignored for that source, while an id pinning exactly one term in each of several sources is kept from all of them — that is agreement, not ambiguity. A disease is never proposed as its own subtype.
Synonyms come from the same data/2-databases/<db>.index.tsv files the predictor uses. Subtypes
come from companion <db>.subtypes.tsv files: one row per direct is_a parent→child edge, built
by scripts/fetch_databases.py for the OBO sources (MONDO, DOID, NCIt). The edge carries ids
only — each child's name is read from that child's own index row, so the two files cannot
disagree about a label, and a child the indexes do not know is skipped rather than proposed under
a name nothing else in the app recognises. That file is the
details.tsv sidecar's hierarchy read the other way round — the sidecar answers "what is this
term a kind of?" for one term and stores parent labels for display, while the engine asks the
inverse across the whole ontology and needs ids, since a label does not identify a term.
Nothing is written implicitly. The ref-edits page's ⚗ enrichment chip previews the additions
(POST /api/v2/enrichment-preview, read-only) and arms an "Apply on publish" checkbox; only then
does publish send apply_enrichment, which applies them and notes them in the PR body. The
server recomputes the additions from the confirmed list at publish time, so the preview can never
go stale.
scripts/import_reports.py folds the curated data/4-reports/ catalogue (diseases, symptoms,
age-of-onset, prevalence, clinical subtypes, authorship, and all cross-references) into
ontologies/ari_t1d.owl. The import is additive and idempotent; the proposed-disease (2_*)
and proposed-change (3_*) reports are intentionally skipped — only the confirmed catalogue
is imported.
The manager reports a git-derived version, 2.<commit-count> (<sha>, <date>), that bumps on
every deployed commit; it is shown in the UI and stamped onto releases.
Copy .env.example to .env (gitignored, chmod 600). Secrets are server-side only and are
never sent to the browser.
| Key | Purpose |
|---|---|
GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET |
OAuth App credentials |
GITHUB_OWNER / GITHUB_REPO |
Repo the app reads ontology data from and writes ontology PRs to |
GITHUB_BASE_BRANCH |
Default PR target and the branch the ontology refresh tracks |
GITHUB_ONTOLOGY_PATH |
Path to the ontology file within that repo (default ontologies/ari_t1d.owl) |
GITHUB_SERVICE_TOKEN |
Optional server-side token for unattended ontology refreshes (see deploy/update-ontology.sh) |
APP_BASE_URL |
Public URL incl. subpath; must match the OAuth callback |
OAUTH_CALLBACK_PATH |
/auth/github/callback |
APP_REPO_BRANCH |
Branch of this app repo that deploy/update.sh tracks |
SESSION_SECRET |
Signs the session cookie (openssl rand -hex 32) |
ALLOWED_LOGINS |
Optional allow-list of GitHub logins (empty = any user with repo access) |
ASSIGN_ADMINS |
Optional allow-list of logins who may queue work for other curators (empty = anyone signed in); filling your own queue is never gated |
PORT |
Default 8001 |
USER_DATA_TTL_DAYS |
Idle per-user copy retention in days (0 = never sweep) |
pip install -r requirements.txt
python run.py # serves http://127.0.0.1:8001 and opens the browserGitHub sign-in needs a .env with OAuth credentials and an OAuth App whose callback is
http://localhost:8001/auth/github/callback; without it the app still browses anonymously.
Custom port / ontology: python run.py --port 8002 --file path/to.owl.
pip install -r requirements-dev.txt # adds pytest + ruff
pytest -q # service-layer suite under tests/
ruff check . # lint (config in ruff.toml)CI (.github/workflows/ci.yml) runs a blocking smoke gate — install, byte-compile, import
(which loads the ontology), then pytest — on every push to main and every PR. A separate
ruff lint job is advisory for now (--exit-zero) while the existing code is brought up to the
ruleset.
See DEPLOY.md: Ubuntu 22.04 Lightsail, uvicorn under systemd (ari-mm), nginx reverse
proxy serving the app at /ari-editor, and Cloudflare free SSL. Two systemd timers keep the
box current: one tracks this app repo's APP_REPO_BRANCH (code), the other refreshes the
ontology file from GITHUB_BASE_BRANCH of the ARI data repo — each restarts the service only
when its target actually changes.
| Method | Path | Purpose |
|---|---|---|
| GET | /api/v2/overview |
Counts + current version |
| GET | /api/v2/diseases |
Flat disease list |
| GET | /api/v2/tree/alphabetical |
Parent→child disease tree |
| GET | /api/v2/tree/tissue |
UBERON tissue tree with diseases attached |
| GET | /api/v2/symptoms |
Flat symptom index |
| GET | /api/v2/schema |
Field schema for every editable data-item category |
| GET | /api/v2/tissues |
Tissue-target individuals (for new-disease forms) |
| GET | /api/v2/disease/{iri} |
Full disease detail |
| POST | /api/v2/disease |
Create a new disease individual |
| PUT | /api/v2/disease/{iri} |
Edit disease fields (appends changelog) |
| POST | /api/v2/disease/{iri}/item |
Add a data item |
| PUT | /api/v2/item/{iri} |
Edit a data item |
| DELETE | /api/v2/item/{iri} |
Delete a data item |
| GET | /api/v2/releases |
Current version + release history |
| POST | /api/v2/releases |
Cut a versioned release |
| GET | /api/v2/xrefs |
Cross-reference matrix for the review page |
| GET | /api/v2/xref-databases |
Cross-reference database registry (labels, prefixes, link-outs) |
| GET | /api/v2/mappings |
Already-curated positive/negative/absent cross-reference judgments |
| GET | /api/v2/id-authors |
Which curator added each cross-reference id (a curator may not confirm their own) |
| GET | /api/v2/concept/{db}/{id} |
Label, synonyms, definition & parents for one target-database id (compare pane); distinguishes a database's own term from a hub cross-reference |
| GET / PUT | /api/v2/ref-session |
Signed-in user's saved cross-reference review session (resume across reloads) |
| GET / POST / DELETE | /api/v2/assignments |
List / assign / unassign curator disease queues; login defaults to the caller (self-assignment is ungated), and a disease another curator holds needs reassign |
| POST | /api/v2/assignments/done |
Mark one assigned disease finished (or reopen it) |
| GET | /api/v2/search?q= |
Full-text search |
| GET / POST / PUT / DELETE | /api/v2/feedback[/{id}] |
Per-term feedback CRUD |
| GET | /api/v2/open-prs |
Open pull requests, matched to diseases by branch name |
| GET | /api/v2/me |
Current user + github_enabled |
| GET | /auth/github |
Start OAuth (optional next) |
| GET | /auth/github/callback |
OAuth callback |
| POST | /api/v2/logout |
Sign out |
| POST | /api/v2/publish |
Commit the curator's diseases onto the source branch + open/append PR (+ SSSOM files) |
| GET | /api/v2/settings |
Source branch, PR target, allowed branches |
| POST | /api/v2/fetch |
Pull latest from the source branch |
| POST | /api/v2/source |
Switch the source branch |
| POST | /api/v2/pr-base |
Set the PR target branch |
| GET | /api/v2/export |
Download current state as 1_Core_ARI_Diseases.xlsx |
App data derives from the ARI catalogue sources previously maintained in
KrishnaTO/ARI and from the generated local OWL file in
ontologies/. No online data sources are pulled into the content at runtime —
external-database identifiers are rendered as link-outs only.