Mastermind

What it is

• A clinical intelligence platform that pairs an iOS app collecting real-time patient telemetry (voice, face, motion) with a web dashboard for researchers and clinicians to monitor, triage, and review that data.
• The iOS app ("Ember") runs an on-device voice model (Gemma 4 via the Cactus runtime) so patient interactions get low-latency responses without a network round trip, with automatic fallback to the Gemini API when cloud access is available.
• Includes a synthetic data pipeline (a small diffusion model trained in PyTorch) that generates realistic patient telemetry vectors for testing and model validation without needing real patient data.

Use cases

• Give researchers a way to monitor patient telemetry (audio, facial, motion signals) and review AI-generated clinical incident reports from a central dashboard.
• Run voice-driven clinical workflows on-device (bedside data capture, triage prompts) even without reliable internet, since the core model runs locally on the device.
• Generate synthetic telemetry data to test and validate the triage/scoring pipeline without real patient records.

Tech stack

• iOS app (Ember): Swift, on-device audio/facial/motion telemetry capture, Cactus runtime running Gemma 4 locally, Gemini API fallback for cloud inference
• Web dashboard: React (Vite), TypeScript, Tailwind CSS, shadcn/ui components, Chart.js/Recharts for telemetry visualization
• Backend API: Python, FastAPI, SQLAlchemy (async, SQLite), scikit-learn for the triage risk-scoring model
• Real-time sync: Convex (bridges the iOS app, backend, and dashboard so data shows up live)
• Synthetic data: PyTorch (custom DDPM-style diffusion model), NumPy, trained on telemetry feature vectors pulled from the same SQLite schema
• LLM: Gemini (clinical report generation, remediation proposals)

Project structure

ember-web-frontend-backend/     Web dashboard + API, the main clinical monitoring app
  src/pages/                    Dashboard views: patient monitor, triage dashboard, neuroscience profile, journals
  backend/                      FastAPI service: ingests device events and iOS incidents, runs the triage model,
                                 generates clinical reports and remediation proposals via Gemini, syncs to Convex
  convex/                       Convex schema and functions: patients, telemetry, incidents, journals, evals

voice-agents-hack/              On-device voice agent
  Ember/Ember/                  iOS app source (Swift): audio/facial/motion telemetry managers, Cactus model
                                 manager, live audio view, journal capture, dashboard views
  index.js                      Minimal Express server used for local testing/health checks during development

synthetic-telemetry-diffusion/  Synthetic data generator
  src/synthetic_telemetry/      Feature extraction, diffusion model (MLP denoiser), training loop
  scripts/                      build_dataset.py (SQLite to feature vectors), train_diffusion.py, sample_diffusion.py
  configs/default.yaml          Data, model, and training hyperparameters

mastermind_model_pseudocode.md  Design notes for the real-time triage/risk-scoring model

Setup

1. Web dashboard (ember-web-frontend-backend)

Requires Node.js and Bun.

cd ember-web-frontend-backend
npm install
npx convex dev        # sets up the Convex backend

In a separate terminal:

npm run dev

The dashboard runs at http://localhost:5173.

Backend API:

cd ember-web-frontend-backend/backend
pip install -r requirements.txt
# set GEMINI_API_KEY in .env (see .env.example)
uvicorn main:app --reload

2. Voice agent / iOS app (voice-agents-hack)

Requires Node.js and a Mac with Cactus installed.

git clone https://github.com/cactus-compute/cactus
cd cactus && source ./setup && cd ..
cactus build --python
cactus download google/functiongemma-270m-it --reconvert
cactus auth   # enter your API key from cactuscompute.com/dashboard/api-keys

Optional cloud fallback via Gemini:

pip install google-genai
export GEMINI_API_KEY="your-key"

Start the local Express server:

cd voice-agents-hack
npm install
node index.js

Open voice-agents-hack/Ember/Ember.xcodeproj in Xcode to run the iOS app on a device or simulator.

3. Synthetic telemetry diffusion (synthetic-telemetry-diffusion)

cd synthetic-telemetry-diffusion
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Smoke test with dummy data:

export PYTHONPATH=src
python scripts/build_dataset.py --dummy-rows 512
python scripts/train_diffusion.py
python scripts/sample_diffusion.py --num-samples 64

To train on real telemetry, point --db at the backend's SQLite database instead. See synthetic-telemetry-diffusion/README.md for details.

Requirements

• Node.js and Bun (web dashboard)
• Python 3.10+ (backend API and diffusion pipeline)
• Xcode and a Mac with Cactus installed (iOS app)
• API keys: Gemini (GEMINI_API_KEY), Cactus account for on-device model access