RecordNote

RecordNote is a self-hosted audio processing API that turns spoken recordings into structured, searchable, and translatable knowledge. Upload an audio file and get back a speaker-diarized transcript, an AI-generated summary, a 200-language translation, and a visual speaker timeline — all running locally.

Features

Capability	Details
Speech-to-text	Powered by faster-whisper (CTranslate2-quantised Whisper). Supports 99 languages.
Speaker diarization	Identifies who spoke using Resemblyzer voice embeddings + agglomerative clustering. No fixed speaker count required.
AI noise reduction	Applies highpass/lowpass bandpass filter + ffmpeg `afftdn` before transcription to reduce background noise.
LLM summarisation	Chunk-and-summarise pipeline with 7 providers: Ollama, OpenAI, Claude, Gemini, Groq, and llama.cpp (GGUF).
Transcript correction	LLM-powered context-aware ASR error correction (`app/utils/fix_transcribe.py`).
Translation	200+ languages via NLLB-200 (CTranslate2 int8, no API key needed).
Timeline visualisation	Generates a horizontal speaker-timeline PNG chart via Matplotlib.
Export formats	JSON (diarized + timestamped), plain TXT, SRT subtitles, HTML summary report.
Chunked upload	Resumable multi-part uploads for large files; re-upload only missing chunks after interruption.
Webhooks	Optional HMAC-signed POST callback on transcription completion.
Processing estimates	`/api/v1/estimate` returns per-step and total time estimates before you submit a job.
Three auth methods	Google OAuth, email/password (JWT), or static API tokens for self-hosted use.
Fully offline	Every AI component runs locally — no cloud calls required unless you choose a cloud LLM provider.

Architecture

RecordNote/
├── main.py                      # FastAPI bootstrap: lifespan, middleware, router registration
├── setup_models.py
├── app/
│   ├── auth.py                  # SQLite-backed auth: JWT, bcrypt, password reset
│   ├── api/
│   │   ├── dependencies/        # DI providers for services
│   │   ├── config.py            # Runtime config endpoints
│   │   └── routes/              # Domain routers (auth/upload/transcribe/...)
│   ├── services/                # Business logic per domain
│   ├── core/                    # Config, logging, rate limit, constants, shared exceptions
│   ├── utils/                   # Shared helpers and model/runtime utilities
│   ├── tools/                   # Conversion and visualization tooling
│   └── schemas/                 # Pydantic request models
└── Data/                        # Runtime job artifacts

Requirements

Python 3.10 or later
FFmpeg (must be on your PATH)
All Python packages listed in requirements.txt
For GPU acceleration: CUDA-capable GPU and matching PyTorch/CUDA drivers (optional)

Installation

1. Clone the repository

git clone https://github.com/Hadi-Univ/RecordNote.git
cd RecordNote

2. Create and activate a virtual environment

# Linux / macOS
python -m venv .venv
source .venv/bin/activate

# Windows
python -m venv .venv
.venv\Scripts\activate

3. Install Python dependencies

pip install -r requirements.txt

4. Install FFmpeg

Platform	Command
Ubuntu / Debian	`sudo apt install ffmpeg`
macOS (Homebrew)	`brew install ffmpeg`
Windows	Download from ffmpeg.org and add to `PATH`

Model Setup

RecordNote ships with a dedicated setup script that downloads every AI model it needs.

# Interactive wizard — choose exactly which models to install
python setup_models.py

# Non-interactive — downloads sensible defaults based on your GPU
python setup_models.py --all

# List every available model without downloading anything
python setup_models.py --list

# Force re-download to refresh existing models
python setup_models.py --update

The script will:

Detect your GPU (NVIDIA / Apple Silicon / AMD) and recommend appropriate model sizes
Download faster-whisper (tiny → large-v3) from HuggingFace
Convert NLLB-200 to CTranslate2 int8 format for fast CPU translation
Download GGUF models for llama.cpp (Qwen 2.5, Llama 3.2, Mistral, Phi, Gemma)
Pull Ollama models via ollama pull
Skip models that are already present (resumable)
Print an installation summary with exact file paths and .env hints

Minimum setup (CPU-only): run python setup_models.py --all — it selects a base Whisper model, NLLB-200 600M, and a 3B GGUF model that comfortably fit in 8 GB of RAM.

Configuration

Copy .env.example to .env and fill in the relevant values:

cp .env.example .env

Required

Variable	Description
`JWT_SECRET`	Long random string used to sign JWT tokens. Must be changed before deploying. Generate one with: `python -c "import secrets; print(secrets.token_hex(32))"`

Google OAuth (optional)

Variable	Description
`GOOGLE_CLIENT_ID`	Your Google OAuth 2.0 client ID. Required for Google Sign-In.

Email / Password Auth

Variable	Default	Description
`AUTH_DB_PATH`	`auth.db`	SQLite database file path
`JWT_EXPIRES_HOURS`	`24`	Login session duration in hours
`JWT_ALGORITHM`	`HS256`	JWT signing algorithm
`REFRESH_TOKEN_EXPIRES_DAYS`	`90`	Refresh token lifetime in days
`RESET_TOKEN_EXPIRES_MINUTES`	`60`	Password reset token lifetime in minutes
`AUTH_MAX_FAILED_LOGINS`	`5`	Failed login attempts before temporary lockout
`AUTH_LOCKOUT_MINUTES`	`15`	Account lockout duration after too many failed attempts
`USERNAME_CHANGE_COOLDOWN_DAYS`	`30`	Minimum days between username changes
`APP_BASE_URL`	`http://localhost:5173`	Frontend URL (used in password-reset links)
`SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASS`, `SMTP_FROM`	—	SMTP settings for reset emails. If unset, reset links are printed to the server console.

Personal API Tokens (self-hosted)

# Format: comma-separated token[:user_id] entries
API_TOKENS=mypersonaltoken:alice,anothertoken

Webhooks

# Optional HMAC secret for signing outbound webhook payloads
WEBHOOK_SECRET=

llama.cpp Provider

# Path to your GGUF model file (set after running setup_models.py)
LLAMA_CPP_MODEL_PATH=/path/to/model.gguf
LLAMA_CPP_N_CTX=4096        # Context window size (tokens)
LLAMA_CPP_N_GPU_LAYERS=0    # GPU layers (0 = CPU only; 99 = offload all)
LLAMA_CPP_N_THREADS=0       # CPU threads (0 = auto-detect)
LLAMA_CPP_CHAT_FORMAT=      # Chat template override (blank = auto-detect)

Global LLM Backend Switch

Set LLM_PROVIDER to route all LLM features (summarize, chat, flashcards, transcript correction) through a single backend, overriding the provider field in individual API requests.

# Route every LLM call through the same backend
LLM_PROVIDER=gemini          # ollama | openai | chatgpt | claude | gemini | groq | llama_cpp
LLM_MODEL=gemini-2.0-flash   # optional: override the provider's default model
LLM_API_KEY=<api-key>        # optional: API key (falls back to provider-specific env var)

Supported providers and their per-provider key fallbacks:

`LLM_PROVIDER`	Alias	Key env var fallback
`ollama`	—	— (local)
`openai`	`chatgpt`	`OPENAI_API_KEY`
`claude`	—	`ANTHROPIC_API_KEY`
`gemini`	—	`GOOGLE_API_KEY`
`groq`	—	`GROQ_API_KEY`
`llama_cpp`	`llamacpp`	— (local, uses `LLAMA_CPP_MODEL_PATH`)

Leave LLM_PROVIDER blank to let each request choose its own provider (default behaviour).

Processing-Time Estimate Tuning

The /api/v1/estimate endpoint uses conservative CPU baselines. Override any rate for your hardware:

ESTIMATE_TRANSCRIBE_RATE=90   # compute-seconds per audio-minute
ESTIMATE_TRANSLATE_RATE=15
ESTIMATE_SUMMARIZE_OLLAMA_RATE=30
# ... (see .env.example for the full list)

Running the Server

Recommended one-command startup:

python main.py

Useful startup modes:

python main.py --setup
python main.py --healthcheck
python main.py --diagnostics
python main.py --dev
python main.py --prod
python main.py --backend-only
python main.py --frontend-only

Direct Uvicorn launch is still supported:

uvicorn main:app --host 0.0.0.0 --port 8000

The interactive API documentation is available at:

Swagger UI: http://localhost:8000/docs
ReDoc: http://localhost:8000/redoc

Runtime Configuration API (secured)

GET /api/v1/config
PUT /api/v1/config
POST /api/v1/config/reload
POST /api/v1/config/reset

Use Authorization: Bearer <CONFIG_ADMIN_TOKEN> or X-Config-Token: <CONFIG_ADMIN_TOKEN>.

Testing

pip install -r requirements-dev.txt
pytest
coverage run -m pytest && coverage report -m

See docs/architecture_and_operations.md for deployment, systemd, nginx, ngrok, WSL workflow, and security hardening guidance.

Authentication

Every protected endpoint expects a google_token parameter (form field, query parameter, or JSON body field depending on the endpoint). Despite the name, it accepts any of the three supported token types:

Token type	How to obtain	Pass as
Google OAuth	Google Sign-In flow	`google_token`
JWT (email/password)	`POST /api/v1/auth/login`	`google_token`
Static API token	Set `API_TOKENS` in `.env`	`google_token`

Register a new account

curl -X POST http://localhost:8000/api/v1/auth/register \
  -H "Content-Type: application/json" \
  -d '{"name": "Alice", "email": "alice@example.com", "password": "s3cr3tpass"}'

Log in

curl -X POST http://localhost:8000/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email": "alice@example.com", "password": "s3cr3tpass"}'
# Returns {"token": "<jwt>", "user": {...}}

Use the returned token as the google_token value in all subsequent requests.

API Reference

All endpoints are prefixed with /api/v1.

Health

GET /api/v1/health

Returns {"status": "ok"}. No authentication required.

Processing-Time Estimate

GET /api/v1/estimate

Query param	Type	Default	Description
`google_token`	string	—	Auth token
`audio_duration_seconds`	float	—	Audio length
`steps`	string	`transcribe,summarize,visualize,translate`	Comma-separated steps
`provider`	string	`ollama`	LLM provider (affects summarize estimate)

curl "http://localhost:8000/api/v1/estimate?google_token=TOKEN&audio_duration_seconds=120&steps=transcribe,summarize"

Transcribe (single upload)

POST /api/v1/transcribe

Accepts a multipart/form-data upload. Converts the audio to 16 kHz mono WAV, applies noise reduction, and runs diarization + transcription.

Form field	Type	Description
`file`	file	Audio file (any format FFmpeg can read)
`google_token`	string	Auth token
`transcribe_lang`	string	Optional language hint (e.g. `id`, `en`)
`webhook_url`	string	Optional callback URL on completion

curl -X POST http://localhost:8000/api/v1/transcribe \
  -F "file=@meeting.mp3" \
  -F "google_token=TOKEN" \
  -F "transcribe_lang=en"
# Returns: {"folder_name": "meeting_1700000000", "file_name": "meeting", ...}

Chunked Upload (for large files)

Use the three-step chunked upload flow when your file is too large to send in one request.

Step 1 — Initialise a session

POST /api/v1/upload/init

{
  "google_token": "TOKEN",
  "filename": "recording.mp3",
  "total_chunks": 10
}

Returns {"upload_id": "up_abc123", "total_chunks": 10, "received_chunks": []}.

Step 2 — Upload chunks (repeat for each chunk, resumable)

POST /api/v1/upload/chunk

Multipart fields: google_token, upload_id, chunk_index (0-based), file (binary chunk).

Check progress after an interruption

GET /api/v1/upload/status/{upload_id}?google_token=TOKEN

Returns {"received_chunks": [...], "missing_chunks": [...], "complete": false}.

Step 3 — Assemble and transcribe

POST /api/v1/upload/complete

{
  "google_token": "TOKEN",
  "upload_id": "up_abc123",
  "webhook_url": "https://your.app/hook"
}

Summarise

POST /api/v1/summarize

{
  "google_token": "TOKEN",
  "folder_name": "meeting_1700000000",
  "file_name": "meeting",
  "provider": "ollama",
  "model": "llama3.1:8b"
}

Supported providers: ollama, openai / chatgpt, claude, gemini, groq, llama_cpp / llamacpp.

Translate

POST /api/v1/translate

{
  "google_token": "TOKEN",
  "folder_name": "meeting_1700000000",
  "file_name": "meeting",
  "source_language": "Indonesian",
  "target_language": "English",
  "files": ["json", "txt", "summary_txt"]
}

source_language / target_language accept common names (English), ISO codes (en), or FLORES-200 codes (eng_Latn). 200+ languages are supported. Leave files empty to translate all three artifact types.

Visualise

POST /api/v1/visualize

{
  "google_token": "TOKEN",
  "folder_name": "meeting_1700000000",
  "file_name": "meeting"
}

Generates a horizontal speaker-timeline PNG from the JSON transcript.

Job History & Details

GET /api/v1/history?google_token=TOKEN
GET /api/v1/job/{folder_name}?google_token=TOKEN

Download Artifacts

GET /api/v1/download/{folder_name}/{file_type}?google_token=TOKEN

`file_type`	File
`audio`	Original WAV
`transcript_json`	Diarized JSON transcript
`transcript_txt`	Plain text transcript
`summary_txt`	Final summary (text)
`summary_html`	Final summary (HTML report)
`image`	Speaker timeline PNG

Auth Endpoints

Method	Endpoint	Description
`POST`	`/api/v1/auth/register`	Create account
`POST`	`/api/v1/auth/login`	Email/password login → JWT
`POST`	`/api/v1/auth/forgot-password`	Request password-reset email
`POST`	`/api/v1/auth/reset-password`	Complete password reset
`POST`	`/api/v1/auth/change-password`	Change password (requires `Authorization: Bearer <token>`)

CLI Utilities

Transcript Correction

Improve ASR quality with context-aware LLM correction:

python app/utils/fix_transcribe.py \
  --input  output/recording.json \
  --output output/recording-corrected.json \
  --provider ollama \
  --model llama3.1:8b

Provider and model default to FIX_TRANSCRIBE_PROVIDER / FIX_TRANSCRIBE_MODEL env vars.

Transcript Summarisation (standalone)

python app/utils/summarize_conv.py transcript.txt --provider ollama --model qwen3:8b
python app/utils/summarize_conv.py transcript.txt --provider openai --save

YouTube Audio Download

python app/tools/fetch_youtube_mp3.py
# Prompts for a YouTube URL and downloads as WAV

Convert JSON Transcript → SRT Subtitles

python app/tools/json2srt.py
# Reads output/tester.json, writes output/tester.srt

Generate Speaker Timeline (standalone)

python app/tools/visualized_diarization.py \
  -i output/recording.json \
  -o output/timeline.png \
  -s "Alice" "Bob"

Output Files

Each job produces a folder under Data/<user_id>/<recording>_<timestamp>/:

File	Description
`<name>.wav`	16 kHz mono WAV used for transcription
`<name>_denoised.wav`	Noise-reduced WAV (when noise reduction was applied)
`<name>.json`	Diarized transcript with per-speaker Whisper segments and timestamps
`<name>.txt`	Clean readable transcript grouped by speaker
`<name>.png`	Horizontal speaker-timeline visualisation
`<name>_final_summary.txt`	Full AI summary
`<name>_final_summary.html`	Shareable HTML report
`<name>_<lang_pair>.json/txt`	Translated versions (after translation step)
`manifest.json`	Job metadata: status of each step, file index, timing

Contributing

Contributions are welcome!

Fork the repository and create a feature branch.
Make your changes, keeping them focused and well-commented.
Verify the server starts cleanly: uvicorn main:app.
Open a pull request with a clear description of what changed and why.

Coding guidelines

Python 3.10+ type hints throughout.
Lazy imports for heavy dependencies (model libraries are only imported when actually used).
All filesystem operations go through _safe_join / _sanitize_name to prevent path traversal.
New LLM providers should be added to the PROVIDERS registry in app/utils/summarize_conv.py.
Secrets must never be committed — always use environment variables.

License

This project does not currently include a LICENSE file. Please contact the repository owner for licensing information before using RecordNote in a production or commercial context.

Name		Name	Last commit message	Last commit date
Latest commit History 144 Commits
.github/workflows		.github/workflows
app		app
tests		tests
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
main.py		main.py
pytest.ini		pytest.ini
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt
setup_models.py		setup_models.py

Folders and files

Latest commit

History

Repository files navigation