arXiv Recommender

A web application that ingests new arXiv papers daily, embeds them using LLMs, and serves ranked paper recommendations to registered users based on their reading history.

Features

• Daily paper ingestion — new arXiv papers are fetched automatically each evening and queued for embedding.
• User library — users can like or dislike papers and bulk-import arXiv IDs from a NASA ADS export.
• Personalised recommendations — papers from the last day, week, or month are ranked by a per-user logistic regression model trained on the user's liked papers.
• LLM-generated summaries — each paper has a structured 7-section summary (Keywords, Scientific Questions, Data, Methods, Results, Conclusions, Key Takeaway) generated by an LLM and displayed alongside the abstract.
• Admin interface — activate/deactivate user accounts, inspect the task queue, and search ingested papers.
• Email verification — optional; new accounts can be set to require email verification before becoming active (requires a Resend API key and email_config.json).
• Password reset — users can request a password-reset link sent to their registered email address.

How It Works

Embedding

The recommender converts each paper into a vector embedding based on its contents.

• The arXiv metadata (title, author list, and abstract) and the raw LaTeX source of the paper are fed into an LLM (currently Qwen3-Next-80B-A3B-Thinking) to produce a structured 7-section summary (Keywords, Scientific Questions, Data, Methods, Results, Conclusions, Key Takeaway).
• The structured summary is then fed into an embedding model (currently Qwen3-Embedding-8B) with two slightly different prompts to produce two 4096-dimensional embedding vectors: one for paper recommendations and one for semantic search. These embeddings are truncated to 512 dimensions (making use of the Matryoshka representation, which allows for flexible dimensionality) before storage in embeddings_cache.db. The recommendation model is further truncated to 128 dimensions at recommendation time to save compute and improve generalization.

Both models are accessed via any OpenAI-compatible inference endpoint (currently routed through the Hugging Face and OpenRouter inference routers). The endpoint, model name, and API key are configured in llm_config.json.

Recommendation

Given the user's set of liked papers ("my papers") and a pool of recent candidate papers, the recommender:

• Computes a distance matrix between "my papers" and the candidate papers in the 128-dimensional embedding space.
• For each candidate paper, extracts features: the minimum distance to any of "my papers"; and the mean RBF kernel values at several logarithmically-spaced distance scales. The same features are also computed in a low-variance subspace of "my papers" to measure how out-of-distribution each candidate is.
• Feeds the features into a logistic regression model trained to distinguish "my papers" from a random background sample. The model score is used to rank the candidate papers.

The recommendation step is fast (milliseconds). The expensive part is embedding, which requires two remote LLM API calls per paper.

Architecture

Backend

Component	Technology
Web framework	FastAPI + uvicorn
Database	SQLite (app.db + embeddings_cache.db)
Authentication	JWT (HS256, HTTP-only cookie) + bcrypt via passlib
Rate limiting	slowapi
ML / numerics	numpy, scipy, scikit-learn
LLM API client	openai SDK (OpenAI-compatible)
Retry logic	tenacity
Reverse proxy	Caddy (auto-TLS)

Frontend

Component	Technology
UI framework	React 19 + TypeScript
Build tool	Vite
Styling	Tailwind CSS 4
Routing	React Router 7
Math rendering	KaTeX

Databases

• app.db — users, user libraries, paper metadata, subscribed categories, recommendation cache, task queue, import rate-limit log, admin audit log.
• embeddings_cache.db — stores separate tables for search and recommendation embeddings of each arXiv ID.

Ingest Pipeline

Ingest happens in two ways: automatically via a daily cron job, and on-demand when a user imports a paper. In both cases, tasks flow through a shared SQLite-backed task queue and are processed by background daemons.

Daily ingest

scripts/cron_daily.py runs nightly after the arXiv mailing is published:

1. Queries the arXiv OAI-PMH interface for papers announced that UTC date, for each category in DAILY_INGEST_CATEGORIES (default: ["astro-ph"]). Author names are parsed from the structured OAI-PMH fields (keyname, forenames, suffix).
2. Writes metadata (title, abstract, authors, categories, announced date) to the papers table in app.db.
3. Enqueues an embed task for each paper not already known.

The embed tasks are then processed by the embed daemon (see below).

# Recommended cron entry (fires at 21:00 US Eastern, which is ≥ 01:00 UTC):
CRON_TZ=America/New_York
0 21 * * 1-5  cd /path/to/arxiv_recommender && python3 scripts/cron_daily.py

Use --date YYYY-MM-DD to backfill a missed day, or --rss to fall back to the RSS Atom feed (legacy; comma-separated author names).

User-activated ingest

When a user adds a paper via the web interface (single arXiv ID or bulk NASA ADS export), the backend:

1. Checks the user's daily import rate limit (Tier A: 16 papers/day for users with < 32 lifetime imports; Tier B: 4 papers/day thereafter, rolling 24-hour window).
2. If the paper is not yet in app.db, enqueues a fetch_meta task.
3. Records the import in user_import_log.
4. Links the paper to the user's library with the given liked/disliked flag.

The meta daemon (daemons/meta_daemon.py) processes fetch_meta tasks:

1. Fetches paper metadata in batches of up to 256 via the Semantic Scholar batch API (with arXiv Atom API as fallback).
2. Writes title, abstract, authors, and published date to the papers table.
3. Enqueues an embed task for each successfully fetched paper.

Embedding (shared pathway)

The embed daemon (daemons/embed_daemon.py) processes embed tasks one at a time:

1. Calls summarize_arxiv_paper() — fetches the paper's LaTeX source, truncates to the model's context limit, and asks the summary LLM to produce the 7-section structured summary. The result is cached in arxiv_summary_cache/.
2. Calls gen_arxiv_embedding() — builds a prompt from the metadata and summary, then calls the embedding model to produce two 4096-dimensional vectors - one for recommendation and one for scoring. Both are truncated to 512 dimensions and stored in embeddings_cache.db. At recommendation scoring time, the relevant embeddings are further truncated to 128 dimensions (these choices are configurable).
3. Marks the task done; failed tasks are retried up to 3 times.

Both daemons run as long-lived background processes:

python3 daemons/meta_daemon.py &
python3 daemons/embed_daemon.py &

Running the Application

Local development

# 1. Create and activate a Python virtualenv
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# 2. Start background daemons
python3 daemons/meta_daemon.py &
python3 daemons/embed_daemon.py &

# 3. Start the web server
SECRET_KEY=$(python3 -c "import secrets; print(secrets.token_hex(32))") python3 -m uvicorn web.app:app --reload --port 8000

# 4. Start the frontend dev server (separate terminal)
cd web/frontend && npm install && npm run dev

Vite proxies /api requests to localhost:8000, so the full app is available at http://localhost:5173.

Production deployment

Production uses Caddy as a reverse proxy (handles HTTPS + TLS certificate renewal automatically) and systemd to manage the three long-running processes.

1. Install Caddy

Follow the official package-manager instructions for your distro at https://caddyserver.com/docs/install. For example, on Debian/Ubuntu:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudflare.com/public/gpan/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/cloudflare-main.gpg
curl -1sLf 'https://dl.cloudflare.com/public/Repositories/deb/PUBLIC_KEY.gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudflare.com/public/Repositories/deb/deb.cloudflare.com/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update && sudo apt install caddy

Caddy requires ports 80 and 443 to be open in your firewall and a DNS A record pointing your domain at the server's IP address.

2. Create the config files

These files are not committed to the repository and must be created manually in the project root.

api_keys.json — API credentials:

{
  "summary_api_key":  "your-summary-llm-api-key",
  "embed_api_key":    "your-embedding-llm-api-key",
  "semantic_scholar": "your-semantic-scholar-api-key",
  "resend_api_key":   "your-resend-api-key"
}

The resend_api_key is only required when email verification is enabled (see email_config.json below).

llm_config.json — LLM endpoints and models:

{
  "summary": {
    "base_url": "https://router.huggingface.co/v1",
    "api_key_name": "summary_api_key",
    "model": "Qwen/Qwen3-Next-80B-A3B-Thinking",
    "max_input_tokens": 98304,
    "cot_closing_tags": ["</think>"]
  },
  "embedding": {
    "base_url": "https://router.huggingface.co/v1",
    "api_key_name": "embed_api_key",
    "model": "Qwen/Qwen3-Embedding-8B",
    "max_input_tokens": 24576
  }
}

email_config.json — Email verification settings (optional; omit the file to disable):

{
  "verification": {
    "enabled": true,
    "email_from": "noreply@mail.yourdomain.com",
    "app_base_url": "https://yourdomain.com"
  }
}

When email verification is enabled, new accounts must verify their email address before they can sign in. When disabled, new accounts are created inactive and require manual activation by an admin.

3. Run the deploy script

Edit the three variables at the top of deploy/deploy.sh:

USER="yourlinuxusername"
PROJECT_DIR="/home/yourlinuxusername/projects/arxiv_recommender"
DOMAIN="arxiv.yourdomain.com"

Then run it as root:

sudo bash deploy/deploy.sh

The script will:

• Create a Python virtualenv and install all dependencies
• Build the React frontend
• Generate a SECRET_KEY and CORS_ALLOW_ORIGINS and write them to .env (keep a backup of this file)
• Install the systemd service and timer files
• Install and validate the Caddyfile
• Enable and start all services

It is safe to re-run for updates (e.g. after pulling new code). The .env file is never overwritten once created.

4. Activate the first admin user

Register an account via the web UI, then promote it from the command line:

sudo -u $USER .venv/bin/python3 scripts/activate_user.py your@email.com --make-admin

New registrations are inactive by default. Admins can activate users via the web UI or CLI.

User & Admin Management

New accounts are inactive by default and must be approved by an admin. When email verification is enabled (email_config.json), accounts become active automatically after the user verifies their email address; manual admin activation is not required in that mode. All admin-privilege changes are CLI-only (not available via the web UI).

python3 scripts/activate_user.py --list                  # list all users
python3 scripts/activate_user.py user@example.com        # activate
python3 scripts/activate_user.py user@example.com --deactivate
python3 scripts/activate_user.py user@example.com --make-admin
python3 scripts/activate_user.py user@example.com --remove-admin

General user approval can be handled via the admin web interface.

Configuration

The config files api_keys.json and llm_config.json are described in the deployment section above.

Key constants in arxiv_lib/config.py:

Constant	Default	Description
EMBEDDING_STORAGE_DIM	512	Dimensions stored in embeddings_cache.db
SEARCH_EMBEDDING_DIM	512	Dimensions used for semantic search
RECOMMENDATION_EMBEDDING_DIM	128	Dimensions used by the recommendation scoring model
DAILY_INGEST_CATEGORIES	["astro-ph"]	arXiv categories for daily cron
RECOMMEND_MIN_LIKED	4	Minimum liked papers to generate recommendations
MAX_LIKED_PAPERS_TO_USE	256	Cap on training data per user
MAX_MODEL_AGE_DAYS	90	Force model retraining after this many days
IMPORT_DAILY_LIMIT_TIER_A	16	Papers/day for new users (< 32 lifetime imports)
IMPORT_DAILY_LIMIT_TIER_B	4	Papers/day for established users

Environment variables read at startup:

Variable	Required	Description
SECRET_KEY	Yes	JWT signing key; must be ≥ 32 characters. Generate with python3 -c "import secrets; print(secrets.token_hex(32))"
CORS_ALLOW_ORIGINS	No	Comma-separated list of allowed CORS origins. Defaults to http://localhost:5173,http://localhost:3000. Set automatically by deploy.sh to https://<DOMAIN>.