Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,6 @@ AI_API_KEY=
# Optional: local AI endpoint for local AI mode
LOCAL_AI_URL=http://127.0.0.1:3456

# --- Turso (libSQL) ---
TURSO_DATABASE_URL=
TURSO_AUTH_TOKEN=

# --- better-auth ---
# Generate with: openssl rand -base64 32
BETTER_AUTH_SECRET=
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
REQUIRED_SECRETS: BETTER_AUTH_SECRET GOOGLE_CLIENT_ID GOOGLE_CLIENT_SECRET TURSO_AUTH_TOKEN TURSO_DATABASE_URL
REQUIRED_SECRETS: BETTER_AUTH_SECRET GOOGLE_CLIENT_ID GOOGLE_CLIENT_SECRET
run: |
set -euo pipefail
secret_names="$(pnpm exec wrangler secret list --format json | node -e "let input='';process.stdin.on('data',c=>input+=c);process.stdin.on('end',()=>{const parsed=JSON.parse(input);console.log(parsed.map((item)=>item.name).join('\n'))})")"
Expand Down
8 changes: 4 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ auto-summarise the saved material. Companion Chrome MV3 extension. See
## Stack (one-liner)

Vite + React 19 SPA (single `app.html` entry) + Hono Worker on Cloudflare
Workers (`src/worker.ts`), Turso (libSQL) via Drizzle ORM, better-auth Google
Workers (`src/worker.ts`), Cloudflare D1 via Drizzle ORM, better-auth Google
OAuth, Cloudflare R2 for PDFs, free-ai-gateway + BYOK + local-ai dev bridge.
No SSR, no Next.js, no Firebase.

Expand All @@ -34,8 +34,8 @@ pnpm test # vitest run
pnpm test:e2e # playwright
pnpm lint # biome check .
pnpm format # biome format --write .
pnpm db:push # drizzle-kit push (schema sync)
pnpm db:studio # drizzle-kit studio
pnpm db:generate # generate a tracked D1 migration
pnpm db:migrate:local # apply migrations to isolated local D1
pnpm docs:check # validate docs/ links + structure
pnpm docs:dev # blume dev (local docs site; requires `pnpm add -D blume`)
pnpm docs:build # blume build (presentation only; not part of production build)
Expand All @@ -62,7 +62,7 @@ Full command map: [docs/development/commands.md](docs/development/commands.md).
and `/api/*` to reach the Worker before the `ASSETS` binding.
- **BYOK provider keys live in the browser only** — never persist or log
server-side. `rdr_*` API keys are hashed at rest; plaintext shown once.
- **Schema changes are additive + deliberate.** `pnpm db:push` is schema-sync;
- **Schema changes are additive + deliberate.** Generate and inspect SQL before applying it;
read the SQL under `drizzle/` before applying to production. See
[docs/operations/runbooks/migrate-schema.md](docs/operations/runbooks/migrate-schema.md).
- **Pre-commit hook (Husky + lint-staged)** runs `biome check --write` on
Expand Down
10 changes: 8 additions & 2 deletions PROJECT_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,17 @@ maintenance-first support.

## Dependencies

- Vite + React 19 SPA, Hono Worker, Turso + Drizzle, Cloudflare R2, better-auth
Google OAuth, and optional AI providers.
- Vite + React 19 SPA, Hono Worker, Cloudflare D1 + Drizzle, Cloudflare R2,
better-auth Google OAuth, and optional AI providers. Turso remains
temporarily preserved as the rollback source for the D1 cutover observation
window.

## Timeline

- **2026-08-01:** Migrated production relational persistence from Turso to a
project-owned Cloudflare D1 database after schema, row-count, aggregate,
ownership, auth/session, and document-path parity checks. R2 remains the PDF
object store; Turso is preserved for the bounded rollback observation window.
- **2026-07-31:** Kept the public sitemap HTML-only and added source-level
sitemap/catalog/Markdown parity coverage.
- **2026-07-31:** Added locally verified Open Graph/Twitter image metadata and
Expand Down
34 changes: 15 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ A modern web application for capturing and annotating articles with a distractio
| Concern | Service |
| ------------ | -------------------------------------------------------------------------------------------- |
| Hosting | Cloudflare Workers (`reader`) via Hono Worker (`src/worker.ts`) |
| Database | Turso (libSQL) via Drizzle ORM |
| Database | Cloudflare D1 via Drizzle ORM |
| Auth | better-auth + Google OAuth |
| File storage | Cloudflare R2 (`reader-pdfs`, bound as `PDFS_BUCKET`) |
| AI | free-ai-gateway (Workers AI chokepoint); BYOK providers (OpenAI/Anthropic/Gemini) + local AI |
Expand Down Expand Up @@ -74,9 +74,9 @@ graph TB
end

subgraph "Backend Services"
Turso[Turso libSQL]
Turso --> Drizzle[Drizzle ORM]
Turso --> BetterAuth[better-auth - Google OAuth]
D1[Cloudflare D1]
D1 --> Drizzle[Drizzle ORM]
D1 --> BetterAuth[better-auth - Google OAuth]

R2[Cloudflare R2 - PDFS_BUCKET]

Expand All @@ -91,7 +91,7 @@ graph TB
end

UI --> API
API --> Turso
API --> D1
API --> R2
API --> External

Expand All @@ -100,14 +100,14 @@ graph TB
classDef external fill:#f59e0b,stroke:#d97706,color:#fff

class UI,Styling,State frontend
class API,Articles,Projects,Auth,AI,Search,Tags,PDF,Snapshot,Turso,Drizzle,BetterAuth,R2 backend
class API,Articles,Projects,Auth,AI,Search,Tags,PDF,Snapshot,D1,Drizzle,BetterAuth,R2 backend
class External,Readability,AIProviders,OpenAI,Anthropic,Gemini,Gateway,LocalAI external
```

### Tech Stack

- **Frontend**: Vite + React 19 SPA (single `app.html` entry, client-side routing via `react-router-dom`), TypeScript, Tailwind CSS v4
- **Database**: Turso (libSQL) via Drizzle ORM
- **Database**: Cloudflare D1 via Drizzle ORM
- **Auth**: better-auth (Google OAuth, Drizzle adapter)
- **Storage**: Cloudflare R2 (PDFs) via Workers binding
- **AI Integration**: Vercel AI SDK + AI Gateway (preferred), BYOK chat providers, local AI support
Expand All @@ -119,7 +119,7 @@ graph TB
### Prerequisites

- Node.js 22+
- A Turso database (`turso db create`) and auth token
- Wrangler-managed local D1 (configured in `wrangler.local.toml`)
- A Cloudflare account with an R2 bucket bound as `PDFS_BUCKET`
- A Google OAuth client (Cloud Console > APIs & Services > Credentials)

Expand All @@ -142,10 +142,6 @@ graph TB
Edit `.env.local`:

```env
# Turso
TURSO_DATABASE_URL=libsql://your-db.turso.io
TURSO_AUTH_TOKEN=...

# better-auth
BETTER_AUTH_SECRET=$(openssl rand -base64 32)
BETTER_AUTH_URL=http://localhost:8787
Expand All @@ -162,10 +158,10 @@ graph TB
LOCAL_AI_URL=http://127.0.0.1:3456
```

3. Push the schema to Turso:
3. Apply the schema to local D1:

```bash
pnpm db:push
pnpm db:migrate:local
```

4. Run the development server:
Expand Down Expand Up @@ -197,8 +193,8 @@ pnpm deploy # validate env + cf:build + wrangler deploy
pnpm lint # Run ESLint
pnpm format # Format code with Biome
pnpm typecheck # tsc --noEmit (app + worker tsconfigs)
pnpm db:push # Apply schema to Turso (Drizzle)
pnpm db:studio # Open Drizzle Studio
pnpm db:generate # Generate a tracked D1 migration
pnpm db:migrate:local # Apply migrations to local D1 only
```

## Deployment
Expand All @@ -211,8 +207,8 @@ pnpm deploy

This runs `cf:build` (Vite SPA build + Astro landing overlay into `dist/`) and `wrangler deploy`.
Configure secrets via `wrangler secret put` for `BETTER_AUTH_SECRET`,
`TURSO_AUTH_TOKEN`, `GOOGLE_CLIENT_SECRET`, etc., and bind the R2 bucket as
`PDFS_BUCKET` in `wrangler.toml`.
`GOOGLE_CLIENT_SECRET`, etc., and bind D1 as `DB` plus R2 as `PDFS_BUCKET` in
`wrangler.toml`.

## Project Structure

Expand All @@ -231,7 +227,7 @@ web-annotator/
│ ├── lib/
│ │ ├── auth.ts # better-auth server config
│ │ ├── auth-client.ts# better-auth browser client
│ │ ├── db/ # Drizzle schema + Turso client
│ │ ├── db/ # Drizzle schema + D1 client
│ │ └── storage.ts # R2 helpers
│ └── types.ts # TypeScript definitions
├── packages/
Expand Down
9 changes: 4 additions & 5 deletions docs/architecture/data-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,8 @@ accept API keys (e.g. key management itself).

## Database (`src/lib/db/`)

- `createDb(env)` builds a Drizzle instance over `@libsql/client/web` using
`TURSO_DATABASE_URL` (rewritten `libsql://` → `https://`) and
`TURSO_AUTH_TOKEN`.
- `createDb(env)` builds a Drizzle instance directly over the request's
Cloudflare D1 `DB` binding.
- A module-level `db` Proxy defers client creation until first property access
(`getDb()`), so the Worker can `bindWorkerEnv()` before any query runs.
- All reads/writes are scoped by `userId` from the auth step. JSON columns
Expand All @@ -61,8 +60,8 @@ accept API keys (e.g. key management itself).
- `api_keys` — `rdr_*` tokens; `token_hash` unique; `revoked_at` nullable.

Migrations live in `drizzle/` (`0000_baseline.sql`, `0001_memories.sql`,
`0002_first_green_goblin.sql` for RSS). Schema sync uses `drizzle-kit push`
(`pnpm db:push`); see [operations/runbooks/migrate-schema.md](../operations/runbooks/migrate-schema.md).
`0002_first_green_goblin.sql` for RSS). Apply them locally with
`pnpm db:migrate:local`; see [operations/runbooks/migrate-schema.md](../operations/runbooks/migrate-schema.md).

## PDF storage (`src/lib/storage.ts` + `src/worker/routes/pdf.ts`)

Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/decisions/0002-turso-drizzle.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# ADR-0002: Turso (libSQL) via Drizzle ORM

**Date:** 2026-04-25 (Firebase → Turso migration); carried forward post-Vite migration
**Status:** Current
**Status:** Superseded by [ADR-0010](0010-cloudflare-d1.md); retained as migration history.
**Supersedes:** [archive/decisions.md ADR-02](../../archive/decisions.md) (same decision; this record updates the runtime context to Vite + Hono)

## Context
Expand Down
28 changes: 28 additions & 0 deletions docs/architecture/decisions/0010-cloudflare-d1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# ADR-0010: Cloudflare D1 for structured data

**Status:** Accepted for migration; production cutover remains operator-gated.

## Decision

Use one Cloudflare D1 database, bound as `DB`, for Reader's application and
better-auth tables. Keep Drizzle as the query and schema layer. Generate tracked
SQLite migrations and apply them through Wrangler's D1 migration ledger.

Local development uses the isolated binding in `wrangler.local.toml`. Remote
resource creation, data import, production binding, deployment, and Turso
retirement are separate approved operations.

## Why

Reader already runs as a Cloudflare Worker and stores PDFs in R2. D1 removes
the external libSQL client and database credentials from each request while
keeping the existing SQLite schema and Drizzle query surface.

## Consequences

- `src/lib/db/client.ts` receives a request-scoped D1 binding.
- Better Auth and all application tables remain in the same database.
- The legacy Firestore import is local by default and requires both `--apply`
and `--remote` before it can write remotely.
- Production Turso remains the authority until parity checks and explicit
cutover approval pass; its retirement is not implied by cutover.
14 changes: 7 additions & 7 deletions docs/architecture/how-it-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ each choice.
A **Vite + React 19 single-page app** (no SSR) runs entirely in the browser and
talks to a **single Hono Worker** on Cloudflare, which routes `/api/*` to
resource handlers, serves the built SPA over the `ASSETS` binding, and fans out
to three backends: **Turso** (libSQL, all structured data), **Cloudflare R2**
to three backends: **Cloudflare D1** (all structured data), **Cloudflare R2**
(PDF binaries), and an **AI gateway** (Workers AI, with BYOK and a local-dev
bridge as alternatives).

Expand All @@ -45,8 +45,8 @@ Every claim below is grounded in the file named next to it.
with Google OAuth for the webapp, plus an `rdr_*` API-key path for the Chrome
extension. `getAuthenticatedUserId()` is the single choke point every
protected handler calls first.
- **Data layer** (`src/lib/db/`, `src/lib/*-db.ts`) — Drizzle over a Turso
libSQL client, wrapped in `*-db.ts` helpers (`articles-db.ts`, `lists-db.ts`,
- **Data layer** (`src/lib/db/`, `src/lib/*-db.ts`) — Drizzle over the Worker's
D1 binding, wrapped in `*-db.ts` helpers (`articles-db.ts`, `lists-db.ts`,
…) that own queries, ownership scoping, and edge-cache invalidation.
- **PDF storage** (`src/lib/storage.ts`, `src/worker/routes/pdf.ts`) — binaries
live in the R2 `PDFS_BUCKET`; the DB stores only a `blob://<key>` reference,
Expand All @@ -72,7 +72,7 @@ flowchart TD
SNAP --> SF["validateExternalUrl + safe-fetch<br/>(SSRF boundary)"]
SF --> READ["Readability + linkedom<br/>extract clean article"]
READ --> SAVE["POST /api/articles<br/>createArticleRecord (articles-db.ts)"]
SAVE --> DB[(Turso libSQL<br/>articles, scoped by userId)]
SAVE --> DB[(Cloudflare D1<br/>articles, scoped by userId)]
U2[User opens reader/:id] --> GET["GET /api/articles/:id"]
GET --> CACHE{"caches.default<br/>5-min TTL?"}
CACHE -->|hit| RENDER[ReaderView renders]
Expand Down Expand Up @@ -107,7 +107,7 @@ flowchart TD

3. **Read.** Opening `reader/:id` calls `GET /api/articles/:id`. Reads are
served from Cloudflare's `caches.default` with a 5-minute TTL when available,
falling back to Turso; writes explicitly bust that cache. `ReaderView` renders
falling back to D1; writes explicitly bust that cache. `ReaderView` renders
the sanitised HTML with the user's theme/font/size preferences.

4. **Annotate.** Selecting text surfaces *Add note* / *Ask AI*. A note is a
Expand All @@ -134,11 +134,11 @@ mental model.
server rendering to justify SSR. A static SPA served from one Worker is
cheaper, simpler to reason about, and avoids the OpenNext build complexity the
project ran into. → [0001-vite-spa-hono-worker.md](decisions/0001-vite-spa-hono-worker.md).
- **Turso + Drizzle for structured data.** A SQLite-compatible edge database
- **D1 + Drizzle for structured data.** A SQLite-compatible edge database
keeps latency low near the Worker while giving typed, migratable schema.
JSON-in-text columns (`notes`, `aiChat`, `summary`, …) avoid a table sprawl
for data that is always read with its parent article. →
[0002-turso-drizzle.md](decisions/0002-turso-drizzle.md).
[0010-cloudflare-d1.md](decisions/0010-cloudflare-d1.md).
- **R2 for PDFs, DB stores only a reference.** Large binaries do not belong in a
row-oriented DB; R2 is cheap object storage bound directly to the Worker, and
proxied downloads keep the bucket private. →
Expand Down
8 changes: 4 additions & 4 deletions docs/architecture/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,12 @@ Vite entry) and routes client-side via `react-router-dom`; the Worker handles
│ │ data-export, ext chat, browser-mem)│
│ └── env.ASSETS.fetch() ← built SPA + landing (dist/) │
│ │
│ Bindings: PDFS_BUCKET (R2), ASSETS (dist/)
│ Bindings: DB (D1), PDFS_BUCKET (R2), ASSETS (dist/) │
└────────────────────────────────┬─────────────────────────────────────────┘
┌──────────────────────────────┼──────────────────────────────┐
▼ ▼ ▼
Turso (libSQL) Cloudflare R2 free-ai-gateway
Cloudflare D1 Cloudflare R2 free-ai-gateway
via Drizzle ORM reader-pdfs bucket (AI_BASE_URL)
(articles, boards, (PDF binaries, + BYOK providers
lists, memories, rss, proxied downloads) + local-ai (dev)
Expand All @@ -47,7 +47,7 @@ Vite entry) and routes client-side via `react-router-dom`; the Worker handles
`/api/<resource>`.
- `src/lib/db/schema.ts` — Drizzle schema (app tables + better-auth tables +
legacy NextAuth tables kept for reference).
- `src/lib/db/client.ts` — Turso libSQL client. Lazy proxy so the client is
- `src/lib/db/client.ts` — D1 Drizzle client. Lazy proxy so the client is
not created at module load (required for the Workers runtime).
- `src/lib/auth.ts` — better-auth server config (`createAuth`), Drizzle
adapter, Google OAuth, `oneTap` plugin, rate limiting disabled.
Expand All @@ -59,7 +59,7 @@ Vite entry) and routes client-side via `react-router-dom`; the Worker handles
- `src/lib/url-validation.ts` + `src/lib/safe-fetch.ts` — SSRF protection and
redirect-safe fetch used by snapshot/proxy/RSS refresh.
- `src/router.tsx` — client-side routes (lazy-loaded pages).
- `wrangler.toml` — Worker config: `main = src/worker.ts`, `ASSETS` + `PDFS_BUCKET`
- `wrangler.toml` — Worker config: `main = src/worker.ts`, `DB` + `ASSETS` + `PDFS_BUCKET`
bindings, `placement.mode = "smart"`, `nodejs_compat_v2`, custom domain.
- `vite.config.ts` — Vite SPA build (React, Tailwind v4, Lightning CSS).
- `app.html` — single SPA HTML entry (Vite input; carries inline shell CSS).
Expand Down
7 changes: 4 additions & 3 deletions docs/development/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,10 @@ ordering; run `pnpm run` to see the live list.
| `pnpm test:coverage` | `vitest run --coverage` |
| `pnpm test:e2e` | `playwright test` |
| `pnpm memory:demo` | `tsx scripts/memory-capture-demo.ts` |
| `pnpm db:push` | `drizzle-kit push` (schema sync) |
| `pnpm db:studio` | `drizzle-kit studio` |
| `pnpm migrate:firestore` | Legacy Firestore → Turso migration (`tsx scripts/migrate-firestore-to-turso.ts`) |
| `pnpm db:generate` | Generate a tracked D1 migration from `src/lib/db/schema.ts` |
| `pnpm db:migrate:local` | Apply tracked migrations to isolated local D1 |
| `pnpm db:migrate:remote` | Explicitly apply tracked migrations to the configured remote D1 |
| `pnpm migrate:firestore` | Legacy Firestore → D1 migration (`tsx scripts/migrate-firestore-to-d1.ts`; dry-run by default) |
| `pnpm prepare` | `husky` (installs pre-commit hook) |
| `pnpm format` | `biome format --write .` |
| `pnpm format:check` | `biome format .` |
Expand Down
7 changes: 3 additions & 4 deletions docs/development/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

- Node.js 22+ (engines field in `package.json`; CI uses 24).
- pnpm 10+ (the `packageManager` field pins the exact version).
- A Turso database (`turso db create`) and auth token.
- Wrangler for the isolated local D1 database (already installed by `pnpm install`).
- A Cloudflare account with an R2 bucket `reader-pdfs` bound as `PDFS_BUCKET`.
- A Google OAuth client (Google Cloud Console → APIs & Services → Credentials)
with a redirect URI for `BETTER_AUTH_URL` (e.g.
Expand All @@ -25,7 +25,6 @@ cp .env.example .env.local
Edit `.env.local` (see [operations/env.md](../operations/env.md) for the full
list and validation):

- `TURSO_DATABASE_URL`, `TURSO_AUTH_TOKEN`
- `BETTER_AUTH_SECRET` (`openssl rand -base64 32`), `BETTER_AUTH_URL`
- `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`
- `AI_GATEWAY_API_KEY` (free-ai gateway) — optional for BYOK-only dev
Expand All @@ -34,10 +33,10 @@ list and validation):
R2 credentials are only needed for production / `wrangler dev`; the binding
itself is provided by `wrangler dev` from `wrangler.toml`.

## Push the schema
## Apply the schema

```bash
pnpm db:push # drizzle-kit pushapplies schema to Turso
pnpm db:migrate:local # tracked migrationsisolated local D1
```

See [operations/runbooks/migrate-schema.md](../operations/runbooks/migrate-schema.md)
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ docs/
jobs.md # scheduled jobs (weekly quality check)
runbooks/
migrate-schema.md # applying Drizzle schema changes
rotate-secrets.md # rotating Cloudflare/Turso/Google secrets
rotate-secrets.md # rotating Cloudflare and Google secrets
rollback.md # rollback procedure for a bad deploy
knowledge/
learnings.md # current, applicable engineering lessons
Expand Down
Loading