diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 0000000..728f780 --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,40 @@ +name: Docs + +on: + push: + branches: [main, master] + paths: + - 'docs/**' + - 'AGENTS.md' + - 'STATUS.md' + - 'PROJECT_STATUS.md' + - 'README.md' + - 'blume.config.ts' + - 'scripts/check-docs.mjs' + - '.github/workflows/docs.yml' + pull_request: + branches: [main, master] + paths: + - 'docs/**' + - 'AGENTS.md' + - 'STATUS.md' + - 'PROJECT_STATUS.md' + - 'README.md' + - 'blume.config.ts' + - 'scripts/check-docs.mjs' + - '.github/workflows/docs.yml' + workflow_dispatch: + +permissions: + contents: read + +jobs: + validate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + - uses: actions/setup-node@v6 + with: + node-version: '22' + - name: Validate docs/ links and structure + run: node scripts/check-docs.mjs diff --git a/.github/workflows/weekly.yml b/.github/workflows/weekly.yml index 0ce8494..ca27b28 100644 --- a/.github/workflows/weekly.yml +++ b/.github/workflows/weekly.yml @@ -7,6 +7,7 @@ on: jobs: quality: runs-on: ubuntu-latest + timeout-minutes: 20 permissions: contents: read diff --git a/.gitignore b/.gitignore index 56e3406..5a83ae0 100644 --- a/.gitignore +++ b/.gitignore @@ -68,3 +68,13 @@ node_modules/ # build output dist/ + +# Blume (documentation presentation layer — generated from docs/) +.blume/ +.blume/dist/ +docs-site/ + +# Local agent logs / scratch +.agent-logs/ +*.agent.log + diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..ba58768 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,177 @@ +# AGENTS.md — Reader + +> Agent bootloader. Concise by design — links to [`docs/`](docs/index.md) for +> depth. Also follow the [shared fleet standard](https://github.com/sass-maker/fleet-workspace/blob/main/AGENTS.md): +> treat this repository as owned product code, protect production stability, +> keep changes scoped, verify work, and record durable follow-up tasks when +> something remains incomplete or blocked. + +## Purpose + +Reader is a personal research library: capture web articles and PDFs, read and +annotate them, organise with tags/lists/boards, search, and AI-chat or +auto-summarise the saved material. Companion Chrome MV3 extension. See +[docs/product/overview.md](docs/product/overview.md). + +## Stack (one-liner) + +Vite + React 19 SPA (single `app.html` entry) + Hono Worker on Cloudflare +Workers (`src/worker.ts`), Turso (libSQL) via Drizzle ORM, better-auth Google +OAuth, Cloudflare R2 for PDFs, free-ai-gateway + BYOK + local-ai dev bridge. +No SSR, no Next.js, no Firebase. + +## Essential commands + +```bash +pnpm install +pnpm dev # Worker (:8787) + Vite SPA (:5173) + local-ai, concurrently +pnpm dev:worker # wrangler dev only +pnpm dev:spa # vite only (proxies /api → 8787) +pnpm build # validate env + vite build → dist/ +pnpm cf:build # build + landing-astro + overlay into dist/ +pnpm deploy # validate env + cf:build + wrangler deploy (manual; CI does not auto-deploy) +pnpm typecheck # tsc --noEmit (app + worker tsconfigs) +pnpm test # vitest run +pnpm test:e2e # playwright +pnpm lint # biome check . +pnpm format # biome format --write . +pnpm db:push # drizzle-kit push (schema sync) +pnpm db:studio # drizzle-kit studio +pnpm docs:check # validate docs/ links + structure +pnpm docs:dev # blume dev (local docs site; requires `pnpm add -D blume`) +pnpm docs:build # blume build (presentation only; not part of production build) +``` + +Chrome extension (separate workspace, excluded from root tooling): +`cd packages/chrome-extension && pnpm dev|build|test`. + +Full command map: [docs/development/commands.md](docs/development/commands.md). + +## Critical constraints + +- **Do not commit secrets.** `.env`, `.env.local`, `.dev.vars`, + `firebase-service-account.json`, and any auth credential are gitignored. + Verify `.gitignore` before any push. +- **Do not push, deploy, run migrations, or open PRs without explicit user + approval.** Make changes locally and leave them staged/committed for review. +- **Production deploy is manual** (`workflow_dispatch` on + `.github/workflows/deploy.yml`). CI runs on push but does not deploy. +- **The Worker name `reader` is load-bearing** — the custom domain + (`read.significanthobbies.com`) and all Cloudflare secrets are bound to it. + Do not rename without re-provisioning. +- **`wrangler.toml` `run_worker_first` list is required** for agent surfaces + and `/api/*` to reach the Worker before the `ASSETS` binding. +- **BYOK provider keys live in the browser only** — never persist or log + server-side. `rdr_*` API keys are hashed at rest; plaintext shown once. +- **Schema changes are additive + deliberate.** `pnpm db:push` is schema-sync; + read the SQL under `drizzle/` before applying to production. See + [docs/operations/runbooks/migrate-schema.md](docs/operations/runbooks/migrate-schema.md). +- **Pre-commit hook (Husky + lint-staged)** runs `biome check --write` on + staged `*.{js,jsx,ts,tsx,json,css}`. Re-stage modified files and retry if + the hook reformats. +- **Do not modify agent skills, plugins, or agent-profile directories** + (`.claude/`, `.codex/skills/`, `.symphony/`, `.clawpatch/`, any `SKILL.md`). + They are tooling, not product code. + +## Documentation navigation + +- **[docs/index.md](docs/index.md)** — canonical documentation hub. Start + there. +- **[STATUS.md](STATUS.md)** — short current-state view (objective, active + work, blockers, next steps). +- **[README.md](README.md)** — product readme for humans landing in the repo. +- **[docs/product/](docs/product/)** — purpose, features, surfaces. +- **[docs/architecture/](docs/architecture/)** — overview, data flow, ADRs. +- **[docs/development/](docs/development/)** — setup, commands, conventions, + testing, OpenSpec. +- **[docs/operations/](docs/operations/)** — deploy, env, CI/CD, jobs, + runbooks. +- **[docs/knowledge/](docs/knowledge/)** — current lessons, external + references, failed approaches. +- **[docs/archive/](docs/archive/)** — historical records (pre-Vite ADRs, + lessons, migration plans, security audit). +- **[openspec/](openspec/)** — spec-driven change workflow tooling and + archived change proposals. See + [docs/development/openspec.md](docs/development/openspec.md). +- **[public/](public/)** — runtime agent-indexing surfaces (`llms.txt`, + `index.md`, `api-ai.json`, `robots.txt`, `sitemap.xml`). See + [docs/product/surfaces.md](docs/product/surfaces.md). + +## Documentation-maintenance rules + +1. **Markdown in `docs/` is the source of truth.** Code and executable config + remain authoritative for implementation details; docs explain *why*, not + *what the code does line-by-line*. +2. **One home per fact.** Don't duplicate — link to the canonical home. If a + fact moves, update links rather than copying. +3. **Prefer `docs/archive/` over deletion.** Move superseded docs with + `git mv`, give them a dated filename, and prepend a one-line historical + marker pointing at the current canonical doc. Preserve git rename history. +4. **Mark unresolved questions explicitly** with `TBD:` or an "Open questions" + section. Do not invent answers. +5. **Keep pages focused** (150–300 lines). Split when a page grows beyond + that. +6. **Validate before commit.** Run `pnpm docs:check` (or + `node scripts/check-docs.mjs`) — it catches broken links, missing required + sections, and files outside the canonical structure. CI runs it in + `.github/workflows/docs.yml`. +7. **Blume is presentation only.** `blume.config.ts` renders `docs/`; never + edit generated Blume output. Edit the Markdown and rebuild. + +## Repo structure (high level) + +``` +app.html # Single SPA HTML entry (Vite input) +vite.config.ts # Vite SPA build (React, Tailwind v4, Lightning CSS) +wrangler.toml # Worker config: main=src/worker.ts, ASSETS + PDFS_BUCKET +src/ + worker.ts # Hono Worker entry — security headers, /api/* routing, asset serving + agent-edge.mjs # Generated agent-edge handler (llms.txt, index.md, api/ai) + worker/routes/ # Hono API route modules (articles, boards, lists, ai, keys, pdf, rss, share, memories, misc) + pages/ # Route page components (lazy-loaded via react-router-dom) + components/ # React components (ReaderView, PDFReaderClient, NotesAIChat, board/, reader/, ui/) + hooks/ # Shared React hooks + lib/ # DB, auth, AI, storage, SSRF validation, RSS, memories, etc. +packages/chrome-extension/ # Chrome MV3 extension (separate Vite build) +landing-astro/ # Astro landing page (overlaid into dist/ during cf:build) +openspec/ # Spec-driven change workflow tooling + archived changes +docs/ # Canonical documentation (source of truth) +drizzle/ # Migration SQL files + meta +scripts/ # local-ai.mjs, validate-env.mjs, overlay-astro-landing.mjs, check-docs.mjs +public/ # Agent-indexing surfaces (llms.txt, index.md, api-ai.json, robots.txt, sitemap.xml) +``` + +Detailed file map: [docs/architecture/overview.md](docs/architecture/overview.md). + +## Fleet guidance + + + +### Adding Tasks + +- Add durable work items in SaaS Maker Cockpit Tasks when the task affects + product behavior, deployment, user feedback, or fleet maintenance. +- Include the project slug, a concise title, acceptance criteria, + priority/status, and links to relevant code, issues, traces, or dashboards. +- If task discovery starts locally in an editor or agent session, mirror the + durable next step back into SaaS Maker before handoff. + +### Using SaaS Maker + +- Treat SaaS Maker as the system of record for project metadata, feedback, + tasks, analytics, testimonials, changelog, and fleet visibility. +- Prefer API-first workflows through `fnd api`, the SDK, or widgets instead + of one-off scripts when interacting with SaaS Maker features. +- Keep this agent file aligned with the project record when operating rules, + integrations, or deployment conventions change. + +### Free AI First + +- Prefer free/local AI paths for routine development and analysis: the + `free-ai` gateway, local models, provider free tiers, and cached context. +- Escalate to paid models only when complexity, correctness risk, or missing + capability justifies the cost. +- Note any paid-AI use in the task or handoff when it materially affects cost, + reproducibility, or future maintenance. + + diff --git a/PROJECT_STATUS.md b/PROJECT_STATUS.md index a80c697..3234502 100644 --- a/PROJECT_STATUS.md +++ b/PROJECT_STATUS.md @@ -1,150 +1,23 @@ -# reader — PROJECT STATUS - -Last updated: 2026-07-13 - -## Why / What - -Reader is a personal reading and annotation app for capturing articles and PDFs, organizing them with tags and projects, tracking progress, and running AI-assisted summaries, key points, and chat over saved material. - -**Users:** Individual readers saving articles/PDFs; signed-in users via Google OAuth; operators running Turso migrations and Cloudflare deploys. - -**Constraints:** Same-origin Worker + SPA pattern; rate limits deferred until endpoint-specific evidence. Memory capture is persisted and authenticated (Turso `memories` table, `/api/memories` routes, `/memory` UI page). - -**IN scope:** Article/PDF capture, annotations, tags/projects, full-text search, AI chat/summaries, boards/lists, Turso persistence, R2 PDF storage, free-ai gateway + BYOK + local-ai dev bridge. - -**OUT of scope:** Browser-extension distribution, full personal knowledge-base automation, paid team/library workflows, separate `landing-astro` deployable product. - -Ships as a Vite + React 19 SPA with a Hono Worker backend on Cloudflare Workers. - -## Dependencies - -### External - -- **Turso (libSQL):** Primary persistence for articles, tags, projects, progress, chat, auth sessions. -- **Cloudflare R2:** `reader-pdfs` (`PDFS_BUCKET`) — PDF binary storage; signed URL access. -- **PostHog:** Product analytics via `posthog-js`. -- **Mozilla Readability + linkedom:** Article extraction. -- **pdfjs / react-pdf:** In-browser PDF viewing. -- **Env files:** See `.env.example` / deploy validation — Turso, auth, AI gateway, R2 bindings in `wrangler` config. - -### Internal (fleet) - -| Service | Role | -| ------------------------------ | ---------------------------------------------------------------------------------------------------- | -| **free-ai** | Default AI chokepoint via `AI_BASE_URL` (`https://ai-gateway.sassmaker.com/v1`) | -| **local-ai** | Dev bridge for authenticated local CLI models (`pnpm local-ai`) | -| **SaaS Maker feedback widget** | In-app feedback capture via `@saas-maker/feedback` | - -### Stack & commands - -**Stack:** Vite 8 · React 19 · React Router 7 · Tailwind v4 · TanStack Query · Hono Worker `src/worker.ts` · Turso · Drizzle ORM · better-auth Google OAuth · R2 `reader-pdfs` · free-ai-gateway · PostHog · `@saas-maker/feedback` · Mozilla Readability + linkedom · pdfjs/react-pdf. - -| Command | Purpose | -| ------------------------------------------------ | ----------------------------------------- | -| `pnpm install` | Install deps | -| `pnpm dev` | worker + SPA + local-ai (concurrently) | -| `pnpm dev:spa` | vite only | -| `pnpm dev:worker` | wrangler dev only | -| `pnpm local-ai` | local AI bridge | -| `pnpm build` | vite build (validate env) | -| `pnpm cf:build` | build + landing-astro + overlay | -| `pnpm deploy` | validate env + cf:build + wrangler deploy | -| `pnpm typecheck` / `pnpm test` / `pnpm test:e2e` | TS + vitest + playwright | -| `pnpm check` | biome check | -| `pnpm db:push` / `pnpm db:studio` | Drizzle push / studio | -| `pnpm migrate:firestore` | legacy Firestore → Turso migration | -| `pnpm memory:demo` | memory capture prototype demo | - -CI: GitHub Actions auto-deploy to Cloudflare on push to `main`. - -**Entrypoints:** `src/worker.ts` · route modules under `src/worker/routes/` · optional `landing-astro/` overlay merged into `dist/` during `cf:build`. - -## Timeline - -- **2026-07-13** — Added an authenticated RSS/Atom reader with OPML import, direct add/remove feed management, bounded manual refresh, unread/read inbox state, and save-to-library actions. The additive `0002_first_green_goblin.sql` migration must be applied before deployment. -- **2026-07-03** — Memory capture promoted from prototype to persisted, authenticated flow. `memories` Turso table + `/api/memories` CRUD/search routes + `/memory` UI page + browser-memory import all wired. Global SearchBar routes memory hits to `/memory`. Read-and-remember instead of read-and-forget. -- **2026-07-02** — Added `api.onError()` global error handler + outer try/catch in worker fetch handler; added React `` wrapping `RouterProvider` in `bootstrap.tsx`. -- **Wave 2 migration** — De-OpenNext migration to Vite + React 19 SPA + Hono worker; Worker name `reader` preserved. Turso/Drizzle persistence; better-auth Google; R2 PDF storage. -- **Security audit pass** — Auth on snapshot routes; SSRF validation; signed PDF URLs. Firestore rules addressed during migration; render-time sanitization. Dead middleware removed; critical/high audit findings closed. - -## Products - -| Surface | URL | -| ----------------------- | ---------------------------------------------------------- | -| Production app | `https://read.significanthobbies.com` | -| AI gateway (Worker var) | `https://ai-gateway.sassmaker.com/v1` | -| Canonical / OG | Set in `landing-astro/astro.config.mjs` and built `dist/` | - -Production uses the `read.significanthobbies.com` custom domain on the canonical Cloudflare Worker; Pages remains reverted. - -## Features (shipped) - -### Architecture - -- Browser (React 19 SPA, TanStack Query, React Router) calls same-origin `/api/*` + static `dist` via ASSETS binding. -- Cloudflare Worker `reader` (Hono) routes: articles, boards, lists, pdf, ai, share, keys, misc (snapshot, proxy, browser-memory import). -- Turso (libSQL) + Drizzle ORM for persistence; better-auth Google OAuth for sessions. -- R2 `reader-pdfs` (`PDFS_BUCKET`) stores PDF binaries with signed URL access. -- AI flows through `AI_BASE_URL` → free-ai-gateway, with BYOK (OpenAI/Anthropic/Gemini) and `scripts/local-ai.mjs` dev bridge. -- Landing: optional `landing-astro/` overlay merged into `dist/` during `cf:build`; SPA fallback via `not_found_handling = single-page-application` pattern. -- PostHog analytics via `posthog-js`; SaaS Maker feedback widget embedded in app shell. - -### Reading & capture - -- Article capture from URL via Mozilla Readability; PDF upload, view, annotate, text extraction. -- Rich annotations with optional DOM anchoring; selection actions (Add note / Ask AI). -- Reading time estimates; customizable reader (theme, font, size). -- RSS/Atom inbox at `/rss`: OPML import, direct feed add/remove, manual refresh with partial-failure reporting, unread/read filtering, and save-to-library. - -### Organization & search - -- Tags with color badges, autocomplete, filtering. -- Full-text search across content, notes, AI chat (Cmd/Ctrl+K). -- Projects grouping; reading progress tracking. -- Boards/lists routes (`/board`, `/board/:id`, share links). - -### AI features - -- Per-article AI chat with persistent markdown history. -- Auto-summaries (short/medium/long); key points extraction (3–5 bullets). -- BYOK providers + gateway + local AI mode. - -### Memory capture (prototype) - -- `/memory` route with fixture-backed examples. -- `POST /api/browser-memory/import` import endpoint. -- Not yet authenticated or persisted as full product flow. - -### Security & audit fixes - -- Auth on snapshot routes; SSRF validation; signed PDF URLs. -- Firestore rules addressed during migration; render-time sanitization. -- Dead middleware removed; critical/high audit findings closed. - -## Todo / Planned / Deferred / Blocked - -### Planned - -1. ~~Turn memory capture prototype into authenticated, persisted product flow.~~ **Done** — `memories` table, `/api/memories` routes, `/memory` UI page, browser-memory import. -2. ~~Move prototype search from in-memory behavior to Turso-backed search where it matters.~~ **Done** — `/api/memories/search` is Turso-backed; global SearchBar routes memory hits to `/memory`. -3. ~~Clarify PDF storage behavior.~~ **Paused** at the current documented representation; reopen for a concrete storage defect. -4. ~~Add abuse and rate-limit handling.~~ **Paused** until endpoint-specific evidence exists. - -### Closure - -- **Personal-use support (2026-07-10):** Keep Reader available for direct use. No roadmap expansion; accept only maintenance, reliability, or personally requested workflow fixes. The preexisting generated `dist/` worktree drift is not part of this decision. - -### Deferred - -- RSS background refresh/scheduled triggers, notifications, OPML folder preservation, and feed discovery. Current RSS refresh is manual. `drizzle/0002_first_green_goblin.sql` is the canonical migration and is applied to production. -- Browser-extension distribution until web import and capture flow are reliable. -- Full personal knowledge-base automation behind strong capture, retrieval, and trust primitives. -- Paid team/library workflows. -- `landing-astro` is optional overlay only — not a separate deployable product surface. -- Memory capture is persisted and authenticated: `memories` Turso table, `/api/memories` CRUD + search routes, `/memory` UI page, browser-memory import. Global SearchBar routes memory hits to `/memory`. -- Residual audit: no explicit CORS config (acceptable for same-origin today); no rate limiting on AI/snapshot/proxy endpoints (deferred pending evidence). -- README still describes pre-migration Next.js layout in places; canonical runtime is Vite SPA + Hono worker. - -### Blocked - -- (none) +# PROJECT_STATUS — Reader + +> **Pointer.** The canonical current-state view is now +> [`STATUS.md`](STATUS.md). This file is kept for fleet tooling that reads +> `PROJECT_STATUS.md` by name (e.g. the `name-domains` skill). For the full +> product/feature/architecture record, see [`docs/`](docs/index.md). +> +> Last substantive content: 2026-07-13 (RSS reader shipped). Superseded by +> STATUS.md on 2026-07-18. + +## At-a-glance + +- **Product:** Reader — personal research library (capture, read, annotate, + AI-chat over articles and PDFs) + Chrome MV3 extension. +- **Production:** `https://read.significanthobbies.com` (Cloudflare Worker + `reader`, custom domain). +- **Stack:** Vite + React 19 SPA + Hono Worker · Turso (libSQL) + Drizzle · + better-auth Google OAuth · Cloudflare R2 (`PDFS_BUCKET`) · free-ai-gateway + + BYOK + local-ai dev bridge. +- **Posture:** Personal-use support (closure 2026-07-10). Maintenance and + reliability only; no roadmap expansion. +- **Current state + blockers + next steps:** see [STATUS.md](STATUS.md). +- **Detailed docs:** see [docs/index.md](docs/index.md). diff --git a/STATUS.md b/STATUS.md new file mode 100644 index 0000000..37a201a --- /dev/null +++ b/STATUS.md @@ -0,0 +1,81 @@ +# STATUS — Reader + +Last updated: 2026-07-18 + +## Current objective + +Keep Reader available for direct personal use. No roadmap expansion. Accept +only maintenance, reliability, or personally requested workflow fixes. +(Closure decision 2026-07-10.) + +## Active work + +- **Documentation consolidation** (2026-07-18): built a maintainable, + local-first `docs/` knowledge system with a Blume presentation layer, + link/structure validator, and CI. AGENTS.md slimmed to a bootloader; + STATUS.md introduced; pre-Vite docs archived with historical markers. See + `docs/index.md`. + +## Recent shipped + +- **2026-07-13** — Authenticated RSS/Atom reader with OPML import, direct + feed add/remove, bounded manual refresh, unread/read inbox state, and + save-to-library. Migration `drizzle/0002_first_green_goblin.sql` is + additive and applied to production. +- **2026-07-03** — Memory capture promoted from prototype to persisted, + authenticated flow (`memories` table, `/api/memories` CRUD + search, + `/memory` UI, browser-memory import). +- **2026-07-02** — `api.onError()` global error handler + outer try/catch in + the Worker fetch handler; React `` wrapping + `RouterProvider` in `bootstrap.tsx`. +- **Wave 2 migration** — De-OpenNext migration to Vite + React 19 SPA + + Hono Worker. Worker name `reader` preserved. See + `docs/architecture/decisions/0001-vite-spa-hono-worker.md`. + +## Blockers + +- (none) + +## Unresolved questions / deferred + +- **Drop legacy NextAuth tables** (`account`, `session`, `verificationToken` + in `src/lib/db/schema.ts`) once confirmed no active rows. See + `docs/architecture/decisions/0004-better-auth-google.md`. +- **Switch `drizzle-kit push` → `drizzle-kit generate`** for safer schema + changes as user count grows. See + `docs/operations/runbooks/migrate-schema.md`. +- **Re-enable better-auth rate limiting** if the app becomes public. + Currently disabled (`rateLimit: { enabled: false }`). +- **RSS background refresh via Cloudflare scheduled triggers** — deferred + until manual use demonstrates the need. See + `docs/architecture/decisions/0008-rss-inbox.md`. +- **Rate limiting on AI/snapshot/proxy endpoints** — deferred pending + endpoint-specific abuse evidence. See + `docs/knowledge/failed-approaches.md`. +- **Explicit CORS on share routes** — deferred until cross-origin access is + needed. +- **Browser-extension distribution** — deferred until web import and capture + flow are reliable. +- **PDF storage representation** — paused at the current documented + representation; reopen only for a concrete storage defect. +- **README still describes pre-migration Next.js layout in places** — + canonical runtime is Vite SPA + Hono worker. (Low priority; AGENTS.md and + docs/ are current.) + +## Next steps + +1. Land the documentation consolidation branch + (`docs/consolidate-knowledge-system`) for human review. +2. Wire `pnpm docs:check` into CI (`.github/workflows/docs.yml`) and confirm + it runs green on the next push. +3. Optionally publish the Blume docs site for Reader (one of the "few core + projects" — confirm with the operator before deploying). + +## Pointers + +- Product context, features, surfaces: [`docs/product/`](docs/product/) +- Architecture + ADRs: [`docs/architecture/`](docs/architecture/) +- Operations + runbooks: [`docs/operations/`](docs/operations/) +- Historical PROJECT_STATUS.md (pre-consolidation, 2026-07-13): + [`PROJECT_STATUS.md`](PROJECT_STATUS.md) — kept as a pointer for fleet + tooling that reads it by name. diff --git a/agents.md b/agents.md deleted file mode 100644 index 8dd2fa5..0000000 --- a/agents.md +++ /dev/null @@ -1,209 +0,0 @@ -# agents.md — reader - -## Shared Fleet Standard - -Also read and follow the shared fleet-level agent standard at `../AGENTS.md`. Treat this repository as owned product code: protect production stability, keep changes scoped, verify work, and record durable follow-up tasks when something remains incomplete or blocked. - -## Purpose - -Personal research library — capture, read, annotate, and AI-chat with web articles and PDFs, with a companion Chrome MV3 extension. - -## Stack - -- Framework: Vite + React 19 SPA (single `app.html` entry, client-side routing via `react-router-dom`) -- Backend: Hono Worker (`src/worker.ts`) mounting `/api/*` routes from `src/worker/routes/*.ts` -- Language: TypeScript -- Styling: Tailwind CSS v4 (`@tailwindcss/vite`) + `@tailwindcss/typography`, Radix UI. CSS via Lightning CSS transformer + minifier (see `vite.config.ts`) -- DB: Turso (libSQL) via Drizzle ORM -- Auth: better-auth (Google OAuth, Drizzle adapter on Turso). Server config: `src/lib/auth.ts` (`createAuth`). Browser client: `src/lib/auth-client.ts`. Handler is a Hono route — `api.on(['GET','POST'], '/api/auth/*')` in `src/worker.ts`. -- Storage: Cloudflare R2 (PDFs) via Workers binding `PDFS_BUCKET` -- Testing: Vitest (unit), Playwright (e2e) -- Deploy: Cloudflare Workers — `pnpm deploy` builds the SPA + Astro landing, then `wrangler deploy`. Worker `main = src/worker.ts`; built SPA served via the `ASSETS` binding (`wrangler.toml`). -- Package manager: pnpm workspace - -## Repo structure - -``` -app.html # Single SPA HTML entry (Vite input; carries inline shell CSS) -vite.config.ts # Vite SPA build (React, Tailwind v4, Lightning CSS) -wrangler.toml # Worker config: main=src/worker.ts, ASSETS + PDFS_BUCKET bindings -src/ - worker.ts # Hono Worker entry — security headers, /api/* routing, asset serving - worker/ - routes/ # Hono API route modules mounted under /api/* - articles.ts # /api/articles — article CRUD - boards.ts # /api/boards - lists.ts # /api/lists - ai.ts # /api/ai — AI chat/summary via free-ai gateway - keys.ts # /api/keys — extension API keys (rdr* keys) - pdf.ts # /api/pdfs — PDF upload/download (R2-backed) - share.ts # /api/share — public share endpoints - misc.ts # /api/* — search, tags, data-export, snapshot, proxy, ext chat, session - pages/ # Route page components (LibraryPage, ReaderPage, BoardPage, etc.) - components/ - ReaderView.tsx # Article reading mode (typography, annotations) - PDFReaderClient.tsx # PDF reading via pdfjs-dist / react-pdf - NotesAIChat.tsx # AI chat panel for notes - ArticleSummary.tsx # AI-generated summary - board/ # Board components - reader/ # Reader components - ui/ # Shadcn-style primitives - hooks/ # Shared React hooks - lib/ - db/ - schema.ts # Drizzle schema (articles, boards, lists, plus better-auth tables) - client.ts # Turso libSQL client (createDb) - articles-db.ts # Article CRUD (Drizzle/Turso) - boards-db.ts # Boards CRUD - lists-db.ts # Lists CRUD - auth.ts # better-auth server config (createAuth — Drizzle adapter, Google OAuth) - auth-client.ts # better-auth browser client - ai-server.ts # AI provider config (server) - storage.ts # Cloudflare R2 helpers (PDFS_BUCKET binding) - pdf-service.ts # PDF file validation (size limit) -packages/ - chrome-extension/ # Chrome MV3 extension (separate Vite build) — excluded from root tooling -landing-astro/ # Astro landing page (built into the deploy via cf:build) -plans/ - migrate-off-firebase.md # Historical: Firebase → Turso + better-auth + R2 (DONE) - archive/ # Archived plans with timestamps -scripts/ - local-ai.mjs # Local LLM bridge server (runs alongside the Worker + SPA in dev) - validate-env.mjs # Env validation for build/runtime/deploy -drizzle.config.ts # Drizzle config (Turso, schema at src/lib/db/schema.ts) -``` - -## Key commands - -```bash -# Web app -pnpm dev # Worker (wrangler dev) + Vite SPA + local-ai.mjs, concurrently -pnpm dev:worker # wrangler dev only (Worker, port 8787) -pnpm dev:spa # vite only (SPA dev server, port 5173, proxies /api → 8787) -pnpm build # validate env + vite build → dist/ -pnpm cf:build # build SPA + Astro landing + overlay landing into dist/ -pnpm deploy # validate env + cf:build + wrangler deploy -pnpm test # vitest run -pnpm test:e2e # playwright test -pnpm lint # eslint -pnpm typecheck # tsc --noEmit (app + worker tsconfigs); `type-check` aliases this -pnpm format # biome format --write . -pnpm check # biome check . - -# Database (Turso) -pnpm db:push # drizzle-kit push -pnpm db:studio # drizzle-kit studio - -# Chrome extension (from packages/chrome-extension/) — separate Vite build -pnpm dev # vite build --watch → dist/ -pnpm build # vite build (production) -pnpm test # vitest run -``` - -## Architecture notes - -- **App shape**: Vite + React SPA served from a Hono Worker. The browser loads `app.html` (one entry) and routes client-side via `react-router-dom`; `src/worker.ts` handles `/api/*` and serves built assets via the `ASSETS` binding. No SSR, no Next.js. -- **DB + Auth**: Turso (libSQL) via Drizzle. `src/lib/db/schema.ts` defines articles/boards/lists plus better-auth tables (`users`, `baSessions`, `baAccounts`, `baVerifications`). better-auth uses the Drizzle adapter; only Google OAuth is configured. Auth is mounted as a Hono catch-all route (`/api/auth/*`) in `src/worker.ts`. -- **PDF storage**: PDFs live in Cloudflare R2 (binding `PDFS_BUCKET`). Upload/download go through the Hono `/api/pdfs` routes (`src/worker/routes/pdf.ts`) so auth + ownership are enforced server-side. See `src/lib/storage.ts`. -- **Chrome extension**: Manifest V3. Side panel (not popup) for chat UI. Content script uses `@mozilla/readability` for page extraction. Authenticates with `rdr*` API keys via `/api/keys`. Builds independently in `packages/chrome-extension/`; excluded from root Biome/ESLint tooling. -- **PDF support**: `pdfjs-dist` (+ `react-pdf`) for rendering. `src/lib/pdf-service.ts` only validates uploaded PDFs (size limit). -- **Boards**: Kanban view using `@xyflow/react`. -- **AI**: `@ai-sdk/openai-compatible` + Vercel AI SDK, routed through the `AI_BASE_URL` free-ai gateway (`wrangler.toml`) with an `x-gateway-project-id: reader` header. `scripts/local-ai.mjs` bridges a local LLM in dev. -- **React Query**: `@tanstack/react-query` for client data fetching; `ReactQueryHydrate` is a thin `HydrationBoundary` wrapper for seeding the client query cache. -- **pnpm workspace**: root is the Vite SPA + Hono Worker; `packages/chrome-extension` and `landing-astro` are separate workspace builds. -- **Env vars**: `TURSO_DATABASE_URL`, `TURSO_AUTH_TOKEN`, `BETTER_AUTH_SECRET`, `BETTER_AUTH_URL`, `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, plus the Cloudflare R2 binding. Validated by `scripts/validate-env.mjs`. No Firebase, no NextAuth. -- Do NOT commit `.env` or auth credentials — verify `.gitignore` before any push. -- Pre-commit hook via Husky: lint-staged runs ESLint (`--fix`) + Biome formatter on staged files. - - - -## Fleet Guidance - -### Adding Tasks - -- Add durable work items in SaaS Maker Cockpit Tasks when the task affects product behavior, deployment, user feedback, or fleet maintenance. -- Include the project slug, a concise title, acceptance criteria, priority/status, and links to relevant code, issues, traces, or dashboards. -- If task discovery starts locally in an editor or agent session, mirror the durable next step back into SaaS Maker before handoff. - -### Using SaaS Maker - -- Treat SaaS Maker as the system of record for project metadata, feedback, tasks, analytics, testimonials, changelog, and fleet visibility. -- Prefer API-first workflows through `fnd api`, the SDK, or widgets instead of one-off scripts when interacting with SaaS Maker features. -- Keep this agent file aligned with the project record when operating rules, integrations, or deployment conventions change. - -### Free AI First - -- Prefer free/local AI paths for routine development and analysis: the `free-ai` gateway, local models, provider free tiers, and cached context. -- Escalate to paid models only when complexity, correctness risk, or missing capability justifies the cost. -- Note any paid-AI use in the task or handoff when it materially affects cost, reproducibility, or future maintenance. - - - -## Active context - - -# Memory Context - -# [reader] recent context, 2026-05-04 12:34pm GMT+5:30 - -Legend: 🎯session 🔴bugfix 🟣feature 🔄refactor ✅change 🔵discovery ⚖️decision 🚨security_alert 🔐security_note -Format: ID TIME TYPE TITLE -Fetch details: get_observations([IDs]) | Search: mem-search skill - -Stats: 50 obs (16,724t read) | 577,586t work | 97% savings - -### May 3, 2026 - -734 9:00a 🔵 Reader — HomeClient article cards lack visual distinction for 'link' type -736 9:01a 🟣 Reader — link-type article cards open URL in new tab with distinct visual treatment and context menu -737 " 🔵 Reader — delete confirmation dialog doesn't handle 'link' type, shows "article" for link items -738 " 🔴 Reader — delete confirmation dialog updated to handle 'link' type with correct copy -739 " 🟣 Reader board — AddWebsiteDialog gains PDF upload mode, becomes "Add Source" with 3-way mode selector -740 " 🔵 Reader — AddWebsiteDialog has orphaned <input tag before conditional JSX causing build error -741 9:02a 🔴 Reader — orphaned <input JSX tag in AddWebsiteDialog fixed, onAddReader wired to BoardCanvasClient -742 " 🟣 Reader board — ReaderNode renders PDFViewer for PDF articles instead of ReaderCore -743 " 🔵 Reader — PDF storage uses Cloudflare R2 via PDFS*BUCKET binding, access proxied through /api/pdfs/:id/download -744 " 🟣 Reader — /reader/[id] server route now redirects link-type articles to their URL -745 " 🔵 Reader Chrome extension — popup still branded "Open in Annotator", extension auth uses rdr* API keys -746 9:03a 🟣 Reader Chrome extension popup — "Save to Library" as primary CTA, "Import & Read" as secondary action -748 9:04a 🔵 Reader — pnpm type-check broken: tsconfig.json only includes saas-maker package paths, not project source -749 " ✅ Reader Chrome extension — production build passes with new Save to Library feature (1844 modules, 2.66s) -750 9:05a 🟣 Reader Chrome extension side panel — SaveButton restructured with Save to Library primary + Import & Read secondary -751 " ✅ Reader — full test suite passes (40 tests) and prettier formatting clean after link/PDF/board feature work -752 9:45a 🔵 Fleet monorepo tsconfig architecture — shared @saas-maker/tsconfig package -753 9:46a ✅ reader tsconfig.json — added local overrides for paths, baseUrl, and relaxed strict flag -754 " 🔵 TypeScript 6+ deprecates baseUrl — causes TS5101 error in reader type-check -755 " 🔴 reader tsconfig.json — removed deprecated baseUrl to fix TS5101 error -756 9:47a 🔵 reader type-check reveals TS6133 unused 'request' params in API route handlers -757 " 🔵 reader API route handlers — GET and DELETE methods declare unused 'request' param pattern -758 9:49a 🔵 @saas-maker/tsconfig include paths resolve relative to package dir — TS18003 across 6+ Fleet projects -760 9:50a 🔵 CodeVetter root tsconfig fails — @saas-maker/tsconfig not installed, node.json extends wrong for React apps -761 " ✅ reader — final diff before commit: tsconfig fix + \_request renames + broader feature changes staged -762 9:55a 🔵 SAAS Maker feedback API rejects type "task" — HTTP 400 Invalid type -763 " ✅ Fleet tsconfig fix — maintenance task filed in SAAS Maker feedback as bug ID 07a0216b -768 10:05a 🔵 Chrome reading list empty across all profiles — extension data unavailable -769 " 🔵 reader project data layer architecture — lists-db.ts and articles-db.ts -770 10:07a 🔵 Chrome Profile 2 reading list — complete URL inventory from LevelDB -771 10:08a 🔵 Reader app searchArticles — in-memory full-text search, not FTS5 -772 " 🔵 Reader app schema.ts — full articles + boards table definitions -773 " 🔵 gaurigupta19.github.io Chrome reading list URLs — all 404 -774 10:09a 🔵 Reader project uses dotenvx, not plain dotenv -775 10:11a 🔵 Reader production Turso DB — single user confirmed via live query -776 " 🔵 Chrome reading list — additional article titles resolved from LevelDB -777 " 🔵 Reader app — complete articles-db and lists-db public API surface -778 " 🔵 Chrome reading list URLs — live reachability confirmed for 8 articles -779 10:12a 🔵 Chrome reading list — complete live URL inventory: 14 of 15 articles reachable -780 " 🟣 Chrome reading list bulk-imported into reader app production DB -781 10:13a 🔵 Chrome Reading List import verified — all 28 articles confirmed as type:link in Turso -782 10:14a 🔵 Chrome open tabs session — local-ai GitHub repo and additional articles discovered -783 10:15a 🔵 Chrome open tabs — extended article inventory for potential reader import -786 10:18a ⚖️ Reader app — pivot from import tool to future reading queue -787 11:44a 🔵 Reader app data layer — article creation + list management API surface mapped -788 11:45a 🔵 Reader app Turso DB — local tsx script access pattern requires dotenv pre-load -789 " 🔵 Reader app — all 28 Chrome reading list articles confirmed in DB with correct list assignment -790 11:49a 🔵 Chrome open tabs extracted via osascript — 29 URLs enumerated for potential reader app import -791 11:53a 🟣 29 Chrome open tabs bulk-closed via osascript after enumeration -792 " 🔵 Reader app "Chrome Open Tabs" list has 2 stale URL mismatches — trailing slash differences - -Access 578k tokens of past work via get_observations([IDs]) or mem-search skill. - diff --git a/blume.config.ts b/blume.config.ts new file mode 100644 index 0000000..a41b0aa --- /dev/null +++ b/blume.config.ts @@ -0,0 +1,78 @@ +// Blume configuration — presentation and search layer for docs/. +// +// Markdown under docs/ is the source of truth. Blume only renders it. +// Never edit generated Blume output; edit the Markdown and rebuild. +// +// Usage: +// pnpm docs:dev → blume dev (local docs site) +// pnpm docs:build → blume build (static site → .blume/dist by default) +// +// Blume is NOT part of the production Worker build. `pnpm deploy` does not +// invoke Blume. The generated site is gitignored (see .gitignore). + +import { defineConfig } from 'blume'; + +export default defineConfig({ + title: 'Reader — Documentation', + description: + 'Personal research library: capture, read, annotate, and AI-chat over web articles and PDFs. Architecture, decisions, operations, and knowledge for the Reader project.', + + content: { + // Canonical documentation root. Keep in sync with scripts/check-docs.mjs. + root: 'docs', + include: ['**/*.{md,mdx}'], + exclude: ['**/_*', '**/.*', 'archive/**'], + }, + + // The archive/ folder is excluded from the published site — it is + // historical context for contributors, not user-facing documentation. + // If you want archive pages to appear, remove "archive/**" from exclude. + + search: { + provider: 'orama', + }, + + theme: { + accent: 'teal', + radius: 'md', + mode: 'system', + }, + + markdown: { + imageZoom: true, + code: { + icons: true, + wrap: false, + }, + codeBlocks: { + theme: { + light: 'github-light', + dark: 'github-dark', + }, + }, + }, + + ai: { + // Emit llms.txt / llms-full.txt for the docs site. The product app has + // its own agent surfaces under public/ (see docs/product/surfaces.md); + // this is the docs-site llms.txt, separate from the app's. + llmsTxt: true, + mcp: { + enabled: false, + }, + }, + + seo: { + og: { enabled: true }, + sitemap: true, + robots: true, + structuredData: true, + }, + + deployment: { + output: 'static', + // Update this to the docs site URL when the docs site is published. + // Leave as a placeholder until the operator confirms the docs domain. + site: 'https://docs.significanthobbies.com', + }, +}); diff --git a/docs/architecture/data-flow.md b/docs/architecture/data-flow.md new file mode 100644 index 0000000..9168323 --- /dev/null +++ b/docs/architecture/data-flow.md @@ -0,0 +1,136 @@ +# Data Flow + +How a request moves through the system. Read alongside +[overview.md](overview.md) for the shape. + +## Worker entry (`src/worker.ts`) + +``` +request → export default { fetch } + 1. handleAgentEdge(request) ← /llms.txt, /llms-full.txt, /index.md, /api/ai, /robots.txt + returns early if matched + 2. if pathname.startsWith('/api/') → api.fetch(request, env, ctx) + Hono router; onError → 500 JSON; outer try/catch → 500 JSON + 3. if GET / and has auth cookie → 302 /library + 4. env.ASSETS.fetch(request) ← static assets + SPA + if ok → withSecurityHeaders(response) + 5. if GET / → serve /index.html (landing overlay) + 6. fallback → serve /app (SPA) ← SPA fallback for client-side routes +``` + +`bindWorkerEnv(c.env)` runs on every `/api/*` request so module-level singletons +(the lazy `db` proxy, `pdfsBucket`) can resolve the runtime env. + +## Auth resolution (`src/lib/auth-api.ts`) + +`getAuthenticatedUserId(headers, env)` is called at the top of every protected +route handler. Priority: + +1. **`Authorization: Bearer rdr_*`** — long-lived API key (Chrome extension). + `verifyApiKey()` hashes the token, looks it up in `api_keys` where + `revoked_at IS NULL`, bumps `last_used_at`, returns the owning `userId`. + The plaintext token is shown once at creation and never persisted. +2. **better-auth session cookie** — webapp users. `createAuth(env).api.getSession()` + reads the `session_token` / `session-token` cookie and resolves the user. + +`requireSessionUserId()` is the cookie-only variant for routes that must not +accept API keys (e.g. key management itself). + +## Database (`src/lib/db/`) + +- `createDb(env)` builds a Drizzle instance over `@libsql/client/web` using + `TURSO_DATABASE_URL` (rewritten `libsql://` → `https://`) and + `TURSO_AUTH_TOKEN`. +- A module-level `db` Proxy defers client creation until first property access + (`getDb()`), so the Worker can `bindWorkerEnv()` before any query runs. +- All reads/writes are scoped by `userId` from the auth step. JSON columns + (`tags`, `notes`, `aiChat`, `summary`, `keyPoints`, `pdfMetadata`, + `sessionReview`) are stored as text and typed via `$type()`. + +### Tables + +- `user`, `ba_session`, `ba_account`, `ba_verification` — better-auth. +- `account`, `session`, `verificationToken` — legacy NextAuth tables kept for + reference, not used by better-auth. Safe to drop after manual verification + (see [knowledge/learnings.md](../knowledge/learnings.md)). +- `articles` — core; `type` ∈ `article | link | pdf`; `pdfStorageKey` for R2. +- `boards`, `lists` — grouping. +- `memories` — browser-memory captures; `(user_id, url)` unique. +- `rss_feeds`, `rss_entries` — RSS subscriptions + entries; `(user_id, feed_url)` + unique; `(feed_id, external_id)` unique. +- `api_keys` — `rdr_*` tokens; `token_hash` unique; `revoked_at` nullable. + +Migrations live in `drizzle/` (`0000_baseline.sql`, `0001_memories.sql`, +`0002_first_green_goblin.sql` for RSS). Schema sync uses `drizzle-kit push` +(`pnpm db:push`); see [operations/runbooks/migrate-schema.md](../operations/runbooks/migrate-schema.md). + +## PDF storage (`src/lib/storage.ts` + `src/worker/routes/pdf.ts`) + +- Upload: `POST /api/pdfs/upload` (multipart) → validate size (10 MB) → + validate magic bytes (`%PDF-`) → `PDFS_BUCKET.put('pdfs//.pdf')` + → store `blob://` as `articles.url` → optional `pdf-parse` text + extraction → return article record. +- Download: `GET /api/pdfs/:id/download` → auth + ownership check → + `PDFS_BUCKET.get(storageKey)` → stream bytes with `application/pdf` content + type. Clients never receive a raw R2 URL. +- The R2 binding is unavailable in `next dev`-style local dev; the binding is + set via `bindWorkerEnv()` from `wrangler dev` / production. + +## AI routing (`src/lib/ai-cloudflare.ts` + `src/worker/routes/ai.ts`) + +- `getLanguageModel()` builds an `@ai-sdk/openai-compatible` provider pointing + at `AI_BASE_URL` (default `https://ai-gateway.sassmaker.com/v1`) with the + `x-gateway-project-id: reader` header. The gateway enforces a 9500 Neuron/day + fleet-wide cap. +- Default model: `@cf/meta/llama-3.3-70b-instruct-fp8-fast` (Workers AI). +- BYOK: when the client sends an `Authorization: Bearer ` that is + not a `rdr_*` key, the route uses the user-supplied endpoint/key/model + directly. BYOK keys are normalised (`normalizeApiKey`, length-capped) and + never persisted or logged. +- Local AI: `isLocalCLIEnabled()` is `NODE_ENV === 'development'` only. + `createLocalAITextStream()` bridges `LOCAL_AI_URL` (default + `http://127.0.0.1:3456`) via Server-Sent Events. +- `POST /api/ai/chat` streams text (`streamText` + `toTextStreamResponse`). +- `POST /api/ai/summarize` generates a summary + 3–5 key points + (`generateText`). +- `POST /api/ai/models` proxies `/models` on the configured endpoint. + +## Server-side URL fetches (SSRF boundary) + +`/api/snapshot`, `/api/proxy`, and RSS refresh all funnel through: + +1. `validateExternalUrl(url)` (`src/lib/url-validation.ts`) — rejects + non-HTTP(S), localhost, RFC1918, link-local, cloud metadata (169.254.169.254), + decimal/hex-encoded IPs. Resolves DNS once (note: not a full DNS-rebinding + defence — appropriate for a personal-use reader; see the file's comment). +2. `fetchWithValidatedRedirects(url)` (`src/lib/safe-fetch.ts`) — follows + redirects manually, re-validating every `Location` before the next hop + (max 5). Prevents a public URL bouncing the server into a private target + after the first request. + +RSS refresh adds: 15 s per-feed timeout, bounded concurrency (4), response-size +cap (`MAX_FEED_BYTES`), entry cap, and per-feed error isolation. + +## Extension auth path + +The Chrome extension cannot share the same-origin session cookie (its requests +originate from the extension's origin). Flow: + +1. User creates an API key in the webapp (`POST /api/keys`) → `generateApiKey()` + returns `rdr_<32 chars>` plaintext once; `token_hash` + `prefix` persisted. +2. Extension stores the plaintext in `chrome.storage` and sends it as + `Authorization: Bearer rdr_...` on every request. +3. `getAuthenticatedUserId()` resolves the bearer path before the cookie path. +4. Extension AI chat uses `/api/ext/chat` (`src/worker/routes/misc.ts`), which + also accepts the bearer key and reuses the same gateway/BYOK logic. + +## Caching + +- `caches.default` (Cloudflare edge cache) is used in `articles-db.ts` for + article reads (5 min TTL), guarded by `globalThis.caches?.default` because it + is unavailable in `next dev`-style local dev. Cache must be explicitly busted + on writes (see `lists-db.ts`). +- Security headers set `Cache-Control: public, max-age=300, s-maxage=600, + stale-while-revalidate=86400` on `text/html` responses so deploys propagate + quickly without sacrificing TTFB. +- `/api/auth/client-config` is sent with `Cache-Control: no-store`. diff --git a/docs/architecture/decisions/0001-vite-spa-hono-worker.md b/docs/architecture/decisions/0001-vite-spa-hono-worker.md new file mode 100644 index 0000000..9d8d41f --- /dev/null +++ b/docs/architecture/decisions/0001-vite-spa-hono-worker.md @@ -0,0 +1,70 @@ +# ADR-0001: Vite + React 19 SPA + Hono Worker (migrate off Next.js + OpenNext) + +**Date:** ~2026-05 (Wave 2 migration landed; Worker name `reader` preserved) +**Status:** Current +**Supersedes:** [archive/decisions.md ADR-01](../../archive/decisions.md) (Next.js 16 on Cloudflare Workers via OpenNext) + +## Context + +Reader originally ran as a server-rendered Next.js App Router app deployed to +Cloudflare Workers via `@opennextjs/cloudflare`. That stack required two bespoke +patch scripts (`scripts/patch-opennext.mjs` pre + post, `scripts/fix-opennext-deps.mjs`) +to work around `@libsql/isomorphic-ws` `node.mjs` vs `web.mjs` resolution and +`WeakRef` / `FinalizationRegistry` not being free globals under +`nodejs_compat_v2`. `next build` also needed `ignoreBuildErrors: true` to avoid +timeouts, pushing type-checking out of the build pipeline. + +For a personal-use, client-heavy reading app with no SSR requirement, the +OpenNext layer was pure operational cost. + +## Decision + +Migrate to a Vite + React 19 single-page application backed by a Hono Worker. + +- **Single Vite entry:** `app.html` is the only HTML input; the browser loads + it and routes client-side via `react-router-dom`. +- **Hono Worker:** `src/worker.ts` mounts `/api/*` route modules and serves + built assets via the `ASSETS` binding. No SSR, no RSC, no Next.js. +- **Worker name preserved:** the Cloudflare Worker is still named `reader` so + the custom domain and secrets carry over without re-provisioning. +- **TypeScript split:** `tsconfig.app.json` (SPA) and `tsconfig.worker.json` + (Worker + server libs) are referenced by `tsconfig.json`; `pnpm typecheck` + runs `tsc --noEmit` against both. +- **Build pipeline:** `vite build` → `dist/`; `cf:build` overlays the Astro + landing onto `dist/index.html` and merges `_headers`; `wrangler deploy` + serves `dist/` via `ASSETS`. +- **SPA fallback:** the Worker explicitly falls back to `/app` for unmatched + GETs so client-side routes work on cold loads. + +## Rationale + +- Removes the OpenNext patch scripts and the `ignoreBuildErrors` workaround — + type-checking now runs in CI and locally via `pnpm typecheck`. +- Vite dev server is faster than Next dev for an SPA-only workload. +- Hono is a tiny, idiomatic Workers router; route modules under + `src/worker/routes/` map 1:1 to resources. +- The `ASSETS` binding + `run_worker_first` for `/api/*`, `/`, sitemap, and + agent surfaces gives the Worker first-touch on the routes that need it + without paying for SSR on every request. + +## Tradeoffs + +- No SSR / no streaming RSC — not acceptable for content-driven sites that need + SEO on protected routes, but Reader's public surface is the landing page + (Astro) plus agent-indexing surfaces (`llms.txt`, `index.md`, `api/ai`); the + app itself is auth-walled and not agent-indexed. +- Client-side routing means cold loads of `/library` fetch the SPA shell first. + Mitigated by lazy-loaded pages (`src/router.tsx`) and edge-cached HTML. +- `caches.default` and the R2 binding are unavailable in pure-`vite` dev; the + Worker dev server (`pnpm dev:worker`) provides them and the SPA dev server + (`pnpm dev:spa`) proxies `/api` to it. + +## Alternatives considered + +- **Stay on Next.js + OpenNext:** rejected — operational cost (patch scripts, + build timeouts, `ignoreBuildErrors`) with no SSR benefit for this app. +- **Astro for the whole app:** Astro is used for the landing overlay only; + adopting it for the auth-walled SPA would have required rebuilding the React + component tree as Astro islands. +- **Remix / TanStack Start on Workers:** considered; Hono + Vite kept the + existing React 19 component tree intact and the dependency surface smaller. diff --git a/docs/architecture/decisions/0002-turso-drizzle.md b/docs/architecture/decisions/0002-turso-drizzle.md new file mode 100644 index 0000000..78766a3 --- /dev/null +++ b/docs/architecture/decisions/0002-turso-drizzle.md @@ -0,0 +1,66 @@ +# ADR-0002: Turso (libSQL) via Drizzle ORM + +**Date:** 2026-04-25 (Firebase → Turso migration); carried forward post-Vite migration +**Status:** Current +**Supersedes:** [archive/decisions.md ADR-02](../../archive/decisions.md) (same decision; this record updates the runtime context to Vite + Hono) + +## Context + +The original data layer was Firebase Firestore (NoSQL, deny-all rules, O(n) +full-table scan for search, no real SQL). The migration plan is preserved in +[archive/plans-migrate-off-firebase.md](../../archive/plans-migrate-off-firebase.md) +and the retro in +[archive/retro-firebase-to-cloudflare-2026-04-25.md](../../archive/retro-firebase-to-cloudflare-2026-04-25.md). + +## Decision + +Use Turso (managed libSQL/SQLite) via Drizzle ORM. + +- Schema lives in `src/lib/db/schema.ts`; a lazy client proxy in + `src/lib/db/client.ts` defers `@libsql/client/web` instantiation until first + use so the Worker can bind its env first. +- Schema sync via `drizzle-kit push` (`pnpm db:push`). Migration SQL files are + also kept under `drizzle/` (`0000_baseline.sql`, `0001_memories.sql`, + `0002_first_green_goblin.sql` for RSS). +- JSON columns (`tags`, `notes`, `aiChat`, `summary`, `keyPoints`, + `pdfMetadata`, `sessionReview`) are stored as text and typed via + `$type()` for TS safety. No row-level JSON filtering in current query + patterns. +- `better-auth` uses its first-party Drizzle adapter on the same DB. + +## Rationale + +- libSQL is SQLite-compatible → portable, easy to reason about, queryable with + SQL. +- Drizzle adapter for better-auth exists natively. +- Turso + Workers `placement.mode = "smart"` co-locate the Worker with the + Turso primary, eliminating cross-region RTT on every request (the + pre-Smart-Placement TTFB was the dominant LCP contributor — see + [knowledge/learnings.md](../../knowledge/learnings.md)). +- External access from `drizzle-kit` and `tsx` scripts (migration scripts, + studio) works because Turso is reachable over HTTP, unlike Cloudflare D1 + which is Workers-only. + +## Tradeoffs + +- `drizzle-kit push` is schema-sync, not migration history. Safe for a + single-user DB; fragile if a push runs against production with data in an + incompatible old shape. Mitigation: additive migrations are committed under + `drizzle/` and applied deliberately; see + [operations/runbooks/migrate-schema.md](../../operations/runbooks/migrate-schema.md). +- The legacy NextAuth tables (`account`, `session`, `verificationToken`) remain + in schema as dead weight from the Auth.js → better-auth swap. Safe to drop + after manual verification (open question, tracked in STATUS). + +## Alternatives considered + +- **Cloudflare D1:** SQLite-compatible but no external access from + `drizzle-kit studio` or migration scripts. +- **Postgres (Neon/Supabase):** more powerful but heavier; `pg` driver has + Workers compat friction; not needed at single-user scale. +- **Keep Firestore:** rejected — O(n) search, no real SQL, deny-all rules. + +## Open questions + +- When to switch from `drizzle-kit push` to `drizzle-kit generate` for safer + schema changes as user count grows. Tracked in STATUS.md. diff --git a/docs/architecture/decisions/0003-r2-pdfs.md b/docs/architecture/decisions/0003-r2-pdfs.md new file mode 100644 index 0000000..7c25c94 --- /dev/null +++ b/docs/architecture/decisions/0003-r2-pdfs.md @@ -0,0 +1,56 @@ +# ADR-0003: Cloudflare R2 for PDF Storage + +**Date:** 2026-04-25 (migrated from GCS/Firebase Storage); native binding adopted 2026-04-27 +**Status:** Current +**Supersedes:** [archive/decisions.md ADR-03](../../archive/decisions.md) + +## Context + +PDFs were originally stored in Google Cloud Storage via Firebase Admin (signed +URLs). Firebase teardown required a replacement. A brief `@aws-sdk/client-s3` +interlude (2026-04-25) was replaced with the native Workers R2 binding once +the Cloudflare Paid plan unlocked it (commit `6a702be`). + +## Decision + +Store PDFs in Cloudflare R2 bucket `reader-pdfs`, bound as `PDFS_BUCKET` in +`wrangler.toml`. All access is proxied through authenticated, ownership-checked +Hono routes — clients never receive a raw R2 URL. + +- Upload: `POST /api/pdfs/upload` (`src/worker/routes/pdf.ts`) validates size + (10 MB) and magic bytes (`%PDF-`), then `PDFS_BUCKET.put('pdfs//.pdf')`. +- Download: `GET /api/pdfs/:id/download` checks auth + ownership, then + `PDFS_BUCKET.get(storageKey)` and streams the bytes. +- Helpers in `src/lib/storage.ts`; the bucket is bound to the module-level + singleton via `setPdfBucket(env.PDFS_BUCKET)` in `bindWorkerEnv()`. +- PDFs use a `blob://` sentinel as the article `url` field so they + do not collide with real HTTP article URLs in the `(user_id, url)` unique + index. + +## Rationale + +- Native Workers binding: zero egress cost, no S3-SDK bundle weight, no HTTP + hop. +- Auth and ownership enforced server-side on every download (prevents IDOR). +- Replaces `firebase-admin` which was being removed entirely. + +## Tradeoffs + +- The R2 binding is unavailable in pure-`vite` dev; `pnpm dev:worker` + (wrangler dev) provides it and `pnpm dev:spa` proxies `/api` to it. +- 10 MB per-file limit and magic-byte validation are enforced in + `src/lib/pdf-service.ts` and the upload route; do not trust `file.type` + (browser-controlled and spoofable). + +## Alternatives considered + +- **Vercel Blob:** abandoned when the deploy moved off Vercel. +- **Cloudflare KV:** not suited for large binary blobs. +- **Durable Objects:** over-engineered for object storage. +- **Keep GCS:** would require `firebase-admin`. + +## Paused / deferred + +- PDF storage representation is "paused at the current documented + representation" per the closure decision (2026-07-10). Reopen only for a + concrete storage defect. See STATUS.md. diff --git a/docs/architecture/decisions/0004-better-auth-google.md b/docs/architecture/decisions/0004-better-auth-google.md new file mode 100644 index 0000000..f42756e --- /dev/null +++ b/docs/architecture/decisions/0004-better-auth-google.md @@ -0,0 +1,56 @@ +# ADR-0004: better-auth (Google OAuth via Drizzle Adapter) + +**Date:** 2026-04-25 (replaced Firebase Auth; brief Auth.js v5 detour settled on better-auth) +**Status:** Current +**Supersedes:** [archive/decisions.md ADR-05](../../archive/decisions.md) + +## Context + +Firebase Auth was removed as part of the migration off Firebase. The original +plan (`archive/plans-migrate-off-firebase.md`) targeted Auth.js v5 +(`next-auth@beta`) with the libSQL adapter; at cutover `better-auth` was chosen +instead because of its first-party Drizzle adapter. + +## Decision + +Use `better-auth` v1.6 with its Drizzle adapter, Google OAuth only. + +- Server config: `src/lib/auth.ts` (`createAuth(env)`). Reads + `BETTER_AUTH_SECRET` (falls back to `AUTH_SECRET`), `BETTER_AUTH_URL` + (falls back to `BETTER_AUTH_BASE_URL`, then the production URL), and the + Google client credentials from `env`. +- Browser client: `src/lib/auth-client.ts`. +- Handler: mounted as a Hono catch-all in `src/worker.ts` — + `api.on(['GET','POST'], '/api/auth/*', ...)` returns `auth.handler(c.req.raw)`. +- Plugins: `oneTap()` (Google One Tap sign-in). +- Rate limiting: **disabled** (`rateLimit: { enabled: false }`). Open question: + revisit if the app becomes public. Tracked in STATUS.md. +- Auth resolution for API routes: `getAuthenticatedUserId()` in + `src/lib/auth-api.ts` accepts either a `Bearer rdr_*` API key (extension) or + a session cookie (webapp). See [../data-flow.md](../data-flow.md). + +## Rationale + +- First-party Drizzle adapter on the same Turso DB; no second data store. +- `oneTap()` plugin gives Google One Tap sign-in with no extra wiring. +- Hono catch-all forwards the raw `Request` to `auth.handler`, preserving + multiple `Set-Cookie` headers (the OAuth callback sets session token + state + clear in one response — see the comment in `src/worker.ts`). + +## Tradeoffs + +- Legacy NextAuth tables (`account`, `session`, `verificationToken`) remain in + `src/lib/db/schema.ts` as unused remnants from the Auth.js detour. Safe to + drop after confirming no active rows. Open question, tracked in STATUS. +- `BETTER_AUTH_SECRET` must be set as a Wrangler secret; in non-production + builds `createAuth` falls back to a hardcoded dev string to avoid blocking + the build. +- Rate limiting disabled — acceptable for single-user scale; revisit before + any public launch. + +## Alternatives considered + +- **Auth.js (NextAuth) v5:** original plan-doc choice; `better-auth` chosen at + cutover for the Drizzle adapter. +- **Firebase Auth:** removed — the point of the migration. +- **Clerk / Auth0:** third-party managed; adds cost and an external dependency. diff --git a/docs/architecture/decisions/0005-ai-gateway-byok.md b/docs/architecture/decisions/0005-ai-gateway-byok.md new file mode 100644 index 0000000..5de2705 --- /dev/null +++ b/docs/architecture/decisions/0005-ai-gateway-byok.md @@ -0,0 +1,68 @@ +# ADR-0005: AI SDK + free-ai-gateway + BYOK (no server-side key storage) + +**Date:** 2026-02-13 (AI SDK integrated); gateway pattern formalised ~2026-04-27 +**Status:** Current +**Supersedes:** [archive/decisions.md ADR-04](../../archive/decisions.md) + +## Context + +Reader needs LLM chat and summarisation. Three paths were available: call +provider APIs directly, proxy through a self-hosted gateway, or use the +Workers AI binding directly. + +## Decision + +All server-side AI calls go through the fleet `free-ai-gateway` +(`AI_BASE_URL`, default `https://ai-gateway.sassmaker.com/v1`) using +`@ai-sdk/openai-compatible` + the Vercel AI SDK. The gateway enforces a +9500 Neuron/day fleet-wide cap and attributes spend per project via the +`x-gateway-project-id: reader` header. + +Client-side BYOK: users supply their own OpenAI/Anthropic/Gemini API key, +which the browser sends per-request; the server proxies it directly to the +provider and **never persists it**. A local AI dev path +(`scripts/local-ai.mjs`) bridges a local LLM for zero-cost development. + +- Model factory: `src/lib/ai-cloudflare.ts` (`getLanguageModel()`). +- Server-side streaming + summarisation: `src/worker/routes/ai.ts`. +- BYOK key normalisation: `normalizeApiKey()` in `src/lib/ai-server.ts` + (length-capped, trimmed). +- Local AI bridge: `createLocalAITextStream()` in `src/lib/ai-server.ts`, + gated by `isLocalCLIEnabled()` (`NODE_ENV === 'development'`). +- Extension AI chat: `/api/ext/chat` in `src/worker/routes/misc.ts` reuses the + same gateway/BYOK logic, authenticated via `rdr_*` API keys. + +## Rationale + +- Single budget chokepoint across the fleet: the gateway owns the daily Neuron + budget so no single project can exhaust it. The `x-gateway-project-id` + header is required for per-project attribution. +- BYOK keys stored in the browser only → zero server-side key storage risk. + Aligns with the security note in `README.md`. +- `@ai-sdk/openai-compatible` allows any OpenAI-compatible endpoint (Workers + AI, OpenAI, Anthropic via OpenAI-compat, Gemini) without provider-specific + SDKs. +- Vercel AI SDK `streamText` + `toTextStreamResponse` handles streaming + uniformly across all providers. + +## Tradeoffs + +- BYOK means users must have their own API keys for non-free models. +- The gateway is a fleet dependency; if it is down, free AI is down. BYOK and + local AI are the fallbacks. +- The default model (`@cf/meta/llama-3.3-70b-instruct-fp8-fast`) is a Workers + AI model routed through the gateway; model selection is configurable. + +## Alternatives considered + +- **Direct provider SDKs:** multiple deps, no unified streaming, no budget + control. +- **Workers AI binding directly:** only works on Cloudflare; no dev path and + no multi-provider support. +- **Self-hosted OpenAI proxy (LiteLLM etc.):** operational overhead. + +## Security notes + +- BYOK keys: never written to the database or logs; normalised before use. +- `rdr_*` API keys (extension): only the SHA-256 hash is persisted; plaintext + is shown once at creation. See [0006-mv3-side-panel.md](0006-mv3-side-panel.md). diff --git a/docs/architecture/decisions/0006-mv3-side-panel.md b/docs/architecture/decisions/0006-mv3-side-panel.md new file mode 100644 index 0000000..a18ab8d --- /dev/null +++ b/docs/architecture/decisions/0006-mv3-side-panel.md @@ -0,0 +1,61 @@ +# ADR-0006: Chrome MV3 Side Panel + hashed `rdr_*` API keys + +**Date:** 2026-04-04 (extension scaffolded with side panel from the start) +**Status:** Current +**Supersedes:** [archive/decisions.md ADR-06](../../archive/decisions.md) + +## Context + +The Chrome extension needs a UI surface for chat and save actions. MV3 offers +popup (ephemeral, closes on blur) and side panel (persistent, stays open while +browsing). Extension requests originate from the extension's origin, not the +Worker's origin, so the same-origin session cookie cannot be shared. + +## Decision + +- **Side panel** (`sidePanel` permission, `side_panel.default_path`) as the + primary UI for persistent chat sessions. +- **Popup** (`action.default_popup`) for ephemeral one-click actions + (Save to Library / Import & Read). +- **Auth via hashed `rdr_*` API keys**, not session cookies. The `api_keys` + table stores `token_hash` (SHA-256) + `prefix` + `revoked_at`; the plaintext + is shown once at creation and never persisted. The extension sends the raw + token as `Authorization: Bearer rdr_...`; `verifyApiKey()` hashes it for + lookup. See `src/lib/api-keys.ts` and `src/worker/routes/keys.ts`. +- Content script runs `@mozilla/readability` in the page context on demand; + both surfaces trigger it via Chrome messaging. +- Chrome Reading List sync (URLs, titles, read state) via the `readingList` + permission. + +## Rationale + +- Side panel persists while the user navigates — essential for reading/chatting + across page loads. Popup closes on blur, fine for one-click capture but not + for extended chat. +- Hashed long-lived API keys avoid the complexities of OAuth token refresh + from a service worker context (MV3 service workers are ephemeral). +- `rdr_` prefix lets `getAuthenticatedUserId()` distinguish an extension key + from a BYOK provider key in the same `Authorization` header. + +## Tradeoffs + +- Side panel requires `sidePanel` permission (Chrome 114+); limits to + Chromium-based browsers. +- Both surfaces share the same content script but have separate React app + entry points (`popup/`, `side-panel/`). +- The extension is built independently in `packages/chrome-extension/` and is + excluded from root Biome/ESLint tooling (see `biome.json` `!**/packages`). + +## Alternatives considered + +- **Popup-only:** simpler, but closes on blur — no persistent chat UX. +- **Full-page extension tab:** loses connection to the current browsing + context. +- **Session cookie sharing:** not possible cross-origin in MV3. + +## Distribution + +Extension distribution is **deferred** until web import and capture flow are +reliable (see [product/overview.md](../../product/overview.md) scope). Local +install: `packages/chrome-extension/ → pnpm dev → load unpacked`. See +`packages/chrome-extension/README.md` and `PRIVACY.md`. diff --git a/docs/architecture/decisions/0007-content-extraction.md b/docs/architecture/decisions/0007-content-extraction.md new file mode 100644 index 0000000..447c833 --- /dev/null +++ b/docs/architecture/decisions/0007-content-extraction.md @@ -0,0 +1,59 @@ +# ADR-0007: Content Extraction Stack + +**Date:** 2026-02-13 (`linkedom` adopted replacing Playwright); pdfjs added 2026-02-14 +**Status:** Current +**Supersedes:** [archive/decisions.md ADR-07](../../archive/decisions.md) + +## Context + +Articles are captured server-side from a URL; PDFs are uploaded and must be +viewable with text extraction. The extraction stack must run inside the +Cloudflare Workers runtime (no full browser process). + +## Decision + +- **HTML extraction (server):** `@mozilla/readability` + `linkedom` in the + `/api/snapshot` flow. `linkedom` is a pure-JS DOM parser that replaced + Playwright (too heavy/slow for Workers). HTML is sanitised before storage + and re-sanitised on read. +- **PDF viewing (client):** `pdfjs-dist` + `react-pdf` in + `src/components/PDFReaderClient.tsx`. The pdfjs web worker is loaded from + `public/pdf.worker.min.mjs` (local static asset), not from a CDN. +- **PDF text extraction (server, at upload):** `pdf-parse` (legacy) — text is + stored in `articles.extracted_text`. + +## Rationale + +- Readability is the reference implementation of the Mercury/Readability + algorithm; produces clean article content. +- `linkedom` is lighter than JSDOM and Workers-compatible; Playwright requires + a full browser process which is not viable in the Workers runtime. +- `pdfjs-dist` is the canonical PDF renderer for web; no viable pure-JS + alternative. +- The pdfjs web worker is loaded locally (not from CDN) to satisfy the + extension's strict CSP (`script-src 'self'`) and to avoid sandboxed CF + Workers blocking remote script fetches. + +## Tradeoffs + +- `linkedom` is less complete than JSDOM for edge-case DOM APIs; acceptable + for article extraction where Readability handles the parsing. +- `pdf-parse` runs at upload time and stores extracted text; re-extraction + requires re-upload. PDF metadata (page count, file size, storage path) is + stored in `articles.pdf_metadata`. + +## Alternatives considered + +- **Playwright / Puppeteer:** requires a full browser process; not viable in + Workers runtime; was used briefly before `linkedom`. +- **JSDOM:** heavier than `linkedom`; more node-specific APIs. +- **External PDF API (PDF.co etc.):** adds cost and an external dependency. + +## Security notes + +- All server-side URL fetches (snapshot, proxy, RSS refresh) funnel through + `validateExternalUrl()` + `fetchWithValidatedRedirects()` — see + [../data-flow.md](../data-flow.md). +- HTML is sanitised at ingestion and re-sanitised on read for + defence-in-depth (carried forward from the security audit; see + [archive/security-audit-2026-03-29.md](../../archive/security-audit-2026-03-29.md)). diff --git a/docs/architecture/decisions/0008-rss-inbox.md b/docs/architecture/decisions/0008-rss-inbox.md new file mode 100644 index 0000000..9997034 --- /dev/null +++ b/docs/architecture/decisions/0008-rss-inbox.md @@ -0,0 +1,72 @@ +# ADR-0008: RSS/Atom Inbox (manual refresh, no scheduled triggers) + +**Date:** 2026-07-13 +**Status:** Current +**Specs:** [`openspec/specs/rss-*`](../../../openspec/specs/) · Archived change: [`openspec/changes/archive/2026-07-13-add-rss-reader/`](../../../openspec/changes/archive/2026-07-13-add-rss-reader/) + +## Context + +Reader could save individual articles and PDFs but could not follow publishing +sources or surface new posts. A focused RSS inbox turns the existing library +into a repeatable reading workflow. + +## Decision + +Add an authenticated RSS/Atom inbox scoped per user: + +- **Two dedicated tables** (`rss_feeds`, `rss_entries`) — see + `src/lib/db/schema.ts`. Feeds unique by `(user_id, feed_url)`; entries unique + by `(feed_id, external_id)` where `external_id` is the feed GUID/id or a + deterministic fallback. Read state and saved article ID live on the entry. + This keeps transient inbox items out of `articles` (treating all entries as + articles would inflate search and blur explicit save intent). +- **Parse feeds in the Worker without a new dependency:** `DOMParser` from the + existing `linkedom` dep + small RSS/Atom normalisation helpers + (`src/lib/rss-parser.ts`). Sanitise any HTML before persisting/returning. +- **Manual refresh only:** `POST /api/rss/refresh` refreshes all subscriptions + or a selected feed with bounded concurrency (4), a 15 s per-feed timeout, + response-size cap, entry cap, and per-feed error isolation. One bad feed + does not abort the others. `ETag` / `Last-Modified` are honoured. +- **OPML import server-side:** `POST /api/rss/import` validates payload size, + recursively reads outlines with `xmlUrl`, accepts only HTTP(S) URLs after + SSRF validation, upserts owned subscriptions, reports + imported/existing/rejected counts. +- **Save entry to library:** `POST /api/rss/entries/:id/save` is idempotent — + creates an article record when content is available, otherwise a link-type + article, and links the entry to it. +- **One dense inbox page:** `/rss` with feed sidebar/filter, unread/all + toggle, refresh/import controls, chronological entry list. Opening an entry + marks it read and opens the canonical URL in a new tab. + +## Rationale + +- Dedicated tables avoid overloading `articles` with transient inbox items. +- Reusing `linkedom` avoids a new feed-parser dependency and bundle cost. +- Manual refresh is predictable and sufficient for personal use; scheduled + refresh would require Worker trigger/config changes and operational + behaviour beyond the requested MVP. +- Server-side OPML parsing centralises validation and is reusable by future + clients. + +## Tradeoffs + +- Manual refresh across many feeds can be slow; mitigated by bounded + concurrency and a page-level progress state with partial success details. +- Feed formats vary widely; common RSS 2.0 and Atom patterns are supported + with fixture tests. Unsupported feeds return a per-feed error. +- Untrusted feed URLs can target internal services → reuse SSRF validation + for initial URLs and redirects; reject non-HTTP(S), private, loopback, and + metadata destinations. + +## Migration note + +The additive `drizzle/0002_first_green_goblin.sql` migration must be applied +before deployment. See +[operations/runbooks/migrate-schema.md](../../operations/runbooks/migrate-schema.md). + +## Open questions / deferred + +- Background refresh cadence and Cloudflare scheduled triggers remain deferred + until manual use demonstrates the need. +- Folder preservation from nested OPML outlines remains deferred; the first + version imports every feed into a flat subscription list. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md new file mode 100644 index 0000000..2528e8e --- /dev/null +++ b/docs/architecture/overview.md @@ -0,0 +1,94 @@ +# Architecture Overview + +Reader is a Vite + React 19 single-page application served from a Hono Worker +on Cloudflare Workers. No SSR, no Next.js. The browser loads `app.html` (one +Vite entry) and routes client-side via `react-router-dom`; the Worker handles +`/api/*` and serves built assets via the `ASSETS` binding. + +## Shape + +``` + ┌──────────────────────── Cloudflare Worker (reader) ────────────────────────┐ + │ │ + browser ──────► src/worker.ts │ + │ ├── handleAgentEdge(request) ← /llms.txt, /index.md, /api/ai, … │ + │ ├── api.fetch() ← Hono router under /api/* │ + │ │ ├── /api/auth/* ← better-auth (Google OAuth, Drizzle) │ + │ │ ├── /api/articles ← articles.ts │ + │ │ ├── /api/boards ← boards.ts │ + │ │ ├── /api/lists ← lists.ts │ + │ │ ├── /api/memories ← memories.ts │ + │ │ ├── /api/ai ← ai.ts (chat / summarize / models) │ + │ │ ├── /api/keys ← keys.ts (rdr_* API keys) │ + │ │ ├── /api/pdfs ← pdf.ts (R2-backed) │ + │ │ ├── /api/rss ← rss.ts (feeds + entries + OPML) │ + │ │ ├── /api/share ← share.ts (public share) │ + │ │ └── /api/* ← misc.ts (search, tags, snapshot, proxy, │ + │ │ data-export, ext chat, browser-mem)│ + │ └── env.ASSETS.fetch() ← built SPA + landing (dist/) │ + │ │ + │ Bindings: PDFS_BUCKET (R2), ASSETS (dist/) │ + └────────────────────────────────┬─────────────────────────────────────────┘ + │ + ┌──────────────────────────────┼──────────────────────────────┐ + ▼ ▼ ▼ + Turso (libSQL) Cloudflare R2 free-ai-gateway + via Drizzle ORM reader-pdfs bucket (AI_BASE_URL) + (articles, boards, (PDF binaries, + BYOK providers + lists, memories, rss, proxied downloads) + local-ai (dev) + api_keys, better-auth) +``` + +## Key files + +- `src/worker.ts` — Hono Worker entry. Security headers, `/api/*` routing, + asset serving, SPA fallback, agent-edge handler. +- `src/worker/routes/*.ts` — one Hono router per resource; mounted under + `/api/`. +- `src/lib/db/schema.ts` — Drizzle schema (app tables + better-auth tables + + legacy NextAuth tables kept for reference). +- `src/lib/db/client.ts` — Turso libSQL client. Lazy proxy so the client is + not created at module load (required for the Workers runtime). +- `src/lib/auth.ts` — better-auth server config (`createAuth`), Drizzle + adapter, Google OAuth, `oneTap` plugin, rate limiting disabled. +- `src/lib/auth-api.ts` — `getAuthenticatedUserId()` resolves either a + `Bearer rdr_*` API key (extension) or a better-auth session cookie (web). +- `src/lib/storage.ts` — R2 helpers for `PDFS_BUCKET`. +- `src/lib/ai-cloudflare.ts` — builds a `LanguageModel` from an OpenAI-compatible + endpoint, routed through the free-ai gateway with `x-gateway-project-id: reader`. +- `src/lib/url-validation.ts` + `src/lib/safe-fetch.ts` — SSRF protection and + redirect-safe fetch used by snapshot/proxy/RSS refresh. +- `src/router.tsx` — client-side routes (lazy-loaded pages). +- `wrangler.toml` — Worker config: `main = src/worker.ts`, `ASSETS` + `PDFS_BUCKET` + bindings, `placement.mode = "smart"`, `nodejs_compat_v2`, custom domain. +- `vite.config.ts` — Vite SPA build (React, Tailwind v4, Lightning CSS). +- `app.html` — single SPA HTML entry (Vite input; carries inline shell CSS). + +## Build & deploy pipeline + +``` +pnpm deploy + → validate:env:deploy (scripts/validate-env.mjs) + → cf:build + → pnpm build (validate env + vite build → dist/) + → pnpm --filter ./landing-astro build + → node scripts/overlay-astro-landing.mjs (overlay landing → dist/) + → wrangler deploy (Worker + ASSETS binding serves dist/) +``` + +The landing page (`landing-astro/`) is an **overlay**, not a separate product. +It overwrites `dist/index.html` and merges `_headers`; the SPA lives at +`dist/app.html` and is served at `/app`. SPA fallback uses +`not_found_handling = single-page-application` semantics via the Worker's +explicit fallback to `/app`. + +## Decisions + +The why behind this shape is in [decisions/](decisions/). Start with +[0001-vite-spa-hono-worker.md](decisions/0001-vite-spa-hono-worker.md) for the +migration off Next.js + OpenNext. + +## Data flow + +See [data-flow.md](data-flow.md) for the request lifecycle, auth resolution, +storage paths, and AI routing. diff --git a/docs/archive/decisions.md b/docs/archive/decisions.md index 69e5f88..8b6c818 100644 --- a/docs/archive/decisions.md +++ b/docs/archive/decisions.md @@ -1,4 +1,13 @@ -# Architecture Decision Records — Reader +# Architecture Decision Records — Reader (pre-Vite, historical) + +> **Historical record.** These ADRs were authored while Reader ran on +> Next.js + OpenNext on Cloudflare Workers. The app has since migrated to a +> Vite + React 19 SPA backed by a Hono Worker (see +> [`docs/architecture/decisions/`](../architecture/decisions/) for current +> ADRs). This file is preserved verbatim for context and git history. +> ADR-01 (Next.js + OpenNext) is **superseded** by the Vite/Hono migration; +> ADR-02 through ADR-07 largely still apply but reference the old runtime in +> places. Read with the migration in mind. Decisions are listed in rough chronological order. Rationale is drawn from code comments, plan docs, and git history. Unknown rationale is flagged diff --git a/docs/archive/index.md b/docs/archive/index.md new file mode 100644 index 0000000..0b8db85 --- /dev/null +++ b/docs/archive/index.md @@ -0,0 +1,55 @@ +# Archive + +Historical records preserved verbatim (with a dated historical marker where +needed). These describe decisions, lessons, plans, and audits that were +authoritative at a point in time. Current truth lives in the rest of `docs/`. + +When archiving a superseded doc, move it here with `git mv`, give it a dated +filename, and prepend a one-line historical marker pointing at the current +canonical doc. + +## Pre-Vite architecture (Next.js + OpenNext era) + +- [`decisions.md`](decisions.md) — ADRs authored while Reader ran on Next.js + + OpenNext. ADR-01 (Next.js + OpenNext) is superseded by + [../architecture/decisions/0001-vite-spa-hono-worker.md](../architecture/decisions/0001-vite-spa-hono-worker.md); + ADR-02 through ADR-07 largely still apply but reference the old runtime. +- [`lessons.md`](lessons.md) — Engineering lessons captured pre-Vite. Many + describe code that no longer exists (`scripts/patch-opennext.mjs`, + `next.config.ts`, `WeakRef` patches, `serverExternalPackages`). Current + lessons live in [../knowledge/learnings.md](../knowledge/learnings.md). +- [`learning-pre-vite-external-references.md`](learning-pre-vite-external-references.md) + — External references curated pre-Vite. Current references live in + [../knowledge/external-references.md](../knowledge/external-references.md). +- [`learning-pre-vite-new-things.md`](learning-pre-vite-new-things.md) — + "New things to learn" notes from the pre-Vite era. + +## Migration history + +- [`plans-migrate-off-firebase.md`](plans-migrate-off-firebase.md) — Firebase + → Turso + Auth.js + Vercel Blob migration plan (DONE). The actual cutover + chose better-auth over Auth.js and R2 over Vercel Blob. +- [`retro-firebase-to-cloudflare-2026-04-25.md`](retro-firebase-to-cloudflare-2026-04-25.md) + — Retro for the full infra swap (DB, Auth, Storage, Deployment). +- [`plans-browser-memory-import.md`](plans-browser-memory-import.md) — + Browser-memory import plan (DONE; became `/api/browser-memory/import`). +- [`plans-memory-capture-prototype.md`](plans-memory-capture-prototype.md) — + Memory capture prototype plan (DONE; promoted to persisted/authenticated). +- [`plans-chrome-extension-chat-with-page-2026-04-24.md`](plans-chrome-extension-chat-with-page-2026-04-24.md) + — Chrome extension chat-with-page plan (archived). + +## Audits / context + +- [`security-audit-2026-03-29.md`](security-audit-2026-03-29.md) — Security + audit from the Next.js + Firebase era. Many findings were fixed during the + migration; residual items (CORS, rate limiting) are tracked in + [../knowledge/failed-approaches.md](../knowledge/failed-approaches.md) and + STATUS.md. +- [`project-recommendation-context-2026-06-06.md`](project-recommendation-context-2026-06-06.md) + — CodeVetter Repo Unpacked-style audit for Starboard recommendations. + References the pre-Vite `src/app/...` path layout; preserved for context. + +## Marketing + +Marketing copy iterations live in [`../marketing/`](../marketing/) (current, +not archived). diff --git a/docs/learning/external-references.md b/docs/archive/learning-pre-vite-external-references.md similarity index 100% rename from docs/learning/external-references.md rename to docs/archive/learning-pre-vite-external-references.md diff --git a/docs/learning/new-things.md b/docs/archive/learning-pre-vite-new-things.md similarity index 98% rename from docs/learning/new-things.md rename to docs/archive/learning-pre-vite-new-things.md index dfdf40c..c281fd9 100644 --- a/docs/learning/new-things.md +++ b/docs/archive/learning-pre-vite-new-things.md @@ -1,7 +1,7 @@ # New things to learn — reader Technologies encountered during reader development that are worth understanding in depth. -See also: [external-references.md](./external-references.md) +See also: [external-references.md](./learning-pre-vite-external-references.md) --- diff --git a/docs/archive/lessons.md b/docs/archive/lessons.md index cb64bbc..2477c8c 100644 --- a/docs/archive/lessons.md +++ b/docs/archive/lessons.md @@ -1,4 +1,13 @@ -# Engineering Lessons — Reader +# Engineering Lessons — Reader (pre-Vite, historical) + +> **Historical record.** These lessons were captured while Reader ran on +> Next.js + `@opennextjs/cloudflare`. The app has since migrated to a Vite + +> React 19 SPA with a Hono Worker, so the OpenNext-specific lessons +> (`scripts/patch-opennext.mjs`, `next.config.ts`, `WeakRef` patches, +> `serverExternalPackages`, etc.) describe code that no longer exists in the +> repo. They are preserved verbatim because the underlying failure modes and +> reasoning are still useful context. Current, applicable lessons live in +> [`docs/knowledge/learnings.md`](../knowledge/learnings.md). Concrete lessons evidenced by code, scripts, or git history. Each links to the decision record where relevant. diff --git a/plans/browser-memory-import.md b/docs/archive/plans-browser-memory-import.md similarity index 100% rename from plans/browser-memory-import.md rename to docs/archive/plans-browser-memory-import.md diff --git a/plans/archive/2026-04-24-chrome-extension-chat-with-page.md b/docs/archive/plans-chrome-extension-chat-with-page-2026-04-24.md similarity index 100% rename from plans/archive/2026-04-24-chrome-extension-chat-with-page.md rename to docs/archive/plans-chrome-extension-chat-with-page-2026-04-24.md diff --git a/plans/memory-capture-prototype.md b/docs/archive/plans-memory-capture-prototype.md similarity index 100% rename from plans/memory-capture-prototype.md rename to docs/archive/plans-memory-capture-prototype.md diff --git a/plans/migrate-off-firebase.md b/docs/archive/plans-migrate-off-firebase.md similarity index 100% rename from plans/migrate-off-firebase.md rename to docs/archive/plans-migrate-off-firebase.md diff --git a/docs/PROJECT_RECOMMENDATION_CONTEXT.md b/docs/archive/project-recommendation-context-2026-06-06.md similarity index 100% rename from docs/PROJECT_RECOMMENDATION_CONTEXT.md rename to docs/archive/project-recommendation-context-2026-06-06.md diff --git a/docs/retros/2026-04-25-firebase-to-cloudflare.md b/docs/archive/retro-firebase-to-cloudflare-2026-04-25.md similarity index 100% rename from docs/retros/2026-04-25-firebase-to-cloudflare.md rename to docs/archive/retro-firebase-to-cloudflare-2026-04-25.md diff --git a/AUDIT.md b/docs/archive/security-audit-2026-03-29.md similarity index 100% rename from AUDIT.md rename to docs/archive/security-audit-2026-03-29.md diff --git a/docs/development/commands.md b/docs/development/commands.md new file mode 100644 index 0000000..53d1aee --- /dev/null +++ b/docs/development/commands.md @@ -0,0 +1,69 @@ +# Commands + +Source of truth: `scripts` in `package.json`. This page annotates intent and +ordering; run `pnpm run` to see the live list. + +## Web app (root workspace) + +| Command | Purpose | +| --- | --- | +| `pnpm dev` | Worker (`wrangler dev`, :8787) + Vite SPA (:5173) + `local-ai.mjs`, concurrently | +| `pnpm dev:worker` | `wrangler dev` only (Worker, :8787) | +| `pnpm dev:spa` | `vite` only (SPA dev server, :5173, proxies `/api` → 8787) | +| `pnpm local-ai` | Local AI bridge (`scripts/local-ai.mjs` spawns `../local-ai/index.mjs` or legacy `../cli-bridge/index.mjs`) | +| `pnpm cli-bridge` | Alias for `pnpm local-ai` | +| `pnpm build` | `validate-env.mjs build` + `vite build` → `dist/` | +| `pnpm cf:build` | `pnpm build` + `landing-astro` build + `overlay-astro-landing.mjs` | +| `pnpm preview` | `vite preview` | +| `pnpm deploy` | `validate:env:deploy` + `cf:build` + `wrangler deploy` | +| `pnpm lint` | `biome check .` | +| `pnpm type-check` / `pnpm typecheck` | `tsc --noEmit -p tsconfig.app.json` + `tsc --noEmit -p tsconfig.worker.json` | +| `pnpm validate:env:build` | `node scripts/validate-env.mjs build` | +| `pnpm validate:env:runtime` | `node scripts/validate-env.mjs runtime` | +| `pnpm validate:env:deploy` | `node scripts/validate-env.mjs deploy` | +| `pnpm test` | `vitest run` | +| `pnpm test:watch` | `vitest` | +| `pnpm test:coverage` | `vitest run --coverage` | +| `pnpm test:e2e` | `playwright test` | +| `pnpm memory:demo` | `tsx scripts/memory-capture-demo.ts` | +| `pnpm db:push` | `drizzle-kit push` (schema sync) | +| `pnpm db:studio` | `drizzle-kit studio` | +| `pnpm migrate:firestore` | Legacy Firestore → Turso migration (`tsx scripts/migrate-firestore-to-turso.ts`) | +| `pnpm prepare` | `husky` (installs pre-commit hook) | +| `pnpm format` | `biome format --write .` | +| `pnpm format:check` | `biome format .` | +| `pnpm check` | `biome check .` | +| `pnpm docs:check` | `node scripts/check-docs.mjs` — validate docs/ links + structure | +| `pnpm docs:build` | `blume build` — render docs/ via Blume (presentation only; requires `pnpm add -D blume` first) | +| `pnpm docs:dev` | `blume dev` — local Blume dev server (requires `pnpm add -D blume` first) | + +## Chrome extension (`packages/chrome-extension/`) + +Separate Vite build; excluded from root Biome/ESLint tooling. + +| Command | Purpose | +| --- | --- | +| `pnpm dev` | `vite build --watch` → `dist/` | +| `pnpm build` | `vite build` (production) | +| `pnpm type-check` | `tsc --noEmit` | +| `pnpm test` | `vitest run` | +| `pnpm pack:zip` | `pnpm build` + zip `dist/` → `web-annotator-extension-.zip` | + +## Landing (`landing-astro/`) + +| Command | Purpose | +| --- | --- | +| `pnpm dev` | `astro dev` | +| `pnpm build` | `astro build` → `landing-astro/dist/` (overlaid onto `dist/` by `cf:build`) | +| `pnpm preview` | `astro preview` | + +## Build pipeline ordering + +``` +pnpm build = validate-env(build) → vite build +pnpm cf:build = pnpm build → landing-astro build → overlay-astro-landing.mjs +pnpm deploy = validate-env(deploy) → cf:build → wrangler deploy +``` + +`pnpm docs:build` and `pnpm docs:dev` are **not** part of the production +build — Blume is the presentation layer for the documentation site only. diff --git a/docs/development/conventions.md b/docs/development/conventions.md new file mode 100644 index 0000000..eb78da2 --- /dev/null +++ b/docs/development/conventions.md @@ -0,0 +1,67 @@ +# Conventions + +## Code style + +- **Formatter / linter:** Biome (`biome.json`). `pnpm format` writes; + `pnpm lint` checks. Biome is the authority for formatting and linting at + the root. +- **Indent:** 2 spaces, single quotes, semicolons, ES5 trailing commas, + LF line endings, 100 col width (see `biome.json`). +- **TypeScript:** strict mode, `tsc --noEmit` against `tsconfig.app.json` + (SPA) and `tsconfig.worker.json` (Worker). The root `tsconfig.json` is a + project-references shell that delegates to those two. +- **Path alias:** `@/*` → `./src/*` (configured in both app and worker + tsconfigs). +- **React:** React 19, function components, `react-router-dom` v7 with + lazy-loaded pages (`src/router.tsx`). +- **Data fetching:** `@tanstack/react-query`; `ReactQueryHydrate` is a thin + `HydrationBoundary` wrapper for seeding the client query cache. + +## Pre-commit hook (Husky + lint-staged) + +`.husky/pre-commit` runs `npx lint-staged`. `lint-staged` config in +`package.json` runs `biome check --write` on staged +`*.{js,jsx,ts,tsx,json,css}`. If the hook modifies files, re-stage them and +retry the commit. + +## Tooling scope + +- Biome ignores `.next`, `.open-next`, `.wrangler`, `out`, `dist`, `build`, + `node_modules`, `.astro`, `vite-env.d.ts`, `cloudflare-env.d.ts`, + lockfiles, `tsconfig.tsbuildinfo`, `.cf-pages-bundle`, `packages`, and + `*.html` (see `biome.json` `files.includes`). The Chrome extension has its + own Vite/Vitest config and is excluded from root tooling. +- Prettier config (`.prettierrc`) and `.prettierignore` exist but Biome is + the active formatter for the root workspace. + +## Commit convention + +Conventional Commits: + +``` +feat(reader): add PDF export +fix(auth): resolve token refresh issue +chore: update dependencies +docs: consolidate knowledge system +``` + +## Git rules + +- Do not commit `.env`, auth credentials, or `firebase-service-account.json` + (all in `.gitignore`). Verify `.gitignore` before any push. +- Do not push, deploy, run migrations, or open PRs without explicit user + approval. See `AGENTS.md` and the fleet standard at `../AGENTS.md`. + +## Documentation conventions + +- Markdown under `docs/` is the source of truth. See + [../index.md#maintenance-rules](../index.md#maintenance-rules). +- Run `pnpm docs:check` before committing doc changes — it catches broken + links and files outside the canonical structure. +- Blume (`blume.config.ts`) only renders `docs/`; never edit generated Blume + output. + +## Spec-driven changes + +Non-trivial feature work uses the OpenSpec workflow. See +[openspec.md](openspec.md). diff --git a/docs/development/openspec.md b/docs/development/openspec.md new file mode 100644 index 0000000..1726a4c --- /dev/null +++ b/docs/development/openspec.md @@ -0,0 +1,58 @@ +# OpenSpec Workflow + +Non-trivial feature work in Reader uses the **OpenSpec** spec-driven workflow. +Tooling lives in [`openspec/`](../../openspec/) at the repo root; agent skills +for the workflow are installed under `.codex/skills/openspec-*` (do not modify +those skill directories — they are tooling, not docs). + +## Shape + +``` +openspec/ + config.yaml # OpenSpec project config (schema: spec-driven) + specs/ # current capability specs (source of truth post-archive) + rss-inbox/spec.md + rss-refresh/spec.md + rss-subscriptions/spec.md + changes/ + archive/ # archived, completed change proposals + 2026-07-13-add-rss-reader/ + .openspec.yaml + proposal.md + design.md + tasks.md + specs/rss-*/spec.md +``` + +## Lifecycle + +1. **Explore** — read existing specs and code to ground the proposal. +2. **Propose** — `openspec/changes//proposal.md` describing why, what + changes, capabilities touched, and impact. +3. **Design** — `openspec/changes//design.md` with context, goals, + non-goals, decisions, risks, migration plan, open questions. +4. **Tasks** — `openspec/changes//tasks.md` tracking implementation. +5. **Apply** — implement; specs in `openspec/specs//spec.md` are + the live requirements. +6. **Archive** — move the change folder into + `openspec/changes/archive/-/`. Specs become the canonical + requirements; their `Purpose` should be updated from the `TBD` placeholder + left by archive. + +## Current specs + +- `rss-subscriptions` — authenticated feed management + OPML import. +- `rss-refresh` — safe, bounded RSS/Atom normalisation and refresh. +- `rss-inbox` — feed entry browsing, read state, save-to-library. + +See [architecture/decisions/0008-rss-inbox.md](../architecture/decisions/0008-rss-inbox.md) +for the decision record behind the RSS work. + +## When to use OpenSpec + +Trigger OpenSpec at the start of non-trivial feature work (multi-file, new +surface, behaviour change, cross-repo). Do not wait for the user to ask. The +`spec-driven` skill automates the explore → propose → apply → archive loop. + +For trivial changes (typo, single-line fix, doc edit) skip OpenSpec and use a +direct commit. diff --git a/docs/development/setup.md b/docs/development/setup.md new file mode 100644 index 0000000..1610b72 --- /dev/null +++ b/docs/development/setup.md @@ -0,0 +1,100 @@ +# Development Setup + +## Prerequisites + +- Node.js 22+ (engines field in `package.json`; CI uses 24). +- pnpm 10+ (the `packageManager` field pins the exact version). +- A Turso database (`turso db create`) and auth token. +- A Cloudflare account with an R2 bucket `reader-pdfs` bound as `PDFS_BUCKET`. +- A Google OAuth client (Google Cloud Console → APIs & Services → Credentials) + with a redirect URI for `BETTER_AUTH_URL` (e.g. + `http://localhost:8787/api/auth/callback/google` in dev). + +## Install + +```bash +pnpm install +``` + +## Configure environment + +```bash +cp .env.example .env.local +``` + +Edit `.env.local` (see [operations/env.md](../operations/env.md) for the full +list and validation): + +- `TURSO_DATABASE_URL`, `TURSO_AUTH_TOKEN` +- `BETTER_AUTH_SECRET` (`openssl rand -base64 32`), `BETTER_AUTH_URL` +- `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET` +- `AI_GATEWAY_API_KEY` (free-ai gateway) — optional for BYOK-only dev +- `LOCAL_AI_URL` (optional; defaults to `http://127.0.0.1:3456`) + +R2 credentials are only needed for production / `wrangler dev`; the binding +itself is provided by `wrangler dev` from `wrangler.toml`. + +## Push the schema + +```bash +pnpm db:push # drizzle-kit push → applies schema to Turso +``` + +See [operations/runbooks/migrate-schema.md](../operations/runbooks/migrate-schema.md) +for migration discipline. + +## Run + +```bash +pnpm dev # Worker (wrangler dev, :8787) + Vite SPA (:5173) + local-ai, concurrently +``` + +- Worker-served app: `http://localhost:8787` +- Vite SPA only (proxies `/api` → 8787): `http://localhost:5173` +- Local AI providers are shown only in development mode. + +If you only need the SPA: + +```bash +pnpm dev:spa +``` + +If you only need the Worker: + +```bash +pnpm dev:worker +``` + +## Chrome extension (separate workspace) + +```bash +cd packages/chrome-extension +pnpm install +pnpm dev # vite build --watch → dist/ +``` + +Then in Chrome: `chrome://extensions` → enable Developer mode → **Load +unpacked** → select `packages/chrome-extension/dist`. See +`packages/chrome-extension/README.md`. + +## Landing page (Astro overlay) + +```bash +pnpm --filter ./landing-astro dev +``` + +The landing is overlaid onto `dist/index.html` during `cf:build` — see +[operations/deploy.md](../operations/deploy.md). + +## Common commands + +See [commands.md](commands.md) for the full script map. The essentials: + +```bash +pnpm typecheck # tsc --noEmit (app + worker tsconfigs) +pnpm test # vitest run +pnpm test:e2e # playwright +pnpm lint # biome check . +pnpm format # biome format --write . +pnpm docs:check # validate docs/ links + structure (see scripts/check-docs.mjs) +``` diff --git a/docs/development/testing.md b/docs/development/testing.md new file mode 100644 index 0000000..c355844 --- /dev/null +++ b/docs/development/testing.md @@ -0,0 +1,46 @@ +# Testing + +## Vitest (unit) + +- Config: `vitest.config.ts`. +- Run: `pnpm test` (one-shot) or `pnpm test:watch`. +- Coverage: `pnpm test:coverage`. +- DOM env: `happy-dom`. +- Test discovery: `src/lib/**/__tests__/**/*.test.ts` and + `src/worker/routes/__tests__/**/*.test.ts`. +- Examples: `src/lib/__tests__/browser-memory-import.test.ts`, + `src/lib/__tests__/category-utils.test.ts`, + `src/lib/__tests__/memory-capture.test.ts`, + `src/worker/routes/__tests__/` (route-level tests). + +## Playwright (e2e) + +- Config: `playwright.config.ts`. +- Run: `pnpm test:e2e`. +- Specs: `tests/login.spec.ts`, `tests/mobile.spec.ts`. +- `PLAYWRIGHT_BROWSERS_PATH=0` is recommended in serverless environments (see + `.env.example`). + +## Type-checking + +`pnpm typecheck` runs `tsc --noEmit` against both `tsconfig.app.json` (SPA) +and `tsconfig.worker.json` (Worker + server libs). This is the canonical +type-check; CI runs it on every push/PR. + +## Chrome extension tests + +`packages/chrome-extension/` has its own Vitest config +(`packages/chrome-extension/vitest.config.ts`); run `pnpm test` from that +directory. Excluded from root tooling. + +## CI + +CI (`.github/workflows/ci.yml`) runs on push/PR to `main`/`master`: +`pnpm install --frozen-lockfile --ignore-scripts` → `validate:env:build` → +`lint` → `type-check` → `test`. See [../operations/ci-cd.md](../operations/ci-cd.md). + +## Documentation checks + +`pnpm docs:check` (`scripts/check-docs.mjs`) validates `docs/` link integrity +and structure. CI runs it in `.github/workflows/docs.yml`. See +[../operations/ci-cd.md](../operations/ci-cd.md). diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..753b233 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,95 @@ +# Reader — Documentation Index + +This folder is the canonical source of truth for Reader's product, architecture, +operations, and durable knowledge. Markdown here is authoritative; the +[`blume.config.ts`](../blume.config.ts) at the repo root only renders it. + +For a fast agent briefing, read [AGENTS.md](../AGENTS.md) first, then this index. + +## Where to start + +- **New to the codebase?** → [product/overview.md](product/overview.md) → [product/features.md](product/features.md) → [architecture/overview.md](architecture/overview.md) → [development/setup.md](development/setup.md) +- **On-call / deploying?** → [operations/deploy.md](operations/deploy.md) → [operations/runbooks/](operations/runbooks/) +- **Why is X the way it is?** → [architecture/decisions/](architecture/decisions/) → [knowledge/learnings.md](knowledge/learnings.md) +- **What broke before?** → [knowledge/failed-approaches.md](knowledge/failed-approaches.md) → [archive/](archive/) +- **Current state of the project?** → [STATUS.md](../STATUS.md) +- **Writing code here?** → [development/conventions.md](development/conventions.md) → [development/commands.md](development/commands.md) → [development/testing.md](development/testing.md) → [development/openspec.md](development/openspec.md) +- **Marketing the product?** → [marketing/hooks.md](marketing/hooks.md) → [marketing/iterations/v2/draft-seo-landing-keywords.md](marketing/iterations/v2/draft-seo-landing-keywords.md) + +## Layout + +``` +docs/ + index.md # this file + product/ + overview.md # purpose, users, scope + features.md # shipped feature inventory + surfaces.md # production URLs + agent-indexing surfaces + architecture/ + overview.md # Vite SPA + Hono Worker shape + data-flow.md # request flow, auth, storage, AI + decisions/ # current ADRs (one file per decision) + 0001-vite-spa-hono-worker.md + 0002-turso-drizzle.md + 0003-r2-pdfs.md + 0004-better-auth-google.md + 0005-ai-gateway-byok.md + 0006-mv3-side-panel.md + 0007-content-extraction.md + 0008-rss-inbox.md + development/ + setup.md # local dev environment + commands.md # pnpm scripts and what they do + conventions.md # code style, formatting, pre-commit hooks + testing.md # vitest + playwright + openspec.md # spec-driven change workflow + operations/ + deploy.md # Cloudflare Workers deploy + secrets + env.md # environment variables and validation + ci-cd.md # GitHub Actions workflows + jobs.md # scheduled jobs (weekly quality check) + runbooks/ + migrate-schema.md # applying Drizzle schema changes + rotate-secrets.md # rotating Cloudflare/Turso/Google secrets + rollback.md # rollback procedure for a bad deploy + knowledge/ + learnings.md # current, applicable engineering lessons + external-references.md # curated external docs (one entry per concept) + failed-approaches.md # approaches tried and abandoned, with reasons + archive/ # historical records, preserved verbatim + marketing/ # landing/SEO copy iterations (current) +``` + +## Maintenance rules + +1. **Markdown is the source of truth.** Code and executable config remain + authoritative for implementation details; docs explain *why*, not *what the + code currently does line-by-line*. +2. **One home per fact.** Don't duplicate a fact across files — link to its + canonical home. If a fact moves, update links rather than copying. +3. **Prefer `archive/` over deletion.** When a doc is superseded, move it to + `docs/archive/` with a dated filename and a one-line historical marker at + the top. Preserve git rename history with `git mv`. +4. **Mark unresolved questions explicitly** with `TBD:` or an "Open questions" + section. Do not invent answers. +5. **Keep pages focused.** Target 150–300 lines per file. Split when a page + grows beyond that. +6. **Validate before commit.** Run `node scripts/check-docs.mjs` (or + `pnpm docs:check`) — it catches broken links, missing required sections, + and files outside the canonical structure. +7. **Blume is presentation only.** Never edit generated Blume output; edit the + Markdown and rebuild. `blume.config.ts` points at this folder as content + root. + +## What lives outside this folder + +- [`AGENTS.md`](../AGENTS.md) — concise agent bootloader (purpose, commands, + constraints, navigation). Links here for depth. +- [`STATUS.md`](../STATUS.md) — short current-state view (objective, active + work, blockers, next steps). +- [`README.md`](../README.md) — product readme for humans landing in the repo. +- [`openspec/`](../openspec/) — spec-driven change workflow tooling and + archived change proposals. Referenced from [development/openspec.md](development/openspec.md). +- [`public/`](../public/) — runtime agent-indexing surfaces (`llms.txt`, + `index.md`, `api-ai.json`, `robots.txt`, `sitemap.xml`) served by the Worker. + Documented in [product/surfaces.md](product/surfaces.md). diff --git a/docs/knowledge/external-references.md b/docs/knowledge/external-references.md new file mode 100644 index 0000000..f002739 --- /dev/null +++ b/docs/knowledge/external-references.md @@ -0,0 +1,117 @@ +# External References — Reader + +One entry per concept. "What / why for this project / link." Pre-Vite +references (OpenNext, `next.config.ts`, `serverExternalPackages`, etc.) are +preserved in [archive/learning-pre-vite-external-references.md](../archive/learning-pre-vite-external-references.md). + +## Deployment / runtime + +**Cloudflare Workers — Hono** +Hono is the HTTP router for the Worker (`src/worker.ts`). Covers the Hono +API, bindings, and middleware patterns. +→ https://hono.dev/docs/ + +**Cloudflare Workers — Vite guide** +Official CF guide for SPA-on-Workers with Vite; covers `nodejs_compat_v2`, +smart placement, and the `ASSETS` binding. +→ https://developers.cloudflare.com/workers/frameworks/framework-guides/vite/ + +**Cloudflare Smart Placement** +Routes a Worker to the CF PoP closest to its backend (Turso in this case). +Directly addresses the TTFB problem flagged in `wrangler.toml`. +→ https://developers.cloudflare.com/workers/configuration/smart-placement/ + +**Cloudflare Workers `caches.default`** +Edge cache API used in `articles-db.ts` (5 min TTL). Not available in pure +Vite dev — guarded by `globalThis.caches?.default`. +→ https://developers.cloudflare.com/workers/runtime/apis/cache/ + +## Database + +**Turso (libSQL) docs** +Managed SQLite-compatible edge database. libSQL client, connection strings, +auth token setup. +→ https://docs.turso.tech/ + +**Drizzle ORM docs** +ORM used for schema definition, queries, and migrations. Covers +`drizzle-kit push` vs `generate` — the choice that matters for production +schema safety. +→ https://orm.drizzle.team/docs/overview + +**`@libsql/client` — web target** +The `/web` entry is what the Workers runtime uses; `src/lib/db/client.ts` +imports `@libsql/client/web` explicitly. +→ https://github.com/tursodatabase/libsql-client-ts + +## Auth + +**better-auth docs** +Auth library used for Google OAuth + session management. Covers the Drizzle +adapter, CF Workers environment quirks, and the `oneTap` plugin. +→ https://www.better-auth.com/docs + +## AI + +**Vercel AI SDK reference** +`streamText`, `toTextStreamResponse`, `generateText`, and provider +configuration. `@ai-sdk/openai-compatible` enables the gateway+BYOK pattern. +→ https://sdk.vercel.ai/docs + +**Cloudflare Workers AI** +Free-tier model catalogue (10 k Neurons/day), including +`@cf/meta/llama-3.3-70b-instruct-fp8-fast` which is the default model in +`src/lib/ai-cloudflare.ts`. +→ https://developers.cloudflare.com/workers-ai/ + +## Extension + +**Chrome MV3 Side Panel API** +Covers `sidePanel` permission, `chrome.sidePanel.open()`, and lifecycle +differences vs popup. +→ https://developer.chrome.com/docs/extensions/reference/api/sidePanel + +**MV3 Service Worker lifecycle** +Explains why session cookies don't work from extension origins and why +long-lived API keys are the right auth approach for the extension. +→ https://developer.chrome.com/docs/extensions/develop/migrate/to-service-workers + +## Content extraction + +**Mozilla Readability** +DOM-based article extraction library used in `/api/snapshot`. Documents what +makes a page parseable and common failure modes (SPA pages with no initial +HTML content). +→ https://github.com/mozilla/readability + +**pdfjs-dist** +PDF rendering engine used in `PDFReaderClient.tsx`. `GlobalWorkerOptions.workerSrc` +configuration and local worker loading are covered in the Getting Started guide. +→ https://mozilla.github.io/pdf.js/ + +**linkedom** +Lightweight DOM parser used server-side instead of JSDOM or Playwright. +Workers-compatible; faster than JSDOM. +→ https://github.com/WebReflection/linkedom + +## Storage + +**Cloudflare R2 — Workers binding** +How `PDFS_BUCKET` binding works (`put`, `get`, `delete`), zero-egress +pricing, and the difference from the S3-compatible HTTP API. +→ https://developers.cloudflare.com/r2/api/workers/workers-api-usage/ + +## Documentation + +**Blume** +The presentation/search layer for `docs/`. Reads `blume.config.ts` and +renders Markdown/MDX. Markdown in `docs/` is the source of truth; Blume is +not. +→ https://useblume.dev/docs/ + +## OpenSpec + +**OpenSpec** +Spec-driven development workflow used for non-trivial feature work. See +[development/openspec.md](../development/openspec.md). +→ https://github.com/Fission-AI/OpenSpec diff --git a/docs/knowledge/failed-approaches.md b/docs/knowledge/failed-approaches.md new file mode 100644 index 0000000..e0b22c9 --- /dev/null +++ b/docs/knowledge/failed-approaches.md @@ -0,0 +1,150 @@ +# Failed Approaches — Reader + +Approaches tried and abandoned, with reasons. Each entry links to the +archived material where relevant. Preserved so the same path is not retried +without a new reason. + +## Next.js + OpenNext on Cloudflare Workers (superseded 2026-05) + +**What:** Server-rendered Next.js App Router deployed to Cloudflare Workers +via `@opennextjs/cloudflare`. + +**Why abandoned:** Required two bespoke patch scripts +(`scripts/patch-opennext.mjs` pre + post, `scripts/fix-opennext-deps.mjs`) +to work around `@libsql/isomorphic-ws` `node.mjs` vs `web.mjs` resolution and +`WeakRef` / `FinalizationRegistry` not being free globals under +`nodejs_compat_v2`. `next build` needed `ignoreBuildErrors: true` to avoid +timeouts, pushing type-checking out of the build pipeline. For a personal-use, +client-heavy reading app with no SSR requirement, the OpenNext layer was pure +operational cost. + +**What replaced it:** Vite + React 19 SPA + Hono Worker. See +[architecture/decisions/0001-vite-spa-hono-worker.md](../architecture/decisions/0001-vite-spa-hono-worker.md) +and the pre-Vite ADRs in [archive/decisions.md](../archive/decisions.md). + +**Do not retry unless:** SSR becomes a product requirement (e.g. public SEO on +auth-walled routes — currently not needed; the landing is Astro and the app is +auth-walled). + +## Cloudflare Pages (same-day revert, 2026-04-25) + +**What:** Migrated the deploy from Cloudflare Workers to Cloudflare Pages for +a clean `*.pages.dev` URL. + +**Why abandoned:** Native R2 and Workers AI bindings behave differently under +the Pages Functions model with OpenNext, and the Workers adapter was more +mature. Reverted the same day (commit `434559e` → `6358c03`, ~2h55m). The +`.cf-pages-bundle` entry in `.gitignore` is a leftover from this experiment. + +**What replaced it:** Stayed on Workers. The custom domain +`read.significanthobbies.com` is bound to the Worker. + +**Do not retry unless:** Pages gains first-class R2 binding parity and a +mature Hono/Vite adapter story. + +## Firebase (Firestore + Auth + GCS) (removed 2026-04-25) + +**What:** Original data layer — Firestore (NoSQL), Firebase Auth, GCS via +Firebase Admin for PDF storage. + +**Why abandoned:** Firestore gave "~zero value beyond storage + Auth +integration" — no real-time, no offline, deny-all rules, and O(n) full-table +scan for search. Three Firebase services (Firestore + Auth + GCS) replaced by +Turso (1) + better-auth (reuses Turso) + R2 (native binding). + +**What replaced it:** Turso (libSQL) via Drizzle ORM, better-auth + Google +OAuth, Cloudflare R2. See +[architecture/decisions/0002-turso-drizzle.md](../architecture/decisions/0002-turso-drizzle.md), +[0003-r2-pdfs.md](../architecture/decisions/0003-r2-pdfs.md), +[0004-better-auth-google.md](../architecture/decisions/0004-better-auth-google.md). + +**Do not retry unless:** A use case emerges that needs Firestore's offline +sync or realtime subscriptions at scale — not the case for a personal reader. + +## Auth.js v5 (NextAuth) — brief detour (2026-04-25) + +**What:** The migration plan targeted `next-auth@beta` (Auth.js v5) with the +libSQL adapter. + +**Why abandoned:** At cutover `better-auth` was chosen because of its +first-party Drizzle adapter. The legacy NextAuth tables (`account`, +`session`, `verificationToken`) remain in `src/lib/db/schema.ts` as dead +weight. + +**What replaced it:** better-auth v1.6 with the Drizzle adapter, Google OAuth +only, `oneTap` plugin. + +**Do not retry unless:** better-auth loses maintenance or a feature Auth.js +uniquely provides becomes required. + +## `@aws-sdk/client-s3` for R2 (replaced 2026-04-27) + +**What:** Initially accessed R2 via the S3-compatible `@aws-sdk/client-s3` +with explicit credentials. + +**Why abandoned:** After upgrading to the Cloudflare Paid plan, the native +`PDFS_BUCKET` Workers binding became available — zero HTTP overhead, no +egress cost, no SDK bundle weight. `@aws-sdk` is fully absent from +`package.json` now. + +**What replaced it:** `getCloudflareContext().env.PDFS_BUCKET` / +`setPdfBucket(env.PDFS_BUCKET)` in `src/lib/storage.ts`. + +**Do not retry unless:** R2 access is needed from outside the Workers runtime +(local scripts, non-Workers services) — the S3-compat API is still the right +path there. + +## Playwright for server-side DOM parsing (replaced 2026-02-13) + +**What:** Used Playwright to render and parse pages for article extraction. + +**Why abandoned:** Requires a full browser process; not viable in the +Cloudflare Workers runtime. Too heavy and slow. + +**What replaced it:** `linkedom` (pure-JS DOM parser) + `@mozilla/readability`. +See [architecture/decisions/0007-content-extraction.md](../architecture/decisions/0007-content-extraction.md). + +**Do not retry unless:** Extraction needs JavaScript-rendered content that +`linkedom` cannot parse — at which point a separate browser-service worker +(not the request-handling Worker) would be required. + +## RSS background refresh via Cloudflare scheduled triggers (deferred) + +**What:** Scheduled triggers (`[triggers] crons` in `wrangler.toml`) for +automatic RSS refresh. + +**Why deferred:** Manual refresh is predictable and sufficient for the +personal-use MVP. Scheduled refresh introduces operational behaviour +(trigger management, failure handling, cost) beyond the requested in-app +reader. + +**What is in place instead:** Manual `POST /api/rss/refresh` with bounded +concurrency and per-feed error isolation. See +[architecture/decisions/0008-rss-inbox.md](../architecture/decisions/0008-rss-inbox.md). + +**Reopen when:** manual use demonstrates the need for background refresh. + +## Rate limiting on AI/snapshot/proxy endpoints (deferred) + +**What:** Cloudflare rate limiting on potentially abusable endpoints. + +**Why deferred:** No endpoint-specific abuse evidence. Prefer ownership +checks, input validation, and cost controls first; only add rate limiting +for a specific abused endpoint with explicit approval. + +**What is in place instead:** Auth + ownership on every protected route; SSRF +validation on URL-fetching routes; 10 MB PDF size cap; bounded RSS refresh +concurrency; free-ai gateway 9500 Neuron/day fleet cap. + +**Reopen when:** a specific endpoint shows abuse in observability +(`[observability] enabled = true` in `wrangler.toml`). + +## Explicit CORS configuration (deferred) + +**What:** Explicit CORS headers on share / API routes. + +**Why deferred:** The app is same-origin today; share routes return JSON +consumed same-origin. External URLs are proxied server-side via `/api/proxy`. + +**Reopen when:** share routes need cross-origin access (e.g. embedding in a +third-party site). diff --git a/docs/knowledge/learnings.md b/docs/knowledge/learnings.md new file mode 100644 index 0000000..c2412e4 --- /dev/null +++ b/docs/knowledge/learnings.md @@ -0,0 +1,172 @@ +# Engineering Lessons — Reader (current) + +Concrete lessons evidenced by current code, scripts, or git history. Each +links to the decision record where relevant. Pre-Vite lessons (OpenNext +patch scripts, `next.config.ts`, `WeakRef` patches, etc.) are preserved in +[archive/lessons.md](../archive/lessons.md) — they describe +code that no longer exists but capture useful failure modes. + +## Worker / runtime + +### L1: Bind the Worker env before any module-level singleton touches it + +`src/worker.ts` runs `bindWorkerEnv(c.env)` on every `/api/*` request so the +lazy `db` proxy (`src/lib/db/client.ts`) and the `pdfsBucket` singleton +(`src/lib/storage.ts`) resolve the runtime env before first use. Eager +instantiation at module load fails in the Workers runtime where the env is +not yet available at import time. + +→ [architecture/decisions/0002-turso-drizzle.md](../architecture/decisions/0002-turso-drizzle.md), + [architecture/decisions/0003-r2-pdfs.md](../architecture/decisions/0003-r2-pdfs.md) + +### L2: `caches.default` is unavailable in pure-Vite dev + +`articles-db.ts` caches article reads at the edge (5 min TTL) using +`globalThis.caches?.default`. The guard is required because `caches.default` +does not exist in `next dev`-style local dev. Cache must be explicitly busted +on writes (see `lists-db.ts`). + +### L3: Smart Placement is essential for Turso latency + +`wrangler.toml` has `[placement] mode = "smart"`. Without it, the Worker runs +in a Cloudflare PoP that may be far from the Turso primary, paying a +cross-region RTT on every request. A psi-swarm audit flagged TTFB >1s as the +dominant LCP contributor before Smart Placement was enabled. + +→ [architecture/decisions/0002-turso-drizzle.md](../architecture/decisions/0002-turso-drizzle.md) + +### L4: `run_worker_first` is required for agent surfaces and `/api/*` + +`wrangler.toml` sets `run_worker_first` for `/sitemap.xml`, `/index.md`, +`/llms-full.txt`, `/llms.txt`, `/api/*`, and `/`. Without it, the `ASSETS` +binding serves static files first and the Worker never sees the agent-edge +or API paths. + +→ [product/surfaces.md](../product/surfaces.md) + +## Database / Drizzle + +### L5: `drizzle-kit push` is schema-sync, not migration history + +`pnpm db:push` diffs `schema.ts` against the live DB and applies the diff. +Safe for a single-user DB; fragile if a push runs against production with +data in an incompatible old shape. Additive migrations are committed under +`drizzle/` and applied deliberately. Open question: switch to +`drizzle-kit generate` as user count grows (tracked in STATUS.md). + +→ [operations/runbooks/migrate-schema.md](../operations/runbooks/migrate-schema.md) + +### L6: Legacy NextAuth tables are dead weight + +`src/lib/db/schema.ts` still defines `account`, `session`, `verificationToken` +(leftover from the Auth.js → better-auth swap). They are unused by +better-auth. Safe to drop after confirming no active rows. Open question, +tracked in STATUS.md. + +→ [architecture/decisions/0004-better-auth-google.md](../architecture/decisions/0004-better-auth-google.md) + +### L7: JSON columns are text + `$type()` + +`tags`, `notes`, `aiChat`, `summary`, `keyPoints`, `pdfMetadata`, +`sessionReview` are stored as text and typed via `$type()` for TS safety. +No row-level JSON filtering in current query patterns; `json_each` is +available if needed. + +## R2 / PDFs + +### L8: PDF MIME must be validated by magic bytes + +`src/lib/pdf-service.ts` and the upload route do not trust `file.type` +(browser-controlled and spoofable). Read the first 5 bytes and check for +`%PDF-` before accepting the file. + +→ [architecture/decisions/0003-r2-pdfs.md](../architecture/decisions/0003-r2-pdfs.md) + +### L9: `blob://` sentinel URL prevents article dedup collision + +PDFs don't have an HTTP URL. The upload stores `blob://` as the +`articles.url` field so the `(user_id, url)` unique index does not collide +with real HTTP article URLs. + +## AI / gateway / BYOK + +### L10: BYOK keys must never be persisted server-side + +BYOK provider keys are sent per-request from the client and used immediately +— never written to the database or logs. `normalizeApiKey()` in +`ai-server.ts` trims and length-limits the key before use. Extension AI +calls use hashed long-lived `rdr_*` API keys instead. + +→ [architecture/decisions/0005-ai-gateway-byok.md](../architecture/decisions/0005-ai-gateway-byok.md) + +### L11: `x-gateway-project-id: reader` header is required for fleet budget attribution + +All AI requests include `x-gateway-project-id: reader`. Without it the +free-ai gateway cannot distinguish Reader from other fleet projects and +cannot attribute the 9500 Neuron/day fleet-wide cap. + +## MV3 extension + +### L12: pdfjs web worker must be loaded locally + +pdfjs-dist defaults to loading its web worker from `unpkg.com`. This fails +in the extension context (CSP `script-src 'self'` blocks remote scripts) +and in the CF Workers build sandbox. The worker is loaded from +`public/pdf.worker.min.mjs` (local static asset) via explicit +`GlobalWorkerOptions.workerSrc`. + +→ [architecture/decisions/0007-content-extraction.md](../architecture/decisions/0007-content-extraction.md) + +### L13: Extension auth uses hashed API keys, not session cookies + +MV3 extensions cannot share the same-origin session cookie. The `api_keys` +table stores HMAC-hashed tokens with a visible `rdr_` prefix. The extension +sends the raw token as a Bearer header; the server hashes it for lookup. + +→ [architecture/decisions/0006-mv3-side-panel.md](../architecture/decisions/0006-mv3-side-panel.md) + +## Security + +### L14: All server-side URL fetches funnel through SSRF validation + +`/api/snapshot`, `/api/proxy`, and RSS refresh use +`validateExternalUrl()` (`src/lib/url-validation.ts`) + +`fetchWithValidatedRedirects()` (`src/lib/safe-fetch.ts`). The latter +re-validates every redirect target before following it, so a public URL +cannot bounce the server into a private network hop after the first request. + +DNS rebinding note: validation resolves DNS once, but `fetch` may re-resolve +later. Full protection would require a custom fetch agent that pins the +resolved IP (not available in the Workers fetch API). Appropriate for a +personal-use reader; revisit if the server becomes privileged. + +→ [architecture/data-flow.md](../architecture/data-flow.md), + [archive/security-audit-2026-03-29.md](../archive/security-audit-2026-03-29.md) + +### L15: HTML is sanitised at ingestion AND on read + +Content is sanitised before storage and re-sanitised in +`fetchArticleById()` / `fetchArticleByShareId()` for defence-in-depth. Do +not remove the read-time sanitisation even though ingestion sanitises — it +guards against a future ingestion bypass or stale data. + +→ [archive/security-audit-2026-03-29.md](../archive/security-audit-2026-03-29.md) + +## Build / deploy + +### L16: `pnpm cf:build` overlays the landing; the SPA lives at `/app` + +`scripts/overlay-astro-landing.mjs` copies `landing-astro/dist/*` over +`dist/` except for protected prefixes (`assets/`, `app.html`). `_headers` is +merged. The SPA is served at `/app`; the landing at `/`. SPA fallback for +unknown paths goes to `/app` (handled in `src/worker.ts`). + +→ [operations/deploy.md](../operations/deploy.md) + +### L17: The Worker name `reader` is load-bearing + +The Worker is named `reader` in `wrangler.toml`. The custom domain +(`read.significanthobbies.com`) and all Cloudflare secrets are bound to that +name. Do not rename it without re-provisioning secrets and the route. + +→ [architecture/decisions/0001-vite-spa-hono-worker.md](../architecture/decisions/0001-vite-spa-hono-worker.md) diff --git a/docs/marketing/iterations/v2/draft-seo-landing-keywords.md b/docs/marketing/iterations/v2/draft-seo-landing-keywords.md index cf1bddb..7d86689 100644 --- a/docs/marketing/iterations/v2/draft-seo-landing-keywords.md +++ b/docs/marketing/iterations/v2/draft-seo-landing-keywords.md @@ -1,7 +1,7 @@ # SEO Landing Keywords — Read-it-Later Annotation & Note Capture > **v2** — 30 search intents for read-it-later annotation / note capture. Grouped by problem, alternative, comparison, how-to. -> See also: [comparison-page-draft.md](./create-comparison-page-draft.md) | [use-case-page-draft.md](./create-use-case-page-draft.md) | [how-it-works-page-draft.md](./create-how-it-works-page-draft.md) +> See also: [comparison-page-draft.md](./create-comparison-page-draft.md) | [use-case-page-draft.md](./create-use-case-page-draft.md) | [how-it-works-page-draft.md](./create-how-it-works-page-draft.md) | [objection-handling-faq.md](./create-objection-handling-faq.md) Focus: people who save articles but forget them, want to annotate while reading, or want to capture notes from online research. diff --git a/docs/operations/ci-cd.md b/docs/operations/ci-cd.md new file mode 100644 index 0000000..d490fcb --- /dev/null +++ b/docs/operations/ci-cd.md @@ -0,0 +1,63 @@ +# CI/CD + +GitHub Actions workflows live in `.github/workflows/`. Code is the authority +for triggers and steps; this page documents intent. + +## CI — `.github/workflows/ci.yml` + +**Triggers:** push to `main`/`master`, PR to `main`/`master`. + +**Steps:** checkout → pnpm setup → Node 24 → +`pnpm install --frozen-lockfile --ignore-scripts` → `validate:env:build` → +`lint` (if present) → `type-check` (if present) → `test` (if present). + +No deploy. No secrets beyond what `validate:env:build` needs (none for +`build` mode). + +## Deploy — `.github/workflows/deploy.yml` + +**Triggers:** `workflow_dispatch` only (manual). A `deploy-preview` job runs +on PRs (build only, no deploy). + +**Production steps:** checkout → pnpm setup → Node 24 → +`pnpm install --frozen-lockfile` → `validate:env:build` → validate Cloudflare +runtime secrets (lists `wrangler secret list` and checks each required name +exists) → `cf:build` → `cloudflare/wrangler-action@v3 deploy` → smoke check +`curl https://read.significanthobbies.com/`. + +**Required GitHub secrets:** `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`. +**Required Cloudflare Worker secrets:** `BETTER_AUTH_SECRET`, +`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `TURSO_AUTH_TOKEN`, +`TURSO_DATABASE_URL`. + +See [deploy.md](deploy.md) for the full pipeline. + +## AI Code Review — `.github/workflows/review.yaml` + +**Triggers:** `workflow_dispatch` only (manual). Temporarily manual because +the referenced reviewer action has no `v1` tag, which made every PR show a +failed review check before code ran. + +## Weekly Quality Check — `.github/workflows/weekly.yml` + +**Triggers:** `cron: '0 9 * * 1'` (Mondays 09:00 UTC) + `workflow_dispatch`. + +**Steps:** checkout → Node 22 → corepack/pnpm setup → install → run +`lint`, `typecheck`, `test`, `build` scripts if present. + +See [jobs.md](jobs.md). + +## Docs — `.github/workflows/docs.yml` + +**Triggers:** push/PR affecting `docs/**`, `AGENTS.md`, `STATUS.md`, +`blume.config.ts`, `scripts/check-docs.mjs`; plus `workflow_dispatch`. + +**Steps:** checkout → Node 22 → run `node scripts/check-docs.mjs` to validate +`docs/` link integrity and structure. + +See [../development/testing.md](../development/testing.md) for the validator. + +## Dependabot — `.github/dependabot.yml` + +Weekly (Monday) npm updates, scoped to `@saas-maker/sdk`, one open PR at a +time, `deps:` commit prefix. diff --git a/docs/operations/deploy.md b/docs/operations/deploy.md new file mode 100644 index 0000000..96b8ab4 --- /dev/null +++ b/docs/operations/deploy.md @@ -0,0 +1,78 @@ +# Deploy + +Reader deploys to Cloudflare Workers as the `reader` Worker with a custom +domain `read.significanthobbies.com`. Production deploys are **manual** +(`workflow_dispatch` on `.github/workflows/deploy.yml`) — there is no +auto-deploy on push to `main` (CI runs on push, deploy does not). + +## Pipeline + +``` +pnpm deploy + → pnpm validate:env:deploy (scripts/validate-env.mjs deploy) + → pnpm cf:build + → pnpm build (validate-env(build) + vite build → dist/) + → pnpm --filter ./landing-astro build + → node scripts/overlay-astro-landing.mjs (overlay landing → dist/index.html, merge _headers) + → pnpm exec wrangler deploy +``` + +The Worker `main` is `src/worker.ts`; built assets in `dist/` are served via +the `ASSETS` binding. `wrangler.toml` configures: + +- `compatibility_date = "2025-04-01"`, `compatibility_flags = ["nodejs_compat_v2"]` +- `assets = { directory = "dist", binding = "ASSETS", run_worker_first = ["/sitemap.xml", "/index.md", "/llms-full.txt", "/llms.txt", "/api/*", "/"] }` +- `routes = [{ pattern = "read.significanthobbies.com", custom_domain = true }]` +- `[placement] mode = "smart"` (co-locate Worker with Turso) +- `[observability] enabled = true, head_sampling_rate = 0.1` +- `[limits] cpu_ms = 30000` +- `[[r2_buckets]] binding = "PDFS_BUCKET", bucket_name = "reader-pdfs"` +- `[vars] AI_BASE_URL`, `BETTER_AUTH_URL`, `NODE_ENV = "production"` + +## Required Cloudflare secrets + +Set via `wrangler secret put `: + +- `BETTER_AUTH_SECRET` +- `GOOGLE_CLIENT_ID` +- `GOOGLE_CLIENT_SECRET` +- `TURSO_AUTH_TOKEN` +- `TURSO_DATABASE_URL` +- `AI_GATEWAY_API_KEY` (free-ai gateway; legacy alias `AI_API_KEY`) + +The deploy workflow validates that each required secret exists in +`wrangler secret list` before building. See +[env.md](env.md) for the full env map and [runbooks/rotate-secrets.md](runbooks/rotate-secrets.md) +for rotation. + +## Landing overlay + +`scripts/overlay-astro-landing.mjs` copies `landing-astro/dist/*` over +`dist/`, **except** protected prefixes (`assets/`, `app.html`). `_headers` is +merged (Astro headers first, then Vite build headers). The SPA lives at +`dist/app.html` and is served at `/app`; the landing lives at +`dist/index.html` and is served at `/`. + +If `landing-astro/dist` is missing, the overlay step warns and skips — the +SPA-only build is still deployable. + +## Smoke check + +The deploy workflow runs: + +```bash +curl --fail --silent --show-error --retry 3 --retry-delay 5 --max-time 20 \ + https://read.significanthobbies.com/ > /dev/null +``` + +A non-200 aborts the workflow. + +## Preview deploys + +`.github/workflows/deploy.yml` has a `deploy-preview` job that runs on PRs +(`pnpm cf:build` only, no deploy). It validates the build is green without +pushing to production. + +## Rollback + +See [runbooks/rollback.md](runbooks/rollback.md). diff --git a/docs/operations/env.md b/docs/operations/env.md new file mode 100644 index 0000000..8cc2b8b --- /dev/null +++ b/docs/operations/env.md @@ -0,0 +1,59 @@ +# Environment Variables + +Validated by `scripts/validate-env.mjs`. The script takes a mode argument +(`build` | `runtime` | `deploy`) and exits non-zero if any required variable +for that mode is missing or empty. + +## Map + +| Variable | Required where | Set via | Purpose | +| --- | --- | --- | --- | +| `TURSO_DATABASE_URL` | runtime, deploy | Wrangler secret | Turso libSQL URL (`libsql://...`) | +| `TURSO_AUTH_TOKEN` | runtime, deploy | Wrangler secret | Turso auth token | +| `BETTER_AUTH_SECRET` | runtime, deploy | Wrangler secret | better-auth session signing key (`openssl rand -base64 32`) | +| `BETTER_AUTH_URL` | — | `wrangler.toml [vars]` | OAuth callback base URL (prod: `https://read.significanthobbies.com`) | +| `GOOGLE_CLIENT_ID` | runtime, deploy | Wrangler secret | Google OAuth client ID | +| `GOOGLE_CLIENT_SECRET` | runtime, deploy | Wrangler secret | Google OAuth client secret | +| `AI_BASE_URL` | — | `wrangler.toml [vars]` | free-ai gateway URL (`https://ai-gateway.sassmaker.com/v1`) | +| `AI_GATEWAY_API_KEY` | runtime (free AI) | Wrangler secret | free-ai gateway bearer token (legacy alias `AI_API_KEY`) | +| `LOCAL_AI_URL` | optional | `.env.local` | Local AI bridge URL (default `http://127.0.0.1:3456`) | +| `CLI_BRIDGE_URL` | optional | `.env.local` | Legacy alias for `LOCAL_AI_URL` | +| `NODE_ENV` | — | `wrangler.toml [vars]` | `production` in prod; `development` enables local AI mode | +| `PDFS_BUCKET` | runtime | `wrangler.toml` binding | R2 bucket binding (`reader-pdfs`) | +| `ASSETS` | runtime | `wrangler.toml` binding | Static asset binding (`dist/`) | +| `CLOUDFLARE_ACCOUNT_ID` | deploy (CI) | GitHub secret | Wrangler deploy account | +| `CLOUDFLARE_API_TOKEN` | deploy (CI) | GitHub secret | Wrangler deploy token | +| `R2_ACCESS_KEY_ID` | local dev (R2) | `.env.local` | R2 S3-compat API (only if not using the binding) | +| `R2_SECRET_ACCESS_KEY` | local dev (R2) | `.env.local` | R2 S3-compat API | +| `R2_BUCKET_NAME` | local dev (R2) | `.env.local` | `reader-pdfs` | +| `VITE_POSTHOG_KEY` | optional | `.env.local` | PostHog analytics key (client) | +| `VITE_SAASMAKER_API_KEY` | optional | `.env.local` | SaaS Maker widget key (client) | +| `PLAYWRIGHT_BROWSERS_PATH` | optional | env | `0` for serverless Playwright | + +## Validation modes + +- `build` — no required vars (the build does not need runtime secrets). +- `runtime` — `BETTER_AUTH_SECRET`, `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, + `TURSO_AUTH_TOKEN`, `TURSO_DATABASE_URL`. +- `deploy` — same as `runtime`. + +## Local dev + +`.env.local` is loaded by `drizzle.config.ts` and `vite.config.ts` via +`dotenv/config`. `.env.example` is the committed template; `.env.local` is +gitignored. The Worker dev server (`pnpm dev:worker`) reads Wrangler secrets +from `.dev.vars` (gitignored) and bindings from `wrangler.toml`. + +## Security + +- Never commit `.env`, `.env.local`, `.dev.vars`, `firebase-service-account.json`, + or any auth credential. All are in `.gitignore`. +- BYOK provider keys (OpenAI/Anthropic/Gemini) live in the browser only — + never sent to the server as env vars. See + [architecture/decisions/0005-ai-gateway-byok.md](../architecture/decisions/0005-ai-gateway-byok.md). +- `rdr_*` API keys are hashed at rest; plaintext is shown once at creation. See + [architecture/decisions/0006-mv3-side-panel.md](../architecture/decisions/0006-mv3-side-panel.md). + +## Rotation + +See [runbooks/rotate-secrets.md](runbooks/rotate-secrets.md). diff --git a/docs/operations/jobs.md b/docs/operations/jobs.md new file mode 100644 index 0000000..3034bd9 --- /dev/null +++ b/docs/operations/jobs.md @@ -0,0 +1,35 @@ +# Scheduled Jobs + +The only scheduled job is the GitHub Actions weekly quality check. + +## Weekly Quality Check + +- **Workflow:** `.github/workflows/weekly.yml` +- **Schedule:** `cron: '0 9 * * 1'` (Mondays 09:00 UTC) + `workflow_dispatch` + for manual runs. +- **What it does:** checkout → Node 22 → corepack/pnpm → install + (`--frozen-lockfile --ignore-scripts`) → run `lint`, `typecheck`, `test`, + `build` scripts if present in `package.json`. +- **Why:** catches drift that doesn't surface on push CI (e.g. dependency + regressions, environment drift, weekly quality baseline). +- **Permissions:** `contents: read` only. + +## Other workflows (manual, not scheduled) + +- `deploy.yml` — `workflow_dispatch` only. See [ci-cd.md](ci-cd.md). +- `review.yaml` — `workflow_dispatch` only (AI code review). +- `docs.yml` — push/PR-triggered + `workflow_dispatch`. + +## Cloudflare Workers scheduled triggers + +**None configured.** `wrangler.toml` has no `[triggers] crons` entry. RSS +refresh is manual (`POST /api/rss/refresh`) by design — see +[architecture/decisions/0008-rss-inbox.md](../architecture/decisions/0008-rss-inbox.md). +Background refresh via Cloudflare scheduled triggers is deferred until manual +use demonstrates the need. + +## Local AI bridge + +`scripts/local-ai.mjs` is a dev-only bridge (`pnpm local-ai`); it spawns +`../local-ai/index.mjs` (or legacy `../cli-bridge/index.mjs`) and is not a +scheduled job. diff --git a/docs/operations/runbooks/migrate-schema.md b/docs/operations/runbooks/migrate-schema.md new file mode 100644 index 0000000..f5581ee --- /dev/null +++ b/docs/operations/runbooks/migrate-schema.md @@ -0,0 +1,68 @@ +# Runbook: Apply a Drizzle Schema Change + +Reader uses `drizzle-kit push` (`pnpm db:push`) for schema sync, with additive +migration SQL files committed under `drizzle/` for deliberate application in +production. The migration journal is `drizzle/meta/_journal.json` with +snapshots in `drizzle/meta/`. + +## When to use this runbook + +- You changed `src/lib/db/schema.ts` and need to apply the change. +- You are deploying a change that includes a new `drizzle/*.sql` migration + (e.g. the RSS `0002_first_green_goblin.sql`). + +## Local / dev + +```bash +pnpm db:push # drizzle-kit push → diffs schema.ts against the live DB and applies +``` + +`drizzle.config.ts` loads `.env.local` for `TURSO_DATABASE_URL` and +`TURSO_AUTH_TOKEN`. Push is schema-sync (no migration history); safe for a +single-user DB. + +## Production + +1. **Read the migration SQL** under `drizzle/_.sql`. Confirm it is + additive (new tables, new columns with defaults, new indexes) and + reversible by `DROP TABLE` / `DROP COLUMN` if needed. +2. **Apply the migration** before deploying application code that depends on + it. For additive migrations the order is: migrate → deploy. For + destructive migrations, deploy backward-compatible code first, then + migrate, then remove the old code path. +3. **Apply via Turso** (not `wrangler`): + + ```bash + # Option A: drizzle-kit push against the production DB + TURSO_DATABASE_URL= TURSO_AUTH_TOKEN= pnpm db:push + + # Option B: run the SQL file directly with the Turso CLI + turso db shell < drizzle/0002_first_green_goblin.sql + ``` + +4. **Verify** with `pnpm db:studio` (read-only inspection) or a direct Turso + query (`turso db shell ".tables"`). +5. **Deploy** the application code per [deploy.md](../deploy.md). + +## Example: the RSS migration + +`drizzle/0002_first_green_goblin.sql` adds `rss_feeds` and `rss_entries`. It +is additive and reversible by dropping the two tables. Apply before deploying +the RSS routes. See [architecture/decisions/0008-rss-inbox.md](../../architecture/decisions/0008-rss-inbox.md). + +## Rollback + +- **Additive migration:** drop the new table/column/index. Data in the new + table is disposable (e.g. RSS entries are transient inbox items). +- **Destructive migration:** restore from Turso backup (`turso db shell + ".restore "`). Take a backup before any destructive + change. + +## Discipline + +- Prefer additive migrations (new tables, new nullable columns, new indexes). +- Avoid `drizzle-kit push` against production with data in an incompatible old + shape — read the generated SQL first. +- Open question (tracked in STATUS.md): switch from `drizzle-kit push` to + `drizzle-kit generate` for safer schema changes as user count grows. See + [knowledge/learnings.md](../../knowledge/learnings.md). diff --git a/docs/operations/runbooks/rollback.md b/docs/operations/runbooks/rollback.md new file mode 100644 index 0000000..fef88ed --- /dev/null +++ b/docs/operations/runbooks/rollback.md @@ -0,0 +1,78 @@ +# Runbook: Rollback a Deploy + +Cloudflare Workers does not keep old Worker versions indefinitely, so rollback +is "redeploy a known-good commit." The deploy is manual +(`workflow_dispatch`), so rollback is also manual. + +## Fast rollback (code) + +1. Identify the last known-good commit on `main` (e.g. `git log main` and + find the commit before the regression). +2. Check out that commit on a fresh branch: + + ```bash + git checkout -b rollback/ + ``` + +3. Run the deploy workflow on that branch, or locally: + + ```bash + pnpm deploy + ``` + +4. Smoke check: `curl --fail https://read.significanthobbies.com/` and log + in to verify auth + library load. + +## Rollback a schema migration + +Only needed if a migration broke production. Additive migrations (the common +case) do not require rollback — old code ignores new tables/columns. + +For a destructive migration that removed data: + +1. Restore the Turso DB from the most recent backup: + + ```bash + turso db shell ".databases" # confirm the DB + turso db shell ".restore " + ``` + +2. Redeploy the last known-good application code (above). +3. Verify with `pnpm db:studio` or a direct query. + +Take a Turso backup before any destructive migration in the future. + +## Rollback a landing overlay change + +The landing is built from `landing-astro/` and overlaid onto `dist/index.html` +during `cf:build`. If a landing change breaks `/`: + +1. Revert the `landing-astro/` change on a branch. +2. Redeploy (`pnpm deploy`). + +The SPA at `/app` is unaffected by landing-only changes. + +## Rollback a config-only change (`wrangler.toml`) + +`wrangler.toml` changes (bindings, routes, vars, placement) take effect on the +next `wrangler deploy`. To roll back, revert the file and redeploy. + +**Caution:** removing a binding (e.g. `PDFS_BUCKET`) or changing the route +breaks production immediately. Revert and redeploy; do not edit `wrangler.toml` +in place on `main` without a smoke check. + +## What you cannot roll back + +- **R2 object deletes** — if a deploy deleted PDF objects, they are gone + unless you have R2 replication or a bucket backup. Avoid `delete` calls in + new code without a soft-delete path. +- **Turso row deletes** — restore from backup. +- **OAuth session invalidation** — rotating `BETTER_AUTH_SECRET` invalidates + all sessions; users must sign in again. There is no rollback. + +## Communication + +For a production rollback, record the incident in SaaS Maker (per the fleet +guidance in `AGENTS.md`) and add a dated entry to +[knowledge/learnings.md](../../knowledge/learnings.md) if there is a durable +lesson. diff --git a/docs/operations/runbooks/rotate-secrets.md b/docs/operations/runbooks/rotate-secrets.md new file mode 100644 index 0000000..257268f --- /dev/null +++ b/docs/operations/runbooks/rotate-secrets.md @@ -0,0 +1,68 @@ +# Runbook: Rotate Secrets + +Rotate on a schedule, after a suspected leak, or when personnel changes. The +production Worker reads secrets from Cloudflare Workers secrets +(`wrangler secret list`); Turso and Google OAuth have their own consoles. + +## Cloudflare Worker secrets + +```bash +wrangler secret put BETTER_AUTH_SECRET # openssl rand -base64 32 +wrangler secret put TURSO_AUTH_TOKEN # from Turso console +wrangler secret put GOOGLE_CLIENT_SECRET # from Google Cloud Console +wrangler secret put AI_GATEWAY_API_KEY # from free-ai gateway +``` + +`BETTER_AUTH_URL`, `AI_BASE_URL`, and `NODE_ENV` are not secrets — they live +in `wrangler.toml [vars]`. `GOOGLE_CLIENT_ID` is technically a secret in this +project (the deploy workflow checks for it) but is also safe to commit as a +var if you prefer; current convention keeps it as a secret. + +After rotating `BETTER_AUTH_SECRET`, existing sessions are invalidated — users +must sign in again. There is only one production user today, so this is a +non-event. + +## Turso + +1. Turso console → database → Tokens → create a new token. +2. `wrangler secret put TURSO_AUTH_TOKEN` with the new value. +3. Revoke the old token in the Turso console once the new one is live. + +## Google OAuth + +1. Google Cloud Console → APIs & Services → Credentials → your OAuth client. +2. Reset the client secret (or create a new client). +3. `wrangler secret put GOOGLE_CLIENT_SECRET` with the new value. +4. Update the authorised redirect URI if you changed the client + (`BETTER_AUTH_URL/api/auth/callback/google`). + +## free-ai gateway + +1. Generate a new gateway key (`openssl rand -hex 32`). +2. Update the gateway Worker's secret and any other fleet consumers. +3. `wrangler secret put AI_GATEWAY_API_KEY` (legacy alias `AI_API_KEY`) with + the new value. + +## R2 + +R2 access for local dev uses `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` in +`.env.local` (gitignored). Rotate via the Cloudflare dashboard → R2 → Manage +R2 API tokens. The production Worker uses the native `PDFS_BUCKET` binding, +not the S3-compat API keys, so R2 key rotation does not affect production. + +## Verify + +```bash +pnpm validate:env:deploy # confirms required env vars are present locally +wrangler secret list # confirms each secret is set on the Worker +``` + +The deploy workflow (`.github/workflows/deploy.yml`) re-checks +`wrangler secret list` for each required secret before building. + +## After rotation + +No redeploy is required for secret-only changes — Workers pick up new secret +values on the next request. For `BETTER_AUTH_SECRET` rotation, the next +request that signs a session uses the new key; old sessions fail validation +and the user is prompted to sign in again. diff --git a/docs/product/features.md b/docs/product/features.md new file mode 100644 index 0000000..b179ac2 --- /dev/null +++ b/docs/product/features.md @@ -0,0 +1,90 @@ +# Feature Inventory (shipped) + +Current feature surface. Source of truth for "what does the app do today"; +update when behaviour changes. For the API route map, see +[architecture/data-flow.md](../architecture/data-flow.md). + +## Reading & capture + +- **Article capture from URL** via `@mozilla/readability` + `linkedom` running + server-side in the `/api/snapshot` flow. HTML is sanitised before storage and + re-sanitised on read for defence-in-depth. +- **PDF upload, view, annotate, text extraction.** PDFs are stored in + Cloudflare R2 (`PDFS_BUCKET`); downloads are proxied through + `/api/pdfs/:id/download` so auth + ownership are enforced server-side. PDF + MIME is validated by magic bytes (`%PDF-`); 10 MB per-file limit + (`src/lib/pdf-service.ts`). PDFs use a `blob://` sentinel as the + article `url` to avoid colliding with the user-URL uniqueness index. +- **Link-type articles.** `/reader/:id` redirects link-type articles to their + canonical URL; cards in the library open the URL in a new tab with a distinct + visual treatment and context menu. +- **Rich annotations** with optional DOM anchoring; selection actions + (mouse-up or selection + right-click) expose `Add note` and `Ask AI`. +- **Reading-time estimates** auto-calculated; customisable reader (theme + light/dark/sepia, font sans/serif/mono, text size). + +## RSS / Atom inbox (`/rss`) + +- OPML import that recursively reads outlines with `xmlUrl`, deduplicates per + user, and reports imported/existing/rejected counts. +- Direct feed add/remove with SSRF-validated HTTP(S) URLs. +- Manual refresh with bounded concurrency (4 feeds at a time, 15 s timeout per + feed), per-feed success/error reporting, `ETag` / `Last-Modified` support, + and a response-size + entry cap. One bad feed does not abort the others. +- Unread/read inbox state; opening an entry marks it read and opens the + canonical URL in a new tab. +- Save an entry to the library exactly once (idempotent); creates an article + record when content is available, otherwise a link-type article. +- See [architecture/decisions/0008-rss-inbox.md](../architecture/decisions/0008-rss-inbox.md) + and the OpenSpec specs under [`openspec/specs/`](../../openspec/specs/). + +## Organisation & search + +- **Tags** with colour badges, autocomplete, and filtering. +- **Lists** (grouping) and **Boards** (Kanban-style view via `@xyflow/react`). + Boards and lists have shareable share-link endpoints. +- **Full-text search** across article content, notes, and AI chat history + (Cmd/Ctrl+K). Search is in-memory `LIKE`-style today, not FTS5 — see + [knowledge/learnings.md](../knowledge/learnings.md) for the trade-off. +- **Reading progress** tracking (`in_progress` / `completed` status). +- **Session review** per article (`POST /api/articles/:id/session-review`). + +## AI features + +- **Per-article AI chat** with persistent markdown history. +- **Auto-summaries** (short/medium/long) and **key points** extraction (3–5 + bullets) via `POST /api/ai/summarize`. +- **BYOK providers** (OpenAI/Anthropic/Gemini) + free-ai gateway + local AI + mode. BYOK keys are sent per-request from the browser and never persisted + server-side. See [architecture/decisions/0005-ai-gateway-byok.md](../architecture/decisions/0005-ai-gateway-byok.md). +- **Model listing** via `POST /api/ai/models` (proxies `/models` on the + configured endpoint). + +## Memory capture + +- `/memory` page with persisted, authenticated captures (`memories` Turso + table, `/api/memories` CRUD + `/api/memories/search`). +- `POST /api/browser-memory/import` for browser-memory imports. +- Global SearchBar routes memory hits to `/memory`. + +## Chrome extension (MV3) + +- Side panel (persistent chat surface) + popup (ephemeral one-click capture). +- Content script runs `@mozilla/readability` in the page context on demand. +- Chrome Reading List sync (URLs, titles, read state). +- Authenticates with `rdr_*` API keys via `/api/keys` (hashed at rest). +- See [architecture/decisions/0006-mv3-side-panel.md](../architecture/decisions/0006-mv3-side-panel.md) + and `packages/chrome-extension/README.md`. + +## Security & audit fixes (carried forward from the Firebase era) + +- Auth on snapshot routes; SSRF validation (`src/lib/url-validation.ts`) and + redirect-safe fetch (`src/lib/safe-fetch.ts`) on all server-side URL fetches. +- PDFs accessed only through authenticated, ownership-checked proxy routes + (no public R2 URLs). +- HTML sanitised at ingestion and re-sanitised on read. +- Security headers applied in `src/worker.ts` (`X-Content-Type-Options`, + `X-Frame-Options`, `Referrer-Policy`, `Permissions-Policy`, HSTS). +- Residual / deferred items: no explicit CORS config (acceptable same-origin), + no rate limiting on AI/snapshot/proxy (deferred pending evidence). See + [archive/security-audit-2026-03-29.md](../archive/security-audit-2026-03-29.md). diff --git a/docs/product/overview.md b/docs/product/overview.md new file mode 100644 index 0000000..94cf425 --- /dev/null +++ b/docs/product/overview.md @@ -0,0 +1,59 @@ +# Product Overview + +## What + +Reader is a personal research library: capture web articles and PDFs, read them +in a distraction-free reader, annotate with notes and highlights, organise with +tags/lists/boards, search across everything, and AI-chat or auto-summarise the +saved material. A companion Chrome MV3 extension captures pages from the +browser and syncs with Chrome's native Reading List. + +## Who + +- **End users:** individual readers saving articles and PDFs. Sign-in is Google + OAuth via better-auth; data is per-user isolated at the database level. +- **Operators:** the maintainer running Turso schema migrations and Cloudflare + Workers deploys. Currently single-user in production. + +## Where + +- Production app: `https://read.significanthobbies.com` (Cloudflare Worker + `reader`, custom domain). See [surfaces.md](surfaces.md) for the full list. +- Source: this repository. +- Landing page: built from `landing-astro/` and overlaid onto `dist/index.html` + during `cf:build`; the SPA lives at `dist/app.html` and is served at `/app`. + +## Scope + +**In scope:** article/PDF capture, rich annotations, tags/lists/boards, +full-text search, AI chat and summaries, RSS/Atom inbox with OPML import, +memory capture, Turso persistence, R2 PDF storage, free-ai gateway + BYOK + +local-ai dev bridge. + +**Out of scope (deliberate):** + +- Browser-extension distribution (deferred until web import/capture is + reliable). +- Full personal knowledge-base automation behind strong capture, retrieval, and + trust primitives. +- Paid team/library workflows. +- `landing-astro` as a separate deployable product — it is an overlay only. +- RSS background refresh / scheduled triggers / notifications / feed discovery + (current RSS refresh is manual). +- Rate limiting on AI/snapshot/proxy endpoints (deferred until endpoint-specific + evidence; see [operations/env.md](../operations/env.md) and the residual + audit notes in [archive/security-audit-2026-03-29.md](../archive/security-audit-2026-03-29.md)). + +## Operating posture + +Personal-use support (closure decision 2026-07-10): keep Reader available for +direct use. No roadmap expansion; accept only maintenance, reliability, or +personally requested workflow fixes. See [STATUS.md](../../STATUS.md) for the +current objective and active work. + +## Branding note + +The product is named **Reader** and served at +`read.significanthobbies.com`. The package name and Chrome extension still use +the legacy `web-annotator` / "Web Annotator" string in places; the canonical +product name in new docs is **Reader**. diff --git a/docs/product/surfaces.md b/docs/product/surfaces.md new file mode 100644 index 0000000..94cbc6c --- /dev/null +++ b/docs/product/surfaces.md @@ -0,0 +1,56 @@ +# Production Surfaces + +## App + +| Surface | URL | Notes | +| --- | --- | --- | +| Production app | `https://read.significanthobbies.com` | Cloudflare Worker `reader`, custom domain | +| Landing page | `https://read.significanthobbies.com/` | Static Astro page overlaid onto `dist/index.html` during `cf:build` | +| SPA entry | `https://read.significanthobbies.com/app` | Vite SPA built from `app.html`; served via the `ASSETS` binding | +| Login | `https://read.significanthobbies.com/login` | Google OAuth via better-auth | +| Library | `https://read.significanthobbies.com/library` | Auth-walled | +| RSS inbox | `https://read.significanthobbies.com/rss` | Auth-walled | +| Memory | `https://read.significanthobbies.com/memory` | Auth-walled | +| Boards | `https://read.significanthobbies.com/board` | Auth-walled | +| Reader | `https://read.significanthobbies.com/reader/:id` | Auth-walled; link-type articles 302 to their URL | +| Share | `https://read.significanthobbies.com/share/:shareId` | Public board share | +| Shared article | `https://read.significanthobbies.com/share/article/:shareId` | Public article share | + +Auth-walled routes are not agent-indexed. + +## Agent / crawler surfaces + +Served from `public/` and the `agent-edge.mjs` handler in `src/worker.ts` +(before the SPA/ASSETS fallback). The Worker `run_worker_first` config in +`wrangler.toml` ensures `/api/*`, `/`, `/sitemap.xml`, `/index.md`, +`/llms-full.txt`, and `/llms.txt` hit the Worker first. + +| Surface | Path | Purpose | +| --- | --- | --- | +| `llms.txt` | `/llms.txt` | Concise agent index (links to product + machine surfaces) | +| `llms-full.txt` | `/llms-full.txt` | Full agent brief | +| `index.md` | `/index.md` | Product brief in Markdown (no JS) | +| `api/ai` | `/api/ai` | JSON catalog of public surfaces | +| `robots.txt` | `/robots.txt` | Allows all + lists agent surfaces | +| `sitemap.xml` | `/sitemap.xml` | Public URL inventory | +| IndexNow key | `/fa7259e2e0d942f1a1267b344a75a143.txt` | Bing/Yandex URL submission key | + +The agent-edge payload is generated into `src/agent-edge.mjs` by the fleet +`apply-agent-surfaces` tooling; the Markdown sources in `public/` are the +human-editable mirrors. When updating agent copy, edit the `public/` files +and regenerate `agent-edge.mjs` per the fleet standard +(`fleet-ops/docs/agent-indexing-standard.md`). + +## Internal fleet services used + +| Service | Role | +| --- | --- | +| `free-ai` gateway | Default AI chokepoint via `AI_BASE_URL` (`https://ai-gateway.sassmaker.com/v1`) with `x-gateway-project-id: reader` | +| `local-ai` | Dev bridge for authenticated local CLI models (`pnpm local-ai`) | +| SaaS Maker feedback widget | In-app feedback capture (`@saas-maker/feedback`) | + +## CI/CD + +GitHub Actions: CI on push/PR, manual deploy (`workflow_dispatch`), manual AI +review, and a weekly quality cron. See +[operations/ci-cd.md](../operations/ci-cd.md). diff --git a/landing-astro/src/pages/faq.astro b/landing-astro/src/pages/faq.astro new file mode 100644 index 0000000..0a8d35b --- /dev/null +++ b/landing-astro/src/pages/faq.astro @@ -0,0 +1,172 @@ +--- +import Layout from '@/layouts/Layout.astro'; + +const TITLE = 'Library Reader — FAQ'; +const DESCRIPTION = + 'Frequently asked questions about Library Reader — a personal research library for web articles and PDFs with highlights, notes, AI chat, and a Chrome extension.'; + +const SITE_URL = 'https://read.significanthobbies.com'; + +const faqs = [ + { + q: 'How can I save and annotate web articles for research?', + a: 'Paste a URL into Library Reader and the page is cleaned with Mozilla Readability into a distraction-free reader view. Select any text to add a persistent highlight or open the side panel for notes — annotations anchor to the exact text you marked so they survive across visits. You can also capture the current page in one click from the Chrome extension.', + }, + { + q: "What's the best tool to build a personal research library with PDF support?", + a: 'Library Reader keeps articles and PDFs in one library. Paste a URL and the page is cleaned with Readability; upload a PDF and it goes into the same library with text extraction so it is searchable and chat-able. For signed-in users, PDFs are stored in Cloudflare R2 with ownership enforced through a proxy; in local mode the PDF stays in your browser.', + }, + { + q: 'How do I organize articles with tags and projects for research?', + a: 'Every article can be tagged, added to lists, and placed on boards. Boards are a Kanban view (built on @xyflow/react) for arranging research projects spatially — drag cards between columns as a piece moves from "to read" to "reading" to "done." Lists group related sources, and tags cross-cut both for fast retrieval.', + }, + { + q: 'Can I chat with AI about my saved articles and PDFs?', + a: 'Yes. The notes side panel includes an AI chat that runs over the article or PDF you have open. You bring your own key — OpenAI, Anthropic, or Gemini — stored in your browser. There is also a gateway option and a local-AI mode for development. We never proxy your key through a server.', + }, + { + q: 'What tool lets me highlight text and add notes to web articles?', + a: "Library Reader's reader view supports persistent highlights and a side panel for notes. Annotations anchor to the exact text you marked, so they survive across visits and reloads. The same selection flow lets you send a quote straight to AI chat.", + }, + { + q: 'How can I search across all my saved articles and notes?', + a: 'The library has built-in full-text search across article bodies, titles, and your notes. Results surface matches from both web articles and PDFs (PDFs have their text extracted on upload), so one query covers your whole library.', + }, + { + q: 'Is there a read-it-later app with AI chat capabilities?', + a: 'Library Reader is a read-it-later library with AI chat built in. Save a page (via paste or the Chrome extension), read it in a cleaned typography-first view, then ask questions about it in the side panel using your own AI key. Summaries and Q&A both run over the text you saved.', + }, + { + q: 'How do I extract key points from articles automatically?', + a: 'Open any saved article or PDF and generate an AI summary from the reader view. The summary pulls the key points out of the full text — useful for long articles and PDFs you want to triage before reading in full.', + }, + { + q: "What's the best way to manage research papers and web articles together?", + a: 'Library Reader treats PDFs and web articles as first-class peers in the same library. Upload a PDF and its text is extracted for search and chat; paste a URL and the page is cleaned with Readability. Both appear in the same lists, boards, and search results, so your research papers and web sources live in one place.', + }, + { + q: "Can I get AI summaries of articles I've saved?", + a: 'Yes. Each saved article or PDF has a summary action that generates an AI summary over its full text. Use your own OpenAI, Anthropic, or Gemini key, or the free-ai gateway.', + }, + { + q: 'How do I track reading progress across multiple articles?', + a: 'Use boards to track reading progress Kanban-style — move cards across columns like "to read," "reading," and "done." Lists and tags give you additional ways to slice what is in flight, and the library preserves your highlights and notes so you can pick up where you left off.', + }, + { + q: 'What tool combines article saving, annotation, and AI assistance?', + a: 'Library Reader does all three in one place: save (paste URL, drop PDF, or one-click from the Chrome extension), annotate (persistent highlights + side-panel notes anchored to exact text), and AI assistance (chat and summaries over the open source using your own key).', + }, + { + q: 'How can I organize my research into projects?', + a: 'Boards give each research project a Kanban canvas — drag sources between columns to reflect status. Combine boards with lists (grouping) and tags (cross-cutting labels) so a single source can belong to a project board and still be found by topic.', + }, + { + q: 'Is there a Chrome extension for saving articles with annotations?', + a: 'Yes. The Library Reader Chrome MV3 extension captures the page you are on (cleaned via Readability) and saves it to your library in one click. It also exposes the Listen control so you can have any page read aloud without leaving it, and a side panel for chat.', + }, + { + q: "What's the best personal knowledge base for web content?", + a: 'Library Reader is a personal knowledge base for web content and PDFs: cleaned articles, persistent highlights, side-panel notes, full-text search, tags, lists, and boards — plus AI chat and summaries over everything you save. It works in your browser without an account, and syncs when you sign in with Google.', + }, + { + q: 'How do I generate intelligent summaries of saved articles?', + a: 'Open any saved article or PDF and choose the summary action. The AI reads the full extracted text and returns the key points. Bring your own OpenAI, Anthropic, or Gemini key, or use the free-ai gateway.', + }, + { + q: 'Can I use my own AI API keys with a research tool?', + a: 'Yes. Library Reader lets you bring your own key — OpenAI, Anthropic, or Gemini — stored in your browser. There is also a gateway option and a local-AI mode for development. We never proxy your key through a server.', + }, + { + q: 'What tool provides distraction-free reading with annotation?', + a: "Library Reader's reader view cleans articles with Mozilla Readability into a typography-first layout, then layers persistent highlights and a notes side panel on top. Select text to highlight or annotate; annotations anchor to the exact text so they survive across visits.", + }, + { + q: 'How can I ask questions about my PDF documents?', + a: 'Upload a PDF to your library and its text is extracted on upload. Open it in the PDF reader and use the AI chat side panel to ask questions about the document — answers are grounded in the extracted text. The same chat works on web articles.', + }, + { + q: 'Is there a research library that supports both articles and PDFs?', + a: 'Library Reader is built around exactly that: one library for web articles (cleaned via Readability) and PDFs (text extracted on upload). Both share the same highlights, notes, tags, lists, boards, search, and AI chat, so articles and PDFs are first-class peers.', + }, +]; + +const faqJsonLd = { + '@context': 'https://schema.org', + '@type': 'FAQPage', + mainEntity: faqs.map((f) => ({ + '@type': 'Question', + name: f.q, + acceptedAnswer: { + '@type': 'Answer', + text: f.a, + }, + })), +}; +--- + + +
+
+
+ + + Library Reader + + + +
+
+ +
+
+ FAQ +

Questions about Library Reader

+

+ A personal research library for web articles and PDFs — capture, read, highlight, + listen, and chat with AI over your own sources. Here are the things people ask most. +

+
+
+ +
+

Frequently asked questions

+
+ {faqs.map((f) => ( +
+ {f.q} +

{f.a}

+
+ ))} +
+
+ +
+
+

Start your library

+

+ Save your first article or PDF. It works in your browser right now — sign in later if you + want to sync. +

+ + Save your first article + + +

+ About · Privacy · + Chrome extension +

+
+
+ +
+ Your reading list is not the same as your understanding. Library Reader helps with the gap. +
+
+ +