Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
613 changes: 383 additions & 230 deletions .agents/skills/youtube-direct/scripts/youtube.mjs

Large diffs are not rendered by default.

729 changes: 440 additions & 289 deletions .agents/skills/youtube-watch/scripts/watch.mjs

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
.agents/skills/youtube-direct/scripts/youtube.mjs linguist-generated
.agents/skills/youtube-watch/scripts/watch.mjs linguist-generated
11 changes: 8 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,19 @@ jobs:
with:
node-version: 22
cache: npm
cache-dependency-path: packages/all-things-youtube/package-lock.json
cache-dependency-path: |
packages/all-things-youtube/package-lock.json
packages/youtube-skills/package-lock.json
- run: npm ci
- run: npm run build
- run: npm ci --prefix ../youtube-skills
- run: sudo apt-get update && sudo apt-get install --yes ffmpeg
- run: npm test
- run: npm test --prefix ../youtube-skills
env:
WATCH_FFMPEG_TEST: '1'
- run: npm run build
- run: npm run skill:check
- run: npm run typecheck --prefix ../youtube-skills
- run: npm run bundle:check --prefix ../youtube-skills
- run: test -x ../../.agents/skills/youtube-direct/scripts/youtube.mjs
- run: node ../../.agents/skills/youtube-direct/scripts/youtube.mjs --help
- run: test -x ../../.agents/skills/youtube-watch/scripts/watch.mjs
Expand Down
23 changes: 14 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,18 +126,18 @@ const transcript = await getTranscript({
console.log(transcript.text);
```

Visual context is available through the Node-only watch subpath. FFmpeg is needed only for exact frames:
YouTube's native storyboard contact sheets are available as a package primitive:

```ts
import { getWatchIndex, extractFrames } from 'all-things-youtube/watch';

const outputDir = '/tmp/youtube-watch';
const index = await getWatchIndex({ videoId: '4vItmdk8F_M', outputDir });
const frames = await extractFrames({
videoId: index.videoId,
timestampsMs: [30_000, 686_000],
outputDir,
import { getStoryboard } from 'all-things-youtube';

const storyboard = await getStoryboard({
videoId: '4vItmdk8F_M',
outputDir: '/tmp/youtube-storyboard',
maxSheets: 12,
});

console.log(storyboard.sheets);
```

The package requires Node.js 18 or newer, includes its own types, and does not require a YouTube Data API key. Keep it server-side: browser requests are commonly blocked by CORS and distribute upstream rate-limit pressure across users.
Expand All @@ -159,6 +159,7 @@ git clone https://github.com/devhims/video2ctx.git
cd video2ctx

npm ci --prefix packages/all-things-youtube
npm ci --prefix packages/youtube-skills
npm ci --prefix packages/video2ctx-cli
npm ci --prefix platform
npm ci --prefix platform/youtube-processor
Expand Down Expand Up @@ -257,6 +258,7 @@ For the complete request path and reliability model, see [`reference/engineering
| [`platform/`](./platform) | TypeScript/Hono Cloudflare Worker with auth, API keys, billing, D1, R2, KV, AI, queues, workflows, and OpenAPI |
| [`platform/youtube-processor/`](./platform/youtube-processor) | Private Node 22/Hono container for outbound YouTube operations and optional proxy egress |
| [`packages/all-things-youtube/`](./packages/all-things-youtube) | Publishable normalized YouTube client and public TypeScript data model |
| [`packages/youtube-skills/`](./packages/youtube-skills) | Private source, tests, and bundling for the self-contained direct YouTube skills |
| [`packages/video2ctx-cli/`](./packages/video2ctx-cli) | Independently published CLI for device login and authenticated hosted API access |
| [`docs/`](./docs) | Public Mintlify documentation site |
| [`reference/`](./reference) | Internal architecture, design, deployment, and agent guidance |
Expand Down Expand Up @@ -294,6 +296,8 @@ Common commands, run from the repository root:
| `npm --prefix platform run verify` | Install processor dependencies, type-check the Worker, and run platform and processor tests |
| `npm --prefix platform run test:container` | Run only the processor contract tests |
| `npm --prefix packages/all-things-youtube run test:watch` | Run the library suite in watch mode |
| `npm run test:skills` | Test, type-check, and verify the private direct-skill source and committed bundles |
| `npm run skill:bundle` | Regenerate the committed self-contained direct-skill executables |
| `npm --prefix packages/video2ctx-cli run verify` | Test and build the hosted-service CLI |
| `npm pack ./packages/video2ctx-cli --dry-run` | Verify the public CLI tarball contents before release |
| `npm run docs:dev` | Preview the Mintlify documentation site locally |
Expand All @@ -306,6 +310,7 @@ Before opening a pull request, run:
```bash
npm run build
npm test
npm run test:skills
npm --prefix web test
npm run docs:check
npm run docs:verify
Expand Down
1 change: 1 addition & 0 deletions docs/open-source/local-development.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ The repository contains the Next.js web application, Cloudflare Worker platform,
git clone https://github.com/devhims/video2ctx.git
cd video2ctx
npm install --prefix packages/all-things-youtube
npm install --prefix packages/youtube-skills
npm install --prefix platform
npm install --prefix web
npm install --prefix docs
Expand Down
5 changes: 3 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@
"build:library": "npm --prefix packages/all-things-youtube run build",
"test:library": "npm --prefix packages/all-things-youtube run test",
"pack:library": "npm pack ./packages/all-things-youtube",
"skill:bundle": "npm --prefix packages/all-things-youtube run skill:bundle",
"skill:check": "npm --prefix packages/all-things-youtube run skill:check",
"skill:bundle": "npm --prefix packages/youtube-skills run bundle",
"skill:check": "npm --prefix packages/youtube-skills run bundle:check",
"test:skills": "npm run build:library && npm --prefix packages/youtube-skills run verify",
"pack:cli": "npm pack ./packages/video2ctx-cli",
"test:auth": "npm --prefix packages/video2ctx-cli test && npm --prefix platform run test:auth",
"test:auth:integration": "npm --prefix platform run test:auth:integration",
Expand Down
66 changes: 40 additions & 26 deletions packages/all-things-youtube/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Search YouTube and get transcripts/captions, comments, video details, channels,

YouTube exposes useful public data across several different page experiences. `all-things-youtube` gives that data one small, task-oriented interface:

- **One import, ten focused functions:** Search or get exactly the resource you need instead of constructing a large client or learning a raw response format.
- **One import, eleven focused functions:** Search or get exactly the resource you need instead of constructing a large client or learning a raw response format.
- **Translation made simple:** Call `getTranscript()` with your desired output language. Supports 150+ languages when available.
- **Complete channel and playlist data:** Channel About links and statistics, channel sorting, playlist cards, and continuation-based pagination are represented directly.
- **Bounded networking by default:** Each attempt has a ten-second deadline; transient failures and `429` responses use bounded exponential backoff with jitter and `Retry-After` support.
Expand All @@ -35,7 +35,7 @@ npm install all-things-youtube

Requires Node.js 18 or newer. The package uses the runtime's standard `fetch`; you can supply your own implementation when needed.

The optional `all-things-youtube/watch` entry point uses a system FFmpeg executable for exact frames. Storyboard and transcript indexing does not require FFmpeg.
Storyboard contact sheets are downloaded as JPEG files beneath a caller-owned output directory. The package does not require FFmpeg.

> Keep calls server-side. Direct browser calls are commonly blocked by CORS and make every user's browser responsible for upstream rate limits.

Expand Down Expand Up @@ -100,50 +100,37 @@ All functions are named exports and accept a single options object.
| Search | `search()` | Paginated videos, channels, and playlists |
| Caption catalog | `getTracks()` | Source tracks and available translation languages |
| Transcript | `getTranscript()` | Full text plus timed segments or words |
| Storyboard | `getStoryboard()` | Contact sheets plus timestamp-ready tile mappings |
| Comments | `getComments()` | One page or a bounded complete collection |
| Video | `getDetails()` | Core metadata, channel, keywords, and availability |
| End screen | `getEndscreen()` | Timed video, playlist, and channel elements |
| Channel | `getChannelInfo()` | Identity and the public About view |
| Channel videos | `getChannelVideos()` | One sorted, paginated Videos-tab page |
| Channel playlists | `getChannelPlaylists()` | One sorted, paginated Playlists-tab page |
| Playlist | `getPlaylist()` | Playlist metadata and one page of videos |
| Watch index | `getWatchIndex()` | Timed transcript and storyboard contact sheets |
| Exact frames | `extractFrames()` | Best-effort timestamped JPEG files |

The original task functions remain on the main entry point. The two visual functions are exported from `all-things-youtube/watch`.
Every function is exported from the main `all-things-youtube` entry point.

## Visual context
## Storyboards

Use the storyboard/transcript index to choose focused timestamps before decoding exact frames:
Download YouTube's native storyboard contact sheets when you need a lightweight visual index:

```ts
import { getWatchIndex, extractFrames } from 'all-things-youtube/watch';
import { getStoryboard } from 'all-things-youtube';

const outputDir = '/tmp/youtube-watch';
const index = await getWatchIndex({
const storyboard = await getStoryboard({
videoId: '4vItmdk8F_M',
outputDir,
granularity: 'segment',
outputDir: '/tmp/youtube-storyboard',
maxSheets: 12,
});

for (const sheet of index.storyboard?.sheets ?? []) {
for (const sheet of storyboard.sheets) {
console.log(sheet.path, sheet.firstFrameIndex, sheet.intervalMs);
}

const result = await extractFrames({
videoId: index.videoId,
timestampsMs: [30_000, 686_000],
outputDir,
maxWidth: 1280,
});

console.log(result.frames);
console.log(result.failures);
```

`getWatchIndex()` returns complete contact-sheet paths and tile mappings without exposing YouTube's storyboard URLs. Transcript timing defaults to segments; pass `granularity: 'word'` only when word timing is necessary.

`extractFrames()` accepts one to 30 timestamps, uses the best seekable public media format, and may fall back below 720p. It keeps signed media URLs in memory, gives FFmpeg a localhost proxy URL, and returns successful frames alongside per-timestamp failures. Install FFmpeg on `PATH`, set `FFMPEG_PATH`, or pass `ffmpegPath`.
The returned sheet paths are absolute. A tile's timestamp is
`(firstFrameIndex + row * columns + column) * intervalMs`. The package downloads at most 12 sheets by default; `maxSheets` accepts 1–20. Storyboard URLs remain internal and are never returned.

## Working with responses

Expand Down Expand Up @@ -266,6 +253,33 @@ for (const segment of transcript.segments) {

Omit `lang` to return the default source track. Supplying `lang` asks for that output language regardless of whether the source captions are English, Spanish, or another supported language. If translation is unavailable, the call rejects with `INVALID_INPUT`.

### `getStoryboard(options)`

Downloads YouTube's native storyboard contact sheets and returns the information needed to map every tile to a timestamp.

```ts
import { getStoryboard } from 'all-things-youtube';

const storyboard = await getStoryboard({
videoId: '4vItmdk8F_M',
outputDir: '/tmp/youtube-storyboard',
maxSheets: 12,
});

for (const sheet of storyboard.sheets) {
console.log(sheet.path, sheet.columns, sheet.rows, sheet.intervalMs);
}
```

| Option | Type | Required | Default | Description |
| ----------- | -------- | -------- | ------- | ----------------------------------- |
| `videoId` | `string` | Yes | — | YouTube video ID |
| `outputDir` | `string` | Yes | — | Caller-owned directory for JPEGs |
| `maxSheets` | `number` | No | `12` | Sheet budget from 1 through 20 |

The highest usable storyboard level is selected. A tile at `row` and `column` represents
`(firstFrameIndex + row * columns + column) * intervalMs`. The package creates a `storyboards` child directory but never recursively deletes `outputDir`.

### `getComments(options)`

Fetch one page for interactive pagination, or crawl the available thread and reply pages in one call.
Expand Down
16 changes: 2 additions & 14 deletions packages/all-things-youtube/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 2 additions & 12 deletions packages/all-things-youtube/package.json
Original file line number Diff line number Diff line change
@@ -1,19 +1,14 @@
{
"name": "all-things-youtube",
"version": "0.3.0",
"description": "TypeScript toolkit for YouTube transcripts, metadata, storyboards, and timestamped visual context.",
"version": "0.4.0",
"description": "TypeScript toolkit for YouTube transcripts, metadata, storyboards, comments, channels, and playlists.",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"exports": {
".": {
"types": "./dist/index.d.ts",
"require": "./dist/index.js",
"default": "./dist/index.js"
},
"./watch": {
"types": "./dist/watch/index.d.ts",
"require": "./dist/watch/index.js",
"default": "./dist/watch/index.js"
}
},
"files": [
Expand All @@ -27,8 +22,6 @@
"scripts": {
"clean": "node -e \"require('node:fs').rmSync('dist', { recursive: true, force: true })\"",
"build": "npm run clean && tsc",
"skill:bundle": "node scripts/build-skill-bundle.mjs",
"skill:check": "node scripts/build-skill-bundle.mjs --check",
"test": "vitest run",
"test:watch": "vitest",
"prepack": "npm test && npm run build"
Expand All @@ -42,7 +35,6 @@
"youtube-comments",
"youtube-channel",
"youtube-playlist",
"youtube-frames",
"youtube-storyboard",
"transcript",
"captions",
Expand Down Expand Up @@ -74,9 +66,7 @@
"devDependencies": {
"@types/he": "^1.2.0",
"@types/node": "^25.9.5",
"esbuild": "0.28.2",
"typescript": "^6.0.3",
"undici": "6.28.0",
"vitest": "^3.2.4"
},
"engines": {
Expand Down
10 changes: 10 additions & 0 deletions packages/all-things-youtube/src/index.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ const mocks = vi.hoisted(() => {
getAllComments: vi.fn(),
getVideo: vi.fn(),
getEndscreen: vi.fn(),
getStoryboard: vi.fn(),
getChannel: vi.fn(),
getChannelVideos: vi.fn(),
getChannelPlaylists: vi.fn(),
Expand All @@ -32,6 +33,7 @@ import {
getDetails,
getEndscreen,
getPlaylist,
getStoryboard,
search,
getTracks,
getTranscript,
Expand All @@ -54,11 +56,14 @@ describe('all-things-youtube public API', () => {
getChannelVideos,
getChannelPlaylists,
getPlaylist,
getStoryboard,
search,
]) {
expect(typeof exported).toBe('function');
}
expect(publicApi).not.toHaveProperty('createYouTubeClient');
expect(publicApi).not.toHaveProperty('getWatchIndex');
expect(publicApi).not.toHaveProperty('extractFrames');
});

test('maps stateless search to the normalized client', async () => {
Expand Down Expand Up @@ -87,11 +92,13 @@ describe('all-things-youtube public API', () => {
mocks.client.getTranscript.mockResolvedValue({ segments: [] });
mocks.client.getVideo.mockResolvedValue({ id: 'abcdefghijk' });
mocks.client.getEndscreen.mockResolvedValue([]);
mocks.client.getStoryboard.mockResolvedValue({ sheets: [] });

await getTracks({ videoId: 'abcdefghijk', fetch: fetchMock });
await getTranscript({ videoId: 'abcdefghijk', lang: 'hi', granularity: 'word' });
await getDetails({ videoId: 'abcdefghijk' });
await getEndscreen({ videoId: 'abcdefghijk' });
await getStoryboard({ videoId: 'abcdefghijk', outputDir: '/tmp/storyboards', maxSheets: 3 });

expect(mocks.createYouTubeClient).toHaveBeenCalledWith(expect.objectContaining({ fetch: fetchMock }));
expect(mocks.client.getCaptionTracks).toHaveBeenCalledWith('abcdefghijk');
Expand All @@ -100,6 +107,9 @@ describe('all-things-youtube public API', () => {
});
expect(mocks.client.getVideo).toHaveBeenCalledWith('abcdefghijk');
expect(mocks.client.getEndscreen).toHaveBeenCalledWith('abcdefghijk');
expect(mocks.client.getStoryboard).toHaveBeenCalledWith({
videoId: 'abcdefghijk', outputDir: '/tmp/storyboards', maxSheets: 3,
});
});

test('keeps comment continuation internals out of the public result', async () => {
Expand Down
11 changes: 11 additions & 0 deletions packages/all-things-youtube/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,11 @@ export interface TranscriptRequest extends VideoRequest {
granularity?: 'segment' | 'word';
}

export interface StoryboardRequest extends VideoRequest {
outputDir: string;
maxSheets?: number;
}

export interface CommentsRequest extends VideoRequest {
continuation?: string;
all?: false;
Expand Down Expand Up @@ -157,6 +162,12 @@ export function getTranscript(options: TranscriptRequest): Promise<Transcript> {
});
}

/** Download normalized YouTube storyboard contact sheets and timestamp mappings. */
export function getStoryboard(options: StoryboardRequest) {
const { videoId, outputDir, maxSheets } = options;
return createYouTubeClient(optionsFrom(options)).getStoryboard({ videoId, outputDir, maxSheets });
}

export function getComments(options: AllCommentsRequest): Promise<CommentsCollection>;
export function getComments(options: CommentsRequest): Promise<CommentsPage>;
export async function getComments(
Expand Down
Loading
Loading