Data Connectors

This repository contains two different connector artifact families. They do not share a runtime or authoring workflow.

• For new connector work, start with Connector authoring. Author the connector in PDP-Connect/pdpp, then package its pinned PDPP artifact here.
• For existing *-playwright maintenance, use Legacy Playwright connector maintenance. A new legacy connector requires an explicit exception.

The existing Playwright connectors use browser automation for local desktop collection. The bundled playwright-runner/ runs that legacy format. It does not run PDPP Collection Profiles.

PDPP Collection Profiles are packaged and installed as a different artifact kind. They support only the bindings declared in their own manifest. Do not use the Playwright scaffold or create scripts to start a PDPP profile.

GitHub and ChatGPT are the current PDPP artifact examples. The checked-in connector-index.json intentionally marks both artifacts with releaseId: "unpublished" until CI regenerates the source-tree index. The immutable connectors-48440fead534 release contains the signed artifacts. The connectors-latest release also provides them. DataConnect v0.7.54 includes both PDPP profiles.

Legacy Playwright connector status

Each connector has a status indicating its maturity level:

• Stable -- Production-ready. Well-tested across multiple releases, reliable login and data extraction, complete schemas.
• Beta -- Functional but less tested. Works in most cases but may have edge cases or require more login handling.
• Experimental -- New or untested. May not work reliably; contributed recently with limited real-world usage.

Connectors

This table is an overview, not the public contract. scope-catalog.json is the authoritative machine-readable contract for sources and scopes. SCOPES.md is its generated human-readable view.

The catalog describes only the local desktop collection path. Registered legacy Playwright connectors use your browser session. It does not list PDPP Collection Profiles.

Source	Published scopes	Desktop connector(s) / maturity
amazon	amazon.orders, amazon.profile	amazon-playwright (beta)
chatgpt	chatgpt.conversations, chatgpt.memories	chatgpt-playwright (stable)
claude	claude.conversations, claude.projects	claude-export-playwright (experimental)
doordash	doordash.orders	doordash-playwright (beta)
github	github.contributions, github.events, github.history, github.profile, github.repositories, github.starred	github-playwright (stable)
heb	heb.nutrition, heb.orders, heb.profile	heb-playwright (experimental)
icloud_notes	icloud_notes.folders, icloud_notes.notes	icloud-notes-playwright (experimental)
instagram	instagram.ads, instagram.following, instagram.posts, instagram.profile	instagram-playwright (stable); instagram-ads-playwright (beta)
linkedin	linkedin.connections, linkedin.education, linkedin.experience, linkedin.languages, linkedin.profile, linkedin.skills	linkedin-playwright (stable)
oura	oura.activity, oura.readiness, oura.sleep	oura-playwright (stable)
shop	shop.orders	shop-playwright (beta)
spotify	spotify.playlists, spotify.profile, spotify.savedTracks	spotify-playwright (stable)
steam	steam.friends, steam.games, steam.profile	steam-playwright (experimental)
uber	uber.receipts, uber.trips	uber-playwright (beta)
wholefoods	wholefoods.nutrition, wholefoods.orders, wholefoods.profile	wholefoods-playwright (experimental)
youtube	youtube.history, youtube.likes, youtube.playlistItems, youtube.playlists, youtube.profile, youtube.subscriptions, youtube.watchLater	youtube-playwright (beta)

Running a legacy Playwright connector

# One-time setup: install the bundled runner's dependencies + Chromium
bash skills/pdp-connect/scripts/setup.sh

node run-connector.cjs ./connectors/github/github-playwright.js              # JSON output (for agents)
node run-connector.cjs ./connectors/github/github-playwright.js --pretty      # colored output (for humans)
node run-connector.cjs ./connectors/github/github-playwright.js --inputs '{"username":"x","password":"y"}'

run-connector.cjs resolves the bundled playwright-runner/ in this repo automatically. It runs only legacy Playwright connectors. See skills/pdp-connect/SETUP.md and skills/pdp-connect/ for the legacy connector skill.

Repository structure

Folder	What's inside	Audience
connectors/	Legacy Playwright connectors plus pinned GitHub and ChatGPT PDPP artifacts	Everyone
scripts/	PDPP packaging plus legacy Playwright test and exception tooling	Human developers
skills/	AI agent skill for running and maintaining legacy Playwright connectors (pdp-connect/)	AI agents (Claude, etc.)
schemas/	Shared meta-schemas such as manifest.schema.json	Validation
types/	TypeScript type definitions (connector.d.ts)	TypeScript consumers
fixture-index.json	Public index of schema-valid connector fixtures with main-branch raw GitHub URLs and checksums	Builders and AI agents

connectors/                        # All platform connectors
├── <company>/
│   ├── <name>-playwright.js       #   Connector script (plain JS)
│   ├── <name>-playwright.json     #   Metadata (login URL, selectors, scopes)
│   ├── schemas/
│   │   └── <platform>.<scope>.json
│   ├── fixtures/
│   │   └── <platform>.<scope>.<scenario>.json
│   └── icons/
│
scripts/                           # Developer tooling (human-driven)
├── create-connector.sh            #   Explicit legacy Playwright exception path
├── capture-session.cjs            #   Browser session capture (manual login)
├── test-connector.cjs             #   Run connector against a real browser
├── validate-connector.cjs         #   Structure + output validator
└── reference/                     #   Templates and API docs
│
skills/pdp-connect/                # Agent skill (AI-agent-driven)
├── SKILL.md                       #   Skill entry point (setup, connect, recipes)
├── CREATE.md                      #   Full walkthrough for building connectors
└── scripts/                       #   Agent-facing scripts (runner, validator, etc.)
│
schemas/                           # Shared meta-schemas
├── manifest.schema.json
│
registry.json                      # Central registry (checksums, versions, OTA)
fixture-index.json                 # Public fixture catalog for builders and agents
scope-catalog.json                 # Generated public source/scope contract
run-connector.cjs                  # Symlink → skills/pdp-connect/scripts/run-connector.cjs
test-connector.cjs                 # Standalone test runner
create-connector.sh                # Explicit legacy Playwright exception path

Distribution contract

data-connectors is an open collection of data connectors that any consumer can pin and resolve as a versioned dependency; nothing here is specific to any one consumer.

• connector-index.json is the authoritative release index.
• scope-catalog.json is the generated, schema-validated public contract for source IDs, scope IDs, payload schema paths, desktop fulfillment, material limits, and maturity. SCOPES.md is generated from the same model.
• Pin scope-catalog.json from an immutable GitHub release named connectors-<commit12>, for example https://github.com/PDP-Connect/data-connectors/releases/download/connectors-<commit12>/scope-catalog.json. Release catalogs embed the full source commit and an immutable schema.url for every payload schema. Checked-in catalogs retain schema.path for local repository use. A raw main URL is not a version pin.
• scope-catalog.json.sigstore.json is the catalog's detached Sigstore bundle.
• The same immutable release includes scope-catalog.schema.json and its detached Sigstore bundle. The catalog's catalogSchema object names both the checked-in schema path and the flat release-asset filename.
• fixture-index.json is the public fixture catalog for schema-valid synthetic sample data. Builders and agents should download fixture raw URLs instead of pasting large JSON payloads into prompts or terminals. Raw URLs point at latest main; use the included SHA-256 checksum when exact bytes matter.
• connector-index.json.sigstore.json is the detached Sigstore bundle for the authoritative signed index published at the stable connectors-latest release URL.
• registry.json is the sole publishability gate for connectors and their scopes. Only registry-selected manifests enter generated public catalogs and release indexes.
• artifacts/<connector>/<connector>-<version>.tgz is the immutable bundle format consumed by installers.
• github-pdpp is an opt-in, unpublished PDPP Collection Profile artifact. Its GitHub collector source lives under connectors/github-pdpp/. Generic PDPP runtime source does not. Explicit regeneration resolves that runtime from the exact pinned PDPP worktree and records its full hash inventory in provenance. The bundled entrypoint and provenance are generated. The raw dist/ build output is ignored, while its exact bytes are committed inside the artifact tarball and recorded by the index and provenance digests. Follow the GitHub PDPP maintenance guide. The checked-in index marks this artifact as unpublished.
• artifacts/**/*.tgz.sigstore.json is the detached Sigstore bundle for each immutable artifact.
• @opendatalabs/data-connectors-tools/installer-core exposes the supported install/update contract used by consumer repos.
• connector-installer is the supported CLI wrapper for lock generation, installation, verification, and update checks.

Legacy Playwright connectors

Each connector lives in connectors/<company>/. A connector bundle keeps its runtime assets together inside that directory. A connector usually consists of:

• <name>-playwright.js -- the connector script (plain JS, runs inside the Playwright runner sidecar)
• <name>-playwright.json -- metadata (display name, login URL, selectors, scopes)
• schemas/<platform>.<scope>.json -- connector-owned JSON Schemas for every declared scope
• fixtures/<platform>.<scope>.<scenario>.json -- optional synthetic sample payloads for local builder tests
• icons/... -- canonical local icon assets referenced by the manifest via a path relative to that connector directory

Some connectors also include a README with platform-specific setup instructions (e.g., API keys).

Builder fixtures

Fixtures are public, synthetic scope payloads for app builders and AI agents. Use them when a realistic local sample is needed but a live account export would be too large to paste into an agent prompt.

Fixture files live next to the connector:

connectors/<company>/fixtures/<scope>.<scenario>.json

Start each shipped scope with these scenarios when useful:

• empty -- valid no-data state
• small -- a compact sample for UI smoke tests
• large -- a larger sample for pagination, performance, and summarization tests

Each fixture must be valid JSON, map to a scope declared by a connector manifest in the same directory, and conform to connectors/<company>/schemas/<scope>.json. The generated fixture-index.json records the fixture id, connector id, source id, scope, scenario, record count, repo path, latest-main raw URL, schema path, byte size, and SHA-256 checksum.

npm run fixtures:validate
npm run fixture-index:generate
npm run fixture-index:check

Legacy Playwright scripts vs. skills

The repo has two interfaces for building and running connectors. They serve different audiences but share the same connector format and output:

	scripts/	skills/pdp-connect/
Audience	Human developers at a terminal	AI agents (Claude, etc.)
Entry point	scripts/create-connector.sh	skills/pdp-connect/SKILL.md
Login	Manual browser login via capture-session.cjs	Runner-driven (run-connector.cjs)
Testing	scripts/test-connector.cjs	run-connector.cjs
When to use	Existing legacy maintenance or an approved exception	Existing legacy maintenance and data export

Both produce the same legacy connector files. They use the same schemas, registry, and validation scripts. They do not create PDPP Collection Profiles.

---

How connectors work

Connectors run in a sandboxed Playwright browser managed by the bundled runner (playwright-runner/) or, in the desktop app, an equivalent driver. The runner provides a page API object (not raw Playwright). The browser starts headless; connectors call page.showBrowser() when login is needed and page.goHeadless() after.

Two-phase architecture

Phase 1 -- Login (visible browser)

1. Navigate to the platform's login page (headless)
2. Check if the user is already logged in via persistent session
3. If not, show the browser so the user can log in manually
4. Extract auth tokens/cookies once logged in

Phase 2 -- Data collection (headless)

1. Switch to headless mode (browser disappears)
2. Fetch data via API calls, network capture, or DOM scraping
3. Report structured progress to the UI
4. Return the collected data with an export summary

Canonical result shape

Connectors return a flat result object. Top-level keys are either reserved metadata or scope keys (source.category format, e.g., linkedin.profile, chatgpt.conversations). The frontend auto-detects scoped keys (any key containing a . that isn't a reserved key) and POSTs each scope separately to the Personal Server at POST /v1/data/{scope}.

const result = {
  // ── Reserved metadata (always required) ──
  requestedScopes: ["platform.scope1", "platform.scope2"],
  timestamp: new Date().toISOString(),
  version: "2.0.0-playwright",
  platform: "platform-name",
  exportSummary: {
    count: 42,
    label: "42 items exported",
    details: {/* ... */},
  },
  errors: [],

  // ── Scope keys (one per collected scope) ──
  "platform.scope1": {/* scope data */},
  "platform.scope2": {/* scope data */},
};

Reserved metadata keys

These top-level keys are reserved and must not be used as scope names:

Key	Type	Required	Description
requestedScopes	string[]	yes	The explicit, deduplicated, non-empty execution target for this run.
timestamp	string	yes	ISO 8601 timestamp of when the run completed.
version	string	yes	Connector version string.
platform	string	yes	Platform identifier.
exportSummary	object	yes	Summary with count (number), label (string), and optional details.
errors	ConnectorError[]	yes	Unresolved output-affecting errors. Empty array if none.

All other top-level keys are treated as candidate scope keys and must be members of requestedScopes.

Scope keys

Scope keys use the platform.category format (e.g., github.repositories). Rules:

• Every produced scope key must appear in requestedScopes.
• A scope not listed in requestedScopes must never be produced (protocol violation).
• Empty-but-present scope payloads ({}) are valid data, not failures.
• A requested scope may be: produced cleanly, produced in degraded form, or omitted.

requestedScopes

requestedScopes represents the resolved execution target for this run, not raw caller input.

• If the caller omits scopes, the runtime expands to the connector manifest's full canonical scope set before execution.
• requestedScopes must always be present, explicit, deduplicated, and non-empty.
• Missing, empty, or non-array requestedScopes is a protocol violation.

errors[]: honest outcome reporting

errors[] carries unresolved, output-affecting problems only. It must always be present (empty array is valid).

Each error entry:

{
  errorClass: 'auth_failed',        // from the shared taxonomy (see below)
  reason: 'Session cookie expired',  // human-readable explanation
  disposition: 'omitted',           // 'omitted' | 'degraded' | 'fatal'
  scope: 'platform.scope1',         // required for omitted/degraded; optional for fatal
  phase: 'auth',                    // optional diagnostic context
}

Dispositions:

Disposition	Meaning	scope required?	Scope key present?
omitted	The requested scope did not produce a trustworthy payload.	yes	no
degraded	The scope produced a payload, but completeness or fidelity is materially affected.	yes	yes
fatal	A run-level problem makes the entire run untrustworthy.	no	n/a

What belongs in errors[]:

• Auth failures preventing collection
• Selector drift causing omission
• Pagination failures making a produced scope materially incomplete
• Scope emitted outside requestedScopes

What does NOT belong (if fully recovered):

• A navigation attempt that failed then succeeded with no output lost
• A fetch that failed once, retry succeeded, no output lost

Error taxonomy

Connectors and runners share one error taxonomy:

auth_failed · rate_limited · upstream_error · navigation_error · network_error · selector_error · timeout · protocol_violation · runtime_error · personal_server_unavailable · unknown

Protocol violations

The following are protocol violations and must be treated as hard failures:

• Missing required metadata fields (requestedScopes, timestamp, version, platform, exportSummary, errors)
• requestedScopes missing, empty, or non-array
• errors missing or not an array
• Top-level scope key produced outside requestedScopes
• Invalid error entries (missing errorClass, reason, or disposition)
• omitted/degraded errors missing scope

Collection outcomes

The runner classifies each run into one of:

Outcome	Meaning
success	All requested scopes produced, no unresolved output-affecting errors.
partial	At least one scope produced, but at least one scope was omitted or degraded.
failure	No scopes successfully produced, or a fatal/protocol-level issue makes the run untrustworthy.
cancelled	Run was cancelled.

Data extraction patterns

Pattern	When to use	Example connector
API fetch via page.evaluate()	Platform has REST/JSON APIs	connectors/openai/chatgpt-playwright.js
Network capture via page.captureNetwork()	Platform uses GraphQL/XHR that fires on navigation	connectors/meta/instagram-playwright.js
DOM scraping via page.evaluate()	No API available, data only in rendered HTML	connectors/linkedin/linkedin-playwright.js

---

Legacy Playwright connector maintenance

Use this path to maintain an existing legacy browser connector. New connector work belongs in PDP-Connect/pdpp by default. See Connector authoring.

Creating a new legacy connector requires an explicit exception. The creation tools reject requests without --legacy-exception. See skills/pdp-connect/CREATE.md after the exception is approved.

Summary:

1. Scaffold: node skills/pdp-connect/scripts/scaffold.cjs --legacy-exception <platform> [company] -- generates script, metadata, and connector-local stub schema
2. Implement: Write login + data collection logic (see CREATE.md for auth patterns, extraction strategies, and reference connectors)
3. Validate structure: node scripts/validate-connector.cjs connectors/<company>/<name>-playwright.js
4. Test: node run-connector.cjs connectors/<company>/<name>-playwright.js --inputs '{"username":"x","password":"y"}'
5. Validate output: node scripts/validate-connector.cjs connectors/<company>/<name>-playwright.js --check-result ~/.pdp-connect/desktop/last-result.json
6. Register: node skills/pdp-connect/scripts/register.cjs connectors/<company>/<name>-playwright.js -- adds entry + checksums to registry.json

---

Page API reference

The page object is available as a global in connector scripts. The runner implementation is bundled in this repo at playwright-runner/.

Method	Description
page.evaluate(jsString)	Run JS in browser context, return result
page.screenshot()	Take a JPEG screenshot, returns base64 string
page.requestInput({message, schema?})	Request data from the driver (credentials, 2FA codes, etc.)
page.goto(url, options?)	Navigate to URL
page.sleep(ms)	Wait for milliseconds
page.setData(key, value)	Send data to host ('status', 'error', 'result')
page.setProgress({phase, message, count})	Structured progress for the UI
page.showBrowser(url?)	Escalate to headed mode; returns { headed: true/false }
page.goHeadless()	Switch to headless mode (no-op if already headless)
page.promptUser(msg, checkFn, interval)	Poll checkFn until truthy
page.captureNetwork({urlPattern, bodyPattern, key})	Register a network capture
page.getCapturedResponse(key)	Get captured response or null
page.hasCapturedResponse(key)	Check if a response was captured
page.clearNetworkCaptures()	Clear all captures
page.closeBrowser()	Close browser, keep process for HTTP work
page.httpFetch(url, options?)	Node.js fetch with auto-injected cookies from the browser session

showBrowser: headed escalation

showBrowser switches the browser to headed mode for cases that require live human interaction (e.g., interactive CAPTCHAs). It returns { headed: true } on success or { headed: false } if the driver doesn't support headed mode. Connectors should check the return value and handle the fallback:

const { headed } = await page.showBrowser(url);
if (!headed) {
  // Headed mode is not available. Retry, skip, or report an error.
}

For normal login flows, use requestInput to ask the driver for credentials without showing a browser:

const { email, password } = await page.requestInput({
  message: "Log in to ChatGPT",
  schema: {
    type: "object",
    properties: {
      email: { type: "string", format: "email" },
      password: { type: "string", format: "password" },
    },
    required: ["email", "password"],
  },
});

The runner relays the request to the driver (Tauri app, agent, CLI) and resolves with the response. The schema field uses JSON Schema, the format used by OpenAI, Anthropic, and Google for LLM tool definitions.

Progress reporting

await page.setProgress({
  phase: { step: 1, total: 3, label: "Fetching memories" },
  message: "Downloaded 50 of 200 items...",
  count: 50,
});

• phase.step / phase.total -- drives the step indicator ("Step 1 of 3")
• phase.label -- short label for the current phase
• message -- human-readable progress text
• count -- numeric count for progress tracking

---

Testing locally

Standalone test runner

Test connectors without any other app or repo. run-connector.cjs spawns the bundled playwright-runner/ (see Running a legacy Playwright connector for one-time setup) as a child process and outputs JSON protocol messages.

# Run a connector (headed by default, browser visible)
node run-connector.cjs ./connectors/linkedin/linkedin-playwright.js

# Colored, human-readable output
node run-connector.cjs ./connectors/linkedin/linkedin-playwright.js --pretty

# Pre-supply credentials
node run-connector.cjs ./connectors/linkedin/linkedin-playwright.js --inputs '{"username":"x","password":"y"}'

# Run headless (no visible browser)
node run-connector.cjs ./connectors/linkedin/linkedin-playwright.js --headless

# Override the initial URL
node run-connector.cjs ./connectors/linkedin/linkedin-playwright.js --url https://linkedin.com/feed

# Save result to a custom path (default: ./connector-result.json)
node run-connector.cjs ./connectors/linkedin/linkedin-playwright.js --output ./my-result.json

The runner reads the connector's sibling .json metadata to resolve the connectURL. In headed mode, goHeadless() becomes a no-op so the browser stays visible throughout.

---

Contributing

Adding a legacy Playwright connector by exception

1. Fork this repo
2. Create a branch: git checkout -b feat/<platform>-connector
3. Add your files in connectors/<company>/:
   • connectors/<company>/<name>-playwright.js -- connector script
   • connectors/<company>/<name>-playwright.json -- metadata
   • connectors/<company>/schemas/<platform>.<scope>.json -- data schema (required for shipped connectors)
4. Test locally using the instructions above
5. Update registry.json with your connector entry and checksums
6. Open a pull request

Modifying an existing legacy Playwright connector

1. Fork and branch
2. Make your changes to the connector script and/or metadata
3. Test locally
4. Update the version in the metadata JSON
5. Regenerate checksums and update registry.json
6. Open a pull request

Guidelines

• Credentials stay on-device. Never send tokens or passwords to external servers.
• Use page.setProgress() to report progress during long exports.
• Include exportSummary in the result. The UI uses it to display what was collected.
• Handle errors. Use page.setData('error', message) with clear error messages.
• Prefer API fetch over DOM scraping. APIs are more stable than DOM structure.
• Avoid obfuscated CSS class names. Use structural selectors, heading text, and content heuristics.
• Rate-limit API calls. Add page.sleep() between requests.
• Test pagination edge cases -- empty results, single page, large datasets.

Registry checksums

The registry uses SHA-256 checksums to verify file integrity during OTA updates. Always regenerate checksums when modifying connector files:

shasum -a 256 connectors/<company>/<name>-playwright.js | awk '{print "sha256:" $1}'
shasum -a 256 connectors/<company>/<name>-playwright.json | awk '{print "sha256:" $1}'

---

How consumers get connectors

Any project can consume connectors from this repo as pinned dependencies. A consumer declares version constraints in a connector-dependencies.json file and resolves them with a connectors:resolve script (naming and invocation are up to the consumer's own tooling), for example:

# Example: a consumer's package.json script
npm run connectors:resolve

This fetches matching versions from the signed connectors-latest connector-index.json, verifies the detached Sigstore bundles plus the artifact/manifest/script checksums, and writes the scripts + manifests to a local snapshot directory. It's analogous to npm install.