fix(copilot): stop losing product images to dead browsers and bot walls

[devin-ai-integration]

Summary

From the production logs: extract_product_images on a DigiKey URL returned 0 images and Image extraction failed, with browser.newPage: Target page, context or browser has been closed. Two independent causes, both of which surfaced as the same useless outcome.

Stale browser handle. Chromium runs --single-process on a 512MB dyno; when it is OOM-killed, browserPromise still resolves to a corpse (the disconnected handler clears it, but not before the next job asks for a page). The next job then dies for the previous job's crash. withBrowserPage now checks liveness and relaunches once:

const browser = await getBrowser();
if (!browser.isConnected()) { await closeBrowser(); return (await getBrowser()).newPage(...); }
try { return await browser.newPage(...); }
catch (err) { if (!isDeadBrowser(err)) throw err; await closeBrowser(); return (await getBrowser()).newPage(...); }

The page was a Cloudflare interstitial. DigiKey served a Turnstile challenge, so there was nothing to harvest, and harvesting it anyway would have handed the copilot a shield logo as a component photo. New server/ai/product-images.ts recognises those pages (looksLikeBotChallenge) and parses og:image/twitter:image/JSON-LD straight out of HTML — no browser — which is where distributors put the product photo anyway. So:

extractProductImages(url) -> { images, via: "browser" | "html", problem?: "blocked" | "browser-unavailable" }

browser first (it resolves lazy gallery images), HTTP fetch as the fallback whenever the browser returns nothing or cannot run. A missing Playwright binary still throws — that is a deployment fact for an operator, not a page to work around. The tool and the copilot's tool line now say why it came back empty ("distributor blocked the reader") so the model picks a different distributor instead of retrying the same wall.

Testing

test/product-images.test.ts (6) covers challenge detection, HTML harvesting and ranking; test/render-browser.test.ts grows 3 cases for relaunch-after-death, the HTML fallback, and a blocked page. Full suite 475 passing; prettier, eslint and tsc --noEmit clean. Not verified against the live DigiKey page — the fix is inferred from the attached production log.

Link to Devin session: https://app.devin.ai/sessions/27a2860693d444f8a9a37442a83762d7
Requested by: @alanshen27

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring