Chrome Crawler

This service uses puppeteer(to be deprecated), playwright and axios to crawl pages.

It allows to scroll, take screenshots and get images encoded as Base64.

Out of scope: it allows to get apps from playstore, in the future it will be migrated to a new service.

How to run?

Dev environment

npm start

or

make run

Docker

A Dockerfile is provided which install chrome inside the container.

Docker Compose

A docker-compose.yml is provided for local use. It builds the image and keeps the crawled cookies in a named volume. Locally it runs with JWT_ALG=HS256 and a shared secret, so there is no keypair to generate:

make compose-up                # or: docker compose up --build
make token                     # a token the running container accepts
make compose-down

make token defaults to the same JWT_ALG=HS256 / JWT_SECRET=dev-secret-change-me the compose file uses, and both read .env, so overriding the secret in one place keeps them in sync. For the ES512/production path:

./scripts/generate_keys.sh                                    # creates .secrets/{private,public}.key
JWT_ALG=ES512 JWT_SECRET=.secrets/public.key make token       # signs with JWT_PRIVATE_KEY

The service listens on http://localhost:3000 (loopback only) with the Swagger UI at /docs. Any of WEB_PORT, WEB_TIMEOUT, JWT_ALG, JWT_SECRET and CRAWLER_DOCS can be overridden from a .env file next to the compose file, or inline — use this if something else already holds port 3000:

WEB_PORT=3010 docker compose up --build

The container runs with init: true and shm_size: 1gb; Chromium leaves zombie processes behind and crashes on Docker's default 64MB of shared memory.

Environment variables

WEB_PORT = 3000
WEB_TIMEOUT = 150 # segs
JWT_SECRET = my-secret-hash
JWT_ALG = HS256
PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH=/usr/bin/chromium-browser
HEADLESS = true 
CRAWLER_DOCS = false # set to "true" to serve the Swagger UI at /docs
LOG_HEALTHCHECK = false # set to "true" to also log the container healthcheck and /metrics scrapes
V7_EXECUTOR = false # set to "true" to mount /v7/executor - it runs caller-supplied code, read that section first
V7_EXECUTOR_CONCURRENCY = 2 # scripts at once; each one owns a browser
V7_EXECUTOR_MEM_MB = 512 # heap cap for the executor child process
V7_EXECUTOR_MAX_SCRIPT_BYTES = 65536
V7_EXECUTOR_MAX_RESULT_BYTES = 1048576
REDIS = redis://127.0.0.1:6379 # with localhost it tries ipv6, redis is disabled
COOKIES_STORE = fs # fs | redis - where the cookie jars live
COOKIES_PATH = data/cookies # fs store only
COOKIES_REDIS_URL = redis://127.0.0.1:6379 # redis store only, falls back to REDIS
COOKIES_PASSWORD = # optional, read from REDIS_PASSWORD
COOKIES_TTL = 7200 # redis store only, seconds a jar survives

Cookie storage

useCookies keeps the browser context cookies between calls, keyed by the hostname of the url and, in v7, by sessionId. Where those jars are kept is an operator decision:

COOKIES_STORE	Where	Notes
fs (default)	COOKIES_PATH/<domain>, COOKIES_PATH/sessions/<sessionId>/<domain>	Per instance. Needs a volume to survive a restart
redis	cookies:<sessionId or "shared">:<domain>	Shared by every instance, expires by COOKIES_TTL

redis is what to pick when the service runs several replicas or on an ephemeral filesystem: a jar written by one instance is readable by the next, and jars expire on their own instead of piling up as dead files.

⚠️ Sharing jars across instances is what version 5 did and what version 6 deliberately moved away from (see the 2023-08-10 changelog entry): if the replicas sit behind different egress IPs, replaying one session's cookies from several addresses is exactly what anti-bot systems look for. Use redis when the replicas share an egress IP, or when you namespace per sessionId and pin a session to a proxy. Otherwise stay on fs.

A jar the store cannot read or write is logged and the crawl continues without cookies; a cookie backend being down does not fail a request. A misconfigured COOKIES_STORE, on the other hand, fails at startup rather than silently falling back to the disk.

Adding a third backend is three methods — get(namespace, domain), set(...), del(...) — in src/cookies.js; postgres would be an upsert on (session_id, domain). If JWT_ALG is "ES512", then JWT_SECRET must contain the absolute or relative path to the public key:

JWT_ALG = "ES512"
JWT_SECRET = ".secrets/public.key"

PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH is set by default and is related to where Alpine install the chromiun browser. This variable doesn't belong to playwright, and it is used becose playwright uses their own browser binaries, which are not compatible for alpine.

⚠️ In the future a Dockerfile.debian version could be provided. It requires a change in how the dockerfile is structured.

API docs

Setting CRAWLER_DOCS=true mounts a Swagger UI at /docs, with the OpenAPI 3 document at /docs/openapi.json. Request bodies in the spec are generated from the same Joi schemas the handlers validate with, so they cannot drift from the implementation.

CRAWLER_DOCS=true npm start

The docs and the spec are served without authentication (Swagger UI cannot send a bearer token for its own page load), which is why they are opt-in and off by default: don't enable them on an internet-exposed instance. The endpoints themselves are still protected — use the Authorize button with a token from make token to try them from the UI.

Swagger UI assets are served from the bundled swagger-ui-dist package, so /docs works in the container without outbound internet access.

Versioning

The project follows a calendar version release as YY.MM.DD_PATCH[-MODIFIER], where:

• YY short year: 23, 24..
• MM short month: 01, 02...
• DD Short day: 01, 02, 03
• PATCH: bug fix or changes 1, 2, 3
• MODIFIER: lts, rc0, rc2...

As example, the version 23.08.10_2 means from the year 2023 of month August from day 10th, patch 0. MODIFIER field is optional.

API

Entities

BrowserConfType

const browserConfType = 
  {
    headless: Joi.boolean().default(headless),
    emulation: Joi.object().keys(emulationType).default(emulationDefault),
    proxy: Joi.object().keys(proxyType).allow(null).default(null),
  }

EmulationType

const emulationType =
  {
    locale: Joi.string().default("en-US"),
    timezoneId: Joi.string().default("America/New_York"),
    isMobile: Joi.boolean().default(false),
    viewport: Joi.object().keys(viewPortType),
    geoEnabled: Joi.boolean().default(false),
    geolocation: Joi.object().keys(geoType).optional().allow(null),
  }

emulationDefault:

const emulationDefault = {
  locale: "en-US",
  timezoneId: "America/New_York",
  isMobile: false,
  viewport: { width: 1280, height: 720},
  geoEnabled: false,
  geolocation: {
    longitude: 40.6976312,
    latitude: -74.1444858
  }
}

ProxyType

const proxyType =
  {
    server: Joi.string().required(),
    username: Joi.string(),
    password: Joi.string()
  }

CrawlPageType

const crawlPageType = Joi.object(
  {
    // Valid formed url to open
    url: Joi.string().required(),
    // Timeout in secs
    ts: Joi.number().default(defaultTs),
    //  Visible text of an element to wait
    waitElement: Joi.string().optional().allow(null),
    // Take a screenshot of the fullpage
    screenshot: Joi.bool().default(false),
    // Save cookies for the domain of the url
    useCookies: Joi.bool().default(false),
    // If ture, the browser will have a fresh start
    cleanCookies: Joi.bool().default(false),
    // Derecated
    cookieId:  Joi.string().allow(null).default(null),
    // Headers used only for axios
    headers: Joi.any().allow(null),
    // Browser configuration see BrowserConfType
    browser: Joi.object().keys(browserConfType).optional().allow(null).default(defaultBrowserConf),
  }
)

General endpoints

• GET /

  • 200 Root status

• GET /metrics

  • 200 prometheus stats

Axios enpoints

• GET /v6/image

  • Uses axios to get an image encoded as base64
  • Query Params: url

• POST /v6/axios

  • 200 if everything ok, 500 if something went wrong
  • body: CrawlPageType
  • notes: ProxyType not used in axios

Chrome Endpoint

• POST /v6/chrome
  • 200 if everything ok, 500 if something went wrong
  • body: CrawlPageType
  • response 200:
    • fullurl [string]: Raw html of the response
    • content [string]: Raw html of the response
    • headers [object]: not used
    • status [number]: status code, 200 or 500
    • fullLoaded [bool]: if the page was loaded completly
    • screenshot [string]: Base64 encoded image
    • error [string]: any message error
    • cookieId [string]: generated
  • response 500:
    • error [string]: message error

Chrome v7 Endpoint

Opens a url in a real Chromium browser, waits for the page to load and returns what it rendered. Javascript runs, so this works on pages that come back empty from a plain HTTP fetch. It is written for a caller that never looks at the page — an AI agent, a pipeline — and has to decide what to do next from the response fields alone.

Shaping the request

• format decides what content is. html is the raw DOM: complete, and mostly navigation, scripts and inline styles. markdown keeps headings, lists and links; text is prose only. Both are a small fraction of the html — ask for html only to parse it.
• maxChars bounds content. truncated says it was cut, contentLength says what the full length was, so you know whether to come back with a larger bound.
• includeLinks returns {url, text} pairs, absolute and deduplicated, so you can pick the next page without parsing html. maxLinks caps the list.
• waitElement is visible text that must appear before the page is read — for pages that fill themselves in after loading. waitElementFound reports whether it showed up.
• ts is the timeout in seconds (30 by default). Keep it under whatever timeout your own caller is holding.
• profile sets locale, timezone, viewport and geolocation together as one coherent identity. geoEnabled: true grants the geolocation permission using its coordinates.
• useCookies persists cookies between calls, which is what lets a login or a consent banner survive. Always pair it with a sessionId: without one the jar is shared with every other caller on the instance. cleanCookies starts that jar fresh. The jars live on disk by default, or in redis with COOKIES_STORE=redis (see Cookie storage).
• screenshot returns a full-page base64 PNG. It is large, and it turns image blocking off for that request so the picture is not blank where the images should be.
• proxy routes the crawl through a proxy; includeHeaders returns every response header instead of the handful worth acting on.

The request surface is flat by design: no nested browser object, and passing one is a 400. Nested optionals are where a programmatic caller hallucinates fields and burns attention tuning things that do not matter. headless stays with the operator (the HEADLESS env var), which is where debugging belongs.

Reading the response

1. Check ok first. true means the fetch succeeded and the content is worth using. Whenever it is false, error is set — the two never disagree, so either is safe to branch on.
2. outcome says what the network did, and nothing else. status is the upstream HTTP status and finalUrl is the url after redirects — cite that one, not the url you sent.
3. warnings carries judgements about the bytes: ANTI_BOT_CHALLENGE, EMPTY_CONTENT. They are heuristics and can be wrong, which is why they are kept out of outcome. Do not quote or summarise content that arrived with a blocking warning: a challenge page reads like a real page.
4. title, description, lang, contentType and fetchedAt describe the page. fetchedAt is when the content was true, which matters if you cite or cache it.

When it fails

error is {code, message, retryable, retryAfterMs}:

• retryable: false — the same request fails the same way. Change something or give up.
• retryable: true with retryAfterMs — the server asked for that wait. Take it.
• retryable: true with retryAfterMs: null — back off on your own. TIMEOUT often clears with a larger ts, EMPTY_CONTENT with a waitElement.
• BLOCKED, or any ANTI_BOT_CHALLENGE warning — an identical request is blocked identically. Come back with a different sessionId or proxy, not sooner.

CrawlPageV7Type

const crawlPageV7Type = Joi.object({
  // absolute http(s) url to open
  url: Joi.string().uri({ scheme: ["http", "https"] }).required(),
  // shape of `content`: raw DOM html, flattened text, or markdown
  format: Joi.string().valid("html", "text", "markdown").default("html"),
  // truncate `content` to this many characters, 0 means no limit
  maxChars: Joi.number().integer().min(0).default(0),
  // return the page links, resolved against the final url
  includeLinks: Joi.boolean().default(false),
  maxLinks: Joi.number().integer().min(1).max(1000).default(100),
  // timeout in secs, V7_TIMEOUT (30) by default
  ts: Joi.number().min(1).max(300).default(defaultTs),
  // visible text to wait for, reported back in `waitElementFound`
  waitElement: Joi.string().allow(null).default(null),
  screenshot: Joi.boolean().default(false),
  // abort image requests. null = true, or false when screenshot is set
  blockImages: Joi.boolean().allow(null).default(null),
  useCookies: Joi.boolean().default(false),
  cleanCookies: Joi.boolean().default(false),
  // cookie jar namespace, see "Cookie storage" for where the jar lives
  sessionId: Joi.string().pattern(/^[A-Za-z0-9_-]{1,64}$/).allow(null).default(null),
  // desktop-us | desktop-es | desktop-ar | mobile-us | mobile-es | mobile-ar
  // sets locale, timezone, viewport and geolocation coherently
  profile: Joi.string().valid(...).default("desktop-us"),
  // proxy to route the crawl through
  proxy: Joi.object().keys(proxyType).allow(null).default(null),
  // grant the geolocation permission, using the profile's coordinates
  geoEnabled: Joi.boolean().default(false),
  // return every response header instead of only the ones worth acting on
  includeHeaders: Joi.boolean().default(false),
})

• POST /v7/chrome
  • The HTTP status says whether the API worked, not whether the page did:
    • 200: a crawl ran, check outcome
    • 400: the body did not validate
    • 502: the crawl could not run at all (browser launch, bad proxy, ...)
  • body: CrawlPageV7Type
  • every response, success or failure, carries the same keys:
    • url [string]: the url that was requested
    • finalUrl [string]: the url after redirects
    • outcome [string]: what the fetch did. ok | http_error | timeout | nav_error | invalid_request
    • ok [bool]: outcome is ok and no blocking warning was raised. False if and only if error is set
    • warnings [array]: {code, detail, blocking} heuristics about the content. ANTI_BOT_CHALLENGE, EMPTY_CONTENT. blocking marks the ones that make ok false
    • status [number|null]: upstream HTTP status of the navigation
    • navigated [bool]: the navigation itself completed
    • waitElementFound [bool|null]: null when no waitElement was asked for
    • title, description, lang, contentType [string|null]: page metadata
    • format [string]: the format content is in
    • content [string|null]: the page in the requested format
    • truncated [bool] / contentLength [number]: whether content was cut, and its length before cutting
    • textLength [number]: length of the extracted text regardless of format
    • links [array|null]: {url, text}, absolute and deduplicated. Only with includeLinks
    • screenshot [string|null]: base64 PNG
    • headers [object]: the response headers worth acting on (content-type, retry-after, location, last-modified, etag, content-length, content-language). The full set only with includeHeaders: true
    • fetchedAt [string]: when the crawl started, ISO 8601. What a caller citing or caching the content needs
    • elapsedMs [number]
    • error [object|null]: {code, message, retryable, retryAfterMs}, null exactly when ok. retryAfterMs comes from the upstream Retry-After header and is null when the server did not say — no backoff is invented. Codes: INVALID_REQUEST, HTTP_ERROR, BLOCKED, EMPTY_CONTENT, TIMEOUT, DNS, CONNECTION_REFUSED, CONNECTION_RESET, TLS, PROXY_ERROR, TOO_MANY_REDIRECTS, NAV_FAILED, CONTENT_READ, BROWSER_LAUNCH, UNKNOWN

ANTI_BOT_CHALLENGE is a heuristic: known challenge markers in the page, plus weaker ones (a captcha widget, "access denied") when the page is tiny or the status is 401/403/429/503. It can be wrong in both directions, which is exactly why it is a warning and not the outcome — a false positive costs the caller a warning it can ignore, never the real status of the response. But an honest guess beats silently handing a challenge page back as if it were the article that was asked for.

curl -X POST localhost:3000/v7/chrome \
  -H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{"url": "https://example.com", "format": "markdown", "includeLinks": true, "maxChars": 4000}'

Executor v7 Endpoint

• POST /v7/executor

The escape hatch for flows /v7/chrome and /v7/search cannot express: multi-step forms, infinite scroll, a login followed by an internal API call, scraping shaped by what the previous page said. You send the body of an async function; it runs against a real Chromium page and whatever it returns comes back as result.

In scope inside the script, and nothing else:

Name	What it is
page, context	The real Playwright Page and BrowserContext. Already open on url if you gave one
args	The object you sent, so the script can stay constant across calls
http (alias axios)	axios with a timeout, a size cap and validateStatus off, so a 404 is data. Private and link-local addresses are refused unless allowPrivateNetwork
cheerio, extract	Parsing, and the same helpers /v7/chrome uses (htmlToText, htmlToMarkdown, extractLinks, detectBlocked, ...)
require(name)	Allowlisted only: axios, cheerio, playwright, google-play-scraper, crypto, url, querystring, util, zlib
console.*	Captured into logs and returned. You cannot see stdout, so this is how you debug
step(name)	A named checkpoint with a timestamp. On a timeout, steps says how far the script got
sleep, Buffer, URL, URLSearchParams, setTimeout

eval and new Function throw. There is no fs, no process, no child_process.

curl -X POST localhost:3000/v7/executor \
  -H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{
        "url": "https://example.com",
        "args": {"min": 15},
        "ts": 60,
        "script": "step(\"parse\"); const $ = cheerio.load(await page.content()); const items = []; $(\".item\").each((_, el) => items.push({name: $(el).text(), price: Number($(el).attr(\"data-price\"))})); console.log(\"found\", items.length); return items.filter(i => i.price >= args.min);"
      }'

Reading the answer works like the rest of v7 — check ok first, error is set exactly when ok is false — with two differences. outcome describes the script (ok, script_error, timeout, nav_error), not the fetch, so an upstream 404 is a non-blocking HTTP_STATUS warning your script is free to be expecting. And logs and steps come back on every response, because they are your only window into a run you cannot watch. A SCRIPT_ERROR carries a stack trimmed to your own frames, with line numbers matching the script you sent.

url is a convenience: it is opened before the script starts, and if it fails to load the script never runs (nav_error). Leave it null and call page.goto yourself when you want to handle navigation failures in your own code. ts is one deadline for the whole request — whatever the pre-navigation spends comes out of the script's budget.

Before you enable it

This endpoint is remote code execution, and it is off unless V7_EXECUTOR says otherwise.

vm is not a security boundary: it isolates the global object, not the realm, and every object handed to a script leaks a path back through .constructor.constructor. So nothing here pretends otherwise — the script gets the real page, not a facade that would cost you half of Playwright and buy nothing. What contains a run is the process: every request is a forked child with an allowlisted environment, a heap cap, its own Chromium, and a parent holding SIGKILL — the only thing that can stop while (true) {}.

The known gap, written down rather than hidden: that child is a fork, so it shares the server's uid, and an escaped script can read /proc/<ppid>/environ to recover the parent's real environment. Closing it means running the executor as a separate container with no secrets mounted, which is the next phase. Until then:

• do not enable it where JWT_ALG=HS256 puts a forgeable signing secret in the environment; with the production ES512 setup the environment holds a public key, which is worth nothing
• treat an executor token as a shell on the crawler, and issue it accordingly
• keep the cookie jars in mind: a script can read every jar the process can

An in-page browser mode — sandboxed by Chromium, no node access, enough for most "the DOM here is weird" cases — is the other half of that phase. mode already rejects anything but node.

Duckduckgo Endpoint

CrawlDuckGoType

const crawlDuckGoType = Joi.object(
  {
    // a query to search in duckduckgo.com
    text: Joi.string().required(),
    // timeout in secs
    ts: Joi.number().default(defaultTs),
    // How many clicks on "More Results"
    moreResults: Joi.number().default(1),
    // "ar-es" by default. 
    region: Joi.string().default("ar-es"),
    // "Any Time", "Past day", "Past week", "Past month", "Past year". Null by default
    timeFilter: Joi.string().default(null).allow(null),
    // Take a screenshot of full rendered page
    screenshot: Joi.bool().default(false),
    // It will store and load cookies
    useCookies: Joi.bool().default(true),
    // Deprecated
    cookieId:  Joi.string().allow(null).default(null),
    browser: Joi.object().keys(browserConfType).optional().allow(null).default(defaultBrowserConf),
  }
)

For regios codes, check see regions codes

• POST /v6/duckduckgo
  • 200 if everything ok, 500 if something went wrong
  • body crawlDuckGoType
  • response 200:
    • query [string]: Parsed query
    • fullurl [string]: Fullurl
    • content [string]: Raw html of the response
    • headers [object]: Empty
    • status [number]: status code, 200 or 500
    • links [List[{href:text}]]: uri of the next page
    • fullLoaded [bool]: if the page was loaded completly
    • screenshot [string]: Base64 encoded image
    • error [string]: any message error
    • cookieId [string]: generated
  • response 500:
    • error [string]: message error

Google Endpoint

crawlGoogleType

const crawlGoogleType = Joi.object(
  {
    // a query to search in google.com
    text: Joi.string().required(),
    // timeout in secs
    ts: Joi.number().default(defaultTs),
    // It will performs a "PgDown" actions for `moreResults` times. 
    moreResults: Joi.number().default(1),
    // region: Joi.string().default("countryAR"),
    region: Joi.string().default("Argentina"),
    // country 
    cr: Joi.string().default("US"),
    // interfaz lang
    hl: Joi.string().default("en"),
    // "Any Time", "Past hour", "Past 24 hours", "Past week", "Past month", "Past year". Null by default
    timeFilter: Joi.string().default(null).allow(null),
    // Take and screenshot
    screenshot: Joi.bool().default(false),
    // use cookies
    useCookies: Joi.bool().default(true),
    // deprecated
    cookieId:  Joi.string().allow(null).default(null),
    browser: Joi.object().keys(browserConfType).optional().allow(null).default(defaultBrowserConf),
  }
)

• POST /v6/google
  • 200 if everything ok, 500 if something went wrong
  • body crawlGoogleType
  • response 200:
    • query [string]: Parsed query
    • fullurl [string]: Fullurl
    • content [string]: Raw html of the response
    • headers [object]: Empty
    • status [number]: status code, 200 or 500
    • links [List[{href:text}]]: uri of the next page
    • fullLoaded [bool]: if the page was loaded completly
    • screenshot [string]: Base64 encoded image
    • error [string]: any message error
    • cookieId [string]: generated
  • response 500:
    • error [string]: message error

Playstore endpoints

⚠️ Playstore API could be very inestable, for more information refer to https://github.com/facundoolano/google-play-scraper

• GET /v1/playstore/:appid

  • Get app detail based on the appid

• POST /v1/playstore/list

  • Get a list of apps
  • body
    • collection: Available options can bee found here
    • category: Available options can bee found here.
    • num: (optional, defaults to 500): the amount of apps to retrieve.
    • country: (optional, defaults to 'us'): the two letter country code used to retrieve the applications.

• POST /v1/playstore/search

  • Perform a search in google playstore
  • body
    • term: the term to search by.
    • num (optional, defaults to 20, max is 250): the amount of apps to retrieve.
    • lang (optional, defaults to 'en'): the two letter language code used to retrieve the applications.
    • country (optional, defaults to 'us'): the two letter country code used to retrieve the applications.
    • fullDetail (optional, defaults to false): if true, an extra request will be made for every resulting app to fetch its full detail.
    • price (optional, defaults to all): allows to control if the results apps are free, paid or both.
      • all: Free and paid
      • free: Free apps only
      • paid: Paid apps only

• POST /v1/playstore/similar

  • Returns a list of similar apps to the one specified
  • body:
    • appId: the Google Play id of the application to get similar apps for.
    • lang (optional, defaults to 'en'): the two letter language code used to retrieve the applications.
    • country (optional, defaults to 'us'): the two letter country code used to retrieve the applications.
    • fullDetail (optional, defaults to false): if true, an extra request will be made for every resulting app to fetch its full detail.

Example:

curl http://localhost:3000/v1/chrome?url=https://www.google.com/doodles/

url must have the protocol screen is a optional param, any value is taked as true

Regions codes

duckduckgo

Check https://duckduckgo.com/settings

Copy the value of the option:

• For "All regions" the value is wt-wt
• For "Argentina" the value is ar-es

google: Check https://www.google.com/preferences Copy as the text shown in Region settings part:

• For "Brazil", the value is Brazil
• For "Agentina", the value is Argentina

Changelog

2026-08-27

• New /v7/executor endpoint: caller-supplied JavaScript run against a real browser, with page, context, a guarded http, cheerio, extract and an allowlisted require in scope. outcome describes the script rather than the fetch, logs and steps come back with every response, and a script error carries a stack trimmed to the caller's own frames. Each run happens in a forked child with an allowlisted environment, a heap cap and a parent holding SIGKILL, so a runaway script dies with a process. Off unless V7_EXECUTOR is set; read "Before you enable it" above, including the deferred same-uid gap.

• New /v7/chrome endpoint, shaped for programmatic/agent callers. v6 is unchanged: v7 reports the real upstream status, distinguishes ok/http_error/blocked/timeout/ nav_error/empty in an outcome field, resolves finalUrl, can return text or markdown bounded by maxChars, returns typed {code, message, retryable} errors, namespaces cookies per sessionId and replaces the nested emulation object with a profile preset. Heuristics about the content are kept in warnings so they cannot overwrite the reported status, ok is false exactly when error is set, retryAfterMs comes from the upstream header, response headers are trimmed to the ones worth acting on and fetchedAt says when the content was true. Its default timeout is 30s instead of 180s.

2023-08-10

• Restarting changelog from now.

• Main endpoints (chrome, axios, image, google and duckduck) will follow simple version schema: Breaking changes in the response or the payload for req/resp will imply a new version in the enpoint.

• calendar version release adopted for the project. More detail in versioning, but the new format is: YY.MM.DD_PATCH[-MODIFIER]. As example, this version will be: 23.08.10_0

• Cookies are now per instance, stored locally. In version 5, cookies were shared cross instances using redis. Because instances could be hosted in different servers with different IPs and having requests with the same cookies from different IPS is not recommended. Instead, each instance store locally their cookies, which is aligned with how actually a browser works.

• Duckduckgo parsing of links disabled by now.y

• Google try first to get the search button using "Search" if it fails, it will try "Buscar"

• A HEADLESS env added. If false it will open the browser. Useful for debugging.

Before

⚠️ From V4 endpoint, it uses Playwright instead of puppeteer

Resources

• Catch timeout error
• Catch error
• BrowserContext
• Using Puppeteer for scrap
• Docker image for puppeteer