Send custom HTTP request headers when downloading a source

[rmx90210]

Closes #102. Implements openaddresses/openaddresses#1745 and openaddresses/openaddresses#8306. Unblocks openaddresses/openaddresses#4550 (Champaign, IL) and likely openaddresses/openaddresses#6226 (Schuylkill County, PA).

What

Reads a source's request.headers (added to the schema in openaddresses/openaddresses#8306) and sends those headers on every download request for that source — e.g. so a source that 403s without a Referer header can finally be fetched.

{
  "name": "champaign",
  "protocol": "http",
  "data": "https://example.gov/addresses.geojson",
  "request": {
    "headers": { "Referer": "https://example.gov/gis/" }
  }
}

Why this was more than a schema-wiring task

DownloadTask.__init__ already accepted a headers dict and merged it over the default User-Agent, and URLDownloadTask.download already sent self.headers on its request — that plumbing predates this PR. But nothing actually reached it:

• DownloadTask.from_protocol_string, the only production constructor path, took no headers argument.
• cache() never read a headers/request key off the source config.
• EsriRestDownloadTask built its EsriDumper without headers, even though pyesridump already supports extra_headers.
• guess_url_file_extension makes its own separate, un-headered GET to sniff the file's Content-Type before the real download runs. Even with everything else fixed, a source gated on Referer would still fail at this pre-flight step — this was the part most easily missed, since it isn't in the obvious download path.

All four are fixed here.

Changes

• openaddr/cache.py
  • guess_url_file_extension() now accepts and sends headers on its pre-flight sniffing request; URLDownloadTask.get_file_path() passes self.headers through.
  • DownloadTask.from_protocol_string() accepts headers and forwards it to both URLDownloadTask and EsriRestDownloadTask.
  • EsriRestDownloadTask.download() passes extra_headers=self.headers to EsriDumper.
• openaddr/__init__.py
  • cache() reads source_config.data_source['request']['headers'] and passes it down.
  • conform() deliberately does not forward headers — it re-downloads from the OA-owned cache artifact (S3), not the contributor's original host, so contributor-supplied headers don't apply there. Left a comment explaining this so it doesn't read as an oversight.
• openaddr/tests/cache.py — new coverage:
  • headers reach task.headers via from_protocol_string() for both http and ESRI
  • the default User-Agent survives alongside custom headers, and can be overridden
  • headers are sent on both the extension-guessing pre-flight request and the real download request
  • EsriRestDownloadTask passes headers to EsriDumper as extra_headers
  • cache() correctly reads request.headers from the source config end-to-end, and a source with no request block at all still works
• openaddr/VERSION / CHANGELOG — bumped to 10.1.0

Naming

Headers are nested under a request object (request.headers) rather than a flat headers key, per discussion in openaddresses/openaddresses#8306 — this leaves room for a future sibling setting (e.g. a query-param token) without a second schema migration. The key went through a couple of names during review (headers → http_request_settings → request); the commit history reflects that back-and-forth in case it's useful context, but the net result is request.headers, matching the schema.

⚠️ Deployment order

The schema side (openaddresses/openaddresses#8306, headers-field-support branch) is already updated to match this shape. This PR should merge and be released before that schema PR merges — otherwise a source could pass schema validation while this repo is still on the old version and silently ignores the field.

Testing

Local Python lacks gdal/shapely; verified against the project's batch-machine:latest Docker image:

docker run --rm -v "$(pwd)":/app -w /app batch-machine:latest python -m unittest openaddr.tests.cache openaddr.tests -v

57 tests, all passing. Also manually confirmed the new cache() test fails correctly when the config key is misspelled, to make sure it actually exercises the wiring rather than passing vacuously.

[rmx90210]

Blocked by #104