add uspstf scraper - #26
Conversation
Adds a US Preventive Services Task Force scraper, a source named in the AMFV project doc and not yet covered. Discovery works around the listing view: it reports a "Hits: N" total but renders only 20 rows, with no working pager or page-size parameter. The same view filtered by one of its twelve topic categories returns that category's full set, so all twelve are walked up front and deduplicated by slug, yielding 90 published recommendations. Two things the site does that the scraper has to correct for: - `topic_status=P` stops being honoured once a category filter is applied, so rows marked Inactive or Referred come back too, some dating to 1996. Those are dropped. Retired guidance in a verification corpus is worse than a smaller corpus: it produces confident, wrong verdicts, which is the failure the project doc warns about for stale indexes. - A category can contribute nothing new, and the shared listing loop reads an empty page as the end of the source. Walking categories lazily one per page would therefore silently drop every later category, so they are walked eagerly instead. A test pins this. Content is the recommendation's own outline: the population/recommendation/ grade summary table, then each content panel, skipping the preamble and mission statement (identical across all 90) and the administrative membership and copyright panels. Licensing is not public domain despite being federal work. AHRQ's notice permits redistribution only "without any changes", bars redistribution for a fee, and asks to be cited. Documents carry license, license_url, and attribution so this travels downstream; the no-changes term is worth attention before the corpus feeds decomposition or training.
| return clean_text(headings[0].text_content()) if headings else "" | ||
|
|
||
|
|
||
| def _recommendation_content(doc: lxml_html.HtmlElement, *, link_mode: LinkMode) -> tuple[str, int]: |
There was a problem hiding this comment.
The _recommendation_content helper in uspstf.py, the content collector, gives html_to_markdown the site root at both conversion calls. That changes the meaning of a link written relative to a recommendation page.
An accepted fix passes the actual recommendation URL into the content helper from both readers and uses it for both table and panel conversion. Add the relative-link fixture. Next we examine which input hostnames are accepted.
TLDR A reference can point to the wrong place because it is resolved from the site root.
| """ | ||
| parsed = urlparse(url.strip()) | ||
| host = parsed.netloc.lower() | ||
| if parsed.scheme not in {"http", "https"} or not host.endswith("uspreventiveservicestaskforce.org"): |
There was a problem hiding this comment.
We move back to the top of the module: the recommendation_slug_from_url helper in uspstf.py, the source collector, validates addresses using a hostname suffix. The offline probe shows that notuspreventiveservicestaskforce.org is accepted because it ends with the expected text.
This is an input-validation issue, not an off-site fetch: recommendation_url reconstructs a canonical address before the reader requests it. An optional improvement accepts the exact expected host or a subdomain with a dot boundary. The closing step gathers the decision.
TLDR A mistyped lookalike domain is accepted, but the actual request still goes to the configured site.
…ost check Two fixes from @zndr27's review. _recommendation_content resolved relative links against the site root at both conversion calls, so a reference written as `grade-definitions` on a recommendation page came out as /grade-definitions instead of /uspstf/recommendation/grade-definitions. Both readers now pass the page URL, and both the summary table and the panel bodies use it. A recommendation URL carries no trailing slash, so a relative reference resolves against /uspstf/recommendation/, which is what a browser on that page does. recommendation_slug_from_url validated the host with a bare suffix match, so notuspreventiveservicestaskforce.org was accepted. It now takes the domain itself or a subdomain of it, on a dot boundary. As the review notes this was input validation rather than an off-site fetch, since recommendation_url rebuilds a canonical address before anything is requested. Reading the host from `hostname` rather than `netloc` also drops a port, so an explicit :443 no longer fails the check. Tests cover relative links on both readers across both conversions, the lookalike domain, the domain appearing as a subdomain of somewhere else, and the bare-domain and explicit-port forms that have to keep working. The three regression cases fail against the previous code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2 participants
Summary
Adds a USPSTF scraper, following the shared scraping base from #21. USPSTF is named in the project doc's data-sources list and isn't covered by any open PR. It yields 90 published recommendation statements, each carrying a letter grade (A/B/C/D/I) — a small, clinician-facing, high-authority corpus rather than an encyclopedic one.
Two site behaviours the scraper has to correct for
Worth calling out, because both are silent-data-loss traps rather than crashes.
1. The listing under-reports, and its status filter is ignored. The published-recommendations view shows a
Hits: 108counter but renders only 20 rows, with no working pager or page-size parameter (page=,items_per_page=all no-op). Filtering by one of the twelve topic categories does return that category's full set, so discovery walks all twelve and dedupes by slug.Separately,
topic_status=Pstops being honoured once a category filter is applied: rows marked Inactive or Referred come back too, some dating to 1996 (retired guidance, or topics handed to another body). Those are dropped. Stale guidance in a verification corpus is worse than a smaller corpus, since it produces confident wrong verdicts — the exact failure the project doc flags when it says a stale index produces wrong "Refuted" labels.statusis still in metadata if anyone wants them back.That accounts for 108 → 107 reachable → 90 published. The 107/108 gap is the site's own: category 21 reports 21 hits but renders 20 rows, and the missing row has no link, so there is nothing to fetch.
2. A category can contribute nothing new.
scrape_listing_documentstreats an empty page as end-of-source. Walking categories lazily, one per listing page, would mean a category whose recommendations all appeared earlier terminates discovery and silently drops every later category. My first draft did exactly that and a test caught it, so categories are now walked eagerly up front. There's a regression test pinning that the last category is still reached when a middle one contributes nothing.Content
Assembled from the recommendation's own outline: the population/recommendation/grade summary table, then each content panel (Importance, Assessment of Net Benefit, Practice Considerations, Supporting Evidence, tables and figures, References).
Skipped: the Preamble / Mission Statement (identical boilerplate across all 90 — 90 verbatim copies is noise) and the administrative Task Force membership and copyright panels. Verified no page chrome leaks into output.
Licensing
Not public domain, despite being federal work. AHRQ's copyright notice permits reproduction and redistribution "provided that it is reproduced without any changes to the work or portions thereof, except as permitted as fair use", prohibits redistribution for a fee or incorporation into a profit-making venture without written permission, and asks that the USPSTF page be cited when parts are quoted.
Every document carries
license,license_url, andattributionin metadata, matching the shape used in #23 and #25 so a corpus can be filtered by source rights. The no-changes term deserves a look before this feeds claim decomposition or training, both derivative uses — that call belongs to whoever owns those pipelines, not to this scraper. Happy to gate it behind a flag or exclude it from default runs if that's the safer default.Compliance
robots.txt allows the paths used and sets
Crawl-delay: 5, applied both between recommendation pages and between the twelve category listing requests.Components affected
datasets/amfv_datasets/scraping/uspstf.py(new)datasets/amfv_datasets/scraping/cli.py— one import, oneSCRAPERSentrydatasets/test/test_scraping_uspstf.py(new)datasets/test/test_scraping_cli.py— the unknown-source test asserts the registered source listTesting
uv run ruff check/uv run ruff format --checkcleanuv run pytest— 51 passing, on Python 3.13 to match CI--urlmode, boilerplate exclusion, and no chrome leakage