Skip to content

safe_url_string(): add the path for problematic scenarios - #297

Open
AdrianAtZyte wants to merge 1 commit into
scrapy:masterfrom
AdrianAtZyte:url-path
Open

safe_url_string(): add the path for problematic scenarios#297
AdrianAtZyte wants to merge 1 commit into
scrapy:masterfrom
AdrianAtZyte:url-path

Conversation

@AdrianAtZyte

Copy link
Copy Markdown
Contributor

URLs without a path but with a query have shown to be problematic, have safe_url_string() force the URL path to prevent such issues.

Fixes scrapy/scrapy#1133, closes scrapy/scrapy#3494, fixes scrapy/scrapy#6574.

@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.47%. Comparing base (6704653) to head (b93e1de).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #297   +/-   ##
=======================================
  Coverage   98.47%   98.47%           
=======================================
  Files           9        9           
  Lines         853      855    +2     
  Branches      177      178    +1     
=======================================
+ Hits          840      842    +2     
  Misses          5        5           
  Partials        8        8           
Files with missing lines Coverage Δ
w3lib/url.py 99.53% <100.00%> (+<0.01%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed-hq

codspeed-hq Bot commented Aug 4, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 45 untouched benchmarks


Comparing AdrianAtZyte:url-path (b93e1de) with master (6704653)

Open in CodSpeed

@AdrianAtZyte
AdrianAtZyte marked this pull request as ready for review August 4, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Always get a 400 error when requesting local resources Scrapy doesn't normalize URLs returned in redirects which leads to Bad Requests (400)

1 participant