Skip to content

Strip a leading colon before parsing (: 24 February 2019) - #1355

Open
DMZ22 wants to merge 2 commits into
scrapinghub:masterfrom
DMZ22:strip-leading-colon
Open

Strip a leading colon before parsing (: 24 February 2019)#1355
DMZ22 wants to merge 2 commits into
scrapinghub:masterfrom
DMZ22:strip-leading-colon

Conversation

@DMZ22

@DMZ22 DMZ22 commented Jul 26, 2026

Copy link
Copy Markdown

Closes #889.

: 24 February 2019 returns None, while 24 February 2019 parses correctly:

>>> dateparser.parse(": 24 February 2019")
None
>>> dateparser.parse("24 February 2019")
datetime.datetime(2019, 2, 24, 0, 0)

sanitize_date already trims trailing colons — RE_TRIM_COLONS = r"(\S.*?):*$" — so "2019:""2019". But a leading colon is left in place, becomes a stray token, and the parse fails.

The fix adds RE_TRIM_LEADING_COLONS = r"^\s*:+\s*" and applies it right after the existing trailing-colon trim:

": 24 February 2019"   -> "24 February 2019"   -> 2019-02-24
":24 February 2019"    -> "24 February 2019"   -> 2019-02-24
":: 2019-02-24"        -> "2019-02-24"         -> 2019-02-24

The important constraint is that this must not touch a colon inside a time — and it doesn't, because a time has digits before the colon (^\s*: only matches a colon at the very start):

"12:30"              -> unchanged  (still parses as a time)
"2019-02-24 12:30"   -> unchanged
"5:45 pm"            -> unchanged
"11:59:59"           -> unchanged

Scope

I deliberately kept this to the leading colon, which is what the issue reports. The related label-prefix cases mentioned in passing (at: …, Date: …) are a broader and riskier change — stripping an arbitrary word: prefix could swallow real tokens — so I left them out. on: is already handled separately by RE_SANITIZE_ON; happy to add a general label-prefix sanitizer in a follow-up if you'd like one.

Tests

Extended TestSanitizeDate.test_sanitize_date_colons with a test_sanitize_date_leading_colons covering the leading-colon cases and asserting "12:30" / "2019-02-24 12:30" are untouched. Reverting the source leaves sanitize_date(": 24 February 2019") unchanged, so the test fails without the fix.

ruff check and ruff format --check (v0.9.3, matching .pre-commit-config.yaml) are clean. tests/test_date_parser.py + tests/test_search.py are 781 passed / 3 failed locally — the 3 are pre-existing Windows-only datetime.fromtimestamp OSErrors in test_parse_negative_timestamp, unrelated to this change. (tests/test_date.py, which holds the new unit test, only collects on POSIX because of its from time import tzset import, so it runs in CI rather than on my Windows box.)

sanitize_date trimmed trailing colons via RE_TRIM_COLONS but left a
leading colon in place, so ": 24 February 2019" parsed as None while
"24 February 2019" parsed fine. Add RE_TRIM_LEADING_COLONS to remove a
leading colon (and any run of them). A time such as "12:30" has digits
before the colon, so it is untouched.
@AdrianAtZyte

Copy link
Copy Markdown
Contributor

Opus 5 feedback:

Two consequences the PR body gets wrong:

1. at: 24 February 2019 is the same bug, not a riskier separate feature. Translation drops the unknown word and hands the parser ": 24 february 2019" — identical to the leading-colon input. Sanitizing the raw input can't reach it because the colon only becomes leading after translation. The PR frames this as needing an "arbitrary word: prefix stripper"; it doesn't.
2. New false positive: parse(":30") was None, now returns day 30 of the current month (":30" → "30" → bare number → day-of-month). Small, but it's a previously-safe None turning into a confidently wrong date.

I tried the parser-level fix — skip numeric tokens containing no actual digit:

             if token in skip_tokens:
                 continue

+            if type == 0 and not any(c.isdigit() for c in token):
+                continue

Result: : 24 February 2019 ✓, at: 24 February 2019 ✓ (bonus), :30 stays None (better), all times unaffected, and the full suite passes (24045 passed). Four lines, fixes a strictly larger set, no new false positives.

":30" was parsed as day 30 of the current month after the colon strip,
where it previously returned None. A colon followed by only digits looks
like a time fragment, not a prefixed date, so it is left alone now
(":30", ": 30" -> None again). The guard allows for whitespace inside
the lookahead, since the quantifier before it can backtrack.
@DMZ22

DMZ22 commented Jul 29, 2026

Copy link
Copy Markdown
Author

Thanks — measured both points on the branch.

Point 2 is right and is fixed in 3190fb8. :30 (and : 30) were parsed as day 30 of the current month, where master returns None. The strip now skips a remainder that is only digits — a colon followed by a bare number looks like a time fragment, not a prefixed date. One subtlety: the lookahead has to allow leading whitespace ((?!\s*\d+\s*$)), because the \s* before it can backtrack and hand the space to the remainder, which let : 30 slip through the first version of the guard. Both cases are pinned in the test.

Current behaviour, measured:

input master branch
: 24 February 2019 None 2019-02-24
:24 February 2019 None 2019-02-24
:: 2019-02-24 None 2019-02-24
:30 / : 30 / :2019 None None
12:30, 5:45 pm time time (unchanged)

Point 1 I could not reproduce. at: 24 February 2019 returns None on this branch, both with default settings and with languages=["en"] (and on master). So at least for English, the translation path doesn't reduce the label prefix to a leading colon that this strip would catch. If you have a locale/settings combination where it does, I'm happy to look — but as it stands the at: case still needs its own handling, which is why I'd left it out.

@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.07%. Comparing base (315e396) to head (3190fb8).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1355   +/-   ##
=======================================
  Coverage   97.07%   97.07%           
=======================================
  Files         236      236           
  Lines        2975     2977    +2     
=======================================
+ Hits         2888     2890    +2     
  Misses         87       87           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Date after colon is not parsed (": 24 February 2019")

2 participants