Parse dates wrapped in harmless extra text (Close #518)

[serhii73]

Summary

dateparser.parse() returns None for strings that are a perfectly good date wrapped in a bit of harmless leading (or trailing) text, e.g. 'Actualisé le 17 avril 2019' (issue #518). The problem is not French-specific — the same happens in any language.

This PR adds an explicit, opt-in IGNORE_SURROUNDING_TEXT setting (default: False) implemented inside DateDataParser.get_date_data():

>>> import dateparser
>>> dateparser.parse('Actualisé le 17 avril 2019', languages=['fr'])
>>> dateparser.parse('Actualisé le 17 avril 2019', languages=['fr'],
...                  settings={'IGNORE_SURROUNDING_TEXT': True})
datetime.datetime(2019, 4, 17, 0, 0)
>>> dateparser.parse('Published on 16/04/2019', settings={'IGNORE_SURROUNDING_TEXT': True})
datetime.datetime(2019, 4, 16, 0, 0)

Closes #518.

Root cause

parse() requires the whole string to be recognised: before a locale is even tried, Locale.is_applicable() → Dictionary.are_tokens_valid() demands that every token be a number, a relative expression, or a dictionary word. A single unknown word such as Actualisé makes the locale non-applicable, so parsing gives up before it starts.

Design

The setting is honored by get_date_data() itself, so parse() and get_date_data() always agree, and it works the same regardless of how the language is known (languages=, locales=, region=, DEFAULT_LANGUAGES, detect_languages_function, or full-scan when no hint is given).

When the setting is enabled and the regular, whole-string parse found nothing, get_date_data() retries the same locale iteration with ignore_surrounding_text=True:

1. Locale.is_applicable(..., ignore_surrounding_text=True) validates the tokens after Dictionary._strip_unknown_edge_tokens() removes unrecognised tokens from the leading/trailing edges. An unrecognised token in the interior still makes the locale non-applicable, so '17 foobar avril 2019' stays unparsed.
2. Locale.translate(..., ignore_surrounding_text=True) drops the same edge tokens inside the single, normal translation pass (numerals → normalization → simplifications → split → strip → translate → join), then parsing proceeds exactly as usual. The result is identical to parsing the wrapped date without its surrounding text.

Because stripping happens inside the regular pipeline, there is no re-simplification, no re-joining of raw tokens, and date_formats, STRICT_PARSING, REQUIRE_PARTS, PREFER_* etc. compose naturally. Purely numeric dates ('published 2019-04-16 ok') and relative expressions ('asdf 3 hours ago asdf') are handled too — no month/weekday anchor is required, since enabling the setting is an explicit request for this behavior.

Why this is safe

• Default behavior is byte-for-byte unchanged: with the setting off (the default), no new code path runs. search_dates() and language detection keep their existing strictness; tests lock this in.
• Fully parseable strings are unaffected even with the setting on: the fallback only runs after the normal parse found nothing, so e.g. '23 March 2000, 1:21 PM CET' still parses timezone-aware.
• The ignored-text behavior is documented in docs/settings.rst, including the caveat that strings merely containing date-like words (e.g. 'Chapter 12 March of the Penguins') produce a date when the setting is enabled, and the suggestion to combine it with STRICT_PARSING/REQUIRE_PARTS.

Tests

• tests/test_clean_api.py: parse()-level tests — the issue examples, numeric/date_formats/relative cases, every language-hint source, default-off locks, non-date guards, the documented caveat, and a search_dates strictness lock.
• tests/test_date.py: get_date_data()-level tests, including try_previous_locales and the default-strict lock.
• tests/test_languages.py: Locale.is_applicable/translate unit tests for ignore_surrounding_text, asserting the stripped translation equals the translation of the unwrapped date.

Full suite: 24114 passed, 20 skipped, 1 xfailed. 100% patch coverage; ruff/ruff-format clean; docs build clean.

Known limitations

• No-word-spacing languages can still fail when the surrounding text itself contains a date word: in '更新日 2019年4月17日 です' (ja), 更新日 splits into 更新 + the known token 日, which leaves a dangling "day" unit the parser cannot drop (same class of problem as the parse() facet of search_dates cannot find complete date #521, out of scope here). '2019年4月17日 です' works.
• region='FR' without languages cannot parse French with or without this setting, because the loader has no <lang>-FR locale for fr (plain fr covers France); that is pre-existing loader behavior, unrelated to this PR. languages=['fr'], region=... and locales=['fr-CA'] work with the setting and are tested.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.10%. Comparing base (315e396) to head (7495119).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1356      +/-   ##
==========================================
+ Coverage   97.07%   97.10%   +0.03%     
==========================================
  Files         236      236              
  Lines        2975     3008      +33     
==========================================
+ Hits         2888     2921      +33     
  Misses         87       87              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[AdrianAtZyte]

Opus 5 feedback

I checked out the PR (f2f5a03) against its real base (315e396) and ran it. Here's what I found.

Bottom line

The diagnosis in the PR is correct and the containment strategy is genuinely well thought out — but I'd not merge it as-is. It silently turns parse() into a fuzzy matcher, and it only half-works.

Verified facts first: full suite passes (24087 passed, 22 skipped, 1 xfailed in 101 s), ruff/ruff-format clean, no new exceptions in a 4000-string fuzz across 7 languages, and on a 2801-string adversarial corpus 0 previously-parsing inputs changed result (the "no currently-working input changes behaviour" claim holds). Failure-path cost goes 0.80 ms → 0.96 ms. The claim that search_dates/detection are untouched also holds — both go through get_date_data, which is byte-identical.

The main objection: parse() and get_date_data() now disagree

dateparser.parse('Actualisé le 17 avril 2019', languages=['fr'])                       # 2019-04-17
DateDataParser(languages=['fr']).get_date_data('Actualisé le 17 avril 2019')            # None

The very property that makes the patch "safe" — hiding the logic in dateparser/__init__.py — is what makes it inconsistent. parse() is documented as the convenience wrapper for get_date_data(); after this they have different notions of "is this a date". Anyone using get_date_data (for period/locale) gets no fix, and the fallback is reachable only by calling a private method from the public API.

And the leniency isn't reachable consistently even from parse():

┌─────────────────────────────────────┬────────────────────────────────────────────────────────────────────┐
│      how the language is known      │                               result                               │
├─────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ languages=['fr']                    │ ✅ parses                                                          │
├─────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ detect_languages_function=…         │ ✅ parses (only because _get_applicable_locales mutates            │
│                                     │ self.languages as a side effect)                                   │
├─────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ region='FR'                         │ ❌ None                                                            │
├─────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ settings={'DEFAULT_LANGUAGES':      │ ❌ None                                                            │
│ ['fr']}                             │                                                                    │
└─────────────────────────────────────┴────────────────────────────────────────────────────────────────────┘

That last row is a bug: _get_locales_to_try has a whole DEFAULT_LANGUAGES block (date.py:713-719) that the gate at date.py:612 makes unreachable — you can only get there by also passing explicit languages.

New false positives

Everything below returns None on base and a date on the PR (languages=['en'], today 2026-07-27):

'Chapter 12 March of the Penguins'  -> 2026-03-12      # core: '12 march of the'
'Room 12 May Hall'                  -> 2026-05-12
'Order #4 June delivery'            -> 2026-06-04
'version 3 May'                     -> 2026-05-03
'Page 3 of 12 March'                -> 2026-03-03      # core '3 12 march'; the 12 is silently dropped
'temperature -17 April 2019'        -> 2019-04-17
'17 April 2019 Eastern Standard Time'-> 2019-04-17     # tz text silently discarded

The "interior tokens must be known" rule is a decent guard, but skip/pertain words (of, the, by, le) are "known", so noise adjacent to them survives. On a synthetic corpus of date-ish words, 340/2801 strings flipped None → date. The common scraping idiom is "parse the field; None means it wasn't a date" — this weakens that materially, and STRICT_PARSING: True doesn't opt out.

…while the fix is also too narrow

The month/weekday/relative "anchor" requirement means numeric dates and word-joined languages get nothing:

'published 2019-04-16 ok'   (en) -> None
'xx 16/04/2019 yy'          (en) -> None
'更新日 2019年4月17日 です'   (ja) -> None      # 年/月/日 simplify to '-', so no anchor survives
'Published on 16/04/2019', date_formats=['%d/%m/%Y'] -> None

That last one is telling: the heuristic overrides an explicit user-supplied format. So the outcome is "leading/trailing noise is tolerated, but only for languages with word-form months, and only when you passed languages=" — hard to document and hard to defend.

My recommendation: make it an explicit opt-in setting implemented inside get_date_data (e.g. settings={'IGNORE_SURROUNDING_TEXT': True}, default off). That fixes the parse/get_date_data split, makes region/DEFAULT_LANGUAGES/detection consistent for free, keeps search_dates strict (it just doesn't enable it), and lets the anchor heuristic be relaxed for numeric/CJK cases without endangering the default path. If you'd rather not grow the settings surface, the alternative is to decline and document search_dates — it already handles the issue example ([('le 17 avril 2019', 2019-04-17)]).

Code-level issues

- dictionary.py:45 — _MONTH_AND_WEEKDAY_TOKENS = frozenset(KNOWN_WORD_TOKENS[:19]) is a magic slice over a list whose order is incidental. Anyone inserting a token at the top silently breaks it. Define explicit WEEKDAY_TOKENS/MONTH_TOKENS lists and build KNOWN_WORD_TOKENS from them.
- locale.py:216-222 — the stripped core is re-joined into a string and handed back to _DateLocaleParser.parse, which runs _translate_numerals + normalize_unicode + _simplify a second time. Non-idempotent simplifications are a real hazard here (_process_russian_compound_ordinals does arithmetic on adjacent number pairs). Better to strip and translate in one pass, or return the token list.
- "".join(core) works only because Dictionary.split() happens to embed adjacent whitespace inside unknown tokens (['actualisé ', 'le', ' ', '17', …]). translate() uses _join(..., separator=" ") for the same job; relying on that incidental behaviour is fragile.
- _get_locales_to_try (date.py:698) duplicates the locale-enumeration half of _get_applicable_locales, including the use_given_order derivation. Drift risk — parameterise the existing generator with a "skip applicability filter" flag instead.
- date.py:625-627 — the previous_locales write can never run: try_previous_locales defaults to False and parse() never sets it (and _get_locales_to_try doesn't consult previous_locales anyway, so it would be a no-op even if it did).
- Locale.strip_extra_text is new public API whose settings=None default crashes. Same wart as translate/is_applicable, so not a blocker, but as an internal helper it should be _strip_extra_text.

Tests and docs

- The new tests live in TestDateDataParser but exercise dateparser.parse, and use bare assertEqual instead of the surrounding given/when/then BaseTestCase style. tests/test_clean_api.py is the natural home.
- No unit tests for Locale.strip_extra_text / Dictionary.strip_extra_edge_tokens in tests/test_languages.py — which is exactly why the dead branches went unnoticed.
- The PR's central safety claim isn't locked in by a test: add assertions that get_date_data and search_dates still reject 'Actualisé le 17 avril 2019'.
- Zero doc changes for a behaviour change to the most-used function in the library. Needs a README/docs/settings.rst note either way.

Is 100% coverage feasible? Yes — and each miss is a design signal, not a testing gap

The 5 uncovered patch lines (I reproduced them exactly: date.py 610, 627, 714; dictionary.py 162, 173):

┌───────────────────┬─────────────────────────────────┬────────────────────────────────────────────────────┐
│       line        │              what               │                        fix                         │
├───────────────────┼─────────────────────────────────┼────────────────────────────────────────────────────┤
│                   │ raise TypeError("Input type     │ Unreachable — the only caller is parse(), which    │
│ date.py:610       │ must be str")                   │ already went through get_date_data's identical     │
│                   │                                 │ check. Delete.                                     │
├───────────────────┼─────────────────────────────────┼────────────────────────────────────────────────────┤
│ date.py:627       │ self.previous_locales[locale] = │ Unreachable (try_previous_locales=False, never set │
│                   │  None                           │  by parse()) and a no-op. Delete.                  │
├───────────────────┼─────────────────────────────────┼────────────────────────────────────────────────────┤
│                   │                                 │ Unreachable because of the gate at :612 — this is  │
│ date.py:714       │ yield from … DEFAULT_LANGUAGES  │ the bug above. Fix the gate (then it's testable)   │
│                   │                                 │ or delete the block.                               │
├───────────────────┼─────────────────────────────────┼────────────────────────────────────────────────────┤
│ dictionary.py:162 │ if match_relative_regex is      │ Dead: the sole caller always passes it. Drop the   │
│                   │ None: default                   │ default parameter.                                 │
├───────────────────┼─────────────────────────────────┼────────────────────────────────────────────────────┤
│                   │ return True                     │ Genuine missing test. One case covers it — I       │
│ dictionary.py:173 │ (relative-expression anchor)    │ confirmed parse('asdf 3 hours ago asdf',           │
│                   │                                 │ languages=['en']) hits line 173.                   │
└───────────────────┴─────────────────────────────────┴────────────────────────────────────────────────────┘

So: delete three dead lines, fix one gate, add one test → 100% patch coverage, with the codebase actually improved rather than padded. Worth telling the contributor that the coverage report was pointing at real problems.

[serhii73]

Thanks @AdrianAtZyte for the thorough review — the verification work (fuzz corpus, the reachability table, mapping each uncovered line to a design problem) made it very clear what to change. I've reworked the PR in 8bb8acb following your main recommendation, and updated the PR description to match. Point by point:

The parse()/get_date_data() split — gone. The behavior is now an explicit opt-in setting, IGNORE_SURROUNDING_TEXT (default off), implemented inside get_date_data(); dateparser/__init__.py is back to byte-identical with master. With the setting off, no new code runs, so search_dates and language detection keep their strictness — both are now locked in by tests (get_date_data returns nothing for 'Actualisé le 17 avril 2019' by default, and search_dates still returns [('le 17 avril 2019', …)]).

Inconsistent reachability — the fallback reuses the same _get_applicable_locales() iteration as the normal pass (parameterised with an ignore_surrounding_text flag instead of the duplicated _get_locales_to_try()), so languages=, locales=, region=, DEFAULT_LANGUAGES, detect_languages_function, and even the no-hint full scan behave the same; each source has a test. One honest caveat: region='FR' without languages can't parse French even for clean strings on master, because the loader has no fr-FR locale (plain fr covers France). That row is a pre-existing loader/data limitation this PR doesn't touch; languages=['fr'], region='CA' and locales=['fr-CA'] work with the setting and are tested.

False positives — now opt-in and documented. docs/settings.rst states the semantics (the ignored text is discarded and the remainder is parsed as if it were the whole input), shows 'Chapter 12 March of the Penguins' as the caveat (also pinned by a test), and recommends combining with STRICT_PARSING/REQUIRE_PARTS, which now compose naturally (tested). Related: with the anchor gone, an opted-in "Ceci n'est pas une date" (fr) does yield a date (the surviving est pas une is skip words plus the word-number une, consistent with parse('une') today). I kept that rather than reintroduce a guard, since it follows from the documented semantics and STRICT_PARSING is the principled way to reject incomplete remainders — happy to revisit if you'd prefer a stricter core requirement.

Too narrow — the month/weekday anchor heuristic (and the magic KNOWN_WORD_TOKENS[:19] slice) is dropped, as the opt-in allows. Numeric dates ('published 2019-04-16 ok', 'xx 16/04/2019 yy'), date_formats ('Published on 16/04/2019' + ['%d/%m/%Y']), and relative expressions ('asdf 3 hours ago asdf') all work and are tested. The ja example is only half-fixed: '2019年4月17日 です' now works, but '更新日 2019年4月17日 です' still fails because 更新日 splits into 更新 + the known token 日, leaving a dangling "day" unit the absolute parser can't drop — the same dangling-unit class as the parse() facet of #521, left as a documented limitation rather than growing this PR into the parser.

Code-level issues — addressed mostly by deletion:

• No double simplification and no "".join(core): stripping is a token-list operation between split and word-mapping inside the single Locale.translate() / is_applicable() pass, so numerals/normalize/simplify run exactly once and tokens flow through the normal _join. A test asserts the stripped translation equals the translation of the unwrapped date.
• The unreachable TypeError, the dead previous_locales write, and the dead match_relative_regex=None default are gone. previous_locales is genuinely reachable in the fallback now, covered by a try_previous_locales=True test.
• Locale.strip_extra_text is gone; what remains is the private Dictionary._strip_unknown_edge_tokens plus an additive flag on the existing is_applicable/translate signatures.

Tests, docs, coverage — the parse()-level tests moved to tests/test_clean_api.py in its given/when/then style, get_date_data()-level tests live in TestDateDataParser, and the Locale-level unit tests are in tests/test_languages.py. The setting is documented in docs/settings.rst. Patch coverage is 100% this time (24114 passed, 20 skipped, 1 xfailed; ruff, ruff-format and the docs build are clean) — you were right that every uncovered line was pointing at a real problem; they are all either deleted or reachable-and-tested now.

[AdrianAtZyte]

Opus feedback:

I checked out the reworked head (8bb8acb) against master (315e396b) and ran it. This is a solid rework — it addresses every point from the previous round — but I found one behavior I'd fix before merging.

Verified

- Patch applies cleanly to 315e396b; full suite passes locally (24113 passed, 22 skipped, 1 xfailed — the 2 extra skips are my env, not the patch).
- The safety claim holds: with the setting off, no new code runs. With it on, strings that don't parse today still don't: 'Le prix est de 12 euros' (fr), 'Total: 100 USD', 'v1.2.3 released' → None.
- Failure-path cost of the second locale pass was in the noise in my measurement (~0.06 ms/parse either way); no perf objection.
- The parse/get_date_data split, the reachability table, the double-simplification and the "".join(core) fragility from round 1 are all genuinely gone.

Blocker: a recognized timezone at the trailing edge is silently discarded

>>> S = {'IGNORE_SURROUNDING_TEXT': True}
>>> parse('23 March 2000 1:21 PM EST', languages=['en'], settings=S)
datetime.datetime(2000, 3, 23, 13, 21, tzinfo=<StaticTzInfo 'EST'>)
>>> parse('Updated 23 March 2000 1:21 PM EST', languages=['en'], settings=S)
datetime.datetime(2000, 3, 23, 13, 21)          # offset gone, not applied

The token list makes it explicit — _strip_unknown_edge_tokens eats the tz:

['updated ', '23', ' ', 'march', ' ', '2000', ' ', '1', ':', '21', ' ', 'pm', ' est']
['23',       ' ', 'march', ' ', '2000', ' ', '1', ':', '21', ' ', 'pm']

UTC/GMT/Z survive only because they're in PARSER_KNOWN_TOKENS; every other abbreviation is dropped. The docs/settings.rst warning calls this "an unrecognized timezone name", but EST is recognized by dateparser — so the caveat mislabels it, and with RETURN_AS_TIMEZONE_AWARE: True the caller gets a wrong instant (local offset stamped onto an EST wall time) with no signal.

The fix isn't a one-liner, which is worth telling the contributor — I tried two and both failed:

- word_is_tz(token) is case-sensitive and the tokens are already lowercased by _simplify.
- word_is_tz(token.upper()) breaks the flagship case: _search_regex is unanchored, so 'ACTUALISÉ' matches ACT and the noise word is kept.

The direction that works is: treat an edge token as known when token.strip() fullmatches the tz regex case-insensitively (note the embedded whitespace in ' est' — an unstripped check silently misses). Anchored + trimmed + case-insensitive, and arguably only at the trailing edge.

Scope: the setting name promises more than the rule delivers

Stripping stops at the first known token, and digits count as known — so any number in the noise blocks the whole thing. All of these return None with the setting on, while search_dates handles them:

┌──────────────────────────────────────────────────┬────────────┬──────────────┐
│                    input (en)                    │ setting on │ search_dates │
├──────────────────────────────────────────────────┼────────────┼──────────────┤
│ 'invoice 12345 paid on 3 March 2019'             │ None       │ ✅           │
├──────────────────────────────────────────────────┼────────────┼──────────────┤
│ 'Posted by admin on 16 April 2019'               │ None       │ ✅           │
├──────────────────────────────────────────────────┼────────────┼──────────────┤
│ 'Meeting on 5 March 2019 in room 12'             │ None       │ ✅           │
├──────────────────────────────────────────────────┼────────────┼──────────────┤
│ 'Booking 2019-04-16 confirmed at 10:00 by agent' │ None       │ ✅           │
└──────────────────────────────────────────────────┴────────────┴──────────────┘

IGNORE_SURROUNDING_TEXT reads like "the surrounding text doesn't matter"; what it does is "drop unrecognized tokens at the edges, stopping at the first digit or dictionary word." I'd either narrow the name or state that rule in the docs with one of the rows above, and cross-reference search_dates for the general case.

That said, it isn't redundant with search: 'Last updated: 12 March 2019 at 10:30 (GMT)' → 2019-03-12 10:30+00:00 with the setting, while search_dates returns None. Worth keeping — the two cover different shapes.

Minor

- _is_known_token re-states the per-token condition inside are_tokens_valid; have are_tokens_valid call it so they can't drift.
- dictionary._strip_unknown_edge_tokens is called from locale.py — private across objects. There's precedent (locale.py:579), so a naming nit at most, but dropping the underscore would be cleaner.

Also worth confirming what happens to the 'Ceci n'est pas une date' → date case the author raised. I'd accept it as following from the documented semantics, same as 'Chapter 12 March of the Penguins' — reintroducing a core-content guard is what made the previous revision hard to defend.