Skip to content

feat(text-normalizer): add Cyrillic and Greek profiles and multi-script fixtures - #6

Merged
mohaelmrabet merged 2 commits into
mainfrom
feat/v0.3.0-additional-scripts
Aug 14, 2026
Merged

feat(text-normalizer): add Cyrillic and Greek profiles and multi-script fixtures#6
mohaelmrabet merged 2 commits into
mainfrom
feat/v0.3.0-additional-scripts

Conversation

@mohaelmrabet

Copy link
Copy Markdown
Contributor

Summary

Extends the profile set beyond Latin and Arabic, and adds a fixture file so a normalization sample can be added without touching PHP.

Cyrillic

NormalizerProfile::cyrillic() folds the letters no normalization form unifies: Yo (ё → е), the Ukrainian/Belarusian і, ї, ў and ґ. Combining marks are stripped by NFD as elsewhere.

Greek

NormalizerProfile::greek() folds final sigma (ς → σ) and strips tonos through canonical decomposition, so Αθήνα and ΑΘΗΝΑ fold together.

Fixtures

tests/fixtures/normalization_samples.json holds one original/expected pair per script, walked by testMultiScriptJsonFixtures. all() now composes the four profiles, so its name becomes arabic_search_latin_cyrillic_greek.

Test plan

  • composer test — 51 tests, 101 assertions
  • composer analyse — no errors

Note

The second commit fixes two Greek expectations that had ς surviving inside a word while folding when it stood alone — the character map folds a code point wherever it appears, and that is the behaviour the profile wants. It also types the fixture rows, which PHPStan rejected at level max (file_get_contents returns string|false, json_decode returns mixed).

Worth a look

cyrillic() maps і and ї to the Latin i, so Київ normalizes to киiв — a token mixing two scripts, and Киев (киев) stays distinct from Київ even though they are the same city. Folding to the Cyrillic и would collapse them instead. The tests and fixtures currently pin the Latin behaviour; happy to change it if the collision is what you want.

Two expectations had the Greek profile keep 'ς' inside a word while
folding it when it stood alone, which the character map cannot do: it
folds the code point wherever it appears, and that is the point — the
profile exists so 'αγγελος' and 'αγγελοσ' do not index apart.

The fixture loader also tripped PHPStan at level max: file_get_contents
returns string|false and json_decode returns mixed, so nothing about the
decoded rows was known. Asserting the shape as it is read types the rows
and checks the fixture file is well-formed at the same time.
@mohaelmrabet
mohaelmrabet merged commit 4698771 into main Aug 14, 2026
8 checks passed
@mohaelmrabet
mohaelmrabet deleted the feat/v0.3.0-additional-scripts branch August 14, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant