feat(text-normalizer): add Cyrillic and Greek profiles and multi-script fixtures - #6
Merged
Merged
Conversation
Two expectations had the Greek profile keep 'ς' inside a word while folding it when it stood alone, which the character map cannot do: it folds the code point wherever it appears, and that is the point — the profile exists so 'αγγελος' and 'αγγελοσ' do not index apart. The fixture loader also tripped PHPStan at level max: file_get_contents returns string|false and json_decode returns mixed, so nothing about the decoded rows was known. Asserting the shape as it is read types the rows and checks the fixture file is well-formed at the same time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the profile set beyond Latin and Arabic, and adds a fixture file so a normalization sample can be added without touching PHP.
Cyrillic
NormalizerProfile::cyrillic()folds the letters no normalization form unifies: Yo (ё → е), the Ukrainian/Belarusianі,ї,ўandґ. Combining marks are stripped by NFD as elsewhere.Greek
NormalizerProfile::greek()folds final sigma (ς → σ) and strips tonos through canonical decomposition, soΑθήναandΑΘΗΝΑfold together.Fixtures
tests/fixtures/normalization_samples.jsonholds one original/expected pair per script, walked bytestMultiScriptJsonFixtures.all()now composes the four profiles, so its name becomesarabic_search_latin_cyrillic_greek.Test plan
composer test— 51 tests, 101 assertionscomposer analyse— no errorsNote
The second commit fixes two Greek expectations that had
ςsurviving inside a word while folding when it stood alone — the character map folds a code point wherever it appears, and that is the behaviour the profile wants. It also types the fixture rows, which PHPStan rejected at level max (file_get_contentsreturnsstring|false,json_decodereturnsmixed).Worth a look
cyrillic()mapsіandїto the Latini, soКиївnormalizes toкиiв— a token mixing two scripts, andКиев(киев) stays distinct fromКиївeven though they are the same city. Folding to the Cyrillicиwould collapse them instead. The tests and fixtures currently pin the Latin behaviour; happy to change it if the collision is what you want.