feat(text-normalizer): add property tests, NFC documentation, benchmarks and edge cases - #4
Merged
Merged
Conversation
…rks and edge cases
mohaelmrabet
force-pushed
the
feat/v0.1.x-hardening
branch
from
August 14, 2026 18:12
3fc8ba4 to
fb1b878
Compare
testUnicodeEdgeCases asserted that invalid UTF-8 comes back untouched, as both the README and the decompose() doc block promise, but it came back as an empty string. The guard in decompose() only ever covered Normalizer::normalize. It was never reached: mb_strtolower runs first and replaces every malformed byte with '?', which the separator pass then folds to spaces and trim removes. Checking the encoding up front is the only place the input is still intact, so normalize() now returns it as it is. The mixed-script assertion in the same test was wrong on its own terms and had never run, the malformed-UTF-8 assertion failing before it: this profile folds 'à' to 'a' like any other Latin diacritic, and the emoji's variation selector (U+FE0F) is a combining mark the profile does not claim, so it survives by the same rule that keeps Arabic harakat.
mohaelmrabet
force-pushed
the
feat/v0.1.x-hardening
branch
from
August 14, 2026 18:13
fb1b878 to
1227391
Compare
mohaelmrabet
added a commit
that referenced
this pull request
Aug 14, 2026
The working tree had reverted testUnicodeEdgeCases to the expectation that shipped broken on the v0.1.x branch: 'à' kept its accent and the emoji left no trace. Neither holds. This profile folds 'à' to 'a' like any other Latin diacritic, and the variation selector trailing the emoji (U+FE0F) is a combining mark the profile does not claim, so it survives by the same rule that keeps Arabic harakat.
2 tasks
mohaelmrabet
added a commit
that referenced
this pull request
Aug 14, 2026
The working tree had reverted testUnicodeEdgeCases to the expectation that shipped broken on the v0.1.x branch: 'à' kept its accent and the emoji left no trace. Neither holds. This profile folds 'à' to 'a' like any other Latin diacritic, and the variation selector trailing the emoji (U+FE0F) is a combining mark the profile does not claim, so it survives by the same rule that keeps Arabic harakat.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardening pass on
TextNormalizerfor the v0.1.x line: no behaviour change, only additional test coverage, a benchmark harness, and documentation accuracy fixes.Changes
TextNormalizerTest:testPropertyIdempotence—normalize(normalize(x)) === normalize(x)across six diverse samples (French, Arabic with diacritics, mixed Latin diacritics, whitespace/punctuation noise, emoji, Arabic yeh variants).testNegativeCollisions— asserts distinct words stay distinct (hamza preservationماءvsما, andcotevscode).testUnicodeEdgeCases— malformed UTF-8 (\x80\xA0\xA1) is returned untouched rather than silently emptied; mixed Arabic + Latin + emoji input normalizes as expected.tests/benchmark.php, wired as thecomposer benchmarkscript.TextNormalizerdoc block.Test plan
composer test— 47 tests, 68 assertions, green on PHP 8.2 / 8.3 / 8.4 (lowest and highest deps)composer analyse— no errorscomposer benchmark