Keep word boundaries when stripping unrenderable emoji - #714
Open
einanderson wants to merge 1 commit into
Open
Conversation
strip_tofu() removed emoji/symbols outright, which glued the surrounding words together whenever a title used one as a separator, e.g. "WEEKEND<emoji>Type" was shown as "WEEKENDType". Replace a run of unrenderable characters with a single space instead, then tidy up: collapse horizontal whitespace, drop whitespace around line breaks (line breaks themselves are still kept, they matter for plots) and trim the result. Reported by kuraikaze.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
strip_tofu()removes unrenderable emoji and symbols so they do not show up as tofu boxes in theskin font. It deleted them without replacement, so when a title used an emoji as a separator the
surrounding words were glued together:
Reported against a community build by a user who noticed titles losing their word boundaries.
The change
A run of unrenderable characters is replaced by a single space instead of being dropped. Runs
collapse to one space, horizontal whitespace is smoothed afterwards, whitespace around line breaks
is trimmed while the line breaks themselves are kept (they matter for plots), and the result is
stripped.
Verification
Checked against the real function with 11 cases (word boundary, emoji run, ZWJ sequence, emoji at
the edges, blank line, accents and CJK untouched, non-string passthrough), and live against Twitch
titles:
Accents, CJK and non-emoji symbols such as U+25AB and U+2192 are unaffected.