Skip to content

test(assessments): fix flaky title-sort tests via DB collation - #1378

Draft
RichDom2185 wants to merge 1 commit into
masterfrom
fix-flaky-submission-sort-test
Draft

test(assessments): fix flaky title-sort tests via DB collation#1378
RichDom2185 wants to merge 1 commit into
masterfrom
fix-flaky-submission-sort-test

Conversation

@RichDom2185

Copy link
Copy Markdown
Member

Problem

The submissions_by_grader_for_index title-sort tests are flaky. On PR #1377's CI they failed with:

Assertion with >= failed
code:  assert x.title >= y.title
left:  "I will speak daggers to her, but use none."
right: "In my mind's eye."

Root cause

The query sorts by upper(title) in the database, which uses PostgreSQL's locale collation:

order_by: [{^sort_direction, fragment("upper(?)", asst.title)}]

The tests then re-checked the returned order with Elixir's binary string comparison (assert x.title >= y.title). The two disagree on how spaces and punctuation sort relative to letters — e.g. PostgreSQL's collation orders "In my mind's eye." before "I will speak daggers to her, but use none." (ignoring the space), whereas Elixir's binary comparison does the opposite (space 0x20 < n).

Assessment titles are randomly generated Hamlet quotes (Faker.Lorem.Shakespeare.En.hamlet()), and Faker is not seeded, so whenever a run happened to produce two titles straddling this boundary, the test failed. Only the title sort tests are affected (type/xp sort by integers). String.upcase/1 alone would not fix it — the failing pair are both already uppercase-initial.

Fix

Assert the returned order against the database's own sort of the same titles, comparing upper(title) on both sides. This makes the check deterministic and consistent with the query under any collation, and it now verifies the full ordering rather than only that the first element is the extremum.

Verified locally: the helper passes for the DB-correct order (which the old binary check would have wrongly rejected) and fails for a reversed order. Ran the two tests across multiple random seeds and the full assessments_test.exs (90 tests) — all green.

Note

Stacked on #1377 (fix-course-creation-500) since it is unrelated to that course-creation fix. GitHub will retarget this PR to master once #1377 merges.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Aug 1, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2950af00-dd29-40fb-9d87-ee9ab06c9897

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

Coverage Status

coverage: 88.354% (-0.02%) from 88.376% — fix-flaky-submission-sort-test into master

@RichDom2185 RichDom2185 self-assigned this Aug 1, 2026
@RichDom2185
RichDom2185 marked this pull request as draft August 1, 2026 12:42
@sayomaki
sayomaki force-pushed the fix-flaky-submission-sort-test branch from 7b04348 to 98c4498 Compare August 11, 2026 05:27
Base automatically changed from fix-course-creation-500 to master August 11, 2026 13:32
The submissions_by_grader_for_index title-sort tests re-checked ordering
with Elixir's binary string comparison, but the query sorts by
`upper(title)` using PostgreSQL's locale collation. The two disagree on
how spaces and punctuation sort relative to letters, so the randomly
generated Hamlet-quote titles intermittently failed the assertion
(e.g. "I will speak daggers to her, but use none." vs "In my mind's eye.").

Assert the returned order against the database's own sort of the same
titles instead, comparing `upper(title)` on both sides so the check is
deterministic and consistent with the query under any collation.
@sayomaki
sayomaki force-pushed the fix-flaky-submission-sort-test branch from 98c4498 to 14fb7d1 Compare August 11, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant