Skip to content

ci(tessl-review): distinguish tooling failures from low scores - #308

Merged
trieloff merged 1 commit into
mainfrom
fix/tessl-review-workflow
Aug 21, 2026
Merged

ci(tessl-review): distinguish tooling failures from low scores#308
trieloff merged 1 commit into
mainfrom
fix/tessl-review-workflow

Conversation

@trieloff

@trieloff trieloff commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Rewritten. The first version of this PR was written against a stale checkout of main and duplicated work that had already landed. It has been rebased and cut down to only what is actually still missing. See the note at the bottom.

Problem

When the Tessl CLI fails to run at all — auth failure, network error, crash — the review job counts it as a review failure and reports:

##[error]N skill(s) scored below 80%

That is misleading. The skill was never scored. On #189 this surfaced as a red "below 80%" check on a skill that actually scores 85%, which reads to a contributor as a quality rejection rather than a broken CI run. It cost a fair amount of time to work out that the real message buried in the log was:

✘ Please authenticate with Tessl to continue.

Changes

Tooling failures are no longer reported as low scores. They get their own counter and are reported as "review could not be run", with a ⚠️ row in the summary table and an explanatory note beneath it. They still fail the job — they're just no longer conflated with a below-threshold score.

Skip cleanly when Tessl isn't configured. If TESSL_TOKEN is absent, the job now skips with an explanatory step summary instead of installing an unauthenticated CLI and failing on the first review call.

Pin actions to commit SHAs. actions/checkout and tesslio/setup-tessl are pinned with # vN comments, so Renovate keeps tracking them.

Add a concurrency group so superseded runs are cancelled.

Silence the one lint warning in this file (SC2129) by using a single redirect block in the skip summary. actionlint is now clean on it.

Deliberately unchanged: the pull_request_target trigger, the fork-only eval environment gating, the merge-ref checkout, and the tessl review run --workspace --json --threshold invocation. Those are all already correct on main.

Verification

actionlint is clean. The exit-code behaviour the error/score split relies on was checked against the real CLI, on a skill in this repo:

Case Exit code Score parseable from output Reported as
Score above threshold 0 yes (89) ✅ pass
Score below threshold 1 yes (89) ❌ below threshold
Not authenticated 1 no ⚠️ review could not be run

I also confirmed main's existing score regex does correctly match the CLI's "reviewScore": 89 field (the -i flag plus the optional underscore make review[_]?score match reviewScore), so that is left alone.

Note on the first version of this PR

I originally believed the workflow still needed the fork-secrets fix and the migration off the deprecated tessl skill review. Both had already landed on main in e56e29f, 756e8d7 and 16184c2 on 2026-07-06/07 — I was diffing against a month-old local main and did not notice. The failing check on #189 is simply a stale run from 2026-07-07T08:49Z, about 72 minutes before the fork fix merged.

That version has been discarded. What remains is the subset that is genuinely still missing, and it no longer touches the fork-gating logic — which is good, because my earlier version would have regressed 756e8d7 by requiring eval approval for same-repo PRs too.

🤖 Generated with Claude Code

Copilot AI lite review requested due to automatic review settings August 21, 2026 10:10
@trieloff
trieloff requested a review from shsteimer as a code owner August 21, 2026 10:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

trieloff commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

⚠️ Outdated — this comment was based on a mistake and no longer applies. Kept for the record. See the rewritten PR description.

The red tessl-review check on this PR is expected...

The whole premise here was wrong. I was diffing against a month-old local main, so the "old workflow" I described had in fact already been replaced upstream on 2026-07-06/07. The refs/pull/308/merge failure came from my own first version of this PR changing a trigger that was already correct on main.

That version has been discarded and this PR rebased. tessl-review now passes here, and no workflow_dispatch trigger or manual approval is needed.

When the Tessl CLI fails to run at all — auth failure, network error,
crash — the job counted it as a review failure and reported it as
"N skill(s) scored below 80%". That is misleading: the skill was never
scored. On #189 this surfaced as a red "below 80%" check on
a skill that actually scores 85%, which reads to a contributor as a
quality rejection.

Track those separately as errors and report them as "review could not be
run", with a ⚠️ row and an explanatory note in the step summary. They
still fail the job — they are just no longer confused with low scores.

Also:

- Skip cleanly with an explanatory summary when TESSL_TOKEN is not
  configured, instead of running an unauthenticated CLI and failing.
- Pin actions/checkout and tesslio/setup-tessl to commit SHAs, with
  `# vN` comments so Renovate keeps tracking them.
- Add a concurrency group so superseded runs are cancelled.
- Use a single redirect block in the skip summary, silencing the one
  actionlint/shellcheck warning (SC2129) in this file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lars Trieloff <lars@trieloff.net>
@trieloff
trieloff force-pushed the fix/tessl-review-workflow branch from ae0ae6e to 3a04a70 Compare August 21, 2026 10:41
@trieloff trieloff changed the title ci(tessl-review): fix fork PRs, drop deprecated CLI, pin checkout ci(tessl-review): distinguish tooling failures from low scores Aug 21, 2026
@trieloff
trieloff merged commit 9a3505f into main Aug 21, 2026
8 of 9 checks passed
@trieloff
trieloff deleted the fix/tessl-review-workflow branch August 21, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants