fix: remove artificial floor of 50 on risk score in scoring.py - #108
Open
danielbennett888 wants to merge 1 commit into
Open
fix: remove artificial floor of 50 on risk score in scoring.py#108danielbennett888 wants to merge 1 commit into
danielbennett888 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi, first PR here. I was looking at the scoring module and noticed the risk score calculation has a floor at 50 that makes it impossible to distinguish mildly risky code from extremely dangerous code.
In
backend/app/code/eval/scoring.py, the_score_staticmethod caps the risk penalty at 50 and floors the score at 50:This means code with 1 risk (score 50, penalty capped at 50) and code with 50 risks (score 50, same) get the same risk score. The penalty cap and the floor work together to compress all risk levels into a single value.
The fix removes the floor and raises the penalty cap so the full 0-100 range is used:
Now the scores are meaningful:
I added tests in
backend/tests/test_pr_13_scoring_risk_floor.pythat verify all these cases, including an explicit assertion that 1 risk and 50 risks produce different scores. All 6 tests pass.