Compute score bounds based on grading conf and warn about discrepancies

[Matistjati]

• Check that no test case gets score.txt and a verdict other than accepted
• Warn if some test data node has infinite max, but we can prove from children's ranges with aggregation that a finite bound can be found
• Warn if any test data node has negative range lower bound. In particular, this will give a warning for every node without an explicitly set range, which I think is good?
• In the new "Kattis" mode, error on objective: min

[gkreitz]

Check that score > max_score never happens at any node in the tree

What check do you think is missing? There is already a check on max (and min) score for every group (which is, AFAIK, the only level which has max scores in legacy).

[Matistjati]

I had 3 bullet points and deleted the wrong one, oops. Updated

[gkreitz]

Warn if some test data node has infinite max, but we can prove from children's ranges with aggregation that a finite bound can be found

Why should this be a warning? It's a reasonable thing to compute and make use of (i.e., if a problem has no limit on the root group, but obviously can't go above 100 due to childrens' max scores and aggregation, we would treat it as having 100 on the root when installing), but why do we want to force a problem setter to manually write files? The default is also +/-inf, so this can get very trigger happy.

I'd be more inclined to give a warning if there is a finite bound we can prove from aggregation rules cannot be reached.

Warn if any test data node has negative range lower bound. In particular, this will give a warning for every node without an explicitly set range, which I think is good?
In the new "Kattis" mode, error on objective: min

If we want to drop minimization and negative scores, why don't we just update the spec? Do we have any reason to believe there are problems out there which people care about that use these? I particularly don't want to add a warning which would be triggered by the default value for a field.

[Matistjati]

Why should this be a warning? It's a reasonable thing to compute and make use of (i.e., if a problem has no limit on the root group, but obviously can't go above 100 due to childrens' max scores and aggregation, we would treat it as having 100 on the root when installing), but why do we want to force a problem setter to manually write files? The default is also +/-inf, so this can get very trigger happy.

I hold a perhaps extreme position on this. Every single time I've seen a problem where range was omitted, it was unintentional and buggy. In the edge-case where you do want infinite max_score, it's trivial to simply specify this explicitly in the group.

IMO, one should therefore always specify the range in every node. testdata_tools already does this, and given the ambiguities in the spec (that I tigtened maybe 6 months ago), any sane generator ought to do the same. If you are hand-editing testdata.yaml files, it is very easy to get it wrong, so I prefer forcing authors to spend 2 minutes deciding this explicitly.

If we want to drop minimization and negative scores, why don't we just update the spec? Do we have any reason to believe there are problems out there which people care about that use these? I particularly don't want to add a warning which would be triggered by the default value for a field.

There are mixed opinions on backporting. Perhaps I should make an attempt. But Kattis has in fact killed off support for minimization and negative scores IIRC, and I would therefore be very happy to explicitly forbid it, so authors discover it using problemtools and not during a running contest.

[gkreitz]

Check that no test case gets score.txt and a verdict other than accepted

Lol, this actually gets a bit annoying with the mulitpass test problem I built. I was accumulating score between passes in score.txt. Now I suddenly need to rm that file on a WA. I guess the recommended approach would be to accumulate score in a different file and rename to score.txt on the final verdict (if AC).

[Matistjati]

Well, score.txt and non-AC verdict is broken in Kattis as far as I'm aware 🙂

[gkreitz]

There are mixed opinions on backporting. Perhaps I should make an attempt. But Kattis has in fact killed off support for minimization and negative scores IIRC, and I would therefore be very happy to explicitly forbid it, so authors discover it using problemtools and not during a running contest.

Well, "legacy" is the current version. I strongly dislike that we are in a position where making changes to the current version of the spec is painful (cf. Kattis/problem-package-format#585)

[gkreitz]

Well, score.txt and non-AC verdict is broken in Kattis as far as I'm aware 🙂

I'll be happy to fix that once all installed problems behave properly. :) Adding a fix that causes Judge Errors feels bad.

[gkreitz]

I added a check for score.txt in #441. Good suggestion, thanks!

IMO, one should therefore always specify the range in every node.

I don't want to require that. Your feature request points in a good direction for computing bounds based on given max scores and aggregation rules. I'll think a bit about it a bit more, but my initial take is that I want to:

• Propagate given score constraints based on aggregation (for non-custom graders)
• Warn if an unreachable bound has been set
• Warn if the root group (after propagation) has unbounded score, and is not explicitly setting range

I think that solves several issues while still being user friendly for problem setters who do not use tools.

Regarding minimization aspects of this issue, I feel like those are sort of wontfix until we've tried dropping min from the spec. If not, they are duplicate of your earlier #345.

[gkreitz]

I'll take the liberty to edit your issue description to something more descriptive for what remains.