Heuristically detect if a .interaction sample is inverted

[Matistjati]

Currently, it's easy to mess this up. Example: https://open.kattis.com/problems/absolutedistanceplan

The thing that makes it so hard is that we don't make it clear whether the sample interaction is from the perspective of the judge or the contestant (without reading the spec, which not many authors do). It would be nice if we could give some warning without having a too-high false positive rate.

Some heuristics:

• The judge rarely starts a line with ? or !
• The interaction typically starts with the judge writing something and ends with the contestant writing something. (This is much less reliable, though)

[gkreitz]

The judge rarely starts a line with ? or !

This sounds extremely specific to the particular problem you just linked

The interaction typically starts with the judge writing something and ends with the contestant writing something. (This is much less reliable, though)

This heuristic is probably the right one. Run the validator < /dev/null and see if it prints something. If so, assume the validator should print first.

[Matistjati]

I think we should also complain if the .interaction file contains leading spaces. It's not too uncommon to see

> 5
< ? 1 2 3 4 5
> 8
< ? 3 1 5 2 4
> 0
< ! 3 1 5 2 4

Which is "clearly" wrong.

[mzuenni]

The interaction typically starts with the judge writing something and ends with the contestant writing something. (This is much less reliable, though)

This heuristic is probably the right one. Run the validator < /dev/null and see if it prints something. If so, assume the validator should print first.

the validator might answer with somthing like -1 to indicate that the read data (from /dev/null) is invalid

[gkreitz]

the validator might answer with somthing like -1 to indicate that the read data (from /dev/null) is invalid

For sure. The point here was to come up with a heuristic. I think what I proposed is good enough to use to set a warning.

If it's too common that interactive validators output a message if fed just an EOF (to the point where my proposed heuristic does more harm than good), I suspect this will be a wontfix unless someone comes up with a good idea. I cannot see any rules for content I would trust as a heuristic.

[mzuenni]

the validator might answer with somthing like -1 to indicate that the read data (from /dev/null) is invalid

For sure. The point here was to come up with a heuristic. I think what I proposed is good enough to use to set a warning.

If it's too common that interactive validators output a message if fed just an EOF (to the point where my proposed heuristic does more harm than good), I suspect this will be a wontfix unless someone comes up with a good idea. I cannot see any rules for content I would trust as a heuristic.

you could also run submissions with /dev/null and see if they print something (that is obviously also brittle because there are problems where the submission can already write stuff without needing the first input from the judge)... but those heuristics could be combined to improve reliability.

Or you could let the output validator actually talk with an (accepted) submission and see who prints the last message. I guess that is quite robust?