Move the syntax error generation rules out of Grammar/python.gram into a separate grammar file

[serhiy-storchaka]

Feature or enhancement

Grammar/python.gram contains not only the Python grammar, but also the rules for generating specialized syntax error messages (the invalid_* rules and the alternatives referencing them) -- about a third of the file. They allow producing better error messages, but they are not part of the Python grammar:

• The full grammar specification in the documentation includes python.gram and has to hide the error rules with fragile regular expressions in a custom Pygments lexer (#.*invalid syntax.* up to the end of file, every alternative mentioning invalid_*).
• Alternative Python implementations which consume the grammar get the CPython-specific error machinery mixed in. Some rules exist only to work around this (e.g. invalid_parameters_helper was added in bpo-42860: Remove type error from grammar #24156 "so that alternative implementations written in statically-typed languages can use this grammar without having type errors in the way").
• The error rules obscure the grammar itself: it is hard to see which alternatives define the language and which only improve error reporting, and the order of the alternatives mixes both concerns.

I propose to split it in two files:

• Grammar/python.gram -- the pure Python grammar, without a single reference to the error rules. By itself it produces a complete parser which parses exactly the same language and only emits generic error messages.
• Grammar/python_errors.gram -- the alternatives for generating specialized syntax error messages, together with the information about where they are inserted into the rules of the main grammar.

The pegen parser generator reads both files and merges them. The error alternatives are written (with their actions) in rule extensions:

raise_stmt (extend):
    | a='raise' b='from' {
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "did you forget an expression between 'raise' and 'from'?") }
    | 'raise' expression a='from' {
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "did you forget an expression after 'from'?") }

The position of an inserted alternative is deduced automatically: it is inserted before the leftmost alternative of the base rule which can succeed by matching a proper prefix of the code matched by the inserted alternative (such alternative would shadow the specialized error if it was tried first), or after all alternatives if there is no such alternative. Alternatives which must preempt a base alternative which reports an error itself rather than failing (because it contains forced tokens or rules like block) are written before ..., which stands for the alternatives of the base rule. Several rules can be extended at once ((kwarg_or_starred | kwarg_or_double_starred) (extend):), and a rule can be extended several times. The inserted alternatives are only used in the second parsing pass, like the alternatives referencing invalid_* rules; only a dozen error rules which are shared, memoized or referenced from other alternatives remain defined as named rules.

The generated parser is not changed: the pure split reproduces Parser/parser.c byte for byte (except the generated-from comment), and the whole change has equivalent behavior, with only the order and grouping of the error alternatives changed. This was verified by a differential test which ran both parsers on a corpus of 873 invalid inputs covering every error message in the grammar (identical exception types, messages, locations) and on 157 stdlib files (identical AST, including attributes).

Linked PRs

• gh-153195: Move the syntax error generation rules into a separate grammar file #153197