GH-98831: Refactor generate_cases.py

[gvanrossum]

Consider this afterGH-99313.

• Issue: Generate the interpreter #98831

[gvanrossum]

My expectation at this point is that converting the majority of instructions to non-legacy form (cache and stack effects) will be a big project, even if we skip opcodes needing arrays or conditional stack effects.

The good news is that it can be parallelized -- each family can be done independently from any others, so anyone who wants to can pick a family and convert it, as long as it doesn't require fixing the generator (it's probably best if I stay on top of that).

I don't know how much we need to do before we can start experimenting with "macro" opcodes (see faster-cpython/ideas#491). Possibly we can start with an example like the one Mark wrote up in the DSL definition.

[markshannon]

I won't attempt a review of this PR, but a few comments on how I'd like the tool(s) to look.

• This is a compiler, so it should be built like one: Dumb IR(s) and multiple passes.
• Any methods on IR classes should be simple query method. Put all processing in the passes.

A possible design:

• Tokenizer: Tokenize the whole input file, so that locations are correct.
• Parser: Parse instructions
• Analyzer: Group and check families, check stack effects. Compute necessary labels and tables, instruction size, etc.
• Code gen: Generate C code as stream of tokens and strings
• Formatter: Format resulting C code, inserting #line directives and whitespace where needed.

Then, when we want to generate a trace interpreter, a register interpreter, compiler front-end, etc. we can re-use the tokenizer, parser, and formatter.

[gvanrossum]

Thanks, I'll try to use that as a north star. In the meantime I am hoping to keep things working the way they were as much as possible to avoid disrupting other work. I have half the analyzer in this refactor, what's missing is a design for the IR to target.

[gvanrossum]

I'm starting over from scratch now GH-99313 is merged.