Parser stack overflow on WASI with `--with-pydebug`

[FFY00]

Bug report

Bug description:

$ python Tools/wasm/wasi.py configure-build-python -- --with-pydebug
...
$ python Tools/wasm/wasi.py make-build-python
...
$ python Tools/wasm/wasi.py configure-host -- --with-pydebug
...
$ python Tools/wasm/wasi.py make-host
...
$ wasmtime run --dir .::/ cross-build/wasm32-wasip1/python.wasm -m test.test_compile
Traceback (most recent call last):
 File "/Lib/runpy.py", line 189, in _run_module_as_main
 mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
 ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
 File "/Lib/runpy.py", line 159, in _get_module_details
 code = loader.get_code(mod_name)
 File "<frozen importlib._bootstrap_external>", line 896, in get_code
 File "<frozen importlib._bootstrap_external>", line 826, in source_to_code
 File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
MemoryError: Parser stack overflowed - Python source too complex to parse

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

NOTE

The failing tests have shifted; see #131770 (comment)

Linked PRs

• GH-131770: increase assumed WASI stack size to 131072 (wasi-sdk default) #131844

[FFY00]

I bisected this to 0142236, specifically, the following call, which uses the new _Py_ReachedRecursionLimitWithMargin function.

cpython/Tools/peg_generator/pegen/c_generator.py

Lines 382 to 386 in 4b3d5b6

	def add_level(self) -> None:
	self.print("if (p->level++ == MAXSTACK || _Py_ReachedRecursionLimitWithMargin(PyThreadState_Get(), 1)){")
	with self.indent():
	self.print("_Pypegen_stack_overflow(p);")
	self.print("}")

PYOS_STACK_MARGIN_BYTES seems a bit low, causing it to fail. In my last run, for example, I am seeing p->level at 209, which is quite far from the MAXSTACK limit. I don't know exactly how the WASM double stack model works, but I suspect that our PYOS_STACK_MARGIN value might be a bit low. I think expecting WASI to have a big enough stack for the parser to be able to parse all regular Python source files in the repo should be a fairly reasonable expectation.

cc @markshannon

[brettcannon]

So what are our options at this point?

• Play with numbers until we find one that passes the test suite (that's normally what I do when our stack protection starts failing)?
• Turn off the stack protection under WASI for what was changed (Implement stack overflow protection for webassembly #130397)?
• Rip it out as I believe @hoodmane has suggested it might not actually work as intended?
• Give up in debug builds in WASI as it's the most constant pain point and I don't know how useful it is?

I just worry the longer we don't fix #131769 the more pain it will take to get debug builds working again. Plus I don't want to go too far into the betas w/ this not working.

[FFY00]

In my opinion, I think we should disable the stack protection until WASM provides a mechanism that allow us to implement it correctly.

Due to WASM being sandboxed, stack overflows are not as big of a concern there compared to native runtimes, so I don't think we are losing a particularly large amount of value by disabling this functionality.

That said, I do understand it might be frustrating for @markshannon, as the one who implemented it, so I'd appreciate his thoughts on this.