3.12 setrecursionlimit is ignored in connection with `@functools.cache`

[PeterLuschny]

Bug report

A minimal example:

# fib.py 
import sys
sys.setrecursionlimit(2000)

from functools import cache

@cache
def fib(n):
 if n<1: return 0
 if n==1: return 1
 return fib(n-1) + fib(n-2)

print(fib(500))

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

Edit: This (simpler) example is from @dimpase .

Linked PRs

• GH-112215: Increase C recursion limit for non debug builds #113397
• [3.12] GH-112215: Increase C recursion limit for non debug builds (GH-113397) #113403
• [3.12] GH-112215: Backport C recursion changes #115083

[dimpase]

also tested on Linux (with the call to sys.getwindowsversion() removed).

As well, there is no discrepancy with 3.11 vs 3.12 if @cache is removed. So the working hypothesis is that @cache is to blame for the regression.

The current master is bad too.
Bisecting currently, last year things were still OK.

In my testing I modified the lines

 sys.set_int_max_str_digits(5000)
 print(sys.version_info, sys.getwindowsversion())

I removed the 1st one (it's very recent), and only left

 print(sys.version_info)

[dimpase]

I found the 1st bad commit, 7644935, by bisecting:

76449350b3467b85bcb565f9e2bf945bd150a66e is the first bad commit
commit 76449350b3467b85bcb565f9e2bf945bd150a66e
Author: Mark Shannon <mark@hotpy.org>
Date: Wed Oct 5 01:34:03 2022 +0100

 GH-91079: Decouple C stack overflow checks from Python recursion checks. (GH-96510)

 Include/cpython/pystate.h | 16 ++++-
 Include/internal/pycore_ceval.h | 21 +++---
 Include/internal/pycore_runtime_init.h | 2 +-
 Lib/test/support/__init__.py | 5 +-
 Lib/test/test_ast.py | 6 +-
 Lib/test/test_call.py | 38 ++++++++++
 Lib/test/test_collections.py | 2 +-
 Lib/test/test_compile.py | 3 +-
 Lib/test/test_dynamic.py | 8 +--
 Lib/test/test_exceptions.py | 8 +--
 Lib/test/test_isinstance.py | 12 ++--
 Lib/test/test_marshal.py | 3 +-
 .../2022-09-05-09-56-32.gh-issue-91079.H4-DdU.rst | 3 +
 Modules/_testinternalcapi.c | 4 +-
 Parser/asdl_c.py | 9 +--
 Python/Python-ast.c | 9 +--
 Python/ast.c | 9 +--
 Python/ast_opt.c | 9 +--
 Python/ceval.c | 81 +++++++++++++++-------
 Python/pystate.c | 5 +-
 Python/symtable.c | 9 +--
 Python/sysmodule.c | 2 +-
 22 files changed, 165 insertions(+), 99 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Core and Builtins/2022-09-05-09-56-32.gh-issue-91079.H4-DdU.rst

[dimpase]

@markshannon - greetings from Oxford, by the way :-)

[sunmy2019]

Yes. I was suggesting exposing an API for the C recursion limit, but not receiving active feedback.

See the discussion here: #107263

[dimpase]

@PeterLuschny - basic Fibonacci

# fib.py 
import sys
sys.setrecursionlimit(2000)

from functools import cache

@cache
def fib(n):
 if n<1: return 0
 if n==1: return 1
 return fib(n-1) + fib(n-2)

print(fib(500))

shows the bug just as well

[dimpase]

it's not "probably in connection with @cache", it's definitely

[dimpase]

The bug is triggered by the use of the C implementation of _lru_cache_wrapper, in Modules/_functoolsmodule.c.
To see this, disable it (then Python code will be used) by applying

--- a/Lib/functools.py
+++ b/Lib/functools.py
@@ -641,7 +641,7 @@ def cache_clear():
 return wrapper
 
 try:
- from _functools import _lru_cache_wrapper
+ from _functools import no_no_lru_cache_wrapper
 except ImportError:
 pass

and see ./python <fib.py computing the answer, instead of erroring out.

In a way, it's to be expected, as C code isn't governed by setrecursionlimit any more in 3.12+.