Skip to content

GH-81061: Fix refcount issue when returning None from a ctypes.py_object callback#13364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kumaraditya303 merged 5 commits into from
Jan 9, 2023
Merged

GH-81061: Fix refcount issue when returning None from a ctypes.py_object callback #13364

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
85026c5
bpo-36880: Add failing test, fix to follow shortly
dgelessus May 16, 2019
2a5aeba
bpo-36880: Fix refcount issue with ctypes callbacks returning None
dgelessus May 16, 2019
611f1b6
Merge branch 'main' into fix_ctypes_callback_none_refcount
dgelessus Jan 1, 2023
fb17c68
Apply suggestions from code review
dgelessus Jan 3, 2023
66ea175
Remove manual sys.getrefcount check in favor of regrtest --huntrleaks
dgelessus Jan 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions Lib/test/test_ctypes/test_refcounts.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -97,5 +97,20 @@ def func(a, b):
f(1, 2)
self.assertEqual(sys.getrefcount(ctypes.c_int), a)

@support.refcount_test
def test_callback_py_object_none_return(self):
Comment on lines +100 to +101
Copy link
Contributor

@eryksuneryksunJan 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICT, this test is rendered pointless in 3.12 due to the adoption of PEP 683 (Immortal Objects, Using a Fixed Refcount). Because None is immortal, executing Py_INCREF(Py_None) and Py_DECREF(Py_None) will no longer modify its refcount.

Copy link
ContributorAuthor

@dgelessusdgelessusJan 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good to know. I assume this isn't implemented yet though, because as of the current state of main (447d061), this test still hard-crashes without the fix.

# bpo-36880: test that returning None from a py_object callback
# does not decrement the refcount of None.

for FUNCTYPE in (ctypes.CFUNCTYPE, ctypes.PYFUNCTYPE):
with self.subTest(FUNCTYPE=FUNCTYPE):
@FUNCTYPE(ctypes.py_object)
def func():
return None

# Check that calling func does not affect None's refcount.
for _ in range(10000):
Copy link
Contributor

@kumaraditya303kumaraditya303Jan 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Single call should be enough to trigger the refleak checker.

Copy link
ContributorAuthor

@dgelessusdgelessusJan 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately not... I tested again - apparently the refleak checker (with default settings: -m test --huntrleaks=: test.test_ctypes.test_refcounts) only detects if None's refcount is too high, not if it's too low. The latter case is only detected once None's refcount gets to zero/negative (which triggers an assertion), and that requires a thousand or so iterations of the broken code.

Copy link
Member

@pablogsalpablogsalJan 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If None refcount goes too small it will crash at some point assuming the rest of the code decrements normally

Copy link
Contributor

@kumaraditya303kumaraditya303Jan 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately not... I tested again - apparently the refleak checker (with default settings: -m test --huntrleaks=: test.test_ctypes.test_refcounts) only detects if None's refcount is too high, not if it's too low. The latter case is only detected once None's refcount gets to zero/negative (which triggers an assertion), and that requires a thousand or so iterations of the broken code.

I see, it is because of #74959.

func()

if __name__ == '__main__':
unittest.main()
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2019-05-13-11-37-30.bpo-36880.ZgBgH0.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
Fix a reference counting issue when a :mod:`ctypes` callback with return
type :class:`~ctypes.py_object` returns ``None``, which could cause crashes.
15 changes: 7 additions & 8 deletions Modules/_ctypes/callbacks.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -275,15 +275,14 @@ static void _CallPythonObject(void *mem,
"of ctypes callback function",
callable);
}
else if (keep == Py_None){
/* Nothing to keep */
Py_DECREF(keep);
}
else if (setfunc != _ctypes_get_fielddesc("O")->setfunc){
if (-1 == PyErr_WarnEx(PyExc_RuntimeWarning,
"memory leak in callback function.",
1))
{
if (keep == Py_None){
/* Nothing to keep */
Py_DECREF(keep);
}
else if (PyErr_WarnEx(PyExc_RuntimeWarning,
"memory leak in callback function.",
1) == -1){
_PyErr_WriteUnraisableMsg("on converting result "
"of ctypes callback function",
callable);
Expand Down