gh-139400: Make sure that parent parsers outlive their subparsers in `pyexpat`#139403

hartwork · 2025-09-28T22:21:08Z

Issue: Heap-use-after-free issue in pyexpat related to .ExternalEntityParserCreate #139400

Within libexpat, a parser created via XML_ExternalEntityParserCreate is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected by CPython. This fixes related reference counting, to stop that from happening.

picnixz

Now, this looks like Tcl/Tk and their callbacks. If you want to check for reference leaks, you should also run the tests with -R (python -m test -R the_test)

Modules/pyexpat.c

@picnixz

Idea by @picnixz

@picnixz

Idea by @picnixz

hartwork · 2025-09-28T23:12:13Z

Now, this looks like Tcl/Tk and their callbacks.

@picnixz I'm not sure what that means 😃

If you want to check for reference leaks, you should also run the tests with -R (python -m test -R the_test)

I cannot figure out why or what to do with python -m test -R the_test. Could you spell it out for me? 🙏

picnixz · 2025-09-28T23:26:35Z

I cannot figure out why or what to do with python -m test -R the_test. Could you spell it out for me? 🙏

Sorry, to be precise the -R : option checks if there are reference leaks. In your case run:

$ ./python -m test test_pyexpat -m test_parent_parser_outlives_its_subparsers -R :

picnixz · 2025-09-28T23:27:40Z

I'm not sure what that means

Tcl/Tk is a graphic library that allows to have delayed callbacks and the callback data is the Python object. Here, for Expat, the handler data is the XML parser itself as well. I wanted to make a parallel between those two constructions.

picnixz

I'm a bit tired (it's 01:33 AM here) so the review might not be perfect. I'll have another look tomorrow.

Modules/pyexpat.c

Lib/test/test_pyexpat.py

Modules/pyexpat.c

hartwork · 2025-09-28T23:44:46Z

I cannot figure out why or what to do with python -m test -R the_test. Could you spell it out for me? 🙏
Sorry, to be precise the -R : option checks if there are reference leaks. In your case run:
$ ./python -m test test_pyexpat -m test_parent_parser_outlives_its_subparsers -R : 

@picnixz after a recompile with --with-pydebug that worked — thank you! 🙏

@picnixz

Idea by @picnixz

@picnixz

Idea by @picnixz

As requested.

hartwork · 2025-09-28T23:54:25Z

I'm a bit tired (it's 01:33 AM here) so the review might not be perfect. I'll have another look tomorrow.

@picnixz no worries, same UTC+2 here. Thank you! 👍 👍

hartwork · 2025-09-28T23:57:42Z

Related: libexpat/libexpat#1066

Misc/NEWS.d/next/Core_and_Builtins/2025-09-29-00-01-28.gh-issue-139400.X2T-jO.rst

Modules/pyexpat.c

@picnixz

As suggested by @picnixz

@picnixz

Suggested by @picnixz

Modules/pyexpat.c

@picnixz

Kudos to @picnixz

@picnixz

As suggested by @picnixz

picnixz · 2025-10-05T15:38:34Z

Lucky me, 3.14 has a clean backport :)

picnixz · 2025-10-05T15:46:01Z

@hartwork Could you make the backports for 3.10-3.13 please? I'm not on my dev session now (sorry for bothering you with this).

hartwork · 2025-10-05T15:46:36Z

@picnixz thanks for your help getting this over the finish line 🙏 🎉 I can look into the backports for this and report back if I hit any roadblocks.

picnixz · 2025-10-05T15:50:18Z

Thank you for writing libexpat :')

…rs in `pyexpat` (python#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

bedevere-app · 2025-10-05T16:01:28Z

GH-139608 is a backport of this pull request to the 3.13 branch.

…ers in `pyexpat` (GH-139403) (#139606) Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

…rs in `pyexpat` (python#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

bedevere-app · 2025-10-05T16:08:16Z

GH-139609 is a backport of this pull request to the 3.12 branch.

…rs in `pyexpat` (python#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

bedevere-app · 2025-10-05T16:21:49Z

GH-139612 is a backport of this pull request to the 3.11 branch.

bedevere-app · 2025-10-05T16:22:13Z

GH-139608 is a backport of this pull request to the 3.13 branch.

bedevere-app · 2025-10-05T16:22:19Z

GH-139609 is a backport of this pull request to the 3.12 branch.

bedevere-app · 2025-10-05T16:22:25Z

GH-139612 is a backport of this pull request to the 3.11 branch.

…rs in `pyexpat` (python#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

bedevere-app · 2025-10-05T16:27:07Z

GH-139613 is a backport of this pull request to the 3.10 branch.

…rs in `pyexpat` (python#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

bedevere-app · 2025-10-05T16:35:02Z

GH-139614 is a backport of this pull request to the 3.9 branch.

…ers in `pyexpat` (GH-139403) (GH-139608) Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

…ers in `pyexpat` (GH-139403) (#139612) * gh-139400: Make sure that parent parsers outlive their subparsers in `pyexpat` (#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd) * Move news item from section "Core and Builtins" to section "Security"

…ers in `pyexpat` (GH-139403) (#139613) * gh-139400: Make sure that parent parsers outlive their subparsers in `pyexpat` (#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd) * Move news item to from section "Core and Builtins" to section "Security"

bedevere-bot · 2025-10-07T01:50:06Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Fedora Stable Refleaks 3.11 (tier-1) has failed when building commit 1459d1f.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1027/builds/823) and take a look at the build logs.
Check if the failure is related to this commit (1459d1f) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1027/builds/823

Failed tests:

test_socket

Failed subtests:

test_aead_aes_gcm - test.test_socket.LinuxKernelCryptoAPI.test_aead_aes_gcm

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last): File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/test_socket.py", line 6557, in test_aead_aes_gcm op.sendall(plain) OSError: [Errno 22] Invalid argument

…rs in `pyexpat` (python#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early.

…rs in `pyexpat` (GH-139403) (GH-139614) Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

…ers in `pyexpat` (GH-139403) (GH-139609) Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

hartwork added 3 commits September 29, 2025 00:17

Add a regression test for issue 139400
3b0afda

Add news item for issue 139400
8f70991

bedevere-appbot added the awaiting review label Sep 28, 2025

bedevere-appbot mentioned this pull request Sep 28, 2025
Heap-use-after-free issue in pyexpat related to .ExternalEntityParserCreate#139400
Closed

picnixz self-requested a review September 28, 2025 22:45

Improve test's docstring
a63bd34

picnixz reviewed Sep 28, 2025
View reviewed changes

Modules/pyexpat.c Outdated Show resolvedHide resolved
Modules/pyexpat.c Outdated Show resolvedHide resolved

hartwork added 2 commits September 29, 2025 01:04

Simplify xmlparseobject_ to xmlparseobject
22417ec
Idea by @picnixz

Leverage Py_CLEAR
20caacd
Idea by @picnixz

hartwork requested a review from picnixz September 28, 2025 23:13

picnixz reviewed Sep 28, 2025
View reviewed changes

Modules/pyexpat.c Outdated Show resolvedHide resolved
Lib/test/test_pyexpat.py Outdated Show resolvedHide resolved
Lib/test/test_pyexpat.pyShow resolvedHide resolved
Modules/pyexpat.c Outdated Show resolvedHide resolved

hartwork added 3 commits September 29, 2025 01:45

Simplify call to Py_CLEAR
f8c58a3
Idea by @picnixz

Streamline docsting of ParentParserLifetimeTest
f1e3e36
Idea by @picnixz

pyexpat.c: Adjust whitespace around pointer
1b2937d
As requested.

hartwork requested a review from picnixz September 29, 2025 14:06

picnixz reviewed Sep 29, 2025
View reviewed changes

Misc/NEWS.d/next/Core_and_Builtins/2025-09-29-00-01-28.gh-issue-139400.X2T-jO.rst Outdated Show resolvedHide resolved
Modules/pyexpat.c Outdated Show resolvedHide resolved
Modules/pyexpat.cShow resolvedHide resolved

Re-write news item
8c5a4e8
As suggested by @picnixz

hartwork requested a review from picnixz September 29, 2025 21:04

hartwork mentioned this pull request Sep 30, 2025
[3.14] gh-90949: add Expat API to prevent XML deadly allocations (CVE-2025-59375) (GH-139234) #139359
Merged

Turn parent into a PyObject pointer
a6f5520
Suggested by @picnixz

picnixz reviewed Sep 30, 2025
View reviewed changes

Modules/pyexpat.c Outdated Show resolvedHide resolved

hartwork added 2 commits September 30, 2025 15:49

Add missing parent visit to xmlparse_traverse
4470cea
Kudos to @picnixz

Re-write comment about mission of "PyObject *parent"
3759ac7
As suggested by @picnixz

bedevere-appbot removed the needs backport to 3.13 bugs and security fixes label Oct 5, 2025

hugovk pushed a commit that referenced this pull request Oct 5, 2025
[3.14]gh-139400: Make sure that parent parsers outlive their subpars…
66ca0d9
…ers in `pyexpat` (GH-139403) (#139606) Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

bedevere-appbot removed the needs backport to 3.12 only security fixes label Oct 5, 2025

bedevere-appbot removed the needs backport to 3.11 only security fixes label Oct 5, 2025

bedevere-appbot removed the needs backport to 3.10 only security fixes label Oct 5, 2025

hartwork mentioned this pull request Oct 6, 2025
[3.9] gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat (GH-139403) #139614
Merged

Uh oh!

gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat#139403

gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat#139403

Uh oh!

Conversation

hartwork commented Sep 28, 2025• edited by bedevere-app botLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picnixz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hartwork commented Sep 28, 2025

Uh oh!

picnixz commented Sep 28, 2025

Uh oh!

picnixz commented Sep 28, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picnixz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hartwork commented Sep 28, 2025

Uh oh!

hartwork commented Sep 28, 2025

Uh oh!

hartwork commented Sep 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

picnixz commented Oct 5, 2025

Uh oh!

picnixz commented Oct 5, 2025

Uh oh!

hartwork commented Oct 5, 2025

Uh oh!

picnixz commented Oct 5, 2025

Uh oh!

bedevere-appbot commented Oct 5, 2025

Uh oh!

bedevere-appbot commented Oct 5, 2025

Uh oh!

bedevere-appbot commented Oct 5, 2025

Uh oh!

bedevere-appbot commented Oct 5, 2025

Uh oh!

bedevere-appbot commented Oct 5, 2025

Uh oh!

bedevere-appbot commented Oct 5, 2025

Uh oh!

bedevere-appbot commented Oct 5, 2025

Uh oh!

bedevere-appbot commented Oct 5, 2025

Uh oh!

bedevere-bot commented Oct 7, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gh-139400: Make sure that parent parsers outlive their subparsers in `pyexpat`#139403

gh-139400: Make sure that parent parsers outlive their subparsers in `pyexpat`#139403

hartwork commented Sep 28, 2025•
edited by bedevere-app bot
Loading

picnixz commented Sep 28, 2025•
edited
Loading