Doc tests action takes too long

[Privat33r-dev]

Issue

Usually doctest's action's "Run documentation doctest"¹ job takes around 12 minutes² to execute. Most of this time is taken by the internal framework doctest.py³. The cause of the issue is GC regression in Python 3.13.

Potential Solution

Fix #124567

Additional Improvements

Improve doctest performance

Since the tests are not interconnected (between different doc files), they can be easily parallelized. I suggest using ProcessPoolExecutor that will be suitable to "bypass" GIL (assuming that tasks are CPU-bound). Github runners have 4 cores available⁴, so, potentially, it can speed up execution by up to 4 times.

@AlexWaygood @erlend-aasland

Footnotes

1. https://github.com/python/cpython/blob/7ac933e2609b2ef9b08ccf9c815b682b0e1ede2a/.github/workflows/reusable-docs.yml#L108 ↩

2. https://github.com/python/cpython/actions/runs/8922903843/job/24506021604 ↩

3. https://github.com/python/cpython/blob/main/Lib/doctest.py ↩

4. https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories ↩

[erlend-aasland]

Sounds good; do you have a PR ready? :)

[Privat33r-dev]

I was waiting for a feedback because I might've missed something :) My schedule is a bit tight lately, but I will try to find a time to implement it (hopefully in a week or two).

[Privat33r-dev]

Sorry, it's going to be delayed. Currently I am loaded with work, but hopefully soon I can start work on the PR.

[hugovk]

No rush, thanks for the update :)

[willingc]

@Privat33r-dev Do you still wish to work on a PR for this? If you are busy, would you mind someone picking up the work to do the PR?

[Privat33r-dev]

I will try to do it within a week, if I can't manage to find time for this, then it will be a good idea for another person to do it. Sorry for the unexpected delay.

[willingc]

No worries on a delay @Privat33r-dev. Seems like it would be a nice addition since the doctests are slow 😉

[Privat33r-dev]

After further investigation I understood the following.
The command to run doctests that takes around 13-16 minutes is:
xvfb-run make -C Doc/ PYTHON=../python SPHINXERRORHANDLING="-W --keep-going" doctest
xvfb-run is needed to simulate graphic environment (my guess it is needed to test turtle and some other modules)
make -C Doc/ doctest is launching Doc/Makefile with doctest builder. Other flags are less relevant, but in short they point out to the Python installation and how the sphinx should handle errors.

Inside of the Makefile, the sphinx-build is being executed with the doctest builder.

So, basically, the sphinx wrapper for the doctests is used. Source

Likely the parallelization should be applied there, I will continue and I might update this comment/add a new comment with further findings.

---

However, what's actually interesting is that speed of the doctest depends on Python version.
Python 3.13 tests are running 3 times slower than 3.12.
Example #1 (Python 3.13) 14m 57s test that ran yesterday
Example #2 (Python 3.12) 5m 18s test that also ran yesterday

Search request:
https://github.com/python/cpython/pulls?q=is%3Apr+label%3Adocs+%5B3.12%5D+

---

Also caching does not work efficiently and it might contribute to the problem, but it's a different topic.

[Privat33r-dev]

I would be glad if someone can join in resolving this issue since currently I am, unfortunately, still occupied by other business. At this weekend I will try to continue.

General conclusion from earlier findings:

1. The multiprocessing change is needed to be done in the Sphinx, rather than in the Python's core library. However, due to the speed degradation comparing to earlier Python version, I think that it would be highly beneficial to investigate it first, instead of just masking the issue by applying multiprocessing.
2. There is a major speed degeneration in Python 3.13 doctest on GH Runner comparing to Python 3.12. I have a few theories (all require further testing/findings to be confirmed/rejected):
   1. Due to some changes, 3.13 is slower on some operations. I just hope that it's not PEP 703-related. How to check: doctest.py performance tests with profiling on both versions.
   2. There were some changes in the doctest library that cause speed loss. How to check: diff doctest.py between both versions.

I can't think of another reason quickly, but it might be as well caused by some Python API changes, building flags or other thing. However, if we can fix and return to near-3.12 speed, it would be a major win already. Applying multiprocessing to it would just skyrocket the speed.

The Plan

1. Investigate the performance issue
2. https://github.com/sphinx-doc/sphinx/blob/master/sphinx/ext/doctest.py#L363C1-L368C1 -> multiprocessing

[hugovk]

@AA-Turner Do you have any ideas?

The doctest on CPython CI is much slower on 3.13 and 3.14 (main) than with 3.12. It's not specifically doctest, but the main Sphinx build is slow before starting the doctest.

For example:

• 3.12: 3m 50s https://github.com/python/cpython/actions/runs/11018010739/job/30597320522?pr=124441
• 3.13: 14m 4s https://github.com/python/cpython/actions/runs/11018009417/job/30597311746?pr=124440

With the example builds, when the full 3.12 Sphinx build + doctest had finished, 3.13 was only at reading sources... [ 73%] library/stdtypes .. library/trace.

[AA-Turner]

Didn't we previously reason that this is as pre-releases don't use an optimised build of Python? What happens if we run CI with the released rc2?

A

[hugovk]

For both of these, we use non-optimised Python built from the branch -- because we're testing the branch's Python.

https://github.com/python/cpython/blob/f4997bb3ac961d6aaf07ce650cd074e28ce6ccd0/.github/workflows/reusable-docs.yml#L95_L118

[willingc]

Hmm...I'm wondering if we use pytest to run these doctests instead of sphinx build plus doctest. We could then parallelize the doctests if needed. @hugovk @AA-Turner

[AA-Turner]

For both of these, we use non-optimised Python built from the branch -- because we're testing the branch's Python.

Right, I was just wondering if the slowdown was soley due to this (which we have no real choice over) or if other factors in Python 3.13+ have caused Sphinx to slow down (which we might be able to resolve).

I'm wondering if we use pytest to run these doctests instead of sphinx build plus doctest.

I don't think pytest supports test{setup,cleanup,code} directives which we make heavy use of.

A