bpo-45949: Pure Python freeze module for cross builds (GH-29899)

[tiran]

Use _bootstrap_python interpreter and pure Python implementation of
freeze_module to generate frozen byte code files. Only importlib
bootstrap files are generated with Programs/_freeze_module.

This simplifies cross building, as the build system no longer needs a
_freeze_module binary. A standard Python installation with same
version is sufficient.

Signed-off-by: Christian Heimes christian@python.org

https://bugs.python.org/issue45949

[gvanrossum]

Where would the frozen importlib files come from? Are you adding those back to git? (Otherwise I don't see how this helps.)

[tiran]

Where would the frozen importlib files come from? Are you adding those back to git? (Otherwise I don't see how this helps.)

• For non-cross builds
  • Python/frozen_modules/importlib._bootstrap* are generated with Program/_freeze_module. The files are needed to create _bootstrap_python.
  • other Python/frozen_modules/*.h are generated with _bootstrap_python and freeze_module.py
• For cross builds all Python/frozen_modules/*.h are generated with an host Python interpreter (must have same version) and freeze_module.py. Cross builds can neither use ./_bootstrap_python nor Program/_freeze_module from the current build because all files are for a different architecture or platform.

[gvanrossum]

Cross builds don't just need the same version -- they need to use the same commit, unless we're in beta or beyond.

I'm not sure what's the point of ever using freeze_module.py for non-cross builds. Is it just so that freeze_module.py gets exercised regularly?

[tiran]

Cross builds don't just need the same version -- they need to use the same commit, unless we're in beta or beyond.

Good point, I'll update the documentation.

I'm not sure what's the point of ever using freeze_module.py for non-cross builds. Is it just so that freeze_module.py gets exercised regularly?

Yes, you are correct. It is not necessary to use freeze_module.py for standard builds. It is the easiest way to test that the code does not regress. I'll add a comment.

[tiran]

The main point of the PR is to get rid of ./configure --with-freeze-module=Programs/_freeze_module again. It makes cross compiling more complicated. Users need to provide the helper tool for the build platform.

This patch removes the dependency. Users will be able to cross build a 3.11 interpreter with a 3.11 host Python (once byte code and API has stabilized in beta).

[kumaraditya303]

How about committing the importlib bootstrap related files to git and generate the rest with pure python version on both standard and cross builds provided the bytecode is same, this would simplify things a lot ?

[tiran]

How about committing the importlib bootstrap related files to git and generate the rest with pure python version on both standard and cross builds provided the bytecode is same, this would simplify things a lot ?

In the past we had the importlib bootstrap files in git. The approach has the downside that it increases the size of git history a lot. Really a lot a lot.

The files are fairly large, about 350k in total. They also must be regenerated every time the source Python files are updated or any aspect of the byte code generator and optimizer is changed. Even a tiny change results in a large diff, often affecting all lines of the header files.

[kumaraditya303]

In the past we had the importlib bootstrap files in git. The approach has the downside that it increases the size of git history a lot. Really a lot a lot.

Yes, it will increase the size of git history but does anyone clones the repo in full ?, for builds most would do a shallow clone.

The files are fairly large, about 350k in total. They also must be regenerated every time the source Python files are updated or any aspect of the byte code generator and optimizer is changed. Even a tiny change results in a large diff, often affecting all lines of the header files.

For diff, these files can be marked as linguist-generated then github would suppress them in PRs

[tiran]

Yes, it will increase the size of git history but does anyone clones the repo in full ?, for builds most would do a shallow clone.

I have a full clone and usually everybody else with a local checkout has a full clone as well.

[arhadthedev]

For diff, these files can be marked as linguist-generated then github would suppress them in PRs

It opens a great possibility to introduce a vulnerability into the Python codebase. Huge (especially suppressed) diffs are hard to analyze so they allow alterations of a generated file that will go unnoticed.

In addition, GitHub is just a part of a development conveyor. Before it there ara local non-Github tools (git diff/gitk, TortoiseGit, Sublime Merge, and so on) that do not support linguist-generated=true extension.

[...] but does anyone clones the repo in full?

Actually this is the very purpose of Git as a distributed version control system, to distribute full clones and work with them offline.

[kumaraditya303]

It opens a great possibility to introduce a vulnerability into the Python codebase. Huge (especially suppressed) diffs are hard to analyze so they allow alterations of a generated file that will go unnoticed.

If generated files were to be added to the repo then it would have been verified on CI that the generated files and files generated on CI are equal.

[tiran]

If generated files were to be added to the repo then it would have been verified on CI that the generated files and files generated on CI are equal.

You would also have to make sure that non of the validating tools rely on the generated files.

[arhadthedev]

If generated files were to be added to the repo then it would have been verified on CI that the generated files and files generated on CI are equal.

If bots are/will be involved, then I agree and retract my note on vulnerability planting.

In fact, I am aware that bootstrapping in a bare environment with no predecessor (so _bootstrap_python minimal interpreter is introduced) is a can of worms that force hardly bearable compromises.

[arhadthedev]

Indeed, maintainability of the diffs is still a question. Even with suppression, unanalysable black boxes carried along with actual proposed changed is not a good idea.

[tiran]

I have added more comments to explain the bootstrap process.

Sorry for the squash merge + force push. I ran into merge conflicts and eventually gave up. Merge was less work.

[ericsnowcurrently]

Mostly LGTM.

I've left a few minor comments. The only matter of consequence is about make rule dependencies in the cross-compiling case, as well as running make regen-frozen as one of the steps for a cross-compiled build.

I'm approving the PR under the assumption that you'll resolve my concerns before merging. 🙂

[ericsnowcurrently]

Instead of "manually", can this be generated now? (added to Tools/scripts/freeze_modules.py)

[tiran]

getpath is not like the others. It would need special handling in Tools/scripts/freeze_modules.py, too. Let's tackle the getpath target in another PR. I only moved existing block to a different location so it is closer to other early bootstrap blocks.

[ericsnowcurrently]

Would you mind adding a short note here (or even a TODO) about the relationship with Tools/freeze/freeze.py?

[tiran]

Good idea!

[ericsnowcurrently]

and sorry for the delay in getting you a review!

[tiran]

@ericsnowcurrently Are you satisfied with the current state of the patch?

[ericsnowcurrently]

LGTM

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows8.1 Non-Debug 3.x has failed when building commit eb483c4.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/405/builds/831) and take a look at the build logs.
4. Check if the failure is related to this commit (eb483c4) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/405/builds/831

Summary of the results of the build (if available):

== Tests result: ENV CHANGED ==

400 tests OK.

10 slowest tests:

• test_peg_generator: 2 min 30 sec
• test_mmap: 2 min 25 sec
• test_io: 2 min
• test_multiprocessing_spawn: 1 min 59 sec
• test_concurrent_futures: 1 min 43 sec
• test_largefile: 1 min 38 sec
• test_mailbox: 1 min 13 sec
• test_asyncio: 59.8 sec
• test_unparse: 45.5 sec
• test_socket: 43.0 sec

1 test altered the execution environment:
test_asyncio

30 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_wait3 test_wait4
test_xxtestfuzz test_zipfile64

Total duration: 10 min 8 sec

Click to see traceback logs

Traceback (most recent call last): File "D:\buildarea\3.x.ware-win81-release.nondebug\build\Lib\asyncio\proactor_events.py", line 116, in __del__self.close() ^^^^^^^^^^^^ File "D:\buildarea\3.x.ware-win81-release.nondebug\build\Lib\asyncio\proactor_events.py", line 108, in closeself._loop.call_soon(self._call_connection_lost, None) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\buildarea\3.x.ware-win81-release.nondebug\build\Lib\asyncio\base_events.py", line 745, in call_soonself._check_closed() ^^^^^^^^^^^^^^^^^^^^ File "D:\buildarea\3.x.ware-win81-release.nondebug\build\Lib\asyncio\base_events.py", line 506, in _check_closedraiseRuntimeError('Event loop is closed') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^RuntimeError: Event loop is closed k