Reproducible pyc: FLAG_REF is not stable.

[methane]

BPO	34093
Nosy	@vstinner, @benjaminp, @methane, @ericsnowcurrently, @serhiy-storchaka, @ctismer, @yan12125
PRs	gh-78214: marshal: Stabilize FLAG_REF usage #8226 bpo-34093: Stablize FLAG_REF usage (two-pass version) #8293 gh-78274: Produce consistent output from marshal.dumps(). #28379
Files	bm_marshal.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-07-11.10:36:33.741>
labels = ['interpreter-core', 'type-bug', '3.11']
title = 'Reproducible pyc: FLAG_REF is not stable.'
updated_at = <Date 2021-09-20.15:21:12.255>
user = 'https://github.com/methane'

bugs.python.org fields:

activity = <Date 2021-09-20.15:21:12.255>
actor = 'eric.snow'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2018-07-11.10:36:33.741>
creator = 'methane'
dependencies = []
files = ['47683']
hgrepos = []
issue_num = 34093
keywords = ['patch']
message_count = 14.0
messages = ['321435', '321448', '321521', '321523', '321524', '321527', '321528', '321529', '321611', '321622', '347970', '401979', '402236', '402244']
nosy_count = 7.0
nosy_names = ['vstinner', 'benjamin.peterson', 'methane', 'eric.snow', 'serhiy.storchaka', 'Christian.Tismer', 'yan12125']
pr_nums = ['8226', '8293', '28379']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34093'
versions = ['Python 3.11']

[methane]

PR-8226 makes marshal two-pass. It may have small overhead.

In case of compiling module, marshal performance is negligible.
But how in other cases? Should this change optional?

And should we backport this to Python 3.7?
Or should distributors cherrypick this?

[methane]

marshal: Mean +- std dev: [master] 123 us +- 7 us -> [patched] 173 us +- 2 us: 1.41x slower (+41%)
compile+marshal: Mean +- std dev: [master] 5.28 ms +- 0.02 ms -> [patched] 5.47 ms +- 0.34 ms: 1.04x slower (+4%)

[serhiy-storchaka]

Look also at alternate patches for bpo-20416. Some of them can solve this problem for simple types. If they have better performance, using them for simple types could save a time. But this will complicate a code, and I'm not sure it is worth.

[vstinner]

According to Serhiy Storchaka, currently marshal.dumps() writes frozenset in arbitrary order, and so frozenset serialization is not reproducible:
https://mail.python.org/pipermail/python-dev/2018-July/154604.html

[vstinner]

What is the time spent in marshal.dumps() at Python startup when Python has to create all .pyc files? For example "./python -c pass" in the master branch with no external dependency? My question is if the PR makes Python startup 5% slower or less than 1% slower.

[methane]

STINNER Victor <vstinner@redhat.com> added the comment:

According to Serhiy Storchaka, currently marshal.dumps() writes frozenset in arbitrary order, and so frozenset serialization is not reproducible:
https://mail.python.org/pipermail/python-dev/2018-July/154604.html

PYTHONHASHSEED can be used to stable frozenset order.

On the other hand, refcnt based approach is more unstable.
Even when x is y, dumps(x) == dumps(y) is not guaranteed.

[methane]

STINNER Victor <vstinner@redhat.com> added the comment:

What is the time spent in marshal.dumps() at Python startup when Python has to create all .pyc files? For example "./python -c pass" in the master branch with no external dependency? My question is if the PR makes Python startup 5% slower or less than 1% slower.

When startup, Python does more than compile()+marshal.dumps().
And as I wrote above, it makes compile()+marshal.dumps() only 4% slower.
So startup must not be slower than 4%.

Additionally, it happens only once if pyc can be writable.
(I don't know if marshal.dumps() is called when open(cache_path, 'wb') failed)

[vstinner]

So startup must not be slower than 4%.

I know. But Python does more than compile()+dumps() at the first run. I'm curious if it is feasible to measure this cost. But it may be hard to get reliable benchmarks, since I expect that the difference will be very small, and I know very well that measuring Python startup is hard since it depends a lot of on the filesystem which is hard to measure.

[ctismer]

Why must this become slower?

To my knowledge, many projects prefer marshal over pickle
for suitable simple objects because it is
so very fast. I would not throw that away:

Would it not be easy to add a named optional keyword
argument, like "stable=True"?

[methane]

Would it not be easy to add a named optional keyword
argument, like "stable=True"?

My pull request did it.

But for now, I get hint on ML and overwrote my PR with another way: Use FLAG_REF for all interned strings.

[vstinner]

According to Serhiy Storchaka, currently marshal.dumps() writes frozenset in arbitrary order, and so frozenset serialization is not reproducible: https://mail.python.org/pipermail/python-dev/2018-July/154604.html

I created bpo-37596 "Reproducible pyc: frozenset is not serialized in a deterministic order" to track this issue.

[ericsnowcurrently]

FYI, I unknowingly created a duplicate of this issue a few days ago, bpo-45186, and created a PR for it: #28379. Interestingly, while I did that PR independently, it has a lot in common with Inada-san's second PR.

My interest here is in how frozen modules can be affected by this problem, particularly between debug and non-debug builds. See bpo-45020, where I'm working on freezing all the stdlib modules imported during startup.