Benchmark importing all standard library modules

[hauntsaninja]

Note this will change if we add or remove modules from the standard library. We could alternatively hardcode a list of standard library modules.

[mdboom]

This is a good idea in general -- the standard library is likely to cover a lot of ground, and is definitely "real code people actually use".

However, this benchmark is likely to change even when the contents of the stdlib changes (not just when files are added/removed). So I think the only way to make a stable benchmark out of this would be to actually include a copy of the stdlib here. That's probably reasonably forward-compatible to new versions of Python (at least as much as any random library).

Given that would be kind of a weird thing to do, I guess I'd like to get some other feedback to see what others think before merging this.

[hauntsaninja]

As a user, I definitely care about stdlib import time, so in my mind that was maybe more of a feature.
But maybe instead we should make this test import a few version-pinned popular third party dependencies; this will reduce noise from incidental changes and weight in favour of stdlib modules that are widely used.

[mdboom]

You make some good points. I think there's a couple of different frames of reference here. If the goal is "how fast is CPython at importing all this code?", copying the code makes sense. If the goal is "how fast does the stdlib take to import for CPython version X.Y", this PR as-is makes a lot of sense. I suppose as long as we document what it's good for to avoid confusion, that's fine. (We don't really have a good place to document those kind of things yet, but a comment in the run_benchmark.py would be good enough for now).

[mdboom]

If the comment I suggested makes sense to you, I think this is good to merge. Feel free to edit it if it's not clear, etc.

[mdboom]

Suggested change

	importos
	"""
	Measuresthetimeittakestoimportallofthemodulesinthestdlib.
	Thisbenchmarkisexpectedtochangeasthestdlibchanges, soisnot
	suitableformeasuringcompilationtime.
	"""
	importos

[hugovk]

Let's merge this.

@hauntsaninja Include the suggested change if you like, otherwise let's just merge.

[AA-Turner]

Agree with Hugo.

One proposed change that might reduce benchmark variance, as stdlib_module_names is an unsorted set:

[AA-Turner]

Suggested change

	withopen(main, "w") asf:
	f.write("""
	importimportlib
	importsys
	forminsys.stdlib_module_names:
	ifmin{"antigravity", "this"}:
	continue
	try:
	importlib.import_module(m)
	exceptImportError:
	pass
	""")
	modules_to_import=sorted(sys.stdlib_module_names-{'antigravity', 'this'})
	withopen(main, 'w', encoding='utf-8') asf:
	formoduleinmodules_to_import:
	f.write(f"""
	try:
	import{module}
	exceptImportError:
	pass
	""")