gh-118761: improve import time for secrets

[picnixz]

Benchmarks were performed on a RELEASE build (no PGO, no LTO).

PR

$ ./python -X importtime -c 'import secrets' import time: self [us] | cumulative | imported package ... import time: 236 | 236 | linecache import time: 415 | 415 | warnings import time: 698 | 698 | _hashlib import time: 133 | 133 | _blake2 import time: 226 | 358 | hashlib import time: 498 | 1968 | hmac import time: 152 | 152 | math import time: 66 | 66 | _operator import time: 252 | 318 | operator import time: 97 | 97 | itertools import time: 79 | 79 | _bisect import time: 99 | 177 | bisect import time: 70 | 70 | _random import time: 418 | 1230 | random import time: 139 | 3336 | secrets

$ hyperfine --warmup 16 "./python -c 'import secrets'" Benchmark 1: ./python -c 'import secrets' Time (mean ± σ): 7.0 ms ± 0.7 ms [User: 6.0 ms, System: 1.2 ms] Range (min … max): 6.5 ms … 14.9 ms 372 runs

Main

$ ./python -X importtime -c 'import secrets' import time: self [us] | cumulative | imported package ... import time: 190 | 190 | linecache import time: 243 | 243 | types import time: 1168 | 1411 | enum import time: 58 | 58 | _sre import time: 187 | 187 | re._constants import time: 269 | 455 | re._parser import time: 67 | 67 | re._casefix import time: 240 | 819 | re._compiler import time: 71 | 71 | itertools import time: 72 | 72 | keyword import time: 41 | 41 | _operator import time: 163 | 203 | operator import time: 101 | 101 | reprlib import time: 40 | 40 | _collections import time: 603 | 1089 | collections import time: 36 | 36 | _functools import time: 423 | 1548 | functools import time: 105 | 105 | copyreg import time: 403 | 4285 | re import time: 100 | 100 | _struct import time: 67 | 166 | struct import time: 113 | 113 | binascii import time: 387 | 4949 | base64 import time: 157 | 157 | warnings import time: 297 | 297 | _hashlib import time: 67 | 67 | _blake2 import time: 113 | 179 | hashlib import time: 113 | 744 | hmac import time: 75 | 75 | math import time: 39 | 39 | _bisect import time: 51 | 90 | bisect import time: 37 | 37 | _random import time: 335 | 535 | random import time: 105 | 6332 | secrets

$ hyperfine --warmup 16 "./python -c 'import secrets'" Benchmark 1: ./python -c 'import secrets' Time (mean ± σ): 10.9 ms ± 0.6 ms [User: 9.3 ms, System: 1.5 ms] Range (min … max): 10.2 ms … 14.7 ms 262 runs

• Issue: Improve import time of various stdlib modules #118761

[AA-Turner]

We should re-evaluate this after base64 no longer imports re.

[picnixz]

Oh yes, you're right. I just uploaded the PRs I had locally and forgot about the dependency.

[vstinner]

LGTM

[picnixz]

Since base64 will not import re anymore, we can perhaps avoid this change as I fear it won't be impactful at all. So I plan to defer this one until #128736 is merged (and re-evaluate the improvements). If there's no improvement, we can just keep the import as it is.

[vstinner]

#128736 was merged.

[picnixz]

I'm actually no more on my dev session so if someone wants to check how long import secrets now takes, I would be happy. Otherwise, I'll do it tomorrow.

[hugovk]

On macOS M2 with release+PGO+LTO.

PR with main merged

❯ ./python.exe -X importtime -c 'import secrets'import time: self [us] | cumulative | imported packageimport time: 238 | 238 | _ioimport time: 31 | 31 | marshalimport time: 324 | 324 | posiximport time: 698 | 1290 | _frozen_importlib_externalimport time: 687 | 687 | timeimport time: 255 | 942 | zipimportimport time: 59 | 59 | _codecsimport time: 655 | 714 | codecsimport time: 504 | 504 | encodings.aliasesimport time: 903 | 2120 | encodingsimport time: 211 | 211 | encodings.utf_8import time: 78 | 78 | _signalimport time: 37 | 37 | _abcimport time: 172 | 208 | abcimport time: 182 | 390 | ioimport time: 43 | 43 | _statimport time: 86 | 128 | statimport time: 832 | 832 | _collections_abcimport time: 75 | 75 | errnoimport time: 71 | 71 | genericpathimport time: 199 | 345 | posixpathimport time: 578 | 1881 | osimport time: 103 | 103 | _sitebuiltinsimport time: 238 | 238 | encodings.utf_8_sigimport time: 569 | 569 | typesimport time: 156 | 156 | importlibimport time: 211 | 211 | importlib._abcimport time: 151 | 517 | importlib.utilimport time: 42 | 42 | importlib.machineryimport time: 399 | 399 | sitecustomizeimport time: 49 | 49 | usercustomizeimport time: 1899 | 5692 | siteimport time: 409 | 409 | linecacheimport time: 474 | 474 | warningsimport time: 2950 | 2950 | _hashlibimport time: 494 | 494 | _blake2import time: 599 | 1092 | hashlibimport time: 542 | 5057 | hmacimport time: 1134 | 1134 | mathimport time: 76 | 76 | _operatorimport time: 330 | 406 | operatorimport time: 86 | 86 | itertoolsimport time: 436 | 436 | _bisectimport time: 316 | 752 | bisectimport time: 365 | 365 | _randomimport time: 742 | 3483 | randomimport time: 2877 | 11416 | secrets

❯ hyperfine --warmup 16 "./python.exe -c 'import secrets'"Benchmark 1: ./python.exe -c 'import secrets' Time (mean ± σ): 14.7 ms ± 1.0 ms [User: 10.9 ms, System: 3.1 ms] Range (min … max): 13.2 ms … 24.5 ms 164 runs Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

main

❯ ./python.exe -X importtime -c 'import secrets'import time: self [us] | cumulative | imported packageimport time: 200 | 200 | _ioimport time: 31 | 31 | marshalimport time: 265 | 265 | posiximport time: 585 | 1079 | _frozen_importlib_externalimport time: 565 | 565 | timeimport time: 237 | 802 | zipimportimport time: 49 | 49 | _codecsimport time: 541 | 590 | codecsimport time: 504 | 504 | encodings.aliasesimport time: 833 | 1926 | encodingsimport time: 222 | 222 | encodings.utf_8import time: 77 | 77 | _signalimport time: 31 | 31 | _abcimport time: 158 | 188 | abcimport time: 170 | 358 | ioimport time: 53 | 53 | _statimport time: 99 | 152 | statimport time: 831 | 831 | _collections_abcimport time: 56 | 56 | errnoimport time: 59 | 59 | genericpathimport time: 132 | 246 | posixpathimport time: 525 | 1752 | osimport time: 85 | 85 | _sitebuiltinsimport time: 290 | 290 | encodings.utf_8_sigimport time: 578 | 578 | typesimport time: 168 | 168 | importlibimport time: 222 | 222 | importlib._abcimport time: 172 | 560 | importlib.utilimport time: 45 | 45 | importlib.machineryimport time: 449 | 449 | sitecustomizeimport time: 53 | 53 | usercustomizeimport time: 2183 | 5991 | siteimport time: 488 | 488 | linecacheimport time: 564 | 564 | _structimport time: 124 | 688 | structimport time: 381 | 381 | binasciiimport time: 251 | 1319 | base64import time: 471 | 471 | warningsimport time: 3136 | 3136 | _hashlibimport time: 807 | 807 | _blake2import time: 788 | 1594 | hashlibimport time: 518 | 5717 | hmacimport time: 610 | 610 | mathimport time: 68 | 68 | _operatorimport time: 317 | 385 | operatorimport time: 73 | 73 | itertoolsimport time: 387 | 387 | _bisectimport time: 461 | 847 | bisectimport time: 334 | 334 | _randomimport time: 873 | 3121 | randomimport time: 2587 | 12742 | secrets

❯ hyperfine --warmup 16 "./python.exe -c 'import secrets'"Benchmark 1: ./python.exe -c 'import secrets' Time (mean ± σ): 15.4 ms ± 1.1 ms [User: 11.3 ms, System: 3.4 ms] Range (min … max): 14.1 ms … 26.0 ms 168 runs Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

[picnixz]

I therefore don't think it's worth the change. Less work for me and less diff with other branches. I'm going to close this PR.

[picnixz]

Thank you Hugo for running the numbers. There is still one module that I had in mind (pickletools) since it imports re but I haven't checked other modules importing re. I'll do that tomorrow if I time, otherwise this w-e.

[vstinner]

I don't think that pickletools import time matters since pickletools is a debug tool.