gh-130167: Minor textwrap.dedent() optimization#131925
Closed
Uh oh!
There was an error while loading. Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
All commit authors signed the Contributor License Agreement. |
Member
AA-Turner commented Mar 31, 2025
It seems for the benchmarks handling large text, there's a consistent slowdown. Could you plot a chart of running the function with inputs of different sizes? Perhaps take the unicodeobject file and profile the first 500 lines, then the first 1000, 1500, etc, for this PR and the version using min/max. A |
ContributorAuthor
Marius-Juston commented Mar 31, 2025
It stays a consistently slower by 5-8% except for long lines test
|
ContributorAuthor
Marius-Juston commented Mar 31, 2025
From additional testing, this way is slightly faster in the ranges where the lines are between 1-18.
after analyzing all the function documentation in the Python codebase the average documentation length seems to be the following https://gist.github.com/Marius-Juston/08c574901317ee40837ab87ce0855685:
( excluding all 1 line comments ) so technically, this implementation is slightly faster for the average documentation length ( now is this performance increase really worth it, no probably not lol ) |
Lib/textwrap.py Outdated
| l1 = min(non_blank_lines, default='') | ||
| l2 = max(non_blank_lines, default='') | ||
| margin = 0 | ||
| l1 = None |
Contributor
There was a problem hiding this comment.
Suggested change
| l1=None | |
| l1='' |
You can then skip the is None check in the loop. Similar for l2 (but you have to pick the default with care)
Lib/textwrap.py Outdated
| if line and not line.isspace(): | ||
| if l1 is None or line < l1: | ||
| l1 = line | ||
| if l2 is None or line > l2: |
Contributor
There was a problem hiding this comment.
Once you skip the if None check, this if can be an elseif
Member
I'm against this change not only because it's harder to read but also because it only offers an overall <10% speed-up. In general, we accept optimizations exceeding that threshold. |
ContributorAuthor
Marius-Juston commented Mar 31, 2025
@eendebakpt with the new changes:
Small values
Large file
So on the large files we get 0 improvement, a 13% slight improvement on the synthetic test benches and on smaller values |
ContributorAuthor
Marius-Juston commented Mar 31, 2025
Agreed that this optimization is most likely not necessary, though this is very slightly more efficient memory wise since it does not have a |
Comment on lines +435 to +453
| val = False | ||
| for i, line in enumerate(lines): | ||
| # Compute min + max concurrently + normalize others | ||
| if line and not line.isspace(): | ||
| if val: | ||
| if line < l1: | ||
| l1 = line | ||
| elif line > l2: | ||
| l2 = line | ||
| else: | ||
| val = True | ||
| l1 = l2 = line | ||
| else: | ||
| lines[i] = '' | ||
| if not val or not l1: | ||
| return '\n'.join(lines) | ||
Contributor
There was a problem hiding this comment.
Suggested change
| val=False | |
| fori, lineinenumerate(lines): | |
| # Compute min + max concurrently + normalize others | |
| iflineandnotline.isspace(): | |
| ifval: | |
| ifline<l1: | |
| l1=line | |
| elifline>l2: | |
| l2=line | |
| else: | |
| val=True | |
| l1=l2=line | |
| else: | |
| lines[i] ='' | |
| ifnotvalornotl1: | |
| return'\n'.join(lines) | |
| l1='z' | |
| l2='' | |
| fori, lineinenumerate(lines): | |
| # Compute min + max concurrently + normalize others | |
| iflineandnotline.isspace(): | |
| ifline<l1: | |
| l1=line | |
| ifline>l2: | |
| l2=line | |
| else: | |
| lines[i] ='' | |
| ifnotl1: | |
| return'\n'.join(lines) | |
| margin=0 | |
| formargin, cinenumerate(l2): | |
| ifc!=l1[margin] orcnotin' \t': | |
| break | |
This saves chechking the val. Performance gain is only a bit though.
I agree with the other reviewers: unless we find out a way to make this much faster, the small gains are not worth the change.
@Marius-Juston Thanks for putting in the work to get all the performance numbers!
ZeroIntensity requested changes Mar 31, 2025
Member
ZeroIntensity left a comment
ZeroIntensity left a comment There was a problem hiding this comment.
Thanks for getting the benchmarks, sometimes its hard to get contributors to quote numbers, and you've gone above and beyond with benchmarking :).
But unfortunately, the numbers make me -1 on this.
min/maxare builtins that are written in C; they're pretty fast, especially compared to multiple lines of Python code. If we're concerned about those being slow, let's try to optimize those on the C level instead.- There is a significant amount of added complexity here, for seemingly not much benefit.
- Optimizing Python code by shifting things around is generally not a good idea, because when the interpreter changes, those optimizations become obsolete. We should definitely not be quoting memory usage, number of (pointer) copies, or number of lookups; those are all things that change on an interpreter-wide level.
I've learned that it can be hard to know when to stop painting. I think we should put down the brush, and enjoy the optimization that's already landed 🙂.
ContributorAuthor
Marius-Juston commented Mar 31, 2025
Makes sense. I am just having so much fun doing this it is definitely hard to stop lol! |
ContributorAuthor
Marius-Juston commented Mar 31, 2025
Closing this down then! ( I will be putting a feature request for the |
Member
I think it's already been proposed but rejected so search through issues and https://discuss.python.org/c/ideas/6 first. |
ContributorAuthor
Marius-Juston commented Mar 31, 2025
I think I found it, the https://discuss.python.org/t/minmax-function-alongside-min-and-max/2834/73 issue got closed down. It seems like it got stale and just ignored? A shame. |
Member
AA-Turner commented Mar 31, 2025
Feel free to open a new thread referencing the one you found. Personally I'd suggest A |
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Minor optimization to @AA-Turner#131792 where you compute the min and max of the non_blank_lines simultaneously, removes additional use of
line.isspace()as well.reuses #130167