Uh oh!
There was an error while loading. Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork 34k
GH-116380: Speed up glob.glob() by removing some system calls#116392
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Uh oh!
There was an error while loading. Please reload this page.
Merged
Changes from all commits
Commits
Show all changes
101 commits Select commit Hold shift + click to select a range
db3c620 GH-116380: Make `glob.glob()` twice as fast
barneygale 9e1f059 Use `os.listdir()` if we don't need to check entry type.
barneygale 10432df A few small speedups.
barneygale 7e389e2 Simplify prefix removal
barneygale 8680a0a Re-implement `glob0()`, `glob1()`, and `has_magic()`.
barneygale 3bf3124 Fix errant `StopIteration`.
barneygale f8fb992 Skip compiling pattern for consecutive `**` segments.
barneygale 50ef080 Clarify regex/path building in literal and recursive selectors.
barneygale ccefacd Simplify code to ignore root_dir.
barneygale fa951f6 Fix possible Windows separator issue.
barneygale 0aec12c Address some review feedback.
barneygale 72691ba Use assignment expressions in a couple of places
barneygale c58dd21 Replace lambda with `operator.not_`.
barneygale c361ec9 Merge branch 'main' into gh-116380
barneygale 22b30db Speed up `_add_trailing_slash()`
barneygale 83b70bd Speed up `select_literal()`
barneygale 1d32d14 Speed up `select_recursive()`
barneygale 1e5aacc Merge branch 'main' into gh-116380
barneygale a038bb8 Merge branch 'main' into gh-116380
barneygale f1440a9 Cache compiled patterns rather than selectors.
barneygale 9c64643 Remove a bit of code duplication.
barneygale b0e8ba6 Fix stray newline
barneygale 1b1233e Merge branch 'main' into gh-116380
barneygale 0e02ec5 Remove tests for glob0 and glob1
barneygale be4865e Add a bunch of comments explaining the more subtle parts.
barneygale 203e8ef Merge branch 'main' into gh-116380
barneygale 13355a0 Clarify variable naming in iglob()
barneygale 2e5cebd Use keyword arguments to pass True/False/None literals, for clarity.
barneygale 5eba2eb Speed up recursive globbing very slightly
barneygale b0a99b7 Merge branch 'main' into gh-116380
barneygale ad0ece8 Implement recursive wildcards with a stack
barneygale cafe9be Add argument defaults, simplify code slightly.
barneygale 301d922 Also make rel_path optional
barneygale beb2507 Optimise _add_trailing_slash
barneygale 312c73a Remove use of os.listdir() -- doesn't generalise
barneygale ae820e2 Add `_Globber` class; prepare for merger with pathlib globbing.
barneygale dcfe11d Unify with pathlib implementation \o/
barneygale 123a0f6 Use literal selector only if no case sensitivity preference is given.
barneygale 0ed7b9c Fix a few tests
barneygale aceb85f Fix a few more tests.
barneygale b04de9d Merge commit '689ada79150f28b0053fa6c1fb646b75ab2cc200' into gh-116380
barneygale 3eb2d19 Merge branch 'main' into gh-116380
barneygale 8a15db0 Fix select() argument order.
barneygale 7eb3e61 Merge branch 'main' into gh-116380
barneygale 316ea56 Merge branch 'main' into gh-116380
barneygale 2018027 Support `include_hidden` and `dir_fd` in `pathlib._glob`.
barneygale 2f21626 Fix stray newline
barneygale 339df68 Update Lib/pathlib/_glob.py
barneygale 28aa95f Fix docs
barneygale abcb1f8 Test for unique results
barneygale 71387a6 Spacing
barneygale de22de6 Merge branch 'main' into gh-116380
barneygale 8b08374 Merge branch 'main' into gh-116380
barneygale 54efa7c Merge branch 'main' into gh-116380
barneygale cf11922 Update whatsnew
barneygale 6710924 Merge branch 'main' into gh-116380
barneygale a547cd2 Merge branch 'main' into gh-116380
barneygale 14ae438 Close file descriptors when `recursive_selector` is finalized.
barneygale 69d7a86 Make `iglob()` a generator.
barneygale 3b84a1d Make `_iglob()` a generator.
barneygale f9f9a8d Make `_relative_glob()` a generator.
barneygale 24a9ee4 Simplify skipping empty string
barneygale d05d58d Merge branch 'main' into gh-116380
barneygale 27c463e Merge branch 'main' into gh-116380
barneygale a94f2a7 Make `_GlobberBase` fully abstract.
barneygale d19bb89 Address review feedback
barneygale 1677588 Typo fix
barneygale 539f044 Speed up pattern parsing.
barneygale 70a1b42 Add test for globbing above recursion limit.
barneygale 1560712 Merge branch 'main' into gh-116380
barneygale 099e86e Apply suggestions from code review
barneygale ee76faf Test that `iglob().close()` closes file descriptors.
barneygale 4cf8a4d Address some review feedback
barneygale 8a118a7 Merge branch 'main' into gh-116380
barneygale 3ad9367 Address more review comments
barneygale 66af33d Drop parse_entry
barneygale ce74ef1 Address review feedback
barneygale a69a060 Add comment.
barneygale a10a1e0 Merge branch 'main' into gh-116380
barneygale e2e0a2e Merge branch 'main' into gh-116380
barneygale be260c1 Merge branch 'main' into gh-116380
barneygale 0d1dc7e Merge branch 'main' into gh-116380
barneygale f1c5326 Merge branch 'main' into gh-116380
barneygale e113457 Merge branch 'main' into gh-116380
barneygale d8ec2d4 Merge branch 'main' into gh-116380
barneygale 203b78b Merge branch 'main' into gh-116380
barneygale ffeb81a Windows test fixes
barneygale ec157a1 Speed up suppression of initial empty path.
barneygale 7a620db Anticipate the removal of glob0() and glob1()
barneygale e9fa3ac Merge branch 'main' into gh-116380
barneygale 46429ad Handle _initial_path_exists more explicitly in select_special()
barneygale f160e8c Add comment, remove redundant check
barneygale 47846ca Avoid slicing prefix when root_dir is None
barneygale e55d946 Replace `_initial_path_exists` sentinel with `empty` boolean arg
barneygale 47499c7 Slightly simplify recursive globbing
barneygale 849de87 Revert "Slightly simplify recursive globbing"
barneygale 10a6cf4 Split out method for scanning cwd
barneygale 413b12e Merge branch 'main' into gh-116380
barneygale acf0f9b Merge branch 'main' into gh-116380
barneygale a89211f Merge branch 'main' into gh-116380
barneygale e4c519c Merge branch 'main' into gh-116380
barneygale File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading. Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading. Please reload this page.
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -75,10 +75,6 @@ The :mod:`glob` module defines the following functions: | ||
| Using the "``**``" pattern in large directory trees may consume | ||
| an inordinate amount of time. | ||
| .. note:: | ||
| This function may return duplicate path names if *pathname* | ||
| contains multiple "``**``" patterns and *recursive* is true. | ||
| .. versionchanged:: 3.5 | ||
| Support for recursive globs using "``**``". | ||
| @@ -88,6 +84,11 @@ The :mod:`glob` module defines the following functions: | ||
| .. versionchanged:: 3.11 | ||
| Added the *include_hidden* parameter. | ||
| .. versionchanged:: 3.14 | ||
| Matching path names are returned only once. In previous versions, this | ||
barneygale marked this conversation as resolved. Show resolvedHide resolvedUh oh!There was an error while loading. Please reload this page. | ||
| function may return duplicate path names if *pathname* contains multiple | ||
| "``**``" patterns and *recursive* is true. | ||
| .. function:: iglob(pathname, *, root_dir=None, dir_fd=None, recursive=False, \ | ||
| include_hidden=False) | ||
| @@ -98,10 +99,6 @@ The :mod:`glob` module defines the following functions: | ||
| .. audit-event:: glob.glob pathname,recursive glob.iglob | ||
| .. audit-event:: glob.glob/2 pathname,recursive,root_dir,dir_fd glob.iglob | ||
| .. note:: | ||
| This function may return duplicate path names if *pathname* | ||
| contains multiple "``**``" patterns and *recursive* is true. | ||
| .. versionchanged:: 3.5 | ||
| Support for recursive globs using "``**``". | ||
| @@ -111,6 +108,11 @@ The :mod:`glob` module defines the following functions: | ||
| .. versionchanged:: 3.11 | ||
| Added the *include_hidden* parameter. | ||
| .. versionchanged:: 3.14 | ||
| Matching path names are yielded only once. In previous versions, this | ||
| function may yield duplicate path names if *pathname* contains multiple | ||
| "``**``" patterns and *recursive* is true. | ||
| .. function:: escape(pathname) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Oops, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Oops, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.