pathlib.Path.glob does not follow symlinks

[BrianMSheldon]

BPO	33428
Nosy	@pfmoore, @pitrou, @tjguk, @jreese, @zware, @zooba, @emilyemorehouse, @BrianMSheldon

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-05-05.06:23:00.558>
labels = ['type-bug', 'library', 'OS-windows']
title = 'pathlib.Path.glob does not follow symlinks'
updated_at = <Date 2020-04-29.11:12:49.344>
user = 'https://github.com/BrianMSheldon'

bugs.python.org fields:

activity = <Date 2020-04-29.11:12:49.344>
actor = 'Danya.Alexeyevsky'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2018-05-05.06:23:00.558>
creator = 'brianmsheldon'
dependencies = []
files = []
hgrepos = []
issue_num = 33428
keywords = []
message_count = 4.0
messages = ['316197', '316724', '316880', '367640']
nosy_count = 9.0
nosy_names = ['paul.moore', 'pitrou', 'tim.golden', 'jreese', 'Danya.Alexeyevsky', 'zach.ware', 'steve.dower', 'emilyemorehouse', 'brianmsheldon']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33428'
versions = ['Python 3.6']

Linked PRs

• GH-77609: Add follow_symlinks argument to pathlib.Path.glob() #102616
• GH-77609: Support following symlinks in pathlib.Path.glob() #104176
• GH-77609: Add recurse_symlinks argument to pathlib.Path.glob() #117311

[BrianMSheldon]

Given a pathlib.Path that contains symlinked subfolders, Path.glob (and .rglob) do not follow symlinks. This is not consistent with glob.glob which does.

For example given the following:
C:\Folder
C:\Folder\Subfolder -> D:\Subfolder
D:\Subfolder\File.txt

pathlib.Path('C:/Folder').glob('**/*') yields the following paths:
WindowsPath('C:/Folder/Subfolder')

glob.glob('C:/Folder/**/*') yields the following paths:
'C:/Folder\Subfolder'
'C:/Folder\Subfolder\File.txt'

Notice how the contents of Subfolder are present in the glob.glob results but not for Path.glob.

I would expect Path.glob to be consistent with glob.glob. This is not the only inconsistency (e.g. bpo-22276, bpo-31202) and perhaps Path.glob should be re-implemented using glob.glob.

[jreese]

This looks like an issue specific to Windows? I can't replicate on Mac, and given Windows' method of implementing "symlinks" as junctions.

[BrianMSheldon]

Windows does not implement symlinks as junctions. Windows has hardlinks, symlinks and junctions which are all distinctly different in behaviour.

I don't doubt that this is a Windows-specific issue, although I have not tested other platforms. Path.glob and .rglob does work for junctions and hardlinks but glob.glob works consistently for all three.

[DanyaAlexeyevsky]

I can reproduce the bug with Linux and python 3.7.5:

Python 3.7.5 (default, Apr 19 2020, 20:18:17) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathlib import Path
>>> Path('a/b').mkdir(parents=True)
>>> Path('c/d').mkdir(parents=True)
>>> Path('a/c').symlink_to('../c')
>>> Path('e').symlink_to('c')
>>> list(Path('.').rglob('*'))
[PosixPath('e'), PosixPath('c'), PosixPath('a'), PosixPath('c/d'), PosixPath('a/c'), PosixPath('a/b')]

Expected result:

[PosixPath('e'), PosixPath('e/d'), PosixPath('c'), PosixPath('a'), PosixPath('c/d'), PosixPath('a/c'), PosixPath('a/c/d'), PosixPath('a/b')]

[dlukes]

Following symlinks was disabled on purpose as a fix for #70200:

cpython/Lib/pathlib.py

Line 319 in 69f6cc7

if entry_is_dir and not entry.is_symlink():

To re-enable it, we'd have to come up with a different mitigation for the symlink loops problem.

Or possibly, if hidden behind a follow_symlinks arg which would be False by default, it might be enough to just document it as a limitation when follow_symlinks=True? I don't know.

On Ubuntu 22 I have the same problem

[barneygale]

Or possibly, if hidden behind a follow_symlinks arg which would be False by default, it might be enough to just document it as a limitation when follow_symlinks=True? I don't know.

glob() still follows symlinks when matching segments like foo/, */ and so forth; it only refuses to follow symlinks when it encounters a ** wildcard. So the current behaviour is roughly follow_symlinks=Sometimes.

I realised this because I've been trying to implement an iterative version of rglob(), building upon walk() and filtering paths with a compiled re.Pattern object. walk() has its own follow_symlinks argument that accepts True and False, but neither option aligns with the existing behaviour, so it seems like a dead end unless this "bug" is "fixed".

[barneygale]

zsh has a *** wildcard that recurses into symlinks to directories, see Recursive Globbing here: https://linux.die.net/man/1/zshexpn

[barneygale]

Having thought about this some more, I'd like to propose that we add follow_symlinks arguments to glob() and rglob(), where:

• follow_symlinks=False treats symlinks as files (default)
• follow_symlinks=True follows symlinks to directories

Note that the default behaviour would therefore change: a pattern like foo/bar or */bar would no longer match foo/bar if foo is a symlink. This is consistent with how **/bar works today.

With that in place, we could implement glob() iteratively atop walk(), which would address #89727 and substantially improve performance!

Thoughts?