GH-101362: Optimise pathlib by deferring path normalisation

[barneygale]

PurePath now normalises and splits paths only when necessary, e.g. when .name or .parent is accessed. The result is cached. This speeds up path object construction by around 4x.

PurePath.__fspath__() now returns an unnormalised path, which should be transparent to filesystem APIs (else pathlib's normalisation is broken!). This extends the earlier performance improvement to most impure Path methods, and also speeds up p.joinpath('bar') and p / 'bar'. edit: will fix separately.

This also fixesGH-76846 and GH-85281 by unifying path constructors and adding an __init__() method. edit: will fix separately.

• Issue: Optimize pathlib path construction #101362

[barneygale]

Constructing path objects is up to 4x faster with one argument:

$ ./python -m timeit -n 1000000 -s 'from pathlib import PurePath''PurePath("foo/bar")' 1000000 loops, best of 5: 2.01 usec per loop # before 1000000 loops, best of 5: 495 nsec per loop # after

More than 2x faster with two arguments:

$ ./python -m timeit -n 1000000 -s 'from pathlib import PurePath''PurePath("foo", "bar")' 1000000 loops, best of 5: 2.28 usec per loop # before 1000000 loops, best of 5: 1.02 usec per loop # after

~~And ~25% faster when joining arguments:~~

[edit: no longer true! ]

$ ./python -m timeit -n 1000000 -s 'from pathlib import PurePath; p = PurePath("foo")''p.joinpath("bar")' 1000000 loops, best of 5: 1.66 usec per loop # before 1000000 loops, best of 5: 1.3 usec per loop # after

But it's 12% slower when the path needs normalization, as with str()

$ ./python -m timeit -n 1000000 -s 'from pathlib import PurePath''str(PurePath("foo/bar"))' 1000000 loops, best of 5: 2.96 usec per loop # before 1000000 loops, best of 5: 3.31 usec per loop # after

And 25% slower when when walking directories (where pathlib keeps everything normalized):

[edit: resolved! see comment]

$ ./python -m timeit -n 20 -s 'from pathlib import Path''list(Path().rglob("*"))' 20 loops, best of 5: 53.4 msec per loop # before 20 loops, best of 5: 66.5 msec per loop # after

But still faster for filesystem operations that don't require normalization:

[edit: no longer true! this can't be properly fixed until other stuff lands]

$ ./python -m timeit -n 100000 -s 'from pathlib import Path''Path("README.rst").read_text()' 100000 loops, best of 5: 26.1 usec per loop # before 100000 loops, best of 5: 21.2 usec per loop # after $ ./python -m timeit -n 100000 -s 'from pathlib import Path''Path("README.rst").exists()' 100000 loops, best of 5: 5.45 usec per loop # before 100000 loops, best of 5: 2.97 usec per loop # after

[barneygale]

I've found a couple other small optimizations which are best tackled in other PRs, so I'm marking this PR as a 'draft' for now.

[barneygale]

I've undone the change to _from_parsed_parts(), which has restored directory-walking performance:

$ ./python -m timeit -n 20 -s 'from pathlib import Path' 'list(Path().rglob("*"))' 20 loops, best of 5: 146 msec per loop # before 20 loops, best of 5: 152 msec per loop # after 

Still a tiny bit slower than pre-PR.

The rest of the speedups/slowdowns mentioned in my previous comment are still there.

[barneygale]

The change to importlib is necessary because it's relying on a bug in pathlib's path normalization:

• pathlib strips trailing slash #65238

I think I need to solve that issue first, so I'm going to mark this PR as a draft (again!)

[barneygale]

This PR has strayed too far from the original implementation. I'm going to abandon it. New PR here:

• GH-76846, GH-85281: Call __new__() and __init__() on pathlib subclasses #102789