bpo-45582: Port getpath[p].c to Python

[zooba]

https://bugs.python.org/issue45582

https://bugs.python.org/issue45582

[zooba]

This is veryvery WIP, but I wanted to share it now for discussion.

My plan is to set it up so that getpath isn't actually a module, but is frozen bytecode that we execute during startup. The top of the file has the expected globals, and I'll use C code to create the dict for it. Then we exec the code, and pull the results out of it.

The next step is to write tests, which can execute it directly without having to modify C code and prove correctness/etc. Because it's all possible to stub out, we can isolate it entirely from any real filesystem and simulate whatever layout/build settings we want.

It may be logical to also merge in some of site.py's functionality, and potentially sysconfig (looking at the install schemes), but those can be deferred to a later PR.

[zooba]

Ping @vstinner

[vstinner]

If I understood correctly, this code doesn't remove getpath.c yet. It's a WIP.

[vstinner]

For me, the most important part is to write unit tests to test that the same always produce the same . The code should be stateless, but can use the current working directory, environment variables, etc.

I wrote down all I know about the "Python Path Configuration" at: https://docs.python.org/dev/c-api/init_config.html#python-path-configuration

[zooba]

If I understood correctly, this code doesn't remove getpath.c yet.

Correct. I just started removing it, and obviously everything is broken :) But I'll come back to it tomorrow.

I wrote down all I know ...

Thanks. I think I've re-discovered most of that by parsing the logic from getpath.c, but what you've got there will help me understand how you designed the config mechanism. I want to remove as much as possible, but hopefully without breaking anyone trying to use it directly.

[zooba]

@ericsnowcurrently FYI, if my change doesn't make it in, you'll want to fix this check. On Windows, executable and stdlib have the same basename in a release build (e.g. C:\Python39\{Lib//python.exe}), but different in a dev build (C:\cpython\Lib // C:\cpython\PCbuild\amd64\python.exe)

[zooba]

This file is a complete rewrite - don't try and analyse the diff. Just go to "View File" under the triple-dot menu for it instead.

[zooba]

This is definitely not ready to merge yet, even if by some chance all the tests pass, but I'm happy to have reviews and comments/discussion.

Any significant discussion should go to the bpo issue, please. Also, if I need to be pinged, please do it on bpo.

[ericsnowcurrently]

Here's a partial review. I haven't dug into the bigger pieces of this PR.

[ericsnowcurrently]

I purposefully left it -1 if we couldn't figure it out. So this change implies that we are always able to figure it out. Is that the case?

[zooba]

Given the rules we've got for figuring out if we're in a dev build or not, yes. Those rules may be insufficient, but it's all a heuristic anyway - if you put those files into a regular install, we'll assume it's a dev build.

[ericsnowcurrently]

we'll assume it's a dev build.

That's what I did at first too. However, I found that it was helpful to know if we were not able to figure it out, which is what the -1 indicates in this case.

[zooba]

I mean, we either find the build landmarks or we don't... what other checks do you have? If the build landmarks aren't there, it's not a dev build, and if they are but the stdlib isn't, it's all going to crash down anyway.

[zooba]

I'm expecting another dumb error (on my part) or two, but I'm very close to having this working.

Reviews would be appreciated! Bear in mind that I'm trying to match the current (quirky) behaviour, rather than streamline anything by changing it (yet). We can do those once we know we've got something working.

[zooba]

There was nothing helpful in that history anyway ;-)

Tests all pass now, reviews appreciated!

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @zooba for commit 5962947 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[zooba]

Found the leak! So it's ready to merge once these tests pass.