Be more purposeful in the Python version in `pyvenv.cfg`

[brettcannon]

In pyvenv.cfg, there's a version field that records ".".join(sys.version_info[:3]). The interesting thing about this is it isn't loose enough to reflect the version accurately if the Python interpreter is updated in-place (e.g. going from Python 3.11.1 to 3.11.2). But it's also not accurate enough to reflect the exact Python version (e.g. Python 3.12.0a6 is recorded as 3.12.0).

Because of this I'm not sure what the number is meant to represent. If it's meant to record what the version was when the virtual environment was created then we should either change the field to record the release level or add another field that records that. But if it's just meant to record what the version might be, then it's a bit too specific.

[brettcannon]

/cc @vsajip

[vsajip]

Perhaps just add the release level in another field, to avoid breakage in case someone's code is parsing the current version field and expecting it to always have 3 components.

[brettcannon]

One interesting wrinkle is the version number is incomplete at creation time, but it can also become inaccurate even today if you upgrade your Python version in-place since the symlinks are only major.minor specific (e.g. 3.11.2 to 3.11.3). As such, this really acts as a recording of what was used when the virtual environment was created.

Regardless, making the release level a separate key/field is fine with me!

[FFY00]

Regardless, making the release level a separate key/field is fine with me!

Do you mean adding a new key that reflects the version of the current interpreter?

If so, do you have an use-case for it?

[vsajip]

Do you mean adding a new key that reflects the version of the current interpreter?

That's how I read it. For it to be useful, it needs IMO to reflect the current interpreter - i.e. as well as setting it up on creation, it should also be updated if the Python in the venv is upgraded.

[FFY00]

I think the issue with that is that we would have to update it then, which I think can only ever happen when the interpreter is run, meaning there's always the possibility the field is out of date.

That's why I asked for an use-case. I think that's the key thing we should be looking at here.

[vsajip]

meaning there's always the possibility the field is out of date

There's no reason why the code that does the --upgrade of the venv can't invoke the updated interpreter to get the release field and then update pyvenv.cfg with it. So the field needn't be out of date.

That's why I asked for an use-case. I think that's the key thing we should be looking at here.

Agreed.

[FFY00]

There's no reason why the code that does the --upgrade of the venv can't invoke the updated interpreter to get the release field and then update pyvenv.cfg with it. So the field needn't be out of date.

But this already happens.

$ python3.9 -m venv test
$ cat test/pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.9.9
$ python3.10 -m venv --upgrade test
$ cat test/pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.10.9

I think the issue here is when the interpreter gets upgraded, but you don't run venv --upgrade.

side-note: We probably should be raising a warning when we are in a virtual environment created for a different interpreter version, right?

[carljm]

You can safely "upgrade" a virtualenv without using venv --upgrade, by just changing the underlying Python in-place, as long as it's only a minor version upgrade (e.g. 3.11.2 to 3.11.3), since all the directory names will still be the same. I think that's why Brett was initially suggesting that if we want the version information to always be correct, it needs to be less precise, and instead only specify the range of compatible versions for this venv (i.e. record only 3.11). If we record anything more than that, it will always be possibly out of date.

[vsajip]

But this already happens.

Yes, but not for any new "release" field that would be added.

I think the issue here is when the interpreter gets upgraded, but you don't run venv --upgrade.

You can safely "upgrade" a virtualenv without using venv --upgrade, by just changing the underlying Python in-place, as long as it's only a minor version upgrade (e.g. 3.11.2 to 3.11.3), since all the symlink target directories will still be the same.

Why wouldn't you use venv --upgrade to upgrade the interpreter? That's the purpose for which it was provided, after all. Is the alternative invocation somehow simpler? Does it do something differently to venv --upgrade?

[carljm]

Why wouldn't you use venv --upgrade to upgrade the interpreter? That's the purpose for which it was provided, after all. Is the alternative invocation somehow simpler? Does it do something differently to venv --upgrade?

I'm not sure what you mean by the "alternative invocation." On Linux, by default, the interpreter is symlinked. So a minor version "upgrade" of a venv can happen silently and automatically just by virtue of the underlying Python installation being upgraded in-place (which is generally considered safe, and distros do it). There need not be any invocation of anything specific to the venv at all.

[brettcannon]

Do you mean adding a new key that reflects the version of the current interpreter?

Either that or another key that only records the release level (e.g. "final", "a6", etc.).

If so, do you have an use-case for it?

If it was completely accurate then I have a use-case for the Python Launcher for Unix (being able to know what the full version of Python is without having to actually run the interpreter). But otherwise the value is a bit misleading, so I figured maybe acting as a registration of what version was used to create the environment would help make it more useful.

I think the issue here is when the interpreter gets upgraded, but you don't run venv --upgrade.

Yep, at least when you're not on Windows.

You can safely "upgrade" a virtualenv without using venv --upgrade, by just changing the underlying Python in-place, as long as it's only a minor version upgrade (e.g. 3.11.2 to 3.11.3), since all the directory names will still be the same. I think that's why Brett was initially suggesting that if we want the version information to always be correct, it needs to be less precise, and instead only specify the range of compatible versions for this venv (i.e. record only 3.11).

Which is what also helped me realize the current version is not accurate either for when the virtual environment was created or what is currently running.

For it to be useful, it needs IMO to reflect the current interpreter - i.e. as well as setting it up on creation, it should also be updated if the Python in the venv is upgraded.

Updating the file as appropriate is not necessarily the worst thing since we already have to read pyvenv.cfg to get the system-site-packages value anyway.

[vsajip]

On Linux, by default, the interpreter is symlinked. So a minor version "upgrade" of a venv can happen silently and automatically just by virtue of the underlying Python installation being upgraded in-place (which is generally considered safe, and distros do it)

Ah, right, I didn't think of that.

[FFY00]

If it was completely accurate then I have a use-case for the Python Launcher for Unix (being able to know what the full version of Python is without having to actually run the interpreter). But otherwise the value is a bit misleading, so I figured maybe acting as a registration of what version was used to create the environment would help make it more useful.

Gotcha. And running the interpreter is not possible here?

I think the key issue is that we simply cannot guarantee the accuracy of the version logged in the file, as the underlying interpreter is updated independently (on Unix at least).

A possible solution would be installing a file that logs the interpreter version as part of the interpreter installation, that way it will always be up-to-date. The Python Launcher for Unix would then be able to simply read that file, allowing it to get the Python version without having to run the interpreter.

Updating the file as appropriate is not necessarily the worst thing since we already have to read pyvenv.cfg to get the system-site-packages value anyway.

But between the interpreter being updated and it being run in the virtual environment for the first time afterwards, won't the value be outdated?