venv does not create lib64 directory and appropriate symlinks

[BinaryKhaos]

BPO	21197
Nosy	@vsajip, @pitrou, @ned-deily, @merwok

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-04-15.12:56:48.617>
created_at = <Date 2014-04-10.13:50:32.233>
labels = ['library']
title = 'venv does not create lib64 directory and appropriate symlinks'
updated_at = <Date 2014-06-03.15:51:52.387>
user = 'https://bugs.python.org/BinaryKhaos'

bugs.python.org fields:

activity = <Date 2014-06-03.15:51:52.387>
actor = 'vinay.sajip'
assignee = 'none'
closed = True
closed_date = <Date 2014-04-15.12:56:48.617>
closer = 'python-dev'
components = ['Library (Lib)']
creation = <Date 2014-04-10.13:50:32.233>
creator = 'BinaryKhaos'
dependencies = []
files = []
hgrepos = []
issue_num = 21197
keywords = []
message_count = 12.0
messages = ['215883', '215902', '215920', '215922', '215924', '215928', '215930', '215964', '216284', '219256', '219533', '219703']
nosy_count = 6.0
nosy_names = ['vinay.sajip', 'pitrou', 'ned.deily', 'eric.araujo', 'python-dev', 'BinaryKhaos']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue21197'
versions = ['Python 3.4']

[BinaryKhaos]

Creating a new venv on a multilib system does not install an appropriate link from lib to lib64 or the other way around. Currently, venv creates a single lib dir, but everything installed with pip within the venv, will be installed to lib64/... instead while lib will remain empty. For some third party apps, this is confusing.

virtualenv, for example, installs a lib64 symlink to lib automatically. IMHO, more correctly would be a lib to lib64 symlink.

[pitrou]

Which third party apps are these? AFAIK, venv's directory layout is private - it shouldn't be "interpreted" by other programs (as opposed to the system-level lib/lib64 layout).

[BinaryKhaos]

Take for example the case that you have to juggle with several venvs and mix them together. This has happened to me in the past in very legitimate cases. You have to add those venvs to the path yourself. I am not talking about having a shell where you do your work but some other automated tasks.

Or jedi (http://jedi.jedidjah.ch/) which has become a popular autocompletion third party tool that supports venvs. You pass it the base directories and it adds their site-packages to the appropriate places. Naturally it has to know if those are under lib or lib64.

Just to name two examples. I guess there are more tools out there which have to juggle with venvs and would stumble upon this inconsistency in comparison with virtualenv. Both are naturally fixable on the user side here but since virtualenv provided those symlinks since quite a while and since IMHO venv should stay as compatible with it as possible, it would be nice if venv provided those too.

Besides all of that, creating a "lib"-dir during the venv creation while for all later installations via pip "lib64" is used (which wasn't created initially), sounds like a bug all in itself. ;-)

If you want, I could have a look and come up with a patch, if you would be willing to consider this "feature" (adding a lib64 to lib symlink).

[vsajip]

What does pip do under Windows? The symlink feature doesn't work on all Windows versions and is not quite the same as on POSIX.

What is it that actually requires lib64? As venvs are specific to a single interpreter, it seems like it is inherently not a good idea to add site-package links to venvs in other venvs willy-nilly, though of course it would work in homogeneous scenarios.

Do we know *why* virtualenv added a lib64 symlink? What (apart from possibly pip) fails if it doesn't do this? Does distutils require it?

[BinaryKhaos]

The problem is: Some Linux dists install Python under /usr/lib64 on a multilib systems and patch Python accordingly, e.g. Gentoo or Fedora. Thus, Python looks for lib64/pythonX.Y/... which is also why pip installs to lib64/... by default on those systems.

This has no effect on Windows systems btw.

virtualenv introduced the symlink sometime 2007, if I researched that correctly (pypa/virtualenv@244b23b) and kept and tuned it ever since.

Again, it is debatable where this should be fixed but since it is such an easy fix to venv while potentially keeping compatibility with tools relying on finding everything under lib/... as well as virtualenv itself, I'd think it would be nice to have it here.

@vinay.sajip: I never said anything about adding links to the site-packages itself-- even though those can be useful and safe in a very limited number of cases. It is just about the fact that on systems where Python is installed under lib64, pip also installs to lib64 while the venv only creates a lib/ dir, no lib64/ symlink to it and some tools (rightfully so) rely on finding everything under lib/ itself... which is empty in that case. For the global python that is irrelevant because there is always a lib to lib64 symlink on multilib systems.

[vsajip]

Thanks for the info.

I'm not opposed to adding a symlink on POSIX systems - I just wanted to make sure that was the right solution.

My comment about adding links willy-nilly was about tools like Jedi that you mentioned (perhaps I misunderstood how it works).

Clearly, if the idea is to support Pythons patched by distros, then it seems that there is little alternative available apart from the symlink.

As you say, lib64 should be the symlink name, pointing to the lib directory. Should the symlink be created on 64-bit Linux venvs on all POSIX platforms, and should OS X be excluded?

[ned-deily]

"should OS X be excluded?"

Yes. OS X uses universal binaries, with multiple CPU archs automatically combined at build time into one file, so there generally is no need for arch-specific directories.

But if the problem is being caused by Pythons patched by third-party distributors, shouldn't the onus be on them to properly patch venv, too, rather than having upstream CPython trying to guess what distributors are doing?

[BinaryKhaos]

@vinay.sajip: No problem. Jedi is a auto-completion library. It does not add any links anywhere. It naturally has to know about which venvs you use so it can find all modules and their sources to process them. Thus, you (or the implementation using Jedi) pass it the base directories of the venvs and it constructs the appropriate path to the site-packages and uses those internally. Rightfully, it assumes a lib/ directory to contain everything which works just fine for virtualenv created venvs due to the symlink but not for venv created venvs.

@ned.deily: I know what you mean. And I am very inclined to fully agree with you that dists should patch this as well. But the problem I see with this is: Except for Debian-based dists, all of the other major ones (Gentoo, Fedora/RedHat/CentOS, OpenSUSE, ...) and their derivatives, put Python under lib64/ on multilib systems. And the patches they use and the changes they introduce already differ here and there in important key details. It would be no different with this issue as well. Besides, this scenario (Python under lib64/) is not so uncommon anymore and imho, should be supported by Python out of the box by making those things properly configurable during compilation. But that is a different subject altogether.

IMHO, this feature is small enough to be added to Python's venv to standardize it across dists. Besides, it makes it more compatible and a better drop-in replacement for virtualenv itself.

[python-dev]

New changeset 4d9b24b2f1b8 by Vinay Sajip in branch '3.4':
Issue bpo-21197: Add lib64 -> lib symlink in venvs on 64-bit non-OS X POSIX.
http://hg.python.org/cpython/rev/4d9b24b2f1b8

New changeset 947515cc8957 by Vinay Sajip in branch 'default':
Closes bpo-21197: Add lib64 -> lib symlink in venvs on 64-bit non-OS X POSIX.
http://hg.python.org/cpython/rev/947515cc8957

[vsajip]

Issue bpo-18807 relates to symlinks not being available, or not being wanted by the user creating the environment. The lib64 symlink is (currently) the only case where we symlink to a directory (in the other cases, such as aliases for the interpreter, we can use copies rather than symlinks). I propose to make a change such that if copying rather than symlinking is specified, the lib64 link simply isn't created - a copy would be of no use here.

[ned-deily]

The added 64-bit test unnecessary adds an import for struct. The documented preferred test is:

sys.maxsize > 2**32

https://docs.python.org/dev/library/platform.html#platform.architecture

[vsajip]

The added 64-bit test unnecessary adds an import for struct.

I corrected this when fixing bpo-21643.