_tkinter built on macOS 10.14 does not link to Tcl and Tk in /Library/Frameworks

[wordtech]

BPO	34956
Nosy	@ronaldoussoren, @ned-deily, @tonybaloney, @miss-islington, @dimpase
PRs	bpo-34956: Fix macOS _tkinter use of Tcl/Tk in /Library/Frameworks #20171 bpo-34956: edit and format NEWS item in 3.9.0b1 changelog #20255 [3.9] bpo-34956: edit and format better NEWS item in 3.9.0b1 changelog (GH-20255) #20257

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = None
created_at = <Date 2018-10-11.03:29:07.952>
labels = ['OS-mac', '3.7', '3.8']
title = '_tkinter built on macOS 10.14 does not link to Tcl and Tk in /Library/Frameworks'
updated_at = <Date 2021-11-04.14:23:12.227>
user = 'https://bugs.python.org/wordtech'

bugs.python.org fields:

activity = <Date 2021-11-04.14:23:12.227>
actor = 'erlendaasland'
assignee = 'ned.deily'
closed = False
closed_date = None
closer = None
components = ['macOS']
creation = <Date 2018-10-11.03:29:07.952>
creator = 'wordtech'
dependencies = []
files = []
hgrepos = []
issue_num = 34956
keywords = ['patch']
message_count = 8.0
messages = ['327508', '327522', '327524', '327556', '337562', '369195', '369434', '369435']
nosy_count = 6.0
nosy_names = ['ronaldoussoren', 'wordtech', 'ned.deily', 'anthonypjshaw', 'miss-islington', 'dimpase']
pr_nums = ['20171', '20255', '20257']
priority = 'high'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue34956'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[wordtech]

I'm trying to build Python 3.7.0 on macOS 10.14, and Tkinter is not linking to my installation of Tcl/Tk 8.6.8 in /Library/Frameworks. Instead it is linking to the ancient 8.5 Tk installed in /System/Library/Frameworks. My usual way of forcing Python to link to my installation is to edit setup.py and comment out all search directories except /Library/Frameworks, but that seems to be ignored here. My basic invocation is "./configure --enable-framework" which, along with omitting the system libraries from setup.py, has always been sufficient in the past. Please advise.

[ned-deily]

macOS 10.14 appears to have introduced some subtle differences in the search order for finding header files and/or frameworks and libraries. I'm not 100% sure I understand completely what has changed but I'm confident that the workaround outlined below should get you going until we have a better solution (but beware of the potential gotcha).

Python has never fully supported building only from an Xcode-only installation, i.e. we have required at a minimum installing system header files into /usr/include et al by using "xcode-select --install". As of 10.14, xcode-select no longer installs header files in /usr/include. So that further cripples Python builds in that header files for several third-party libraries shipped with macOS are no longer found, like zlib and sqlite3. When using an Xcode-only installation (no Command Line Tools) in previous releases of macOS, I believe it was the case that essentially the system root directory ('/') was searched by the compiler tool chain for header files, for libraries, and for frameworks, and for frameworks the long standard fallback framework search path order was honored by default: first search /Library/Frameworks, then /System/Library/Frameworks. That worked fine if you were installing frameworks like Tcl and TK into /Library/Frameworks and wanting them to override the system ones. If you did not install the Command Line Tools, then the tool chain used the MacOSX.sdk embedded in Xcode.app as the system root for searching. By default, that's found in:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

With an unmodified Xcode install, that directory contains usr and System directories but no Library. So the contents of /Library were ignored by the tool chain with the net effect that _tkinter builds would also link with the Apple-supplied Tcl and Tk 8.5 in /System/Library/Frameworks.

With 10.14, Xcode-only installs still work the same way but now Command Line Tools seem to effectively use the SDK embedded in /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk as the system root, rather than the system /. Just like the Xcode copy of the sdk, the CLT copy does not have a Library directory so now the results are the same as Xcode-only builds: _tkinter only links with the /System/Library Tcl and Tk. From the viewpoint of developers trying to build applications for distribution to others, it makes sense to try to ensure that /Library does not influence the build because /Library is under user control, not the system (Apple).

Until we "recently" started shipping our private version of Tcl and Tk 8.6.x, which we do not install to /Library btw, it made sense to default to linking to /Library and falling back to /System/Library, allowing optional use of a third-party Tcl/Tk (like from ActiveState) and it all sort of worked if you used the tool chain from the CLT. But with 10.14, that no longer works. Python's build system complicates this all in large part because the top-level setup.py tries to guess what directories the compiler tool chain are going to search; that was always kind of iffy and now it's much more broken with macOS 10.14. There are ways to work around most of the issues but we need to have better default behavior. That's kind of a bigger deal.

For the specific case of building _tkinter to link with Tcl and Tk frameworks in /Library, I think the easiest workaround is to add a Library symlink to the SDK in use, either under /Library/Developer/CommandTools or /Applications/Xcode.app. "xcode-select --print-path" will show which one is in use, but, alas, not the full path to the SDK.

either:
$ xcode-select --print-path
/Library/Developer/CommandLineTools
$ cd /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/

or:
$ xcode-select --print-path
/Applications/Xcode.app/Contents/Developer
$ xcodebuild -version -sdk macosx Path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
$ cd $(xcodebuild -version -sdk macosx Path)

You can then carefully create a symlink from the SDK to /Library.

$ sudo ln -s /Library .

Now the Python builds should find the Current version of Tcl and Tk frameworks in /Library.

*But* be aware that this might affect other builds if there are other frameworks you have installed in /Library. So this workaround should be used with caution and you might consider removing the symlink once you are done building Python.

There are other, more complicated workarounds but this seems to me to be the simplest and most foolproof as long as one is aware of the risks and is prepared to potentially recreate the Library symlink after software updates to Xcode or the Command Line Tools.

[ned-deily]

Correction: in the original second paragraph, the fourth sentence obviously has the case backwards. It should read:

"When using a *Command Line Tools installation (no Xcode)* in previous releases of macOS, I believe it was the case that essentially the system root directory ('/') was searched by the compiler tool chain for header files, for libraries, and for frameworks, and for frameworks the long standard fallback framework search path order was honored by default: first search /Library/Frameworks, then /System/Library/Frameworks. [...]"

[wordtech]

Thank you, this helped.

[dimpase]

On https://bugs.python.org/issue36231 I propose a way to build without creating any symlinks.

[ned-deily]

New changeset 1731d6d by Ned Deily in branch 'master':
bpo-34956: Fix macOS _tkinter use of Tcl/Tk in /Library/Frameworks (GH-20171)
1731d6d

[ned-deily]

New changeset bac170c by Ned Deily in branch 'master':
bpo-34956: edit and format better NEWS item in 3.9.0b1 changelog (GH-20255)
bac170c

[miss-islington]

New changeset e7bf868 by Miss Islington (bot) in branch '3.9':
bpo-34956: edit and format better NEWS item in 3.9.0b1 changelog (GH-20255)
e7bf868