Prepare for removing the legacy Unicode C API

[serhiy-storchaka]

BPO	36346
Nosy	@malemburg, @ronaldoussoren, @pitrou, @scoder, @vstinner, @ezio-melotti, @methane, @serhiy-storchaka, @willingc, @corona10, @miss-islington, @shihai1991, @iritkatriel
PRs	bpo-36346: Prepare for removing the legacy Unicode C API. #12409 bpo-36346: array: Don't use deprecated APIs #19653 bpo-36346: Add Py_DEPRECATED to deprecated unicode APIs #20878 bpo-36346: Document removal schedule of deprecate APIs #20879 bpo-36346: Emit DeprecationWarning for PyArg_Parse() with 'u' or 'Z'. #20927 [3.9] bpo-36346: Add Py_DEPRECATED to deprecated unicode APIs (GH-20878) #20932 bpo-36346: Raise DeprecationWarning when creating legacy Unicode #20933 bpo-36346: Make unicodeobject.h C89 compatible #20934 [3.9] bpo-36346: Add Py_DEPRECATED to deprecated unicode APIs (GH-20878) #20941 bpo-36346: Prepare for removing the legacy Unicode C API (AC only). #21223 bpo-36346: Undeprecate private function _PyUnicode_AsUnicode(). #21336 bpo-36346: Do not use legacy Unicode C API in ctypes. #21429 bpo-36346: Make using the legacy Unicode C API optional #21437 bpo-36346: Doc: Update removal schedule of legacy Unicode #21479 [3.9] bpo-36346: Doc: Update removal schedule of legacy Unicode (GH-21479) #21738 [3.8] bpo-36346: Doc: Update removal schedule of legacy Unicode (GH-21479) #21739 [3.9] bpo-36346: Document removal schedule of deprecate APIs (GH-20879) #24625 [3.8] bpo-36346: Document removal schedule of deprecate APIs (GH-20879) #24626
Dependencies	bpo-36387: Refactor getenvironment() in _winapi.c

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-29.03:10:20.373>
created_at = <Date 2019-03-18.14:23:01.818>
labels = ['interpreter-core', 'expert-C-API', '3.8', 'expert-unicode']
title = 'Prepare for removing the legacy Unicode C API'
updated_at = <Date 2022-01-29.03:10:20.372>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2022-01-29.03:10:20.372>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-29.03:10:20.373>
closer = 'methane'
components = ['Interpreter Core', 'Unicode', 'C API']
creation = <Date 2019-03-18.14:23:01.818>
creator = 'serhiy.storchaka'
dependencies = ['36387']
files = []
hgrepos = []
issue_num = 36346
keywords = ['patch']
message_count = 36.0
messages = ['338228', '338284', '338285', '338286', '338289', '338290', '338331', '338340', '338343', '338344', '338565', '339860', '355535', '368615', '368653', '371730', '371731', '371734', '371735', '371745', '371795', '372656', '372658', '373032', '373035', '373450', '373478', '374855', '374856', '374857', '387513', '387545', '387546', '387828', '412025', '412047']
nosy_count = 13.0
nosy_names = ['lemburg', 'ronaldoussoren', 'pitrou', 'scoder', 'vstinner', 'ezio.melotti', 'methane', 'serhiy.storchaka', 'willingc', 'corona10', 'miss-islington', 'shihai1991', 'iritkatriel']
pr_nums = ['12409', '19653', '20878', '20879', '20927', '20932', '20933', '20934', '20941', '21223', '21336', '21429', '21437', '21479', '21738', '21739', '24625', '24626']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36346'
versions = ['Python 3.8']

Linked PRs

• gh-80527: Change support.requires_legacy_unicode_capi() #108438
• [3.12] gh-80527: Change support.requires_legacy_unicode_capi() (GH-108438) #108446

[serhiy-storchaka]

The legacy Unicode C API was deprecated in 3.3. Its support consumes resources: more memory usage by Unicode objects, additional code for handling Unicode objects created with the legacy C API. Currently every Unicode object has a cache for the wchar_t representation.

The proposed PR adds two compile time options: HAVE_UNICODE_WCHAR_CACHE and USE_UNICODE_WCHAR_CACHE. Both are set to 1 by default.

If USE_UNICODE_WCHAR_CACHE is set to 0, CPython will not use the wchar_t cache internally. The new wchar_t based C API will be used instead of the Py_UNICODE based C API. This can add small performance penalty for creating a temporary buffer for the wchar_t representation. On other hand, this will decrease the long-term memory usage. This build is binary compatible with the standard build and third-party extensions can use the legacy Unicode C API.

If HAVE_UNICODE_WCHAR_CACHE is set to 0, the wchar_t cache will be completely removed. The legacy Unicode C API will be not available, and functions that need it (e.g. PyArg_ParseTuple() with the "u" format unit) will always fail. This build is binary incompatible with the standard build if you use the legacy or non-stable Unicode C API.

I hope that these options will help third-party projects to prepare for removing the legacy Unicode C API in future.

[scoder]

Thanks for implementing this, Serhiy.
Since these C macros are public, should they be named PY_* ?

[scoder]

I think this is a good preparation that makes it clear what code will eventually be removed, and allows testing without it.

No idea how happy Windows users will be about all of this, but I consider it quite an overall improvement for the Unicode implementation. Once this gets removed, that is.

Removing the "unicode_internal" codec entirely (which is changed by this PR) is discussed in bpo-36297.

[malemburg]

I'd change the title of this bpo item to "Prepare for removing the whcar_t caching in the Unicode C API".

Note that the wchar_t caching was put in place to allow for external applications and C code to easily and efficiently interface with Python. By removing it you will slow down such code significantly, esp. on Linux and Windows where wchar_t code is fairly common (one of the reasons we added UCS4 in Python was to make the interaction with Linux wchar_t code more efficient).

This should be clearly mentioned as part of the change and the compile time flags.

BTW: You have a few other changes in the PR which don't have anything to do with the intended removal:

- envsize = PySequence_Fast_GET_SIZE(keys);
- if (PySequence_Fast_GET_SIZE(values) != envsize){
+ envsize = PyList_GET_SIZE(keys);
+ if (PyList_GET_SIZE(values) != envsize){

[scoder]

I had also looked through the unrelated changes, and while, yes, they are unrelated, they seemed to be correct and reasonable modernisations of the code base while touching it. They could be moved to a separate PR, but there is a relatively high risk of conflicts, so I'm ok with keeping them in here for now.

[malemburg]

On 18.03.2019 22:33, Stefan Behnel wrote:

I had also looked through the unrelated changes, and while, yes, they are unrelated, they seemed to be correct and reasonable modernisations of the code base while touching it. They could be moved to a separate PR, but there is a relatively high risk of conflicts, so I'm ok with keeping them in here for now.

I don't think changing sequence iteration to list iteration only
is something that should be hidden in a wchar_t removal PR.

My guess is that these changes have made it into the PR by mistake.
They deserve a separate PR and discussion.

[methane]

I'm not sure we need two options.
Does USE_UNICODE_WCHAR_CACHE=0 really helps preparing to the removal?

[serhiy-storchaka]

I wrote this PR just to see how much code should be changed after removing the wchar_t cache, and what be performance impact. Get it, experiment with it, run tests and benchmarks. I think we could set USE_UNICODE_WCHAR_CACHE to 0 by default. If this will cause significant troubles, it is easy to set it to 1.

I am going to add configure options for switching these options. On Windows you will still need to edit the config file manually.

I'm not sure we need two options.
Does USE_UNICODE_WCHAR_CACHE=0 really helps preparing to the removal?

Currently some of the legacy functions are not decorated with Py_DEPRECATED, because this would cause compiler warnings in the code that uses these functions. If USE_UNICODE_WCHAR_CACHE is 0, these functions will no longer used, so we can add compiler warnings for them.

I don't think changing sequence iteration to list iteration only
is something that should be hidden in a wchar_t removal PR.

getenvironment() is the function that has been rewritten to the new API without preserving the old variant. Since the code was rewritten so much, I performed some code clean up. PyMapping_Keys() and PyMapping_Values() always return a list now, so that using the PySequence_Fast API is superfluous. They could return a tuple in the past, but this provoked bugs because the user code used PyList API for it.

I'll open a separate issue for this.

Since these C macros are public, should they be named PY_* ?

CPython configuration macros (like HAVE_ACOSH or USE_COMPUTED_GOTOS) do not have the PY_ prefix.

[methane]

FYI, I had created PR 12340 which removes use of deprecated API in ctypes.

[ronaldoussoren]

One thing to keep in mind: HAVE_UNICODE_WCHAR_CACHE == 1 and HAVE_UNICODE_WCHAR_CACHE == 0 have a different ABI due to a different struct layout. This should probably affect the ABI tag for extension modules.

[pitrou]

The proposed PR adds two compile time options: HAVE_UNICODE_WCHAR_CACHE and USE_UNICODE_WCHAR_CACHE

I don't think this is a good approach. Most projects and developers don't recompile Python. It's especially a chore when you have many dependencies with C extensions, because you'll have to recompile them all as well.

I would recommend simply removing that cache.

[methane]

I think these ABI incompatible options are used many people.
But it is helpful to find extensions which using legacy APIs before Python 3.10 is released.

I had found ujson and MarkupSafe used legacy APIs. I fixed MarkupSafe.
I don't care ujson because it is wrapper of wchar_t based C library
and there are enough json libraries.

I suppose there are some other packages in PyPI, but I'm not sure.