Deprecate lib2to3 (and 2to3) for future removal

[gpshead]

BPO	40360
Nosy	@gvanrossum, @gpshead, @vstinner, @carljm, @ambv, @ericsnowcurrently, @hroncok, @corona10, @miss-islington, @tirkarthi, @isidentical, @iritkatriel
PRs	bpo-40360: Prepare to deprecate lib2to3. #19645 bpo-40360: Deprecate lib2to3 module in light of PEP 617 #19663 bpo-40360: Add a What's new entry for lib2to3 pending deprecation #19898 bpo-40360: Handle PendingDeprecationWarning in test_lib2to3. #21694 [3.9] bpo-40360: Handle PendingDeprecationWarning in test_lib2to3. (GH-21694) #21696 [3.9] bpo-40360: Handle PendingDeprecationWarning in test_lib2to3. (GH-21694) #21697 bpo-40360: Deprecate the lib2to3 package #28116 bpo-40360: [doc] Rephrase deprecation note about lib2to3 #28122 [3.10] bpo-40360: [doc] Rephrase deprecation note about lib2to3 (GH-28122) #28127 bpo-40360: Make the 2to3 deprecation more obvious. #29064

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-02.16:03:42.312>
created_at = <Date 2020-04-22.04:40:54.230>
labels = ['type-feature', 'expert-2to3', '3.11']
title = 'Deprecate lib2to3 (and 2to3) for future removal'
updated_at = <Date 2021-10-20.22:28:52.390>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2021-10-20.22:28:52.390>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-02.16:03:42.312>
closer = 'vstinner'
components = ['2to3 (2.x to 3.x conversion tool)']
creation = <Date 2020-04-22.04:40:54.230>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 40360
keywords = ['patch']
message_count = 51.0
messages = ['366973', '367005', '367031', '367051', '367208', '367209', '367230', '367235', '367716', '367726', '367730', '367743', '367744', '367766', '367767', '367770', '367771', '367884', '368077', '368388', '373185', '373198', '373327', '373332', '373334', '373444', '373538', '374348', '374349', '374566', '374572', '374573', '374574', '374611', '374634', '374643', '379014', '400876', '400877', '400879', '400883', '400885', '400892', '400905', '400906', '400923', '400927', '400933', '400937', '404324', '404532']
nosy_count = 14.0
nosy_names = ['gvanrossum', 'gregory.p.smith', 'vstinner', 'carljm', 'lukasz.langa', 'eric.snow', 'davidhalter', 'hroncok', 'corona10', 'miss-islington', 'xtreak', 'BTaskaya', 'Peter Ludemann', 'iritkatriel']
pr_nums = ['19645', '19663', '19898', '21694', '21696', '21697', '28116', '28122', '28127', '29064']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40360'
versions = ['Python 3.11']

[gpshead]

Based on the PEP-617 acceptance thread on python-dev, lib2to3 is eventually going to run into trouble parsing modern syntax a few releases from now.

It would be better off maintained outside of the standard library. It gets used by a lot of things and is generally useful, but would make a lot more sense as a PyPI project than as something only quasi-maintained within the stdlib (it only gained the ability to parse a couple modern syntax features in via bugfix contributions to the stdlib the past month or two... meaning a lot of versions of it out there cannot)

Black has already forked it.

goal: PendingDeprecationWarning and documentation as such in 3.9. Move to DeprecationWarning in 3.10 or 3.11 and remove it by ~3.12. Subject to our existing deprecation process guidelines.

[gvanrossum]

I am in favor of this. We could promote LibCST, which is based on Parso, which uses a forked version of pgen2 (the parser in lib2to3). I believe one of these could switch to a fork of pegen as its parser, so it will be able to handle new PEG based syntax in 3.10+.

Removal by 3.12 might be feasible.

[carljm]

I volunteered in the python-dev thread to write a patch to the docs clarifying future status of lib2to3; happy to include the PendingDeprecationWarning as well.

Re linking to alternatives, we want to make sure we link to alternatives that are committed to updating to support newer Python versions' syntax. This definitely includes LibCST; I can inquire with the parso maintainer about whether it also includes parso. In future it could also include a third-party-maintained copy of lib2to3, if someone picks that up.

[carljm]

I opened a PR. It deprecates the lib2to3 library to discourage future use of it for Python3, but not the 2to3 tool. This of course means that the lib2to3 module will in practice stick around in the stdlib as long as 2to3 is still bundled with Python.

It seems like the idea in this issue is to deprecate and remove both. I'm not sure what we typically do to deprecate a command-line utility bundled with Python. Given warnings are silent by default, the deprecation warning for lib2to3 won't be visible to users of 2to3. Should I add something to its --help output? Or something more aggressive; an unconditionally-printed warning?

[gpshead]

New changeset 503de71 by Carl Meyer in branch 'master':
bpo-40360: Deprecate lib2to3 module in light of PEP-617 (GH-19663)
503de71

[gpshead]

Okay,the pending deprecation is in. Keeping open as a reminder to turn that into a real DeprecationWarning in 3.10 after the 3.9 branch is cut.

We'll then want to track reminding us to remove it in 3.12.

[carljm]

@gregory.p.smith

What do you think about the question I raised above about how to make this deprecation visible to users of the 2to3 CLI tool, assuming the plan is to remove both?