GH-133711: Enable UTF-8 mode by default (PEP 686)#133712

AA-Turner · 2025-05-08T21:21:32Z

Issue: Enable UTF-8 mode by default (implement PEP 686) #133711

📚 Documentation preview 📚: https://cpython-previews--133712.org.readthedocs.build/

Doc/library/os.rst

Doc/library/sys.rst

zooba · 2025-05-09T10:19:05Z

Doc/whatsnew/3.15.rst

+* Python UTF-8 mode is now enabled by default.
+ It may be disabled with by setting :envvar:`PYTHONUTF8=0 <PYTHONUTF8>` as
+ an environment variable or by using the :option:`-X utf8=0 <-X>` flag.
+ See :pep:`686` for further details.


I feel like we can probably put some more explanation in here, such as that it affects TextIOWrapper and hence open(). The current description doesn't sound as scary as it needs to, in my opinion.
Along the lines of: "Python UTF-8 mode is now enabled by default. This means that (files/console/etc.) will now use UTF-8 regardless of system settings, unless specifically overridden in code (typically with an encoding= argument)."

To be clear, it's nothing new. But we shouldn't assume that everyone already knows what UTF-8 mode implies. There are many more people out there who haven't ever thought about it than those who are waiting for it to be the default.

Another effect of the UTF-8 Mode is that Python ignores the locale encoding.

I've expanded the What's New entry, please see https://cpython-previews--133712.org.readthedocs.build/en/133712/whatsnew/3.15.html#other-language-changes

Doc/using/windows.rst

vstinner · 2025-05-10T07:25:22Z

Doc/whatsnew/3.15.rst

 ======================

-
+* Python now uses UTF-8_ as the default encoding, independent of the system's


You might mention the UTF-8 Mode earlier since it has other side effects documented in the UTF-8 Mode section, such as changing sys.stdout error handler and ignoring the locale encoding.

"default encoding" is UTF-8 since Python 3.0. (e.g. sys.getdefaultencoding())

Lib/test/test_cmd_line.py

# Conflicts: # Doc/whatsnew/3.15.rst

Lib/test/test_cmd_line.py

vstinner · 2025-05-28T14:50:55Z

test_regrtest failed:

I wrote #134839 to fix test_regrtest when PYTHONUTF8=1 env var is set.

vstinner · 2025-05-28T15:25:01Z

I can also reproduce the [test_readline] issue in the main branch using the fr_FR locale and the command:
PYTHONUTF8=1 LANG=fr_FR ./python -m test -v test_readline -u all

I wrote #134841 to fix test_readline for the Python UTF-8 Mode.

vstinner · 2025-05-28T16:31:58Z

buildbot/wasm32-wasi Non-Debug PR — Build done.

Configure host Python fails with:

checking for /dev/ptmx... not set configure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling

It seems to be unrelated to this PR.

AA-Turner · 2025-05-29T16:52:07Z

buildbot/wasm32-wasi Non-Debug PR — Build done.
Configure host Python fails with:
checking for /dev/ptmx... not set configure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling 
It seems to be unrelated to this PR.

cc @brettcannon, this builder seems to be consistently failing since ~15/05.

vstinner

LGTM.

Doc/library/os.rst

Include/cpython/initconfig.h

vstinner · 2025-06-16T14:17:05Z

Ping @AA-Turner, I made two suggestions.

vstinner · 2025-07-13T17:06:01Z

vstinner disabled auto-merge

In fact, it would be safe to run this PR on buildbots before merging it. I will wait until GitHub CIs complete first.

bedevere-bot · 2025-07-13T18:32:32Z

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 3130318 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133712%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

AA-Turner · 2025-07-15T09:45:28Z

Thanks @vstinner!

A

Co-authored-by: Victor Stinner <vstinner@python.org>

Enable UTF-8 mode by default (PEP 686)
fc19177

AA-Turner requested a review from methane May 8, 2025 21:21

AA-Turner requested review from FFY00, ericsnowcurrently and gpshead as code owners May 8, 2025 21:21

bedevere-appbot mentioned this pull request May 8, 2025
Enable UTF-8 mode by default (implement PEP 686) #133711
Closed

bedevere-appbot added the awaiting core review label May 8, 2025

AA-Turner requested a review from vstinner May 8, 2025 21:21

This comment was marked as outdated.
Sign in to view

This comment was marked as resolved.
Sign in to view

vstinner reviewed May 9, 2025
View reviewed changes

Doc/library/os.rst Outdated Show resolvedHide resolved
Doc/library/os.rstShow resolvedHide resolved
Doc/library/sys.rst Outdated Show resolvedHide resolved

zooba reviewed May 9, 2025
View reviewed changes

AA-Turner added 3 commits May 10, 2025 01:47

Victor's documentation comments
c706cfa

Fix test_cmd_line
f095ab0

Improve What's New entry
ff066b1

AA-Turner force-pushed the utf-8-mode branch from 036555d to ff066b1Compare May 10, 2025 02:09

AA-Turnerand others added 2 commits May 10, 2025 03:20

Add references to UTF-8 and Unicode
abfb864

Merge branch 'main' into utf-8-mode
f40b986

FFY00 removed their request for review May 10, 2025 02:28

vstinner reviewed May 10, 2025
View reviewed changes

Doc/using/windows.rstShow resolvedHide resolved

vstinner reviewed May 10, 2025
View reviewed changes

methane reviewed May 10, 2025
View reviewed changes

Lib/test/test_cmd_line.py Outdated Show resolvedHide resolved

AA-Turner added 2 commits May 10, 2025 23:50

Check stream buffer type in test_python_legacy_windows_stdio
3226e1d

Merge branch 'main' into utf-8-mode
b83c966
# Conflicts: # Doc/whatsnew/3.15.rst

This comment was marked as resolved.
Sign in to view

methane reviewed May 10, 2025
View reviewed changes

Lib/test/test_cmd_line.py Outdated Show resolvedHide resolved

methane reviewed May 11, 2025
View reviewed changes

Lib/test/test_cmd_line.pyShow resolvedHide resolved

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 26, 2025

This comment was marked as resolved.
Sign in to view

Merge branch 'main' into utf-8-mode
9b3ebf7

AA-Turner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2025

This comment was marked as duplicate.
Sign in to view

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2025

vstinner approved these changes May 30, 2025
View reviewed changes

Doc/library/os.rst Outdated Show resolvedHide resolved
Include/cpython/initconfig.h Outdated Show resolvedHide resolved

bedevere-appbot added awaiting merge and removed awaiting core review labels May 30, 2025

vstinner added 3 commits July 13, 2025 19:01

Update Doc/library/os.rst
6ecea69

Update Include/cpython/initconfig.h
e87ce6c

Merge branch 'main' into utf-8-mode
3130318

vstinner enabled auto-merge (squash) July 13, 2025 17:04

vstinner disabled auto-merge July 13, 2025 17:04

vstinner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 13, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 13, 2025

AA-Turner merged commit c89a66f into python:mainJul 15, 2025
127 checks passed

bedevere-appbot removed the awaiting merge label Jul 15, 2025

github-project-automationbot moved this to Done in lavitaconnect@MOSTAFAAMMER Jul 15, 2025

AA-Turner deleted the utf-8-mode branch July 15, 2025 09:45

taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request Aug 4, 2025
pythonGH-133711: Enable UTF-8 mode by default (PEP 686) (python#133712)
4c6acfb
Co-authored-by: Victor Stinner <vstinner@python.org>

Agent-Hellboy pushed a commit to Agent-Hellboy/cpython that referenced this pull request Aug 19, 2025
pythonGH-133711: Enable UTF-8 mode by default (PEP 686) (python#133712)
e7da33e
Co-authored-by: Victor Stinner <vstinner@python.org>

		======================


		* Python now uses UTF-8_ as the default encoding, independent of the system's

Uh oh!

GH-133711: Enable UTF-8 mode by default (PEP 686)#133712

GH-133711: Enable UTF-8 mode by default (PEP 686) #133712

Uh oh!

Conversation

AA-Turner commented May 8, 2025• edited by github-actions botLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

zoobaMay 9, 2025

Choose a reason for hiding this comment

Uh oh!

zoobaMay 9, 2025

Choose a reason for hiding this comment

Uh oh!

vstinnerMay 9, 2025

Choose a reason for hiding this comment

Uh oh!

AA-TurnerMay 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinnerMay 10, 2025

Choose a reason for hiding this comment

Uh oh!

methaneJun 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

This comment was marked as resolved.

vstinner commented May 28, 2025

Uh oh!

vstinner commented May 28, 2025

Uh oh!

This comment was marked as duplicate.

vstinner commented May 28, 2025

Uh oh!

AA-Turner commented May 29, 2025

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vstinner commented Jun 16, 2025

Uh oh!

vstinner commented Jul 13, 2025

Uh oh!

bedevere-bot commented Jul 13, 2025

Uh oh!

AA-Turner commented Jul 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

AA-Turner commented May 8, 2025•
edited by github-actions bot
Loading