gh-130804: Fix support of typing unicode chars in pyrepl

[sergey-miryanov]

Fixed typing of unicode chars (I tested on Cyrillic chars on Windows)

• Issue: [Windows] New REPL doesn't allow to input non-ASCII Unicode characters #130804

[encukou]

This does fix the issue on Windows. I've also tested on Linux and saw no breakage.

[pablogsal]

LGTM

Thanks a lot for the fix @sergey-miryanov

[miss-islington-app]

Thanks @sergey-miryanov for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @sergey-miryanov and @pablogsal, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 7c98b0674daa3e4eb3e8f35afb61a0dba61d1780 3.13 

[StanFromIreland]

Tests have failed.
workflow

[pablogsal]

How was the PR landed then?

[pablogsal]

Test were ok in https://github.com/python/cpython/actions/runs/13775445246/job/38523588646

[StanFromIreland]

It's odd, it's the same set up yet somehow they are all failing now?

[pablogsal]

This PR also only modifies test_multiple_statements_fail_early so I don't see how is this related

[pablogsal]

Reverting doesn't fix anything:

~/github/python/main main ⇡ ❯ git revert 7c98b0674daa3e4eb3e8f35afb61a0dba61d1780 [main 46a7449b8e9] Revert "gh-130804: Fix support of typing unicode chars in pyrepl (#130805)" 3 files changed, 3 insertions(+), 17 deletions(-) delete mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2025-03-10-21-46-37.gh-issue-130804.0PpcTx.rst ~/github/python/main main ⇡ ❯ make -j -s Checked 112 modules (34 built-in, 77 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import) ~/github/python/main main ⇡ ❯ ./python -m test test_repl -vvv == CPython 3.14.0a5+ (heads/main:7c98b0674da, Mar 10 2025, 22:34:19) [GCC 14.2.1 20250207] == Linux-6.13.5-arch1-1-x86_64-with-glibc2.41 little-endian == Python build: debug == cwd: /home/pablogsal/github/python/main/build/test_python_worker_1544359æ == CPU count: 36 == encodings: locale=UTF-8 FS=utf-8 == resources: all test resources are disabled, use -u option to unskip tests Using random seed: 1616529274 0:00:00 load avg: 9.94 Run 1 test sequentially in a single process 0:00:00 load avg: 9.94 [1/1] test_repl test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early) ... FAIL test_toplevel_contextvars_async (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_async) ... ok test_toplevel_contextvars_sync (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_sync) ... ok test_asyncio_repl_is_ok (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_is_ok) ... ok test_asyncio_repl_reaches_python_startup_script (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_reaches_python_startup_script) ... ok test_close_stdin (test.test_repl.TestInteractiveInterpreter.test_close_stdin) ... ok test_interactive_source_is_in_linecache (test.test_repl.TestInteractiveInterpreter.test_interactive_source_is_in_linecache) ... ok test_interactive_traceback_reporting (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting) ... ok test_interactive_traceback_reporting_multiple_input (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting_multiple_input) ... ok test_multiline_string_parsing (test.test_repl.TestInteractiveInterpreter.test_multiline_string_parsing) ... ok test_no_memory (test.test_repl.TestInteractiveInterpreter.test_no_memory) ... ok test_runsource_show_syntax_error_location (test.test_repl.TestInteractiveInterpreter.test_runsource_show_syntax_error_location) ... ok test_interactive_syntax_error_correct_line (test.test_repl.TestInteractiveModeSyntaxErrors.test_interactive_syntax_error_correct_line) ... ok ====================================================================== FAIL: test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pablogsal/github/python/main/Lib/test/test_repl.py", line 304, in test_multiple_statements_fail_early self.assertNotIn("afterwards", output) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ AssertionError: 'afterwards' unexpectedly found in 'asyncio REPL 3.14.0a5+ (heads/main:7c98b0674da, Mar 10 2025, 22:34:19) [GCC 14.2.1 20250207] on linux\nUse "await" directly instead of "asyncio.run()".\nType "help", "copyright", "credits" or "license" for more information.\n>>> import asyncio\n>>> Traceback (most recent call last):\n File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 448, in result\n return self.__get_result()\n ~~~~~~~~~~~~~~~~~^^\n File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 393, in __get_result\n raise self._exception\n File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 42, in callback\n coro = func()\n File "<stdin>", line 1, in <module>\n 1 / 0; print(\'afterwards\')\n ~~^~~\nZeroDivisionError: division by zero\n>>> \nexiting asyncio REPL...\nTraceback (most recent call last):\n File "/home/pablogsal/github/python/main/Lib/runpy.py", line 198, in _run_module_as_main\n return _run_code(code, main_globals, None,\n "__main__", mod_spec)\n File "/home/pablogsal/github/python/main/Lib/runpy.py", line 88, in _run_code\n exec(code, run_globals)\n ~~~~^^^^^^^^^^^^^^^^^^^\n File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 204, in <module>\n sys.exit(return_code)\n ~~~~~~~~^^^^^^^^^^^^^\nSystemExit: 0\n>>> \n' ---------------------------------------------------------------------- Ran 13 tests in 1.223s FAILED (failures=1) test test_repl failed 0:00:01 load avg: 9.94 [1/1/1] test_repl failed (1 failure) == Tests result: FAILURE == 1 test failed: test_repl Total duration: 1.3 sec Total tests: run=13 failures=1 Total test files: run=1/1 failed=1 Result: FAILURE 

[StanFromIreland]

Its this PR: 6ab5c4a

[pablogsal]

Doesn't seem like it:

~/github/python/main main ⇡ ❯ git revert 6ab5c4aa05bf35832a3ccd1e71b28b8475fa30f4 [main 1428e541980] Revert "gh-124927: Fix conversion issue between coordinates and position in REPL (#125001)" 4 files changed, 1 insertion(+), 12 deletions(-) delete mode 100644 Misc/NEWS.d/next/Library/2024-10-05-13-25-07.gh-issue-124927.uzNA32.rst ~/github/python/main main ⇡ ❯ make Checked 112 modules (34 built-in, 77 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import) ./python -E ./Tools/build/generate-build-details.py `cat pybuilddir.txt`/build-details.json ~/github/python/main main ⇡ ❯ ./python -m test test_repl -vvv == CPython 3.14.0a5+ (bisect/good-ecdf6b15b0c0570c3c3302ab95bdbfd3007ea941-2-g6ab5c4aa05b:6ab5c4aa05b,) [GCC 14.2.1 20250207] == Linux-6.13.5-arch1-1-x86_64-with-glibc2.41 little-endian == Python build: debug == cwd: /home/pablogsal/github/python/main/build/test_python_worker_1588881æ == CPU count: 36 == encodings: locale=UTF-8 FS=utf-8 == resources: all test resources are disabled, use -u option to unskip tests Using random seed: 4038269957 0:00:00 load avg: 3.85 Run 1 test sequentially in a single process 0:00:00 load avg: 3.85 [1/1] test_repl test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early) ... FAIL test_toplevel_contextvars_async (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_async) ... ok test_toplevel_contextvars_sync (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_sync) ... ok test_asyncio_repl_is_ok (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_is_ok) ... ok test_asyncio_repl_reaches_python_startup_script (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_reaches_python_startup_script) ... ok test_close_stdin (test.test_repl.TestInteractiveInterpreter.test_close_stdin) ... ok test_interactive_source_is_in_linecache (test.test_repl.TestInteractiveInterpreter.test_interactive_source_is_in_linecache) ... ok test_interactive_traceback_reporting (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting) ... ok test_interactive_traceback_reporting_multiple_input (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting_multiple_input) ... ok test_multiline_string_parsing (test.test_repl.TestInteractiveInterpreter.test_multiline_string_parsing) ... ok test_no_memory (test.test_repl.TestInteractiveInterpreter.test_no_memory) ... ok test_runsource_show_syntax_error_location (test.test_repl.TestInteractiveInterpreter.test_runsource_show_syntax_error_location) ... ok test_interactive_syntax_error_correct_line (test.test_repl.TestInteractiveModeSyntaxErrors.test_interactive_syntax_error_correct_line) ... ok ====================================================================== FAIL: test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pablogsal/github/python/main/Lib/test/test_repl.py", line 304, in test_multiple_statements_fail_early self.assertNotIn("afterwards", output) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ AssertionError: 'afterwards' unexpectedly found in 'asyncio REPL 3.14.0a5+ (bisect/good-ecdf6b15b0c0570c3c3302ab95bdbfd3007ea941-2-g6ab5c4aa05b:6ab5c4aa05b,) [GCC 14.2.1 20250207] on linux\nUse "await" directly instead of "asyncio.run()".\nType "help", "copyright", "credits" or "license" for more information.\n>>> import asyncio\n>>> Traceback (most recent call last):\n File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 448, in result\n return self.__get_result()\n ~~~~~~~~~~~~~~~~~^^\n File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 393, in __get_result\n raise self._exception\n File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 42, in callback\n coro = func()\n File "<stdin>", line 1, in <module>\n 1 / 0; print(\'afterwards\')\n ~~^~~\nZeroDivisionError: division by zero\n>>> \nexiting asyncio REPL...\nTraceback (most recent call last):\n File "/home/pablogsal/github/python/main/Lib/runpy.py", line 198, in _run_module_as_main\n return _run_code(code, main_globals, None,\n "__main__", mod_spec)\n File "/home/pablogsal/github/python/main/Lib/runpy.py", line 88, in _run_code\n exec(code, run_globals)\n ~~~~^^^^^^^^^^^^^^^^^^^\n File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 204, in <module>\n sys.exit(return_code)\n ~~~~~~~~^^^^^^^^^^^^^\nSystemExit: 0\n>>> \n' ---------------------------------------------------------------------- Ran 13 tests in 1.400s FAILED (failures=1) test test_repl failed 0:00:01 load avg: 3.85 [1/1/1] test_repl failed (1 failure) == Tests result: FAILURE == 1 test failed: test_repl Total duration: 1.5 sec Total tests: run=13 failures=1 Total test files: run=1/1 failed=1 Result: FAILURE 

[StanFromIreland]

Weird? Only one left then would be yours a931a8b maybe it has something to do with the traceback modifications?

[pablogsal]

I'm pretty sure it's a931a8b

[pablogsal]

#131065

[pablogsal]

@sergey-miryanov Can you check #130805 (comment) ?

Seems the backport needs some adaptations to 3.13. Could you take a look?

[sergey-miryanov]

Seems the backport needs some adaptations to 3.13. Could you take a look?

@pablogsal
There is a conflict because a65366e doesn't backported to 3.13 and this PR fixes regression from this commit. Should we backport a65366e first?

[pablogsal]

That's tricky. That may be considered a feature so I am not sure we can. @ambv what do you think?

[bedevere-app]

GH-133462 is a backport of this pull request to the 3.13 branch.