GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe#132440

chris-eibl · 2025-04-12T14:31:29Z

E.g. on my keyboard with German layout, { is usually entered via pressing the AltGr key and 7, i.e. AltGr+7.

Likewise, }, [, ], \ and some more can only be entered via AltGr.

But since #128388 / #128389 these are swallowed by the REPL on Windows and can no longer be entered.

This happens in legacy Windows terminals, where the virtual terminal mode is turned off (e.g. cmd.exe).

In virtual terminal mode there are other issues, see #131878.

Issue: New REPL on Windows swallows characters entered via AltGr #132439

Lib/_pyrepl/windows_console.py

chris-eibl · 2025-04-12T14:42:29Z

Lib/_pyrepl/windows_console.py

 elifself.__vt_support:
 # If virtual terminal is enabled, scanning VT sequences
-self.event_queue.push(rec.Event.KeyEvent.uChar.UnicodeChar)
+self.event_queue.push(raw_key)


A missed opportunity when main was merged during development of #124119.

Which @sergey-miryanov takes care of here as part of #131901.
So maybe I should remove this one, but I'd really like to keep the other two raw_key changes (not only because I'd have to adapt almost all tests :)

IMHO you should merge my branch here :)

I can undo the change - then you won't have conflicts. I don't think we should (partially) merge between our two PRs?

Yeah, I think you are right - we shouldn't merge our PRs. You can keep your changes - I'm OK if here will be conflict.

chris-eibl · 2025-04-12T14:46:39Z

Lib/_pyrepl/windows_console.py

+self.event_queue.insert(Event(evt="key", data=key, raw=raw_key))
 returnEvent(evt="key", data="\033") # keymap.py uses this for meta
-returnEvent(evt="key", data=key, raw=key)
+returnEvent(evt="key", data=key, raw=raw_key)


Looking at the diff , previously
returnEvent( evt="key", data=code, raw=rec.Event.KeyEvent.uChar.UnicodeChar )
was used, which in the new code should have been raw_key?

Actually, I was wondering why this change did not break anything, but AFAICT the raw member is not used in the whole code base except
cpython/Lib/_pyrepl/unix_console.py
Line 513 in 5d8e432
defgetpending(self):

where the two getpending implementations dutifully respect e.raw += e.raw :)

When searching in the code base for getpending I found
cpython/Lib/_pyrepl/windows_console.py
Lines 518 to 521 in 5d8e432
defgetpending(self) ->Event:
"""Return the characters that have been typed but not yet
processed."""
returnEvent("key", "", b"")
which clearly seems to be a bug to me? Because even though WindowsConsole._read_input() only reads one Windows INPUT_RECORD per call, get_event could have put something into self.event_queue.
I've addressed this in https://github.com/python/cpython/pull/132889/files#r2059235071.

Lib/_pyrepl/windows_console.py

chris-eibl · 2025-04-12T14:53:03Z

Adding @vstinner, @eendebakpt and @paulie4 , since they were involved in #128389 and already talked about AltGr.

chris-eibl · 2025-04-12T14:56:22Z

PS: switching to an english keyboard layout, I can use the AltGr key like right-Alt, since it is not used in this keyboard layout :)

chris-eibl · 2025-04-12T17:15:46Z

The failing of Ubuntu (free-threading) is definitely unrelated, since this is a Windows specific change.

tomasr8 · 2025-04-17T19:21:45Z

Do you think it'd be possible to add some tests for this change to ensure we don't accidentally regress again?

chris-eibl · 2025-04-17T19:43:44Z

Yeah, I've already thought about it, but am still unsure how to best do it.

ATM, I am thinking of mocking

cpython/Lib/_pyrepl/windows_console.py

Line 411 in b87189d

def_read_input(self, block: bool=True) ->INPUT_RECORD|None:

to be able to test

cpython/Lib/_pyrepl/windows_console.py

Lines 426 to 432 in b87189d

defget_event(self, block: bool=True) ->Event|None: 
"""Return an Event instance. Returns None if |block| is false
 and there is no event pending, otherwise waits for the
 completion of an event."""
whileself.event_queue.empty(): 
rec=self._read_input(block) 

AFAICT, all the existing tests just mock get_event, but for this, IMHO get_event itself must be tested.
Mocking _read_input seems to be the best way forward.

WDYT?

tomasr8 · 2025-04-17T20:17:22Z

Agreed that we shouldn't be mocking get_event since that's where we're making changes. _read_input seems like a good option, we can capture some real inputs and then replay them with it.

Lib/test/test_pyrepl/test_windows_console.py

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

because it will be part of 131901, anywayy

because it is unused

Lib/test/test_pyrepl/test_windows_console.py

ambv

Excellent work again! Very well thought through.

miss-islington-app · 2025-05-05T16:45:48Z

Thanks @chris-eibl for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington-app · 2025-05-05T16:45:51Z

Sorry, @chris-eibl and @ambv, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 07f416a3f063db6b91b8b99ff61a51b64b0503f1 3.13

sergey-miryanov · 2025-05-05T17:12:53Z

Will repeat here too - problem with backport because VT support was not backported #130805 (comment) (for completeness)

ambv · 2025-05-05T17:32:09Z

Yeah, I am backporting the VT support too because it's the only way we can have a sensibly similar codebase for bug fixes.

…ltGr on cmd.exe (pythonGH-132440) (cherry picked from commit 07f416a) Co-authored-by: Chris Eibl <138194463+chris-eibl@users.noreply.github.com> Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

bedevere-app · 2025-05-05T17:46:04Z

GH-133460 is a backport of this pull request to the 3.13 branch.

…n cmd.exe (GH-132440) (GH-133460) (cherry picked from commit 07f416a) Co-authored-by: Chris Eibl <138194463+chris-eibl@users.noreply.github.com> Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

… cmd.exe (pythonGH-132440) Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

chris-eibl added 2 commits April 12, 2025 16:01

do not swallow characters that have been entered via AltGr
59d71b0

blurb it
cca6155

chris-eibl requested review from ambv, lysnikolaou and pablogsal as code owners April 12, 2025 14:31

bedevere-appbot added the awaiting review label Apr 12, 2025

bedevere-appbot mentioned this pull request Apr 12, 2025
New REPL on Windows swallows characters entered via AltGr #132439
Closed

chris-eibl commented Apr 12, 2025
View reviewed changes

Lib/_pyrepl/windows_console.pyShow resolvedHide resolved

chris-eibl added OS-windows 3.13 bugs and security fixes 3.14 bugs and security fixes topic-repl Related to the interactive shell labels Apr 12, 2025

unrelated stuff worth to do (?)
1e61540

chris-eibl commented Apr 12, 2025
View reviewed changes

Lib/_pyrepl/windows_console.py Outdated Show resolvedHide resolved

chris-eibl requested a review from vstinner April 12, 2025 14:49

picnixz added needs backport to 3.13 bugs and security fixes and removed 3.13 bugs and security fixes 3.14 bugs and security fixes labels Apr 12, 2025

chris-eibl added 3 commits April 19, 2025 08:30

Merge remote-tracking branch 'origin/main' into fix_altgr
c60a727

fix missing con.restore() in test_multiline_ctrl_z
ed035e5

add WindowsConsoleGetEventTests
69e8779

chris-eibl commented Apr 20, 2025
View reviewed changes

Lib/test/test_pyrepl/test_windows_console.pyShow resolvedHide resolved

sergey-miryanov mentioned this pull request Apr 24, 2025
Brackets "[" and "]" writing or inserting does not work in Python 3.13 interactive shell (Portuguese keyboard) #132871
Closed

This was referenced Apr 24, 2025
Pasting in new REPL is slow #130328
Closed
GH-130328: pasting in new REPL is slow on Windows #132884
Merged

chris-eibl commented Apr 25, 2025
View reviewed changes

Lib/test/test_pyrepl/test_windows_console.py Outdated Show resolvedHide resolved

chris-eibland others added 6 commits April 25, 2025 21:40

Apply suggestions from code review
e67452b
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

Update Lib/test/test_pyrepl/test_windows_console.py
396f48e

Merge branch 'main' into fix_altgr
40b339a

revert change in virtual terminal path
1fbe240
because it will be part of 131901, anywayy

Merge remote-tracking branch 'origin/main' into fix_altgr
2d59563

remove raw part of Event in get_event() in case of windows_console
adb89ba
because it is unused

chris-eibl commented May 5, 2025
View reviewed changes

Lib/test/test_pyrepl/test_windows_console.pyShow resolvedHide resolved

ambv changed the title ~~GH-132439: REPL on Windows swallows characters entered via AltGr~~GH-132439: Fix REPL swallowing characters entered with AltGrMay 5, 2025

ambv approved these changes May 5, 2025
View reviewed changes

bedevere-appbot added awaiting merge and removed awaiting core review labels May 5, 2025

ambv merged commit 07f416a into python:mainMay 5, 2025
52 checks passed

bedevere-appbot removed the awaiting merge label May 5, 2025

miss-islington-appbot assigned ambv May 5, 2025

ambv changed the title ~~GH-132439: Fix REPL swallowing characters entered with AltGr~~GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exeMay 5, 2025

bedevere-appbot removed the needs backport to 3.13 bugs and security fixes label May 5, 2025

chris-eibl deleted the fix_altgr branch May 5, 2025 18:14

eendebakpt mentioned this pull request Jun 3, 2025
Dead keys on Windows with German keyboard layout #135105
Closed

Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
pythonGH-132439: Fix REPL swallowing characters entered with AltGr on…
bace14d
… cmd.exe (pythonGH-132440) Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

	defgetpending(self) ->Event:
	"""Return the characters that have been typed but not yet
	processed."""
	returnEvent("key", "", b"")

Uh oh!

GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe#132440

GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe #132440

Uh oh!

Conversation

chris-eibl commented Apr 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

chris-eiblApr 12, 2025

Choose a reason for hiding this comment

Uh oh!

chris-eiblApr 20, 2025

Choose a reason for hiding this comment

Uh oh!

sergey-miryanovApr 20, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chris-eiblApr 20, 2025

Choose a reason for hiding this comment

Uh oh!

sergey-miryanovApr 21, 2025

Choose a reason for hiding this comment

Uh oh!

chris-eiblApr 12, 2025

Choose a reason for hiding this comment

Uh oh!

chris-eiblApr 20, 2025

Choose a reason for hiding this comment

Uh oh!

chris-eiblApr 20, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chris-eibl commented Apr 12, 2025

Uh oh!

chris-eibl commented Apr 12, 2025

Uh oh!

chris-eibl commented Apr 12, 2025

Uh oh!

tomasr8 commented Apr 17, 2025

Uh oh!

chris-eibl commented Apr 17, 2025

Uh oh!

tomasr8 commented Apr 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ambv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-appbot commented May 5, 2025

Uh oh!

miss-islington-appbot commented May 5, 2025

Uh oh!

sergey-miryanov commented May 5, 2025

Uh oh!

ambv commented May 5, 2025

Uh oh!

bedevere-appbot commented May 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

chris-eibl commented Apr 12, 2025•
edited
Loading

sergey-miryanovApr 20, 2025•
edited
Loading

chris-eiblApr 20, 2025•
edited
Loading