Python 3 raises Unicode errors with the C locale

[Sworddragon]

BPO	19846
Nosy	@malemburg, @loewis, @terryjreedy, @ncoghlan, @pitrou, @vstinner, @larryhastings, @jwilk, @abadger, @bitdancer, @serhiy-storchaka
Files	test.py: Example script asciilocale.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-12-09.10:42:16.755>
created_at = <Date 2013-11-30.21:40:45.188>
labels = ['type-bug', 'invalid', 'expert-IO']
title = 'Python 3 raises Unicode errors with the C locale'
updated_at = <Date 2017-12-18.14:38:09.078>
user = 'https://bugs.python.org/Sworddragon'

bugs.python.org fields:

activity = <Date 2017-12-18.14:38:09.078>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2013-12-09.10:42:16.755>
closer = 'vstinner'
components = ['IO']
creation = <Date 2013-11-30.21:40:45.188>
creator = 'Sworddragon'
dependencies = []
files = ['32914', '33026']
hgrepos = []
issue_num = 19846
keywords = ['patch']
message_count = 68.0
messages = ['204849', '204850', '204852', '205418', '205419', '205454', '205459', '205462', '205465', '205472', '205497', '205498', '205505', '205538', '205545', '205547', '205548', '205549', '205550', '205554', '205555', '205564', '205611', '205615', '205623', '205625', '205637', '205640', '205642', '205646', '205654', '205655', '205669', '205670', '205671', '205672', '205673', '205675', '205688', '205690', '205691', '205693', '205694', '205727', '205747', '205748', '205749', '205751', '205772', '205783', '205848', '205855', '205859', '205871', '206055', '206065', '206068', '206071', '206098', '206101', '206107', '206109', '206112', '206116', '206169', '232290', '283717', '308567']
nosy_count = 14.0
nosy_names = ['lemburg', 'loewis', 'terry.reedy', 'ncoghlan', 'pitrou', 'vstinner', 'larry', 'jwilk', 'a.badger', 'r.david.murray', 'Sworddragon', 'serhiy.storchaka', 'bkabrda', 'editor-buzzfeed']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19846'
versions = ['Python 3.3', 'Python 3.4']

[Sworddragon]

It seems that print() and write() (and maybe other of such I/O functions) are relying on sys.getfilesystemencoding(). But these functions are not operating with filenames but with their content. In the attachments is an example script which demonstrates this problem. Here is what I get:

sworddragon@ubuntu:~/tmp$ echo $LANG
de_DE.UTF-8
sworddragon@ubuntu:~/tmp$ python3 test.py
sys.getdefaultencoding(): utf-8
sys.getfilesystemencoding(): utf-8
ä
sworddragon@ubuntu:~/tmp$ LANG=C
sworddragon@ubuntu:~/tmp$ python3 test.py
sys.getdefaultencoding(): utf-8
sys.getfilesystemencoding(): ascii
Traceback (most recent call last):
 File "test.py", line 4, in <module>
 print('\xe4')
UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 0: ordinal not in range(128)

[bitdancer]

Victor can correct me if I'm wrong, but I believe that stdin/stdout/stderr all use the filesystem encoding because filenames are the most likely source of non-ascii characters on those streams. (Not a perfect solution, but the best we can do.)

[vstinner]

"Filesystem encoding" is not a good name. You should read "OS encoding" or
maybe "locale encoding".

This encoding is the best choice for interopability with other (python2 or
non python) programs. If you don't care of interoperabilty, force the
encoding using PYTHONIOENCODING environment variable.

[terryjreedy]

Unless there is an actually possibility of changing this, which I doubt since it is a choice and not a bug, and changing might break things, this issue should be closed.

[pitrou]

I think the ship has sailed on this. We can't change our heuristic everyone someone finds a flaw in the current one.

In the long term, all sensible UNIX systems should be configured for utf-8 filenames and contents, so it won't make a difference anymore.

[vstinner]

If you want to avoid the encoding errors, you can also use PYTHONIOENCODING=:replace or PYTHONIOENCODING=:backslashreplace in Python 3.4 to use the locale encoding, but use an error handler different than strict.

[Sworddragon]

Using an environment variable is not the holy grail for this. On writing a non-single-user application you can't expect the user to set extra environment variables.

If compatibility is the only reason in my opinion it would be much better to include something like sys.use_strict_encoding() which decides if print()/write() will use sys.getfilesystemencoding() or sys.getdefaultencoding().

[pitrou]

Using an environment variable is not the holy grail for this. On
writing a non-single-user application you can't expect the user to set
extra environment variables.

I am not understanding why the user would have to set anything at all.
What is the use case for per-user encoding settings?

I understand that passing LANG=C (e.g. to disable a program's
translations) forces ASCII instead of UTF-8, which is a flaw. Perhaps
the filesystem encoding should be set to UTF-8 when the system locale
says ASCII.

(OTOH, it's IMHO a system bug that LANG=C forces the ASCII charset;
we're not in the 80s anymore)

[ncoghlan]

Antoine's suggestion of being a little more aggressive in choosing utf-8 over ascii as the OS API encoding sounds reasonable to me.

I think we're getting to a point where a system claiming ASCII as the encoding to use is almost certainly a misconfiguration rather than a desired setting. If someone *really* means ASCII, they can force it for at least the std streams with PYTHONIOENCODING.

[pitrou]

Here is a patch.

$ LANG=C ./python -c "import os, sys, locale; print(sys.getfilesystemencoding(), sys.stdin.encoding, os.device_encoding(0), locale.getpreferredencoding())"

-> Without the patch:
ascii ANSI_X3.4-1968 ANSI_X3.4-1968 ANSI_X3.4-1968

-> With the patch:
utf-8 utf-8 utf-8 ANSI_X3.4-1968

[vstinner]

There was a previous try to use a file encoding different than the locale encoding and it introduces too many issues:
https://mail.python.org/pipermail/python-dev/2010-October/104509.html
"Inconsistencies if locale and filesystem encodings are different"

Python uses the fact that the filesystem encoding is the locale encoding in various places. For example, Python uses the C codec (mbstowcs) to decode byte string from the filesystem encoding before Python codecs can be used. For example, the ISO 8859-15 codec is implemented in Python and so you need something during Python startup until the import machinery is ready and the codec is loaded (using ascii encoding is not correct).

The C locale may use a different encoding. For example on AIX, the ISO 8859-1 encoding is used. On FreeBSD and Solaris, the ISO 8859-1 encoding is announced but the ASCII encoding is used in practice. Python forces the ascii encoding on FreeBSD to avoid other issues.

I worked hard to have Python 3 working out of the box on all platform. In my opinion, going against the locale encoding in some cases (the C locale) would introduce more issues than it solves.

[pitrou]

Python uses the fact that the filesystem encoding is the locale
encoding in various places.

The patch doesn't change that.

[ncoghlan]

Note that the *only* change Antoine's patch makes is that:

• *if* the locale encoding is ASCII (or an alias for ASCII)
• *then* Python sets the filesystem encoding to UTF-8 instead

If the locale encoding is anything *other* than ASCII, then that will still be used as the filesystem encoding, so environments that use something other than ASCII for the C locale will retain their current behaviour.

The rationale for this approach is based on the assumption that the *most likely* way to get a locale encoding of ASCII at this point in time is to use "LANG=C" on a system where the locale encoding is normally something more suited to a Unicode world (likely UTF-8).

Will assuming utf-8 sometimes cause problems? Quite possibly. But assuming that the platform's claim to only support ASCII is correct causes serious usability problems, too.