ElementTree should use UTF-8 for xml declaration.

[methane]

Feature or enhancement

Currently, ElementTree.tostring(root, encoding="unicode", xml_declaration=True) uses locale encoding.

I think ElementTree should use UTF-8, instead of locale encoding.

Example:

$ LANG=ja_JP.eucJP ./python.exe
Python 3.11.0a7+ (heads/bytes-alloc-dirty:7fbc7f6128, Apr 19 2022, 16:53:54) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import xml.etree.ElementTree as ET
>>> et = ET.fromstring("<t>hello</t>")
>>> ET.tostring(et, encoding="unicode", xml_declaration=True)
"<?xml version='1.0' encoding='eucJP'?>\n<t>hello</t>"

Code:

cpython/Lib/xml/etree/ElementTree.py

Lines 732 to 742 in bcf14ae

	with _get_writer(file_or_filename, enc_lower) as write:
	if method == "xml" and (xml_declaration or
	(xml_declaration is None and
	enc_lower not in ("utf-8", "us-ascii", "unicode"))):
	declared_encoding = encoding
	if enc_lower == "unicode":
	# Retrieve the default encoding for the xml declaration
	import locale
	declared_encoding = locale.getpreferredencoding()
	write("<?xml version='1.0' encoding='%s'?>\n" % (
	declared_encoding,))

Pitch

• UTF-8 is the most common encoding for XML.
• Locale encoding name (e.g. cp932 or eucJP) would be different from XML encoding name recommended by w3c (e.g. Shift_JIS or EUC-JP).

[serhiy-storchaka]

Look at dump(). It writes an element to stdout, which usually uses the locale encoding. I think it is the rationale of using the locale encoding here. You need to change dump() to use the stdout's encoding explicitly.

[serhiy-storchaka]

Or maybe change write() to get the encoding from the output string if available.

[methane]

dump don't use xml_declaration=True. So this issue doesn't affect it.

[methane]

@scoder would you give us an advice? (you are listed as etree expert in expert index).

There is no correct behavior, because output is Unicode and etree don't know what is real output encoding.

There are some cases that current behavior is better (e.g. using default encoding (e.g. open(filename, 'w')).
On the other hand, encoding="cp932" (in Japanese Windows) is non-portable (encoding="Shift_JIS" should be used), and UTF-8 is the most recommended encoding for XML.

I have two ideas:

a. Make UTF-8 default. This is simplest.
b. Keep using locale.getpreferredencoding() and wait PEP 686 accepted. (But it should be replaced with locale.getpreferredencoding(False) anyway.)

[serhiy-storchaka]

Adding encoding="UTF-8" and using cp932 to encode the content would be even worse.

Maybe add a simple mapping from Python encodings to XML encodings (for example we need to write "ascii" as "us-ascii")? Later we can discuss adding a public API for this.

[serhiy-storchaka]

I proposed to get the default encoding from the file object if available. #91812 (comment)

[methane]

Adding encoding="UTF-8" and using cp932 to encode the content would be even worse.

Of course, we should recommend to use UTF-8.
Note that encoding='cp932' and using UTF-8 is possible bug for now already.
Any default value may cause bug. There is no one correct default. But UTF-8 may be the best for now.

Maybe add a simple mapping from Python encodings to XML encodings (for example we need to write "ascii" as "us-ascii")? Later we can discuss adding a public API for this.

We may not know Python encoding because output is Unicode (e.g. Unicode string or StringIO).
Such idea works only when output is TextIOWrapper. (And there are no guarantee that TextIOWrapper.encoding is really the final encoding.)

If we want to support arbitrary encoding, we should add another option like xml_declaration_encoding="Shift_JIS".
But this is not strict necessary.
User can chose xml_declaration=False and prepend <?xml version="1.0" encoding="Shift_JIS" ?> manually when they really need to use encoding other than UTF-8.

[serhiy-storchaka]

Note that encoding='cp932' and using UTF-8 is possible bug for now already.

Yes, it is a bug, and #91903 fixes it.