Non-ASCII characters in os.environ cause silent failures in test_httpservers

[ambv]

BPO	44647
Nosy	@ambv, @miss-islington
PRs	bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows #27161 [3.10] bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) #27169 [3.9] bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) #27170 bpo-44647: Add a permanent Unicode-valued env var to regrtest #27187 [3.10] bpo-44647: Add a permanent Unicode-valued env var to regrtest (GH-27187) #27191

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-07-15.20:13:07.907>
created_at = <Date 2021-07-15.17:50:20.422>
labels = ['type-bug', '3.9', '3.10', '3.11']
title = 'Non-ASCII characters in os.environ cause silent failures in test_httpservers'
updated_at = <Date 2021-07-16.13:55:33.744>
user = 'https://github.com/ambv'

bugs.python.org fields:

activity = <Date 2021-07-16.13:55:33.744>
actor = 'lukasz.langa'
assignee = 'none'
closed = True
closed_date = <Date 2021-07-15.20:13:07.907>
closer = 'lukasz.langa'
components = []
creation = <Date 2021-07-15.17:50:20.422>
creator = 'lukasz.langa'
dependencies = []
files = []
hgrepos = []
issue_num = 44647
keywords = ['patch']
message_count = 6.0
messages = ['397572', '397576', '397578', '397579', '397621', '397630']
nosy_count = 2.0
nosy_names = ['lukasz.langa', 'miss-islington']
pr_nums = ['27161', '27169', '27170', '27187', '27191']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44647'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[ambv]

#67826 introduced a new test for Accept: headers in CGI HTTP servers. This test serializes all of os.environ on the server side. For non-UTF8 locales this can fail for some Unicode characters found in environment variables.

This started failing this week on Azure Pipelines with their rollout of a new Windows 2019 image version that included a "BUILD_SOURCEVERSIONAUTHOR" env variable. For me specifically it includes a leading Unicode character so all my PRs started failing on Azure Pipelines Windows 2019 alone.

The result was truncated output from the CGI HTTP server, like:

======================================================================
FAIL: test_accept (test.test_httpservers.CGIHTTPServerTestCase) [OrderedDict([('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8')])]
----------------------------------------------------------------------

Traceback (most recent call last):
 File "D:\a\1\s\lib\test\test_httpservers.py", line 860, in test_accept
 self.assertIn(expected.encode('ascii'), res.read())
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b"'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'" not found in b''

The root cause is that the CGI script in question (cgi_file6 in test_httpservers.py) is crashing on server-side reaching the print(repr(os.environ)) line. However, this exception isn't visible on the client side where the test is running. The only visible issue is truncated output.

I suggest adding ENSURE_UNICODE_WORKS=Łukasz to the testing env so that this never regresses.

[ambv]

New changeset 82b218f by Łukasz Langa in branch 'main':
bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161)
82b218f

[ambv]

New changeset 356bdff by Miss Islington (bot) in branch '3.10':
bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) (bpo-27169)
356bdff

[ambv]

New changeset 95596d5 by Miss Islington (bot) in branch '3.9':
bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) (bpo-27170)
95596d5

[ambv]

New changeset 7915c96 by Łukasz Langa in branch 'main':
bpo-44647: Add a permanent Unicode-valued env var to regrtest (bpo-27187)
7915c96