gh-109276: libregrtest: add RunTests.work_dir

[vstinner]

• WorkerThread now always creates a temporary directory, even on Emscripten and WASI: it's used as the working directory of the test worker process.
• Fix Emscripten and WASI: start the test worker process in the Python source code directory, where 'python.js' and 'python.wasm' can be found. Then worker_process() goes to the temporary directory created to run tests.

• Issue: Enhance Python regrtest test runner (test.libregrtest) #109276

[vstinner]

This PR fix how libregrtest creates temporary directories for worker processes and which directory is used as the working directory of worker processes. It seems like two tests fail with this change since they leak temporary files/directories!

• test.test_asyncio.test_events
• test_compileall

[vstinner]

It seems like two tests fail with this change since they leak temporary files/directories!
test.test_asyncio.test_events
test_compileall

I created issue #109295 to track this issue and PR #109298 to fix it.

[vstinner]

Oh, and Windows two tests also failed, but other tests: test_generated_cases and test_os.

ERROR: test_access_denied (test.test_os.StatAttributeTests.test_access_denied) ---------------------------------------------------------------------- Traceback (most recent call last): File "D:\a\cpython\cpython\Lib\test\test_os.py", line 742, in test_access_denied create_file(fname, b'ABC') File "D:\a\cpython\cpython\Lib\test\test_os.py", line 94, in create_file with open(filename, "xb", 0) as fp: ^^^^^^^^^^^^^^^^^^^^^^^ FileExistsError: [Errno 17] File exists: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\test_python_worker_a5djsgg5\\@test_5340_tmp�' 

and

 0:19:14 load avg: 6.45 [2/3/2] test_generated_cases failed (uncaught exception) Re-running test_generated_cases in verbose mode test test_generated_cases crashed -- Traceback (most recent call last): File "D:\a\cpython\cpython\Lib\test\libregrtest\single.py", line 139, in _runtest_env_changed_exc _load_run_test(result, runtests) File "D:\a\cpython\cpython\Lib\test\libregrtest\single.py", line 87, in _load_run_test test_mod = importlib.import_module(module_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\a\cpython\cpython\Lib\importlib\__init__.py", line 88, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1381, in _gcd_import File "<frozen importlib._bootstrap>", line 1354, in _find_and_load File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 929, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 1006, in exec_module File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "D:\a\cpython\cpython\Lib\test\test_generated_cases.py", line 11, in <module> import generate_cases File "D:\a\cpython\cpython\Tools\cases_generator\generate_cases.py", line 41, in <module> DEFAULT_INPUT = os.path.relpath(os.path.join(ROOT, "Python/bytecodes.c")) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\a\cpython\cpython\Lib\ntpath.py", line 791, in relpath raise ValueError("path is on mount %r, start on mount %r" % ( ValueError: path is on mount 'D:', start on mount 'C:' 

[vstinner]

Many tests fail on the macOS CI because the AF_UNIX domain name is too longer (more than ~100 bytes):

15 tests failed: test.test_asyncio.test_unix_events test.test_concurrent_futures.test_as_completed test.test_concurrent_futures.test_deadlock test.test_concurrent_futures.test_init test.test_concurrent_futures.test_process_pool test.test_concurrent_futures.test_shutdown test.test_concurrent_futures.test_wait test.test_multiprocessing_forkserver.test_manager test.test_multiprocessing_forkserver.test_misc test.test_multiprocessing_forkserver.test_processes test.test_multiprocessing_forkserver.test_threads test.test_multiprocessing_spawn.test_manager test.test_multiprocessing_spawn.test_misc test.test_multiprocessing_spawn.test_processes test_multiprocessing_main_handling 

The main test process:

tempfile.gettempdir: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T self.tmp_dir: /Users/runner/work/cpython/cpython/build runtests.work_dir: /Users/runner/work/cpython/cpython/build/test_python_4592æ cwd: /Users/runner/work/cpython/cpython/build/test_python_4592æ 

A worker test process:

worker test_turtle (pid 4597): /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/test_python_worker_2eq4te_m 

/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/test_python_worker_2eq4te_m is 84 (ASCII) characters long.

multiprocessing.util.get_temp_dir() creates a temporary directory with pymp-XXXXXXXX (13 characters) name.

multiprocessing.connection.arbitrary_connection(AF_UNIX) creates a listener-XXXXXXXX (17 characters) temporary file in pymp-XXXXXXXX directory.

A UNIX domain name liike /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/test_python_worker_2eq4te_m/pymp-XXXXXXXX/listener-XXXXXXXX is already 117 characters long.

[vstinner]

Windows x64:

• Python source code: D:\a\cpython\cpython (drive D:)
• Temporary directory: C:\Users\RUNNER~1\AppData\Local\Temp (drive C:)

test.pythoninfo:

sysconfig.is_python_build: True sysconfig[prefix]: D:\a\cpython\cpython sysconfig[srcdir]: D:\a\cpython\cpython tempfile.gettempdir: C:\Users\RUNNER~1\AppData\Local\Temp 

Main test process:

tempfile.gettempdir: C:\Users\RUNNER~1\AppData\Local\Temp self.tmp_dir: D:\a\cpython\cpython\build runtests.work_dir: D:\a\cpython\cpython\build\test_python_3212� cwd: D:\a\cpython\cpython\build\test_python_3212� 

A test worker process:

C:\Users\RUNNER~1\AppData\Local\Temp\test_python_worker_xa_vmpox => C:\Users\runneradmin\AppData\Local\Temp\test_python_worker_xa_vmpox 

test_generated_cases was skipped:

0:16:54 load avg: 11.61 [393/463/2] test_generated_cases skipped -- running (1): test_subprocess (1 min 1 sec) test_generated_cases skipped -- the current working directory and the Python source code directory have different mount drives (C: and D:) 

[vstinner]

PR #106219 is awaiting for an Emscripten/WASI fix for libregrtest.

[vstinner]

This change seems to be too disruptive, so I wrote a more conservative change (come back to old behavior): PR #109313.

[vstinner]

This change seems to be too disruptive, so I wrote a more conservative change (come back to old behavior): PR #109313.

Merged. I close this PR.