./Programs/_freeze_module fails with MSAN: Uninitialized value was created by an allocation of 'stat.i'

[vstinner]

BPO	46887
Nosy	@vstinner
PRs	bpo-46887: Work around clang MSAN bug on stat()/fstat() #31633

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-03-01.09:38:30.397>
labels = ['build', '3.11']
title = "./Programs/_freeze_module fails with MSAN: Uninitialized value was created by an allocation of 'stat.i'"
updated_at = <Date 2022-03-01.15:01:07.589>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-03-01.15:01:07.589>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2022-03-01.09:38:30.397>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 46887
keywords = ['patch']
message_count = 6.0
messages = ['414249', '414264', '414267', '414268', '414269', '414271']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['31633']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue46887'
versions = ['Python 3.11']

[vstinner]

"./Programs/_freeze_module zipimport" fails with MSAN:
---

$ make SHELL="bash -x"
(...)
+ ./Programs/_freeze_module zipimport ./Lib/zipimport.py Python/frozen_modules/zipimport.h
==110524==WARNING: MemorySanitizer: use-of-uninitialized-value
==110523==WARNING: MemorySanitizer: use-of-uninitialized-value
==110526==WARNING: MemorySanitizer: use-of-uninitialized-value
==110525==WARNING: MemorySanitizer: use-of-uninitialized-value
 #0 0x4b8cfc in read_text /home/vstinner/python/main/Programs/_freeze_module.c:109:9
 #1 0x4b8cfc in main /home/vstinner/python/main/Programs/_freeze_module.c:224:24
 #2 0x7fe8acb8555f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)
 #3 0x7fe8acb8560b in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60b)
 #4 0x435f44 in _start (/home/vstinner/python/main/Programs/_freeze_module+0x435f44)

Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
#0 0x4b72e0 in main /home/vstinner/python/main/Programs/_freeze_module.c:205

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/vstinner/python/main/Programs/_freeze_module.c:109:9 in read_text
Exiting
make: *** [Makefile:1184: Python/frozen_modules/importlib._bootstrap_external.h] Error 1
make: *** Waiting for unfinished jobs....
#0 0x4b8cfc in read_text /home/vstinner/python/main/Programs/_freeze_module.c:109:9
#1 0x4b8cfc in main /home/vstinner/python/main/Programs/_freeze_module.c:224:24
#2 0x7f64ecd1355f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)
#3 0x7f64ecd1360b in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60b)
#4 0x435f44 in _start (/home/vstinner/python/main/Programs/_freeze_module+0x435f44)

Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
#0 0x4b72e0 in main /home/vstinner/python/main/Programs/_freeze_module.c:205

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/vstinner/python/main/Programs/_freeze_module.c:109:9 in read_text
Exiting
make: *** [Makefile:1176: Python/frozen_modules/getpath.h] Error 1
#0 0x4b8cfc in read_text /home/vstinner/python/main/Programs/_freeze_module.c:109:9
#1 0x4b8cfc in main /home/vstinner/python/main/Programs/_freeze_module.c:224:24
#2 0x7fb2a178d55f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)
#3 0x7fb2a178d60b in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60b)
#4 0x435f44 in _start (/home/vstinner/python/main/Programs/_freeze_module+0x435f44)

#0 0x4b8cfc in read_text /home/vstinner/python/main/Programs/_freeze_module.c:109:9
#1 0x4b8cfc in main /home/vstinner/python/main/Programs/_freeze_module.c:224:24
#2 0x7f4d471ca55f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)
#3 0x7f4d471ca60b in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60b)
#4 0x435f44 in _start (/home/vstinner/python/main/Programs/_freeze_module+0x435f44)

Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
#0 0x4b72e0 in main /home/vstinner/python/main/Programs/_freeze_module.c:205

#0 0x4b72e0 in main /home/vstinner/python/main/Programs/_freeze_module.c:205

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/vstinner/python/main/Programs/_freeze_module.c:109:9 in read_text
SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/vstinner/python/main/Programs/_freeze_module.c:109:9 in read_text
Exiting
Exiting
make: *** [Makefile:1181: Python/frozen_modules/importlib._bootstrap.h] Error 1
make: *** [Makefile:1187: Python/frozen_modules/zipimport.h] Error 1
---

Build Python with clang MSAN:
---
./configure --with-pydebug --with-memory-sanitizer CC=clang LD=clang
make
---

[vstinner]

It looks like a bug in clang MSAN:
llvm/llvm-project#54131

[vstinner]

Hum, later "import ssl" in setup.py also fails. Simplified code:
---

import _ssl
print(_ssl.txt2obj('1.3.6.1.5.5.7.3.1', name=False)) # server OID

---

Error:
---

$ ./python x.py 
Uninitialized bytes in MemcmpInterceptorCommon at offset 0 inside [0x701000000032, 8)
==135651==WARNING: MemorySanitizer: use-of-uninitialized-value
 #0 0x49467c in memcmp (/home/vstinner/python/main/python+0x49467c)
 #1 0x7f5546825adc in OBJ_bsearch_ex_ (/lib64/libcrypto.so.1.1+0x14eadc)
 #2 0x7f5546826ff1 in OBJ_obj2nid (/lib64/libcrypto.so.1.1+0x14fff1)
 #3 0x7f554675b434 (/lib64/libcrypto.so.1.1+0x84434)
 #4 0x7f554675b6af in d2i_ASN1_OBJECT (/lib64/libcrypto.so.1.1+0x846af)
 #5 0x7f5546827702 in OBJ_txt2obj (/lib64/libcrypto.so.1.1+0x150702)
 #6 0x7f5546a78c31 in _ssl_txt2obj_impl /home/vstinner/python/main/Modules/_ssl.c:5276:11
 #7 0x7f5546a78c31 in _ssl_txt2obj /home/vstinner/python/main/Modules/clinic/_ssl.c.h:1216:20
 (...)

Uninitialized value was created by a heap allocation
#0 0x466642 in __interceptor_malloc (/home/vstinner/python/main/python+0x466642)
#1 0x7f55468276b3 in OBJ_txt2obj (/lib64/libcrypto.so.1.1+0x1506b3)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/home/vstinner/python/main/python+0x49467c) in memcmp
Exiting
---

The Python code calls OBJ_txt2obj("1.3.6.1.5.5.7.3.1", 0) in C: the OpenSSL function.

[vstinner]

It looks like a bug in clang MSAN:
llvm/llvm-project#54131

I wrote #75814 to work around the false alarm on stat() and fstat().

[vstinner]

The Python code calls OBJ_txt2obj("1.3.6.1.5.5.7.3.1", 0) in C: the OpenSSL function.

This error is unrelated to Python, but comes from OpenSSL.

[vstinner]

The Python code calls OBJ_txt2obj("1.3.6.1.5.5.7.3.1", 0) in C: the OpenSSL function.

I reported this issue to OpenSSL: openssl/openssl#17784

[vstinner]

It's unclear to me if it's an issue in Python itself, Clang/GCC, or how the glibc was built. For example, openssl/openssl#17784 (comment) was closed OpenSSL must be built with -fsanitize=memory.

Anyway, I close the issue.

[illia-v]

It's unclear to me if it's an issue in Python itself, Clang/GCC, or how the glibc was built. For example, openssl/openssl#17784 (comment) was closed OpenSSL must be built with -fsanitize=memory.

I've recently tried building and compiling CPython with CC=clang ./configure --with-memory-sanitizer && make -j12 against the system OpenSSL.

After making the #102510 change, a slightly different use-of-uninitialized-value error tracing to /lib/x86_64-linux-gnu/libcrypto.so.3 appeared.

So, I tried compiling it against OpenSSL built with -fsanitize=memory, and that fixed an error.

But it looks like there is a sequence of other use-of-uninitialized-value appearing one by one.
The next one looks to be fixable by this

diff --git a/Python/dynload_shlib.c b/Python/dynload_shlib.c
index 6761bba457..ac379d5d98 100644
--- a/Python/dynload_shlib.c
+++ b/Python/dynload_shlib.c
@@ -56,7 +56,7 @@ _PyImport_FindSharedFuncptr(const char *prefix,
{
 dl_funcptr p;
 void *handle;
- char funcname[258];
+ char funcname[258] = "";
 char pathbuf[260];
 int dlopenflags=0;

I stopped digging on this one, but noticed the symbol lookup error: /home/user/openssl/lib64/libcrypto.so.3: undefined symbol: __msan_retval_tls warnings:

Uninitialized bytes in __interceptor_strlen at offset 0 inside [0x701000003040, 1)
==91306==WARNING: MemorySanitizer: use-of-uninitialized-value
/usr/bin/llvm-symbolizer-15: symbol lookup error: /home/user/openssl/lib64/libcrypto.so.3: undefined symbol: __msan_retval_tls
==91306==WARNING: external symbolizer didn't start up correctly!
==91306==WARNING: Can't write to symbolizer at fd 6
/usr/bin/llvm-symbolizer-15: symbol lookup error: /home/user/openssl/lib64/libcrypto.so.3: undefined symbol: __msan_retval_tls
==91306==WARNING: external symbolizer didn't start up correctly!
==91306==WARNING: Can't write to symbolizer at fd 6
/usr/bin/llvm-symbolizer-15: symbol lookup error: /home/user/openssl/lib64/libcrypto.so.3: undefined symbol: __msan_retval_tls
==91306==WARNING: external symbolizer didn't start up correctly!
==91306==WARNING: Can't write to symbolizer at fd 6
/usr/bin/llvm-symbolizer-15: symbol lookup error: /home/user/openssl/lib64/libcrypto.so.3: undefined symbol: __msan_retval_tls
==91306==WARNING: external symbolizer didn't start up correctly!
==91306==WARNING: Can't write to symbolizer at fd 6
/usr/bin/llvm-symbolizer-15: symbol lookup error: /home/user/openssl/lib64/libcrypto.so.3: undefined symbol: __msan_retval_tls
==91306==WARNING: external symbolizer didn't start up correctly!
==91306==WARNING: Failed to use and restart external symbolizer!
 #0 0x7f98fe69104b (/lib/x86_64-linux-gnu/libreadline.so.8+0x3204b) (BuildId: 990587ed58ff96de9f94756458d4aaa2f51aaa3d)
 #1 0x7f98fe880ab2 (/home/user/cpython/build/lib.linux-x86_64-3.12/readline.cpython-312-x86_64-linux-gnu.so+0x3ab2) (BuildId: a50d39ef2c15d0868c490d8e02f64b74edea405b)
 #2 0x557f32c393af (/home/user/cpython/python+0x9a03af) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #3 0x557f32c36cb8 (/home/user/cpython/python+0x99dcb8) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #4 0x557f327b7d68 (/home/user/cpython/python+0x51ed68) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #5 0x557f326002e9 (/home/user/cpython/python+0x3672e9) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #6 0x557f326012b5 (/home/user/cpython/python+0x3682b5) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #7 0x557f326016e3 (/home/user/cpython/python+0x3686e3) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #8 0x557f32afd88f (/home/user/cpython/python+0x86488f) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #9 0x557f32ad0cc2 (/home/user/cpython/python+0x837cc2) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #10 0x557f32cc4ce3 (/home/user/cpython/python+0xa2bce3) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #11 0x557f32cbbefe (/home/user/cpython/python+0xa22efe) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #12 0x557f32cba316 (/home/user/cpython/python+0xa21316) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #13 0x557f32d6975b (/home/user/cpython/python+0xad075b) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #14 0x557f32d6b41a (/home/user/cpython/python+0xad241a) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #15 0x557f32d6b5cd (/home/user/cpython/python+0xad25cd) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #16 0x557f324375d8 (/home/user/cpython/python+0x19e5d8) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #17 0x7f990342350f (/lib/x86_64-linux-gnu/libc.so.6+0x2350f) (BuildId: d1704d25fbbb72fa95d517b883131828c0883fe9)
 #18 0x7f99034235c8 (/lib/x86_64-linux-gnu/libc.so.6+0x235c8) (BuildId: d1704d25fbbb72fa95d517b883131828c0883fe9)
 #19 0x557f323b0e54 (/home/user/cpython/python+0x117e54) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)

 Uninitialized value was created by a heap allocation
 #0 0x557f323e4f00 (/home/user/cpython/python+0x14bf00) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
 #1 0x7f98fe699ddc (/lib/x86_64-linux-gnu/libreadline.so.8+0x3addc) (BuildId: 990587ed58ff96de9f94756458d4aaa2f51aaa3d)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/lib/x86_64-linux-gnu/libreadline.so.8+0x3204b) (BuildId: 990587ed58ff96de9f94756458d4aaa2f51aaa3d)