sequential/test-inspector-port-zero segfaults on macOS 10.10 and below

[gibfahn]

• Version: 8.9.0.0
• Platform: macOS <= 10.10.1
• Subsystem: libuv, getaddrinfo

This test fails consistently on earlier versions of macOS (used code in #16685 to work out that it was a segfault).

Original test case failure:

$ tools/test.py sequential/test-inspector-port-zero
=== release test-inspector-port-zero === 
Path: sequential/test-inspector-port-zero
assert.js:42
 throw new errors.AssertionError({
 ^

AssertionError [ERR_ASSERTION]: exitCode: null, signal: SIGSEGV
 at ChildProcess.proc.on.mustCall (/build/jenkins/n8-test/ab673161/node/test/sequential/test-inspector-port-zero.js:37:59)
 at ChildProcess.<anonymous> (/build/jenkins/n8-test/ab673161/node/test/common/index.js:533:15)
 at emitTwo (events.js:126:13)
 at ChildProcess.emit (events.js:214:7)
 at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
Command: out/Release/node /build/jenkins/n8-test/ab673161/node/test/sequential/test-inspector-port-zero.js
[00:00|% 100|+ 0|- 1]: Done 

Minimal reproduction:

Only one line of the test is failling:

node/test/sequential/test-inspector-port-zero.js

Line 46 in 5df0b9e

test('--inspect=localhost:0');

Which means you can reproduce with:

node --inspect=localhost:0
# Outputs: Segmentation fault: 11 (core dumped)

[gibfahn]

lldb thread list

(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff95e3456a libsystem_kernel.dylib`semaphore_wait_trap + 10, stop reason = signal SIGSTOP
 thread #2: tid = 0x0001, 0x00007fff95e39132 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
 thread #3: tid = 0x0002, 0x00007fff95e39132 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
 thread #4: tid = 0x0003, 0x00007fff95e39132 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
 thread #5: tid = 0x0004, 0x00007fff95e39132 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
 thread #6: tid = 0x0005, 0x00007fff95e3456a libsystem_kernel.dylib`semaphore_wait_trap + 10, stop reason = signal SIGSTOP
 thread #7: tid = 0x0006, 0x00007fff8d1bad14 libsystem_info.dylib`mdns_addrinfo + 335, stop reason = signal SIGSTOP
(lldb) thread backtrace 1
* thread #1: tid = 0x0000, 0x00007fff95e3456a libsystem_kernel.dylib`semaphore_wait_trap + 10, stop reason = signal SIGSTOP
 * frame #0: 0x00007fff95e3456a libsystem_kernel.dylib`semaphore_wait_trap + 10
 frame #1: 0x0000000100b5f6c5 node`uv_sem_wait + 16
 frame #2: 0x0000000100a4222e node`node::inspector::InspectorIo::Start() + 48
 frame #3: 0x0000000100a402cb node`node::inspector::Agent::StartIoThread(bool) + 149
 frame #4: 0x0000000100a401a0 node`node::inspector::Agent::Start(node::NodePlatform*, char const*, node::DebugOptions const&) + 560
 frame #5: 0x00000001009c8d80 node`node::Start(v8::Isolate*, node::IsolateData*, int, char const* const*, int, char const* const*) + 305
 frame #6: 0x00000001009c8b91 node`node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) + 454
 frame #7: 0x00000001009c7fe1 node`node::Start(int, char**) + 469
 frame #8: 0x0000000100001a34 node`start + 52
(lldb) thread backtrace 6
 thread #6: tid = 0x0005, 0x00007fff95e3456a libsystem_kernel.dylib`semaphore_wait_trap + 10, stop reason = signal SIGSTOP
 frame #0: 0x00007fff95e3456a libsystem_kernel.dylib`semaphore_wait_trap + 10
 frame #1: 0x0000000100b5f6c5 node`uv_sem_wait + 16
 frame #2: 0x0000000100a40cf0 node`node::inspector::(anonymous namespace)::StartIoThreadMain(void*) + 28
 frame #3: 0x00007fff97abb2fc libsystem_pthread.dylib`_pthread_body + 131
 frame #4: 0x00007fff97abb279 libsystem_pthread.dylib`_pthread_start + 176
 frame #5: 0x00007fff97ab94b1 libsystem_pthread.dylib`thread_start + 13
(lldb) thread backtrace 7
 thread #7: tid = 0x0006, 0x00007fff8d1bad14 libsystem_info.dylib`mdns_addrinfo + 335, stop reason = signal SIGSTOP
 frame #0: 0x00007fff8d1bad14 libsystem_info.dylib`mdns_addrinfo + 335
 frame #1: 0x00007fff8d1bab85 libsystem_info.dylib`search_addrinfo + 179
 frame #2: 0x00007fff8d1ba8da libsystem_info.dylib`si_addrinfo + 1395
 frame #3: 0x00007fff8d1ba2c3 libsystem_info.dylib`getaddrinfo + 179
 frame #4: 0x0000000100b58ee5 node`uv_getaddrinfo + 461
 frame #5: 0x0000000100a49613 node`node::inspector::InspectorSocketServer::Start() + 155
 frame #6: 0x0000000100a42c9a node`void node::inspector::InspectorIo::ThreadMain<node::inspector::InspectorSocketServer>() + 534
 frame #7: 0x00007fff97abb2fc libsystem_pthread.dylib`_pthread_body + 131
 frame #8: 0x00007fff97abb279 libsystem_pthread.dylib`_pthread_start + 176
 frame #9: 0x00007fff97ab94b1 libsystem_pthread.dylib`thread_start + 13

lldb process status

(lldb) process status
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff95e3456a libsystem_kernel.dylib`semaphore_wait_trap + 10, stop reason = signal SIGSTOP
 frame #0: 0x00007fff95e3456a libsystem_kernel.dylib`semaphore_wait_trap + 10
libsystem_kernel.dylib`semaphore_wait_trap:
-> 0x7fff95e3456a <+10>: retq 
 0x7fff95e3456b <+11>: nop 

libsystem_kernel.dylib`semaphore_wait_signal_trap:
 0x7fff95e3456c <+0>: movq %rcx, %r10
 0x7fff95e3456f <+3>: movl $0x1000025, %eax
 thread #2: tid = 0x0001, 0x00007fff95e39132 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
 frame #0: 0x00007fff95e39132 libsystem_kernel.dylib`__psynch_cvwait + 10
libsystem_kernel.dylib`__psynch_cvwait:
-> 0x7fff95e39132 <+10>: jae 0x7fff95e3913c ; <+20>
 0x7fff95e39134 <+12>: movq %rax, %rdi
 0x7fff95e39137 <+15>: jmp 0x7fff95e34ca3 ; cerror_nocancel
 0x7fff95e3913c <+20>: retq 
 thread #3: tid = 0x0002, 0x00007fff95e39132 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
 frame #0: 0x00007fff95e39132 libsystem_kernel.dylib`__psynch_cvwait + 10
libsystem_kernel.dylib`__psynch_cvwait:
-> 0x7fff95e39132 <+10>: jae 0x7fff95e3913c ; <+20>
 0x7fff95e39134 <+12>: movq %rax, %rdi
 0x7fff95e39137 <+15>: jmp 0x7fff95e34ca3 ; cerror_nocancel
 0x7fff95e3913c <+20>: retq 
 thread #4: tid = 0x0003, 0x00007fff95e39132 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
 frame #0: 0x00007fff95e39132 libsystem_kernel.dylib`__psynch_cvwait + 10
libsystem_kernel.dylib`__psynch_cvwait:
-> 0x7fff95e39132 <+10>: jae 0x7fff95e3913c ; <+20>
 0x7fff95e39134 <+12>: movq %rax, %rdi
 0x7fff95e39137 <+15>: jmp 0x7fff95e34ca3 ; cerror_nocancel
 0x7fff95e3913c <+20>: retq 
 thread #5: tid = 0x0004, 0x00007fff95e39132 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
 frame #0: 0x00007fff95e39132 libsystem_kernel.dylib`__psynch_cvwait + 10
libsystem_kernel.dylib`__psynch_cvwait:
-> 0x7fff95e39132 <+10>: jae 0x7fff95e3913c ; <+20>
 0x7fff95e39134 <+12>: movq %rax, %rdi
 0x7fff95e39137 <+15>: jmp 0x7fff95e34ca3 ; cerror_nocancel
 0x7fff95e3913c <+20>: retq 
 thread #6: tid = 0x0005, 0x00007fff95e3456a libsystem_kernel.dylib`semaphore_wait_trap + 10, stop reason = signal SIGSTOP
 frame #0: 0x00007fff95e3456a libsystem_kernel.dylib`semaphore_wait_trap + 10
libsystem_kernel.dylib`semaphore_wait_trap:
-> 0x7fff95e3456a <+10>: retq 
 0x7fff95e3456b <+11>: nop 

libsystem_kernel.dylib`semaphore_wait_signal_trap:
 0x7fff95e3456c <+0>: movq %rcx, %r10
 0x7fff95e3456f <+3>: movl $0x1000025, %eax
 thread #7: tid = 0x0006, 0x00007fff8d1bad14 libsystem_info.dylib`mdns_addrinfo + 335, stop reason = signal SIGSTOP
 frame #0: 0x00007fff8d1bad14 libsystem_info.dylib`mdns_addrinfo + 335
libsystem_info.dylib`mdns_addrinfo:
-> 0x7fff8d1bad14 <+335>: movw (%rdx), %ax
 0x7fff8d1bad17 <+338>: movw %ax, -0xd2(%rbp)
 0x7fff8d1bad1e <+345>: jmp 0x7fff8d1bad74 ; <+431>
 0x7fff8d1bad20 <+347>: movl %r9d, -0xec(%rbp)

[gibfahn]

(lldb) thread select 7
* thread #7, stop reason = signal SIGSTOP
 frame #0: 0x00007fff8d1bad14 libsystem_info.dylib`mdns_addrinfo + 335
libsystem_info.dylib`mdns_addrinfo:
-> 0x7fff8d1bad14 <+335>: movw (%rdx), %ax
 0x7fff8d1bad17 <+338>: movw %ax, -0xd2(%rbp)
 0x7fff8d1bad1e <+345>: jmp 0x7fff8d1bad74 ; <+431>
 0x7fff8d1bad20 <+347>: movl %r9d, -0xec(%rbp)

(lldb) register read 
General Purpose Registers:
 rax = 0x0000000107000440
 rbx = 0x0000000000000000
 rcx = 0x0000000000001000
 rdx = 0x0000000000000000
 rdi = 0x00007fff7a8357c8 si_module_static_mdns.si
 rsi = 0x0000000102600152
 rbp = 0x00000001070004e0
 rsp = 0x00000001070003c0
 r8 = 0x0000000000000001
 r9 = 0x0000000000000001
 r10 = 0x0000000000000a58
 r11 = 0x00007fff8fe10c00 libsystem_platform.dylib`_platform_memchr$VARIANT$Haswell
 r12 = 0x0000000102600152
 r13 = 0x0000000000000000
 r14 = 0x0000000000000000
 r15 = 0x0000000000000001
 rip = 0x00007fff8d1bad14 libsystem_info.dylib`mdns_addrinfo + 335
 rflags = 0x0000000000010202
 cs = 0x000000000000002b
 fs = 0x0000000000000000
 gs = 0x0000000000000000

So SEGV happened on movw (%rdx), %ax, which moves the value at address %rdx to %ax. However %rdx isn't a valid address (it's 0x0).

Looks related to this Python bug: https://bugs.python.org/issue17269

[gibfahn]

This only fails on earlier versions of macOS

Testing machines:

macOS Version	Pass/Fail
10.8.5	🔴
10.10.1	🔴
10.10.5	💚
10.11.4	💚
10.13.1	💚

Different node versions on 10.10.1

Version	Pass/Fail
8.9.0	🔴
8.6.0	🔴
8.3.0	🔴
8.1.2	💚

8.3.0.0 process.versions

{ http_parser: '2.7.0',
 node: '8.3.0',
 v8: '6.0.286.52',
 uv: '1.13.1',
 zlib: '1.2.11',
 ares: '1.10.1-DEV',
 modules: '57',
 openssl: '1.0.2l',
 icu: '59.1',
 unicode: '9.0',
 cldr: '31.0.1',
 tz: '2017b' }

8.1.2.0 process.versions

{ http_parser: '2.7.0',
 node: '8.1.2',
 v8: '5.8.283.41',
 uv: '1.12.0',
 zlib: '1.2.11',
 ares: '1.10.1-DEV',
 modules: '57',
 openssl: '1.0.2l',
 icu: '59.1',
 unicode: '9.0',
 cldr: '31.0.1',
 tz: '2017b' }

[eugeneo]

Is there a way I could run the code on those bots?

[refack]

I have an intuition that something with the threads is not safe (maybe access to the uv_loop_t)
That could also be the cause of #15558

[mhdawson]

@eugeneo you can run this job on your branch. It is the job we are using to test out new machines for the CI and it includes the earlier versions: https://ci.nodejs.org/view/All/job/node-test-commit-osx-macstadium/

[mhdawson]

And it shows the failure on 10.09 and 10.10.

[mhdawson]

@gibfahn are you investigating this one ? I think its the last blocker before we could enable more of the osx machines. As I look at the current osx test backlog I'm looking forward to that.

[gibfahn]

As I look at the current osx test backlog I'm looking forward to that.

What patch version of 10.10 are you on? I'd assume we want to be on the latest, in which case this shouldn't block 10.10 going into CI.

I am looking at this, but no reason we shouldn't mark the test as flaky on macOS <10.10 in the meantime.

[mhdawson]

@gdams I'm guessing we are on an older version of 10.10 as it would have started with what was installed from the cd. I wonder if we can add to our ansible script so that we upgrade as part of ansible config ?