investigate flaky test-domain-error-types in CI

[Trott]

test/parallel/test-domain-error-types.js is flaky on containered shared-lib debug builds in CI.

Example failures:

https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/15829/consoleText

not ok 486 parallel/test-domain-error-types
 ---
 duration_ms: 3.479
 severity: crashed
 exitcode: -11
 stack: |-
 ...

https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/15824/consoleText

not ok 485 parallel/test-domain-error-types
 ---
 duration_ms: 3.577
 severity: crashed
 exitcode: -11
 stack: |-
 ...

https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/15820/consoleText

not ok 484 parallel/test-domain-error-types
 ---
 duration_ms: 3.53
 severity: crashed
 exitcode: -11
 stack: |-
 ...

Not sure who is in a position to dig in to figure this out so I might be pinging a bit more widely than some people may prefer, so apologies in advance if so. Pings: @nodejs/build @nodejs/testing @nodejs/domains @nodejs/docker @nodejs/diagnostics

[Trott]

This is happening a lot. I'm not sure it's necessary to paste more of these in here at this point, but here's one more.

https://ci.nodejs.org/job/node-test-commit-linux-containered/15862/nodes=ubuntu1804_sharedlibs_debug_x64/consoleText

not ok 592 parallel/test-domain-error-types
 ---
 duration_ms: 2.530
 severity: crashed
 exitcode: -11
 stack: |-
 ...

[Trott]

@addaleax

[gireeshpunathil]

today seen again:

09:00:01 not ok 548 parallel/test-domain-error-types
09:00:01 ---
09:00:01 duration_ms: 1.718
09:00:01 severity: crashed
09:00:01 exitcode: -11
09:00:01 stack: |-

ref: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/15910/consoleFull

[addaleax]

Backtrace with optimized V8 debug:

#0 v8::internal::Bitmap::MarkBitFromIndex (index=<optimized out>, this=<optimized out>) at ../deps/v8/src/heap/marking.h:135
#1 v8::internal::MarkingStateBase<v8::internal::ConcurrentMarkingState, (v8::internal::AccessMode)0>::MarkBitFrom (this=0x7f4d5ea6be20, addr=<optimized out>, p=<optimized out>) at ../deps/v8/src/heap/mark-compact.h:42
#2 v8::internal::MarkingStateBase<v8::internal::ConcurrentMarkingState, (v8::internal::AccessMode)0>::MarkBitFrom (this=0x7f4d5ea6be20, obj=...) at ../deps/v8/src/heap/mark-compact.h:36
#3 v8::internal::MarkingStateBase<v8::internal::ConcurrentMarkingState, (v8::internal::AccessMode)0>::WhiteToGrey (this=0x7f4d5ea6be20, obj=...) at ../deps/v8/src/heap/mark-compact-inl.h:35
#4 v8::internal::ConcurrentMarkingVisitor::MarkObject (object=..., this=0x7f4d5ea6bdf0) at ../deps/v8/src/heap/concurrent-marking.cc:540
#5 v8::internal::ConcurrentMarkingVisitor::ProcessStrongHeapObject<v8::internal::FullHeapObjectSlot> (heap_object=..., slot=..., host=..., this=0x7f4d5ea6bdf0) at ../deps/v8/src/heap/concurrent-marking.cc:115
#6 v8::internal::ConcurrentMarkingVisitor::VisitPointersImpl<v8::internal::FullObjectSlot> (this=this@entry=0x7f4d5ea6bdf0, host=..., host@entry=..., start=start@entry=..., end=...) at ../deps/v8/src/heap/concurrent-marking.cc:158
#7 0x00000000013a8558 in v8::internal::ConcurrentMarkingVisitor::VisitPointers (end=..., start=..., host=..., this=0x7f4d5ea6bdf0) at ../deps/v8/src/heap/concurrent-marking.cc:140
#8 v8::internal::BodyDescriptorBase::IteratePointers<v8::internal::ConcurrentMarkingVisitor> (obj=..., obj@entry=..., start_offset=start_offset@entry=8, end_offset=end_offset@entry=48, v=v@entry=0x7f4d5ea6bdf0) at ../deps/v8/src/objects/objects-body-descriptors-inl.h:127
#9 0x00000000013ac927 in v8::internal::FixedBodyDescriptor<8, 48, 48>::IterateBody<v8::internal::ConcurrentMarkingVisitor> (v=0x7f4d5ea6bdf0, obj=..., map=...) at ../deps/v8/src/objects/objects-body-descriptors.h:93
#10 v8::internal::FixedBodyDescriptor<8, 48, 48>::IterateBody<v8::internal::ConcurrentMarkingVisitor> (object_size=<optimized out>, v=0x7f4d5ea6bdf0, obj=..., map=...) at ../deps/v8/src/objects/objects-body-descriptors.h:99
#11 v8::internal::SubclassBodyDescriptor<v8::internal::FixedBodyDescriptor<8, 48, 48>, v8::internal::FixedBodyDescriptor<56, 72, 72> >::IterateBody<v8::internal::ConcurrentMarkingVisitor> (object_size=<optimized out>, v=0x7f4d5ea6bdf0, obj=..., map=...) at ../deps/v8/src/objects/objects-body-descriptors.h:173
#12 v8::internal::HeapVisitor<int, v8::internal::ConcurrentMarkingVisitor>::VisitSyntheticModule (object=..., map=..., this=0x7f4d5ea6bdf0) at ../deps/v8/src/heap/objects-visiting-inl.h:98
#13 v8::internal::HeapVisitor<int, v8::internal::ConcurrentMarkingVisitor>::Visit (this=this@entry=0x7f4d5ea6bdf0, map=..., object=object@entry=...) at ../deps/v8/src/heap/objects-visiting-inl.h:45
#14 0x00000000013addbc in v8::internal::ConcurrentMarking::Run (this=0x561b5b0, task_id=<optimized out>, task_state=<optimized out>) at ../deps/v8/src/heap/concurrent-marking.cc:815
#15 0x00000000012c005e in v8::internal::CancelableTask::Run (this=<optimized out>) at ../deps/v8/src/tasks/cancelable-task.h:155
#16 0x0000000000fa468d in node::(anonymous namespace)::PlatformWorkerThread (data=0x55a72e0) at ../src/node_platform.cc:45
#17 0x00007f4d5f6406ba in start_thread (arg=0x7f4d5ea6d700) at pthread_create.c:333
#18 0x00007f4d5f37641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

I’ll try to get one from a non-optimized build but this is most likely a V8 GC bug.

@nodejs/v8

[addaleax]

Full stack trace:

#0 0x000000000203cb2f in v8::base::Relaxed_Load (ptr=0xdeadbeedbeac0008) at ../deps/v8/src/base/atomicops_internals_portable.h:183
#1 0x0000000002396aa6 in v8::base::AsAtomicImpl<long>::Relaxed_Load<unsigned long const> (addr=0xdeadbeedbeac0008) at ../deps/v8/src/base/atomic-utils.h:77
#2 0x0000000002396a84 in v8::internal::BasicMemoryChunk::GetFlags<(v8::internal::AccessMode)0> (this=0xdeadbeedbeac0000) at ../deps/v8/src/heap/basic-memory-chunk.h:152
#3 0x0000000002392b3d in v8::internal::BasicMemoryChunk::IsFlagSet<(v8::internal::AccessMode)0> (this=0xdeadbeedbeac0000, flag=v8::internal::BasicMemoryChunk::NEVER_EVACUATE) at ../deps/v8/src/heap/basic-memory-chunk.h:135
#4 0x0000000002393ca5 in v8::internal::MemoryChunk::IsEvacuationCandidate<(v8::internal::AccessMode)0> (this=0xdeadbeedbeac0000) at ../deps/v8/src/heap/spaces.h:806
#5 0x000000000238f280 in v8::internal::MarkCompactCollector::RecordSlot (object=..., slot=..., target=...) at ../deps/v8/src/heap/mark-compact-inl.h:486
#6 0x000000000248be84 in v8::internal::MarkingVisitor<(v8::internal::FixedArrayVisitationMode)0, (v8::internal::TraceRetainingPathMode)0, v8::internal::IncrementalMarkingState>::VisitPointerImpl<v8::internal::FullObjectSlot> (this=0x7ffe2d8e7630, host=..., slot=...) at ../deps/v8/src/heap/mark-compact-inl.h:298
#7 0x0000000002485edf in v8::internal::MarkingVisitor<(v8::internal::FixedArrayVisitationMode)0, (v8::internal::TraceRetainingPathMode)0, v8::internal::IncrementalMarkingState>::VisitPointer (this=0x7ffe2d8e7630, host=..., p=...) at ../deps/v8/src/heap/mark-compact.h:955
#8 0x000000000248bdaf in v8::internal::MarkingVisitor<(v8::internal::FixedArrayVisitationMode)0, (v8::internal::TraceRetainingPathMode)0, v8::internal::IncrementalMarkingState>::VisitPointersImpl<v8::internal::FullObjectSlot> (this=0x7ffe2d8e7630, host=..., start=..., end=...) at ../deps/v8/src/heap/mark-compact-inl.h:323
#9 0x0000000002485d58 in v8::internal::MarkingVisitor<(v8::internal::FixedArrayVisitationMode)0, (v8::internal::TraceRetainingPathMode)0, v8::internal::IncrementalMarkingState>::VisitPointers (this=0x7ffe2d8e7630, host=..., start=..., end=...) at ../deps/v8/src/heap/mark-compact.h:962
#10 0x000000000248bb58 in v8::internal::BodyDescriptorBase::IteratePointers<v8::internal::MarkingVisitor<(v8::internal::FixedArrayVisitationMode)0, (v8::internal::TraceRetainingPathMode)0, v8::internal::IncrementalMarkingState> > (obj=..., start_offset=8, end_offset=48, v=0x7ffe2d8e7630) at ../deps/v8/src/objects/objects-body-descriptors-inl.h:127
#11 0x000000000248eb6f in v8::internal::FixedBodyDescriptor<8, 48, 48>::IterateBody<v8::internal::MarkingVisitor<(v8::internal::FixedArrayVisitationMode)0, (v8::internal::TraceRetainingPathMode)0, v8::internal::IncrementalMarkingState> > (map=..., obj=..., v=0x7ffe2d8e7630) at ../deps/v8/src/objects/objects-body-descriptors.h:93
#12 0x000000000248c316 in v8::internal::FixedBodyDescriptor<8, 48, 48>::IterateBody<v8::internal::MarkingVisitor<(v8::internal::FixedArrayVisitationMode)0, (v8::internal::TraceRetainingPathMode)0, v8::internal::IncrementalMarkingState> > (map=..., obj=..., object_size=72, v=0x7ffe2d8e7630) at ../deps/v8/src/objects/objects-body-descriptors.h:99
#13 0x00000000024869b9 in v8::internal::SubclassBodyDescriptor<v8::internal::FixedBodyDescriptor<8, 48, 48>, v8::internal::FixedBodyDescriptor<56, 72, 72> >::IterateBody<v8::internal::MarkingVisitor<(v8::internal::FixedArrayVisitationMode)0, (v8::internal::TraceRetainingPathMode)0, v8::internal::IncrementalMarkingState> > (map=..., obj=..., object_size=72, v=0x7ffe2d8e7630) at ../deps/v8/src/objects/objects-body-descriptors.h:173
#14 0x000000000247c8b2 in v8::internal::HeapVisitor<int, v8::internal::MarkingVisitor<(v8::internal::FixedArrayVisitationMode)0, (v8::internal::TraceRetainingPathMode)0, v8::internal::IncrementalMarkingState> >::VisitSyntheticModule (this=0x7ffe2d8e7630, map=..., object=...) at ../deps/v8/src/heap/objects-visiting-inl.h:98
#15 0x0000000002472400 in v8::internal::HeapVisitor<int, v8::internal::MarkingVisitor<(v8::internal::FixedArrayVisitationMode)0, (v8::internal::TraceRetainingPathMode)0, v8::internal::IncrementalMarkingState> >::Visit (this=0x7ffe2d8e7630, map=..., object=...) at ../deps/v8/src/heap/objects-visiting-inl.h:45
#16 0x0000000002467e42 in v8::internal::MarkCompactCollector::ProcessMarkingWorklistInternal<(v8::internal::MarkCompactCollector::MarkingWorklistProcessingMode)0> (this=0x700f770) at ../deps/v8/src/heap/mark-compact.cc:1754
#17 0x0000000002451e64 in v8::internal::MarkCompactCollector::ProcessMarkingWorklist (this=0x700f770) at ../deps/v8/src/heap/mark-compact.cc:1721
#18 0x0000000002452c75 in v8::internal::MarkCompactCollector::MarkLiveObjects (this=0x700f770) at ../deps/v8/src/heap/mark-compact.cc:1863
#19 0x000000000244cb43 in v8::internal::MarkCompactCollector::CollectGarbage (this=0x700f770) at ../deps/v8/src/heap/mark-compact.cc:514
#20 0x00000000023f7d25 in v8::internal::Heap::MarkCompact (this=0x6fd29a8) at ../deps/v8/src/heap/heap.cc:2179
#21 0x00000000023f6a3c in v8::internal::Heap::PerformGarbageCollection (this=0x6fd29a8, collector=v8::internal::MARK_COMPACTOR, gc_callback_flags=v8::kNoGCCallbackFlags) at ../deps/v8/src/heap/heap.cc:1955
#22 0x00000000023f4955 in v8::internal::Heap::CollectGarbage (this=0x6fd29a8, space=v8::internal::NEW_SPACE, gc_reason=v8::internal::GarbageCollectionReason::kAllocationFailure, gc_callback_flags=v8::kNoGCCallbackFlags) at ../deps/v8/src/heap/heap.cc:1545
#23 0x00000000024037c3 in v8::internal::Heap::AllocateRawWithLightRetrySlowPath (this=0x6fd29a8, size=16, allocation=v8::internal::AllocationType::kYoung, origin=v8::internal::AllocationOrigin::kRuntime, alignment=v8::internal::kWordAligned) at ../deps/v8/src/heap/heap.cc:4918
#24 0x00000000024038dc in v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath (this=0x6fd29a8, size=16, allocation=v8::internal::AllocationType::kYoung, origin=v8::internal::AllocationOrigin::kRuntime, alignment=v8::internal::kWordAligned) at ../deps/v8/src/heap/heap.cc:4933
#25 0x00000000023ce913 in v8::internal::Heap::AllocateRawWith<(v8::internal::Heap::AllocationRetryMode)1> (this=0x6fd29a8, size=16, allocation=v8::internal::AllocationType::kYoung, origin=v8::internal::AllocationOrigin::kRuntime, alignment=v8::internal::kWordAligned) at ../deps/v8/src/heap/heap-inl.h:269
#26 0x00000000023b21a7 in v8::internal::Factory::AllocateRawWithImmortalMap (this=0x6fc9640, size=16, allocation=v8::internal::AllocationType::kYoung, map=..., alignment=v8::internal::kWordAligned) at ../deps/v8/src/heap/factory.cc:214
#27 0x00000000023bbcc4 in v8::internal::Factory::NewForeign (this=0x6fc9640, addr=31307306) at ../deps/v8/src/heap/factory.cc:1737
#28 0x00000000023c48d0 in v8::internal::Factory::NewSyntheticModule (this=0x6fc9640, module_name=..., export_names=..., evaluation_steps=0x1ddb62a <node::loader::ModuleWrap::SyntheticModuleEvaluationStepsCallback(v8::Local<v8::Context>, v8::Local<v8::Module>)>) at ../deps/v8/src/heap/factory.cc:3079
#29 0x000000000207423b in v8::Module::CreateSyntheticModule (isolate=0x6fc9640, module_name=..., export_names=std::vector of length 15, capacity 15 ={...}, evaluation_steps=0x1ddb62a <node::loader::ModuleWrap::SyntheticModuleEvaluationStepsCallback(v8::Local<v8::Context>, v8::Local<v8::Module>)>) at ../deps/v8/src/api/api.cc:2363
#30 0x0000000001dd2a05 in node::loader::ModuleWrap::New (args=...) at ../src/module_wrap.cc:168
#31 0x000000000215e715 in v8::internal::FunctionCallbackArguments::Call (this=0x7ffe2d8e81c0, handler=...) at ../deps/v8/src/api/api-arguments-inl.h:158
#32 0x0000000002160d18 in v8::internal::(anonymous namespace)::HandleApiCallHelper<true> (isolate=0x6fc9640, function=..., new_target=..., fun_data=..., receiver=..., args=...) at ../deps/v8/src/builtins/builtins-api.cc:111
#33 0x000000000215efb4 in v8::internal::Builtin_Impl_HandleApiCall (args=..., isolate=0x6fc9640) at ../deps/v8/src/builtins/builtins-api.cc:137
#34 0x000000000215ee2f in v8::internal::Builtin_HandleApiCall (args_length=9, args_object=0x7ffe2d8e83d8, isolate=0x6fc9640) at ../deps/v8/src/builtins/builtins-api.cc:147
#35 0x00000000030d8c60 in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit () at ../deps/v8/src/builtins/builtins-async-iterator-gen.cc:268
#36 0x0000000002e5bc01 in Builtins_JSBuiltinsConstructStub () at ../../deps/v8/../../deps/v8/src/builtins/base.tq:412

[addaleax]

I’m not really sure how to debug this correctly … valgrind is impractical here, I can’t run the test file in debug mode under it in less than three hours and even then I’d assume the test would still be flaky.

If the @nodejs/v8 team has any guidance/ideas for next steps, I’m all ears :)

[targos]

ptr=0xdeadbeedbeac0008

This looks a lot like kZapValue:

node/deps/v8/src/common/globals.h

Line 529 in eb30b2c

constexpr uint64_t kZapValue = uint64_t{0xdeadbeedbeadbeef};

[Trott]

Do we think we're likely to have something we can do here soon-ish or should we mark the test as flaky (or, probably better, skip on the one platform where there's a problem, if possible)?

[Trott]

Skip it for now on the one CI configuration where it fails frequently? #30629