Node 20.3 Crashes all the time when executed inside docker

[ronag]

• Node 20.3.0 crashes on start in non-bullseye docker container
• In bullseye container node-gyp-build (yarn add bufferutil) fails with Text file busy
• In bullseye container + UV_USE_IO_URING=0 everything works

[ronag]

Another possible ref electron/rebuild#1085

[ronag]

@nodejs/libuv

[bnoordhuis]

"Text file busy" means trying to write a shared object or binary that's already in use.

My hunch is that node-gyp has some race condition in reading/writing files that wasn't manifesting (much) when everything still went through the much slower thread pool, whereas io_uring is fast enough to make it much more visible.

[ronag]

Is there a way to disable ioring using a env variable as a temporary workaround when running node-gyp?

[bnoordhuis]

Yes, set UV_USE_IO_URING=0 in the environment. Use at your own risk: not a stable thing, will disappear again the future.

[ronag]

test-srv3.hq.lan:~# docker run -it node:20.3.0 bash
root@6f6c66eb5077:/# UV_USE_IO_URING=0 yarn add bufferutil
node[8]: ../src/node_platform.cc:68:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xc8e4a0 node::Abort() [node]
 2: 0xc8e51e [node]
 3: 0xd0a059 node::WorkerThreadsTaskRunner::WorkerThreadsTaskRunner(int) [node]
 4: 0xd0a17c node::NodePlatform::NodePlatform(int, v8::TracingController*, v8::PageAllocator*) [node]
 5: 0xc4bbc4 node::V8Platform::Initialize(int) [node]
 6: 0xc49408 [node]
 7: 0xc497db node::Start(int, char**) [node]
 8: 0x7ff94ec5818a [/lib/x86_64-linux-gnu/libc.so.6]
 9: 0x7ff94ec58245 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
10: 0xba9ade _start [node]
Aborted

[ronag]

This is actually worse than I thought. Node doesn't run at all with 20.3

[ronag]

test-srv3.hq.lan:~# docker run -it node:20.3.0 node
node[1]: ../src/node_platform.cc:68:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xc8e4a0 node::Abort() [node]
 2: 0xc8e51e [node]
 3: 0xd0a059 node::WorkerThreadsTaskRunner::WorkerThreadsTaskRunner(int) [node]
 4: 0xd0a17c node::NodePlatform::NodePlatform(int, v8::TracingController*, v8::PageAllocator*) [node]
 5: 0xc4bbc4 node::V8Platform::Initialize(int) [node]
 6: 0xc49408 [node]
 7: 0xc497db node::Start(int, char**) [node]
 8: 0x7f5c393be18a [/lib/x86_64-linux-gnu/libc.so.6]
 9: 0x7f5c393be245 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
10: 0xba9ade _start [node]
test-srv3.hq.lan:~# docker run -it node:20.2.0 node
Welcome to Node.js v20.2.0.
Type ".help" for more information.
> 

[mcollina]

I can reproduce this, and this is quite critical.

[mcollina]

cc @nodejs/tsc for visibility