`Buffer.from(str)` memory leak

[Daninet]

• Version: v14.16.1, v15.14.0, v16.0.0
• Platform: Microsoft Windows NT 10.0.19042.0 x64
• Subsystem: Buffer

What steps will reproduce the bug?

Run the following code with node --expose-gc

const str = 'x'.repeat(10000);

console.log('start', process.memoryUsage());

for(let i = 0; i < 1e6; i++) {
 Buffer.from(str); // memory leak happens here
 if (i % 1e5 === 0) {
 gc();
 console.log('step', i, 'rss', process.memoryUsage().rss);
 }
}

gc();
console.log('end', process.memoryUsage());

How often does it reproduce? Is there a required condition?

It reproduces consistently with v14.16.1 and v15.14.0.
It only reproduces with a string parameter. Arrays are not causing issues.
I also tested some older versions (v10.22.0, v12.8.1) and the leak is not present there.

What is the expected behavior?

Created buffers should be garbage collected, and memory usage shouldn't increase too much during execution.

What do you see instead?

Memory usage increases to over 10 GB with the supplied code snippet. I get OOM errors when I increase the number of iterations.

start{
 rss: 18243584,
 heapTotal: 4055040,
 heapUsed: 3168160,
 external: 260263,
 arrayBuffers: 10778
}
step 0 rss 19763200
step 100000 rss 1078853632
step 200000 rss 2133364736
step 300000 rss 3189673984
step 400000 rss 4241575936
step 500000 rss 5293998080
step 600000 rss 6354276352
step 700000 rss 7406329856
step 800000 rss 8458436608
step 900000 rss 9510563840
end{
 rss: 10562609152,
 heapTotal: 5980160,
 heapUsed: 3170888,
 external: 21400146,
 arrayBuffers: 10054
}

[Ayase-252]

rss seems stabilized around 5G in my machine (MacOS).

out/Release/node -v 
v16.0.0-pre


out/Release/node --expose-gc index-2.js
start{
 rss: 21245952,
 heapTotal: 4050944,
 heapUsed: 3169480,
 external: 253370,
 arrayBuffers: 10762
}
step 0 rss 22884352
step 100000 rss 1076359168
step 200000 rss 2127024128
step 300000 rss 3178852352
step 400000 rss 4227698688
step 500000 rss 4917829632
step 600000 rss 4834082816
step 700000 rss 4781207552
step 800000 rss 4714610688
step 900000 rss 4624203776
end{
 rss: 4560502784,
 heapTotal: 5894144,
 heapUsed: 3159872,
 external: 21382689,
 arrayBuffers: 10038
}

[bytemain]

Macos 11

nodejs 14:

start{
 rss: 20209664,
 heapTotal: 4206592,
 heapUsed: 2560312,
 external: 828505,
 arrayBuffers: 9898
}
step 0 rss 21798912
step 100000 rss 1076994048
step 200000 rss 2127327232
step 300000 rss 3179413504
step 400000 rss 4227592192
step 500000 rss 4739432448
step 600000 rss 4694577152
step 700000 rss 4600496128
step 800000 rss 4497977344
step 900000 rss 4408086528
end{
 rss: 4287864832,
 heapTotal: 7929856,
 heapUsed: 2226328,
 external: 37181504,
 arrayBuffers: 9898
}

nodejs 12:

start{
 rss: 19390464,
 heapTotal: 4644864,
 heapUsed: 2476832,
 external: 782967,
 arrayBuffers: 9386
}
step 0 rss 21192704
step 100000 rss 128675840
step 200000 rss 141754368
step 300000 rss 138477568
step 400000 rss 141754368
step 500000 rss 129318912
step 600000 rss 133066752
step 700000 rss 144846848
step 800000 rss 141770752
step 900000 rss 153034752
end{
 rss: 147271680,
 heapTotal: 7581696,
 heapUsed: 2168520,
 external: 933915,
 arrayBuffers: 53559386
}

[rensrongen]

I experience this issue running Node v15.14.0 on Ubuntu 20.04 (LTS) x64.

Basically, I have a long-running server process that runs out of memory and crashes periodically. It makes frequent Buffer.from() method calls.

The problem started after upgrading to v15.14.0.

[Ayase-252]

It seems the issue did not happen in v14.0.0. I would do bisect to look which change is suspicious.

v14.0.0

node --expose-gc index.js
start{
 rss: 19959808,
 heapTotal: 4149248,
 heapUsed: 2313384,
 external: 764993,
 arrayBuffers: 9382
}
step 0 rss 21532672
step 100000 rss 127037440
step 200000 rss 129449984
step 300000 rss 141750272
step 400000 rss 133693440
step 500000 rss 136900608
step 600000 rss 135507968
step 700000 rss 128778240
step 800000 rss 133877760
step 900000 rss 130506752
end{
 rss: 132894720,
 heapTotal: 7610368,
 heapUsed: 2092368,
 external: 925930,
 arrayBuffers: 51169382
}

[rensrongen]

FWIW the issue was not present in v15.11.0 from which I upgraded.

[Daninet]

I just tested it, node.js v15.11.0 / Windows 10 / x64 leaks memory on my computer with the supplied code.

[Linkgoron]

I'm seeing a relatively large change between v14.4 and 14.5

[Daninet]

I can confirm, it was broken with v14.5.0.
v14.4.0 works fine.

[XadillaX]

Try this code?

const str = 'x'.repeat(10000);

console.log('start', process.memoryUsage());

function wait(ms) {
 let resolve;
 const promise = new Promise((_resolve, _reject) => { resolve = _resolve; });

 setTimeout(() => {
 resolve();
 }, ms);

 return promise;
}

(async () => {
 for(let i = 0; i < 1e6; i++) {
 Buffer.from(str); // memory leak happens here
 if (i % 1e5 === 0) {
 await wait(100);
 gc();
 console.log('step', i, 'rss', process.memoryUsage().rss);
 }
 }
})().then(() => {
 gc();
 console.log('end', process.memoryUsage());
});

[bytemain]

v14.16.0

XadillaX's code:

start{
 rss: 20185088,
 heapTotal: 4206592,
 heapUsed: 2555408,
 external: 828505,
 arrayBuffers: 9898
}
step 0 rss 21876736
step 100000 rss 251977728
step 200000 rss 279126016
step 300000 rss 287125504
step 400000 rss 297897984
step 500000 rss 304611328
step 600000 rss 257728512
step 700000 rss 280563712
step 800000 rss 270651392
step 900000 rss 264871936
end{
 rss: 1112817664,
 heapTotal: 7667712,
 heapUsed: 2237200,
 external: 33190872,
 arrayBuffers: 9898
}

[Daninet]

Yeah, I does not happen if the control goes back to the event loop. But still, it's not a solution, given that a lot of libraries / code rely on synchronous processing of buffers. It would be a serious disadvantage of using Buffers compared to Uint8Arrays.