Piping readable stream to multiple writeable streams results in crash

[kr3l]

If I execute the following code snippet in node >= 4.2.0, it results in the read stream reading the data very fast but not being able to write them out to wr2 equally fast. The memory usage of the process increases very fast until the process exits with an error.

var fs = require('fs');

//M: and L: are local disks with fast read and write
// the .mov file is several gigabytes in size
var rd = fs.createReadStream('M:/A007C003_141024_R7C6.mov');
var wr = fs.createWriteStream('L:/A007C003_141024_R7C6.mov');

//N: is a NAS drive, slow
var wr2 = fs.createWriteStream('N:/A007C003_141024_R7C6.mov');

//read from fast disk, write to both fast and slow disk
rd.pipe(wr);
rd.pipe(wr2);

The script ends with this error:

.\node-4.2.0 test.js
buffer.js:98
 const ui8 = new Uint8Array(size);
 ^

RangeError: Invalid array buffer length
 at new ArrayBuffer (native)
 at new Uint8Array (native)
 at allocate (buffer.js:98:17)
 at new Buffer (buffer.js:49:12)
 at Function.Buffer.concat (buffer.js:234:16)
 at fromList (_stream_readable.js:840:20)
 at ReadStream.Readable.read (_stream_readable.js:339:11)
 at flow (_stream_readable.js:743:26)
 at WriteStream.<anonymous> (_stream_readable.js:601:7)
 at emitNone (events.js:67:13)
 at WriteStream.emit (events.js:166:7)

Earlier versions of node (4.1.2 and below) do not show the same behaviour.

• Version: v4.2.0
• Platform: Win10 64bit
• Subsystem:

[Fishrock123]

v4.2.0 has this commit: #2325 -- #2325

Sounds like it might be related?

[calvinmetcalf]

though the error in question isn't actually an error related to running out of memory which is the error you'd expect if it was actually due to improper back pressure handling between 2 streams.

[kr3l]

From the error and the observed memory behaviour, maybe the 'size' variable is becoming too big and overflowing before another memory-related error can occur?

[addaleax]

There’s a maximum size of 2^31-1 bytes for a Buffer, so that would definitely make sense, even without an actual overflow occurring.

[addaleax]

I’ve looked a bit into this and it seems @Fishrock123 is right, #2325 effectively – and probably inadvertently – disabled the readableState.awaitDrain functionality that’s supposed to keep the destination streams in sync (if I read the code correctly).

[Fishrock123]

cc @mscdex here too

[addaleax]

PR: #6023