fs.realpath 70x slower than native

[stefanpenner]

repost of nodejs/node-v0.x-archive#7902 to ensure it is not lost, as per @jasnell suggestion.

credit goes to @joliss I am merely transplanting the issue.

---

The fs.realpath function is 70x slower than native C realpath. On my system, fs.realpath takes 32 µs, while C realpath takes 0.45 µs.

This is a real problem in the Broccoli build tool, where we need to resolve symlinks in hot code paths. Resolving 1000 symlinked files - not an unusual case - would take 45 ms, slowing down the build considerably. [1]

As for a solution: I haven't looked at the fs.js source in detail, but it seems we might be able to call the realpath function in the C standard library, where available, instead of using our own implementation.

Benchmark code for Node:

var fs = require('fs')

var start = Date.now()
var n = 10000
for (var i = 0; i < n; i++) {
 if (fs.realpathSync('.') === 'dummy') throw new Error('never happens')
}
console.log(((Date.now() - start) * 1000 / n) + ' us') // => 32 us on Node 0.11.13

Benchmark code for C:

#include <limits.h> /* PATH_MAX */
#include <stdio.h>
#include <stdlib.h>

// Adapted from http://stackoverflow.com/a/1563237/525872

int main(void) {
 char buf[PATH_MAX + 1]; /* not sure about the "+ 1" */
 int i;
 for (i = 0; i < 1000000; i++) {
 char *res = realpath(".", buf);
 if (res) {
 // printf("This source is at %s.\n", buf);
 } else {
 perror("realpath");
 exit(EXIT_FAILURE);
 }
 }
 return 0;
}

Run with gcc -std=gnu99 realpath-benchmark.c -o realpath-benchmark && time ./realpath-benchmark. This yields 0.45 µs per iteration on my Linux system.

[1] We cannot work around this by using naïve string concatenation, because path_resolution(7) requires that we resolve symlinks in all path components. Here is a gist to show why this matters.

[ChALkeR]

@stefanpenner, about your hot path: does it modify those symlinks (or parent directories) in the same hot path? If not, you could pass a cache argument, it would speed up fs.realpath (and fs.realpathSync) several times. Not 70, though.

For example, if you resolve a lot of symlinks in the same dir in a loop, it could help to use a single cache for all those fs.realpath calls.

[ChALkeR]

Ah. @joliss ↑↑

[stefanpenner]

@ChALkeR i was about to implement something (what i believe to be) very similar. Can you share docs on cache argument?

cache is an object literal of mapped paths that can be used to force a specific path resolution or avoid additional fs.stat calls for known real paths.

-- source: the docs

[Fishrock123]

@stefanpenner https://iojs.org/api/fs.html#fs_fs_realpath_path_cache_callback

[ChALkeR]

@stefanpenner Also, each fs.realpath call updates the passed cache object, when needed.
So using a single object (initialized with {} at the start) for several calls would save some time.

[ChALkeR]

@Fishrock123 Btw I think the above fact isn't properly documented.

[stefanpenner]

@Fishrock123, btw, I think the above fact isn't properly documented.

I would agree, but largely due to this pattern not being very common in the stdlib, so i wasn't looking for it.

I'll give this pattern a try in the next day or so and report back. It may prove to be a nice win.

That being said, we shouldn't assume this is the solution and a fast realpath should still be the goal.

[Fishrock123]

Repost of @trevnorris at nodejs/node-v0.x-archive#7902 (comment)

Here's the flamegraph for fs.realpathSync('.'): https://i.cloudup.com/n9pPZFuyc0.svg

[Fishrock123]

Fwiw I'm +1 on using realpath(3) when possible.

[thefourtheye]

Why can't this be proposed to uv and we use it?

[Fishrock123]

Why can't this be proposed to uv and we use it?

Depends if it is available cross-platform or not.

Looks like this would be on windows though? https://msdn.microsoft.com/en-us/library/windows/desktop/aa364963(v=vs.85).aspx

[stefanpenner]

IMHO Libuv would be the ideal place for this.