Help request to compute the amount of disk space and bandwidth wasted by useless published items?

[ericmorand]

Select Topic Area

Question

Body

Dear maintainers,

I'd like to conduct a survey about the amount of disk space and bandwidth wasted on the npm registry by items that are never used by package consumers - i.e. items that are published in the registry and are never imported by projects and / or products, because they technically can't be imported or because they are not supposed to be.

Typically, the following items are a waste of resources, hanging around, only taking space in the registry and in the client systems:

• Source files, sometimes with inline source maps
• Source maps
• Project documentations (not package documentation)
• Artefacts expected to be deployed to CDNs which are published because CDN tools fetch them from the npm registry
• package.json attributes that are never considered by Node.js resolution algorithm or npm package homepage generator - e.g scripts, devDependencies, files, third-party project attributes like prettier one...

If you need an example of what I mean, svgicons2svgfont is a good one:

https://www.npmjs.com/package/svgicons2svgfont?activeTab=code

The package comes with sources, source maps, tests, and a package.json that holds 190 line when only around 80-90 are actually needed to consume the package. It consumes 425KB on my computer, when only 280KB are reachable and required to make it work. Around 30% of its size is just wasted.

My objective is to compute the total amount of space occupied by this uneeded published information, accross all packages (public and, ideally, private) in order to write a paper about it and, hopefully, raise awareness and initiate a change in both npm and client sides.

Is there a way to achieve such a thing, using a public API, or maybe by having access to some data that npm's maintainer are likely to possess? I don't care about the content - except for package.json files. I'm only interested in file types (path and extension) and their size.

Thanks in advance,

Eric.

[SafaSaifudeen]

So, there’s no official way to get per-file sizes for all packages directly from npm — is that because of privacy concerns or just lack of public API support?

[ericmorand]

I really don't know. I was expecting an endpoint dedicated to packages - e.g GET /packages - but the only thing that we have is a search endpoint.

[ericmorand]

The problem is not computing the size of one given package - this is trivial actually. The problem is to get the list of all the packages hosted by the registry.

[Danindu05]

[ericmorand]

If you’re analyzing bandwidth too, you might need to approximate based on tarball sizes (dist.size) since download stats aren’t public.

Actually, they do expose the download stats from their public API that serves HTML representations - aka the www API. The data structure is not documented but it can be inferred.

Anyway, thank you. I may be able to get what I need by mixing all their APIs together (the registry ones, if I happen to find their documentation, and the www ones). It will require quite a few requests to get the job done, but it should work.

[ericmorand]

Well, the thing is there is no way to retrieve the whole list of packages hosted in the registry, at least using one of the documented endpoints.