Dependency research · 13

Do popular packages ship what is in their source?

Reviewers read git. Users install from the registry. We compared the two for the 1,000 most-downloaded npm and PyPI packages. Where the comparison could be made, 98% matched. For about one release in thirteen, it could not be made at all.

Measured 12 Oct 2026 · corrected 9 Nov 2026 · 1,000 packages, 827 compared · npm and PyPI · depproof research

Two copies of every release

Every package release exists twice. There is the source in a public repository, which is what code review, security audits and curious engineers read. And there is the artifact on the registry, which is what npm install and pip install actually fetch. Most of the time nobody checks that the two agree, and that gap is part of the application security risk that sits in your dependencies.

They do not have to be identical. Packages legitimately ship build output that is not in git. But some things decide what runs on your machine at install, and should not change between the two without a reason: the scripts a package runs on install, the dependencies it pulls in, the commands it adds, and on PyPI the .pth files that run at every Python start.

That matters for a practical reason. Several widely reported registry compromises were published straight to the registry with a maintainer's stolen credentials, with no matching change in git. A release that declares something its source does not is a sensible place for review to start.

What we compared

The 500 most-downloaded packages on npm and the 500 most-downloaded on PyPI, each at its latest release on 12 October 2026. For each, we took the artifact users install and the source at the git tag for that release, and compared what each declares at install level: install scripts, dependencies and command-line entry points, and on PyPI also .pth files and setup.py.

Where it can be checked, it almost always matches

Of the 827 releases that could be compared reliably, 810 declare nothing at install level that their source does not. That is 98%. Fifteen of those differ only in fields that do not affect what happens at install.

For 117 PyPI packages that declare their dependencies only inside an executable setup.py, the dependency comparison could not be made; their other checks ran. 110 of them are in the "matches" row. Leaving them out entirely, 700 of 710 match, which is also 98%.

The table shows the 12 October run. The 53 releases our first matching wrongly counted as untraceable were compared on 9 November: 45 match, 2 differ only by a setup.py generated at build time, 1 comes from a repository that publishes several packages, and 5 were too large to fetch. With them added, 855 of 874 releases compared reliably match, which is still 98%, with no new unexplained difference.

The 17 that differ, and why most of them do

Seventeen releases declare something at install level that their source does not. For 13 the cause is an ordinary step in how the package is built and published:

The remaining 4 were not attributed automatically. We have no evidence either way about them, and they are under review. We are not naming them, because an unexplained difference is a question, not a finding.

The useful observation is the first category, not the last. Install scripts and binaries added at publish time are normal practice, and they are also exactly the part of a release that a reviewer reading git never sees.

About one in thirteen cannot be checked at all

The larger finding sits before the comparison. 77 of the 1,000 releases cannot be checked against their source, because the package publishes no source-repository link (24) or the repository has no git tag for the published version (53). That is about 8%: 37 on npm and 40 on PyPI. Up to five more sit on code hosts whose tags we could not read. (The first version of this study said 133; see the correction above.)

That is not evidence of anything wrong. Plenty of projects do not tag every release, and large repositories that publish many packages often tag the repository rather than each package. But for those releases there is no path from what people install to a reviewable commit, and nobody outside the project can say which source a given release was built from.

What this changes

For most popular packages, reviewing the source is a fair proxy for what you install, at least at the level of what runs during installation. That is worth knowing, and it is good news for anyone assessing application security risk in third-party code.

It stops being true in two places. For about one release in thirteen there is no source to review against, so the evidence has to come from the artifact itself, or from what the package does once it is installed. And where a release does differ, the difference sits exactly where install-time behaviour lives. A review process that reads only git sees neither, and a scanner that matches CVEs will not flag it either, because there is no CVE to find. That makes it a software supply chain question as much as an AppSec one.

What this does not prove

The part we have not solved

If about one release in thirteen cannot be traced to its source, what does a reasonable team do with those, short of refusing to use them? Some things we would like argued with: should registries require a repository link and a matching tag? Should a published-only dependency list, like a set of platform binaries, be shown to reviewers explicitly? And if you maintain one of these packages, have we described a difference wrongly?

Where depproof fits

depproof records what each dependency actually does during your own installs and tests, so the evidence does not depend on whether its source can be traced.

If you re-run this and get something different, we would rather hear it. hello@depproof.com.

Related reading: managing application security risk in dependencies · what software supply chain security covers · nobody left to fix it · what your SBOM cannot say

Questions people ask

Cite this