GitLab discovers widespread NPM supply chain attack

This is why you want containerisation or, even better, full virtualisation. Running programs built on node, python or any other ecosystem that makes installing tons of dependencies easy (and thus frustratingly common) on your main system where you keep any unrelated data is a surefire way to get compromised by the supply chain eventually. I don't even have the interpreters for python and js on my base system anymore - just so I don't accidentally run something in the host terminal that shouldn't run there.

It's funny because techies love to tell people that common sense is the best antivirus, don't click suspicious links, etc. only to download and execute a laundry list of unvetted dependencies with a keystroke.

The lesson surely though is 'don't use web-tech, aimed at solving browser incompatibility issues for local scripting'.

When you're running NPM tooling you're running libraries primarily built for those problems, hence the torrent of otherwise unnecessary complexity of polyfills, that happen to be running on a JS engine that doesn't get a browser attached to it.

Same story from a month ago. The moment I saw the sheer number of dependencies artillery wanted to pull I gave up.

I used to run npm only inside docker containers, and I've been regularly laughed at on these forums. I eventually gave up…

Apparently, The Github CLI only stores its oauth token in the HOME directory if you don't have a keyring. They also say it may not work on headless systems. See https://github.com/cli/cli/discussions/7109.

For example, in my macOS machines the token is safely stored in the OS keyring (yes, I double checked the file where otherwise it would've been stored as plain text).

That's true, but the same may already be true of your browser's cookie file. I believe Chrome on MacOS and Windows (unsure about Linux) now does use OS features to prevent it being read from other executables, but Firefox doesn't (yet)

But protecting specific directories is just whack-a-mole. The real fix is to properly sandbox code - an access whitelist rather than endlessly updating a patchy blacklist

this, this, this

All our tokens should be in is protected keychain and there are no proper cross-platform solutions for this. All gclouds, was aww sdks, gh and other tools just store them in dotfile.

And worst thing, afaik there is no way do do it correctly in MacOS for example. I'd like to be corrected though.

I'm also a victim of this. Last time I try and install Backstage.

Have you wiped your laptop/infected machine? If not I would recommend it; part of it created a ~/.dev-env directory which turned my laptop into a GitHub runner, allowing for remote code execution.

I have a read-only filesystem OS (Bluefin Linux) and I don't know quite how much this has saved me, because so much of the attack happens in the home directory.

The hostage naively walked past all the police and into the data centre, and you’re shooing them in the leg. They’ll probably survive, but they knowingly or incompetently made their choice. Sucks to be them.

Compared to the Java ecosystem, I think there's a couple of issues in the NPM ecosystem that makes the situation a lot worse:

1) The availability of the package post-install hook that can run any command after simply resolving and downloading a package[1].

That, combined with:

2) The culture with using version ranges for dependency resolution[2] means that any compromised package can just spread with ridiculous speed (and then use the post-install hook to compromise other packages). You also have version ranges in the Java ecosystem, but it's not the norm to use in my experience, you get new dependencies when you actively bump the dependencies you are directly using because everything depends on specific versions.

I'm no NPM expert, but that's the worst offenders from a technical perspective, in my opinion.

[1]: I'm sure it can be disabled, and it might even be now by default - I don't know. [2]: Yes, I know you can use a lock file, but it's definitely not the norm to actively consider each upgraded version when refreshing the lockfile.

One factor is that node's philosophy is to have a very limited standard library and rely on community software for a ton of stuff.

That means that not only the average project has a ton of dependencies, but also any given dependency will in turn have a ton of dependencies as well. there’s multiplicative effects in play.

Larger attack surface (JS has been the #1 language on GitHub for years now) and more amateur developers (who are more likely to blindly install dependencies, not harden against dev attack vectors, etc).

Npm has weak security boundaries.

Basically any dependency can (used to?) run any script with the develop permissions on install. JVM and python package managers don't do this.

Of course in all ecosystems once you actually run the code it can do whatever with the permissions of the executes program, but this is another hurdle.

As far as I understand, NPM packages are not self-contained like e.g. Python wheels and can (and often need to) run scripts on install.

So just installing a package can get you compromised. If the compromised box contains credentials to update your own packages in NPM, then it's an easy vector for a worm to propagate.

Maybe some technical reasons, but more like the mind set of the JS "community" that if you don't have the latest version of a package 30 seconds after it's pushed you're hopelessly behind.

In other "communities" you upgrade dependencies when you have time to evaluate the impact.

For the last 2 years PyPi (main Python package repository) requires mandatory 2FA.

Last time I did anything with Java, felt like use of multiple package repositories including private ones was a lot more popular.

Although higher branching factor for JavaScript and potential target count are probably very important factors as well.

I feel with Python upgrade cycle is slower. I upgrade dependencies when something is broken or there is known issue. That means any active vulnerabilities propagate slower. Slower propagation means lower risk. And also as there is fewer upstream packages impact of compromised maintainer is more limited.

NPM lets you upload literally anything, without approval

As it turns out, vibe coding started with npm,

not chat bots.

> anyone potentially affected

How does one know one is affected?

What's the point of rotating tokens if I'm not sure that I've been affected - the new tokens will just be ex-filtrated as well.

First step would be to identify infection, then clean up and then rotate tokens.

> if you're like most developers and reused any of those passwords elsewhere

Is this true? God I hope not, if developers don't even follow basic security practices then all hope is lost.

I'd assume this is stating the obvious, but storing credentials in environment variables or files is a big no-no. Use a security key or at the very least an encrypted file, and never reuse any credential for anything.

To me, the worming aspect and taking developers data as hostages against infrastructure take down is most concerning.

Previously, you had isolated places to clean up a compromise and you were good to go again. This attack approaches the semi-distributed nature and attacks the ecosystem as a whole and i am affraid this approch will get more sophisticated in the future. It reminds me a little of malicious transactions written into a distributed ledger.

Even with periodoc rotation of credentials, attacker gets enough time to do sufficient damage. Imo, the best way to solve would be to not handle any sort of credentials at all at the application layer! If at all the application must only handle only very short lived tokens. Let there be a sidecar (for example) that does the actual credential injection.

Also a good reminder that you should be storing secrets in some kind of locker, not in plain text via environment variables or config files. Impossible to get everyone on board but if you can you should as much as possible.

I hate that high profile services still default to plain text for credential storage.

Also the user data destruction if it stops being able to propagate itself.

I would put blame on contemporary GitHub for a few things but this is not one of them. We need better community practices and tools. We can't expect to rely on Microsoft to content-filter.

I love! how Github, as a corporate company now owned by Microsoft, is directly tied to GoLang as the main repository of the vast majority of packages/dependencies.

Imagine the number of things that can go wrong when they try to regulate or introduce restrictions for build workflows for the purpose of making some extra money... lol

The original Java platform is a good example to think about.

Pushing the data to Github was a blessing in disguise. A friend wouldn't have noticed he got caught if it didn't create a repo on his account. It would have been worse if it silently sent the data to some random server.

They're part of the same company, but that's a good point. They both have mediocre security.

Wouldn’t have been that hard to write a rule that matches the repositories being created by this malware. It literally does the same thing to every victim.

The security product marketers ruined “a possible attack” as a brag 25 years ago. Every time a firewall blocks something, it’s a possible attack being blocked, and imagine how often that happens.

SonaType Lifecycle has some magic to prevent these types of attacks. They claim it is AI based. Not sure how it all works as it is proprietary but it is one of the things we use at work. SonaType IQ server powers it

Current "AI" is generative "AI". It can generate bullshit not evaluate anything.

Edit: see the curl posts about them being bombarded with "AI" generated security reports that mean nothing and waste their time.

Is there a way to list all the packages in the dependency tree with preinstall/postinstall hooks? Preferably before doing the installation?

Stupid question, but:

- If it's safe to "ignore scripts", why does this option exist in the first place?

- Otherwise, what kind of cascade breakage in dependencies you risk by suppressing part of their installation process?

Once you run the JavaScript of the npm library you just installed, if it's Node, what's to stop it accessing environment variables and any file it wants, and sending data to any domain it wants?

Or use pnpm

If you ask the user "should I run this script" after installing, they will just hit yes every time. But also, a lot (I'm confident it's "most") of NPM install operations are done on a CI server, which need to run without human interaction.

Why would that make it any less insightfull?

You are not the target then, but people using Gitlab might find insightful.

At least make them run pnpm instead of npm, disabling post-install scripts. https://pnpm.io/supply-chain-security

"a friend" because friend starts with a consonant sound, not a vowel sound. "a project" for the same reason.

HTH.

In this narrow case, using pnpm or something similar that blocks postinstall scripts by default should be sufficient. In general, you probably want to use a container/vm/sandbox of some sort so dev stuff can’t access anything else on your machine.

Phew, thought it was another one.

Do you rewrite fundamental data structures over and over, like maps, of just not use them?

But how do you left pad a string?

How do you create a hashmap?

While you think this is a producer problem, it's simply a userland market.

Just like in the 90s when viruses primarily went to windows, it' wasn't some magical property of windows, it was the market of users available.

Also, following this logic, it then becomes survivorship bias, in that the more attacks they get, the more researchers spend time looking & documenting.

Biome: https://biomejs.dev/

Also the whole ecosystem around OXS looks very promising: https://oxc.rs/

Send them a request to have Trusted publishers support at central-support (at) sonatype.com

I did that a couple of weeks ago and received an acknowledgment "Another request on Trusted Publishing option. Assigning to Product for review and further action." so this is a bit encouraging.

At least Maven dependencies don't execute scripts on install, but Maven plugins could have a big blast radius.

Over a decade ago at Amazon, all third party dependencies needed to be manually imported. On the one hand, it makes importing new versions or packages slow. On the other hand, there is a very explicit intention and log of every external change that made it into internal projects.

At my previous company, I implemented staged dependencies with artifactory so that production could never get packages that had never gone through CR, or staging environments first. They just were never replicated. That eliminated fuzzy dependency matches that showed up for the first time in production (something that did happen). Because dev to production was about 1 week, it also afforded time to identify packages before they had a chance to be deployed. Obviously it was less robust than manually importing.

Maybe self-hosted package caches support these features now, but 6-7 years ago, that was all manual work.

I looked at one and it was doubly encoded base64.

So, EV code signing certificates? Windows has that, and it'll verify that right in the OS. Git for instance, shows as being signed by

CN = Johannes Schindelin O = Johannes Schindelin S = Nordrhein-Westfalen C = DE

Downside is the cost. Certificates cost hundreds of dollars per year. There's probably some room to reduce cost, but not by much. You also run into issues of paying some homeless person $50 to use their identity for cyber crimes.

You don't think bad actors don't have access to entire countries worth of stolen identities to use for supply chain attacks?

This is what macOS codesigning does. Notarization goes one step further and anchors the signature to an Apple-owned CA to attest that Apple has tied the signature to an Apple developer account.

Also layer upon layer of abstractions - to the point where no single person understands the stack from top to bottom.

Perhaps there is a light at the end of the tunnel: with AI coding assistance, the whole application can be written from scratch (like the old days). All the code is there, not buried deep within someone else's codebase.

JS also has a stability issue. The language evolved fast, the tools and the number of tools evolved fast and in different directions. The module system is a mess and trying to make it better caused more mess. There's Node.js, TypeScript and the browser. That's a lot to handle when trying to make something "std".

Meanwhile I have been using Ruby for 15 years and it has evolved in a stable way without breaking everything and without having to rewrite tons of libraries. It's not as powerful in terms of performance and I/O, it's not as far-reaching as JS is because it doesn't support the browser, it doesn't have a typescript equivalent, but it's mature and stable and its power is that it's human-friendly.

This is harder than it sounds. Look at the amount of effort it took to standardise temporal (new time library) and then for all the runtimes to implement it. It’s a lot of work.

And what’s more, people have proposed a standard library through tc39 without success - https://github.com/tc39/proposal-built-in-modules

Of course any large company could create a massive standard library on their own without going through the standards process but it might not be adopted by developers.

If you look at the list of compromised packages, very few of them could reasonably be included in a standard library. It's mostly project-specific stuff like `@asyncapi/specs` or `@zapier/zapier-sdk`. The most popular generic one I see is `get-them-args`, which is a CLI argument parser - which is something Node has in the form of `util.parseArgs` since v16.17.0.

That is literally how the CycloneDX SBOM packages work, well, after the fact and after the disclosure process.

There's an xckd for that :)

The one with 12 competing standards going to 13 competing standards, or something like that.

The US and Israel also have advanced penetration teams. But they wouldn't be this sloppy - they want persistent advanced access. I suspect Iran, Russia and China also wouldn't be this sloppy. This is too wide ranging and easily detectable and scattershot.

This feels like opportunistic cyber criminals, or North Korea (which acts like cyber criminals.)

Proving the attack is state-sponsored is difficult (as any attack you attribute to a country can very well be a false-flag operation), and “state sponsorship” is itself a spectrum; for example, you could argue India’s insufficient action against tech-support scammers is effectively state-sanctioned.

This can of course be resolved, but here’s the kicker: our own governments equally enjoy this ambiguity to do their own bidding; so no government truly has an incentive to actually improve cross-border identity verification and cybercrime enforcement.

Not to mention, even besides government involvement, these malicious actors still “engage” or induce “engagement” which happens to be the de-facto currency of the technology industry, so even businesses don’t actually have any incentive of fighting them.

It shouldn't be a "get the foreigners!" situation. Sure that is a method of solving the symptoms. But what you're really asking for is ... a software bill of materials. Why dont we have that yet? Bc it's cheaper to get ripped off than it is to pay for a bom. Thats the real problem

We are still bound to our primal instincts. If you cut the throat of a baby in the middle of Times Square, the outrage will be insane. Yet, lack of financing to hospitals can do that many times over but people are numb to it.

Take the Jaguar hack, the economic loss is estimated at 2.5bn. Given an average house price in the UK of $300k, that’s like destroying ~8.000 homes.

Do you think the public and international response will be the same if Russia or China leveled a small neighborhood even with no human casualties?

I wonder that, too. Surely, this is a fantastic opportunity to claim that it comes from whoever is declared evil right now, and force a harder us-vs-them mindset. If people don't have a clearly defined "evil bad guy" that is responsible for everything bad, how will you get teenagers to die for your country in war?

Or, in other words; maybe the nature of humans and the inherent pressure of our society to perform, to be rich, to be successful, drives people to do bad things without any state actor behind it?

They aren't, in fact the very true happens, that we are bombarded non stop with information that everything is the fault of actors from these companies even when it isn't.

We should fight this kind of behavior (and our privacy) regardless of whose involved, yet our governments in the west have nurtured this narrative of always pointing at big tech and foreign actors as scape goats for anything privacy or hacking related.

Also, any cyber attack tracker will show you this is a global issue, if you think there aren't millions of attacks carried out from our own countries, you're not looking enough.

majority of these are actually north korea, india and america. the really disappointing ones are usually india and american and ones that lay dormant code are usually north korea.