Lessons learned:
- replace AUR with nixpkgs
- stop using NPM, provision and build javascript with Nix (if you absolutely must use that garbage language and ecosystem)
- stop using Pypi and Python, provision and build using nix if you absolutely must use the Python ecosystem.
- stop trusting package managers with major security flaws in their build systems/ that place too much trust in community volunteers/ have no oversight
I use Nix by the way.
Do you have any tips on how to set up a python environment using nix?
I also use NixOS (btw) but have so far sidestepped this by the expedient of giving up on any project not in nixpkgs whose install process is a pypi script.
People wanting to use AUR helper, you’re better off using
aurutilsoraurtothanyayorparu. Aurto, even with auto update already remove packages when the maintainer changes because aurto trust models was always to check the maintainer first and not the package itselfI know everyone say “use at your own risk,” but in practice that’s not how regular users are using npm, PyPi, AUR, Cargo and such. They’re not manually reviewing every little update to a deluge of dependencies.
…I’m guilty of this.
I don’t know a perfect solution, but it feels like this can’t go on, as package hijacking is en vogue now.
Containerization to contain damage is good, I guess, but still.
I know everyone say “use at your own risk,” but in practice that’s not how regular users are using npm, PyPi, AUR, Cargo and such.
This won’t work any more in the future. Linux is too big and the Internet, or the world as a whole has become an too unfriendly place.
It is like that I once lived in a small village in Belgium in a shared house and I loved that we never needed to lock the door, even when we were away. But you can’t do that in a big city.
Well, as a Linux user, you can’t run untrusted code from strangers. Which is what AUR and PyPy is. As a normal user, you should run only checked code from your distribution. And when you develop software, you need to check the credentials and signatures of upstream software and their developers.
Good luck with checking all dependencies as a developer, bonus points for JavaScript. You’ve just become a 98% less effective. But seriously, how would you check everything? And if you stumble upon malicious code, would you even recognize it?
Nobody sane should be installing js code in their systems. Nor having node or even npm installed.
Good luck with checking all dependencies as a developer, bonus points for JavaScript.
Yes I know well that JavaScript development practices are unsustainable.
And at some point, chickens will come home to roost.
For my part, I focus on minimalist, well defined systems, both as a user and developer. And trust where it is reasonable - not by default.
Exactly, I wouldn’t know what I was looking at probably. We don’t really learn malicious programming at uni.
Some of this can be alleviated by not using the latest version of everything. Granted that’s not always possible, say if updating to fix a vulnerability or bug fix. Stable distros will have some insulation from this at least since there will be more time to discover the issue.
That’s a failed analogy. The AUR is an end-user build-script repo, not a source/binary/source+binary repo for both devs and sometimes users.
If you e.g. install a CLI tool via cargo, there is at least an implicit tree of trust, with each dependant in a dependency tree doing at least some minimal vetting of dependencies. And the source is all there anyway (barring exceptions like
build.rspulling code, or the indirection of proc macros).The same applies to npm and pypi, although there is no distinction between “code” and “binary”, given the scripting nature of the languages. But some pypi packages do actually distribute binaries (e.g. C/C++ compiled libraries). Don’t know about npm.
But, if I’m not mistaken, the py/js tooling wasn’t always there for stuff like full pinning of dep versions like cargo, and that’s a very important technical detail.
With the AUR, there is no trust tree. And often no fixed code (or binaries) to look at (e.g. *-git packages). So the feasibility of doing any sort of global in-tree checking/vetting is not there. On the other hand, source repos are responsible for removing, or at least flagging, malware or otherwise harmful packages once that becomes known.
Incidentally, I commented on both AUR security and cargo trust here and here. So, I will stop blabbing.
there is at least an implicit tree of trust, with each dependant in a dependency tree doing at least some minimal vetting of dependencies.
Depends.
Unless every single dependency is version locked, one update or maintainer change can silently compromise the whole chain. Obviously this is a problem for AUR (as Arch is a single rolling release), but it can hit Rust and PyPi in a similar way.
And a strict “version locking” ethos can lead to security problems down the line, too.
Also, in this case, they exploited PKGBuild an npm, yes. But I think that was just out of convenience. It’s easy to sneak some malware downloader into a long Rust dependency chain, even if it’s technically all open code.
Unless every single dependency is version locked
Which is the case with
Cargo.lock. You can even lock the toolchain itself withrust-toolchain.toml. There a reason whyuvdevelopers knew what good practices to implement in their quest to fix python tooling 😉.one update or maintainer change can silently compromise the whole chain.
That’s another failed analogy. There is no orphaning and total randos taking over in crates.io (same for npm/pypi I would assume). The owner himself must hand over or give permissions to a specific person/account. That’s another implicit chain of trust that doesn’t exist in AUR.
Nothing is fail-proof of course, as the
xzincident showed.I don’t follow npm incidents closely, but the cases I saw involved the legit owners/maintainers themselves, who were either compromised, or genuinely doing the compromising themselves 🤣.
Arch is a single rolling release
This is largely irrelevant to the discussion.
hit Rust and PyPi in a similar way.
It won’t be similar for the all the reasons stated.
And a strict “version locking” ethos can lead to security problems down the line, too.
That’s where language implementation compat guarantees, audit tooling (
cargo auditet al) and good tooling in general, together with a good semver story in the ecosystem itself, ALL come together to make this largely not a problem.This works well enough in the
crates.ioecosystem.From what I hear,
npmhas the audit tooling part covered. But it’s failure in all the other aspects that’s making the situation unattainable.As with
pypi, there is the pain point of python minor release updates breaking shit all the time. But beyond that, I don’t know enough to give an informed take (other than thatuvis a blessing for those who don’t want to get their hands really dirty in that ecosystem).And again, these details and differences are very relevant.
It’s easy to sneak some malware downloader into a long Rust dependency chain, even if it’s technically all open code.
As of now, the number of successful supply chain attacks (where a chain of real dependants exists) in the
crates.iosystem is ZERO.
If you e.g. install a CLI tool via cargo, there is at least an implicit tree of trust, with each dependant in a dependency tree doing at least some minimal vetting of dependencies.
But still weaker than Debian packages, for example, while on the other hand the number of dependencies now often goes into the hundreds.
still weaker than Debian
There is a lot of myths that surround what distros actually do, can do, and have the resources to do. We had this discussion in one of the two threads I linked.
Maybe this is a good time to try and get the top 10% of aur into official repos or 3rd party repos
Top 10%? Why not just top 10 most used? Or top 25? Maybe top 25 accounts for 80% of AUR traffic
I use malware, BTW.
I have 1825 packages installed, 15 of which are from the AUR, and I use for very specific stuff, like VR. I personally know the mantainer of two of them. The AUR is cool, but should always be used with the same care as manual downloads.
Just be aware that the AUR is not vetted. Its like downloading random stuff of the internet and then install said stuff. Always check your sources.
I was thinking that too, like downloading random .exes and complaining your Windows got a virus. At least now a days, Windows .exes have a signing process that warn you, but that’s just like using non-AUR sources; they’re verified.
But we like Linux because it doesn’t give you a big scary full screen warning every time you try to open something… so IDK. If that’s what you need to keep you from installing malware, get off Arch.
That’s not why I like Linux… but on that note anyway: if the source providing said warning is trustworthy, then let’s have it!
The article mentions the potential need for human review. I have no idea how that could be feasible for something as massive as the AUR. Maybe it could work like Nix, where every package goes through a PR/MR process, and then after it gets approved, the submitter is added to the list of contributors. It’s definitely not a perfect process, but it’s better than the zero-review process that the AUR has.
No, the entire point of AUR is to be a repository of unreviewed publicly submitted scripts. Malware has always been an expectation. If you don’t want that risk, don’t use AUR.
I’ve noticed some installers have at least a voting system (e.g. Octopi) which helps… slightly. At least in knowing what the right package name probably is. Crowd source reviewing is probably the only option for such a vast and open system, even if it can be gamed sometimes.
imo, there should be automatic tags like “active”, “abandoned”, “maintainer changed recently”, “updated after hiatus” and a few more.
The arch devs and community can decide on the time frames. It’s not going to be perfect, but it may help warn users of the changes and so they can do a double take.
Anything other than the “active” ones should show what changed (paru already does this) and users should make a conscious choice to install it anyway. (y/N) instead of going through the installation spamming the return key.
There’s a reason why we already called it orphaned. The flag already exists. The AUR helper that auto updates stuff is the problem
AUR does have a voting system, albeit a basic one.
It is my speculation based on experience and direct invitations to review stuff that Google effectively does ranked crowd-sourcing for Google Maps, etc., in which Google secretly tracks and gives heftier vote power to veteran accounts with longer histories of competence and reliability (especially, say, accounts that are a decade old and still regularly contributing and proposing corrections), unbeknownst to said account holders themselves. Perhaps their lead could be the way out of such messes.
This is kind of funny, because in the “reviews” part of Google Maps, a “Local Guide” giving a restaurant ⭐⭐⭐⭐⭐ is usually a strong indicator that you shouldn’t eat from there 😉. That has been my experience, at least.
As I understand it, the attack vector went after orphaned packages primarily. Several of the affected packages would only run the malicious code if it was a fresh install not an update only. So it would have had to be a clean install of an affected package or a newly installed dependency called to invoke this during the approximately 2 day window.
Yes, this is bad, but it’s clearly testing for weaknesses in the chain through AUR.
Yes, I know it’s contributed code by the community and a random actor can cause havoc. Yes, I know how to manually build packages and check for changes. Yes, I am guilty of using helpers. No, I wouldn’t catch everything on my own.
I do limit what I do use from the AUR, because those installs and updates require more scrutiny.
I am reassessing my own threat model as there’s a couple of packages where I’m dependent on the AUR - most notably displaylink drivers.
I do wish communication was better around the event. I found out first through being subscribed to the mailing list. An announcement on the main page would have gone a long way.
I’d love to know what’s going on with this. Arch has its haters but someone’s putting a lot of effort into this
It seems like some person with a bot just asked to maintain a bunch of orphaned packages, abusing the 2-week waiting period. Right?
Thats why they used npm; off the shelf, almost “standard practice” credential harvesting malware. Nothing too fancy.
ok, time to completely stop using AUR.
I’ll only manually git clone packages
Nah, just curl the install script from GitHub pipe it to bash, make sure to run it with sudo.
