Techies vs spies: the xz backdoor debate
Diving into some of the dynamics and the interpretations of the brazen ploy to subvert the liblzma compression library.
Well — we just witnessed one of the most daring infosec capers of my career.
Here’s what we know so far: some time ago, an unknown party evidently noticed that liblzma (aka xz) — a relatively obscure open-source compression library — was a dependency of OpenSSH, a security-critical remote administration tool used to manage millions of servers around the world. This dependency existed not because of a deliberate design decision by the developers of OpenSSH, but because of a kludge added by some Linux distributions to integrate the tool with the operating system’s newfangled orchestration service, systemd.
Equipped with this knowledge about xz, the aforementioned party probably invented the persona of "Jia Tan” — a developer with no prior online footprint who materialized out of the blue in October 2021 and started making helpful contributions to the library. Up to that point, xz had a single maintainer — Lasse Collin — who was dealing with health issues and was falling behind. Shortly after the arrival of “Jia”, several apparent sock puppet accounts showed up and started pressuring Lasse to pass the baton; it seems that he relented at some point in 2023.
Since then, “Jia” diligently continued the maintenance work — culminating in February 2024 with the seamless introduction of a sophisticated, well-concealed backdoor tucked inside one of the build scripts. Full analysis of the payload is still pending, but it appears to have targeted the pre-authentication crypto functions of OpenSSH; it’s probably safe to assume that it added “master key” functionality to let the attackers access all affected servers at will.
Some time after getting the backdoor in, “Jia” — along with a new cast of sock puppet accounts — started pinging Linux distro maintainers to have the backdoored library packaged and distributed to end users. The scheme worked until Andres Freund — a PostgreSQL developer in the employ of Microsoft — reportedly decided to investigate some unexpected SSH latency caused by a minor bug in the backdoor code.
If this timeline is correct, it’s not the modus operandi of a hobbyist. In today’s world, if you have the technical chops and the patience to pull this off, you can easily land a job that would set you for life without risking any prison time. It’s true that we also have some brilliant folks with sociopathic tendencies and poor impulse control — but almost by definition, such “black hat” groups seek instant gratification and don’t plan heists years in advance. In other words, all signs point to this being a professional, for-pay operation — and it wouldn’t be surprising if it was paid for by a state actor.
With attribution up in the air, it’s still tempting to assign blame. Some pundits are pointing fingers at the supposedly exploitative relationship between Big Tech and the open source community; they claim that the lack of adequate compensation is the source of all malaise. I don’t buy this. The relationship with commercial vendors isn’t always healthy, but many major OSS projects are supported to a significant extent. Countless prominent OSS developers are on Big Tech payroll; quite a few projects receive hefty grants.
The real issue with a lot of small, foundational OSS libraries is just that there isn’t enough to do. They were written decades ago by a single person — and beyond bugfixes, they are not really supposed to change much. You don’t do major facelifts of zlib or giflib every year; even if you wave some cash around, it’s hard to build a sustainable community around watching paint dry. After a while, the maintainer just isn’t all that into it anymore; they are eager to pass the baton to anyone with a pulse and some modicum of skill.
Heck, the same happens on the other side of the equation: even with Big Tech staffing and money, if you have a library that almost never needs any attention, the “ownership” of that code becomes pretty theoretical too. It’s hard to build a rewarding career on being very familiar with some boring, old dependency that’s just taken for granted by everyone else.
More fundamentally, the xz backdoor isn’t a technical problem and it probably can’t be solved with technology alone. To a large extent, it’s a counterintelligence challenge — squarely within the competencies of governments and a handful of commercial entities with ecosystem-wide surveillance capabilities. This notably includes Google and Microsoft.
In fact, here’s an interesting thought: perhaps they have known for a while. Would we be able to tell the difference between a carefully-timed disclosure — presumably engineered to conceal “methods and sources” — and a serendipitous discovery?
For a followup article on the backdoor, click here. A thematic catalog of posts on this blog can be found on this page.
Update: a more detailed analysis of the payload is here:
https://bsky.app/profile/filippo.abyssdomain.expert/post/3kowjkx2njy2b
Plus, a good summary of the obfuscation mechanism can be found here:
https://gynvael.coldwind.pl/?lang=en&id=782
An interesting question that I didn't address in the article is whether the group behind xz would be putting all their eggs in one basket - i.e., are there more backdoors of this type waiting to be found?
I think the "one basket" theory is fairly likely. Running multiple similar projects in parallel paradoxically *reduces* the odds of success. Probability of discovery goes up - and once one is found, people start looking for the patterns elsewhere. If my reading of the situation is correct, we're talking about people who understand this.
This is not an argument to stop looking, especially since there were numerous (less sophisticated) attempts to backdoor OSS before, and there will be more in the future. But I wouldn't be surprised if we come up empty-handed.
If there isn't enough to do, being the maintainer of such a library would be a pretty good sinecure using the income of a small endowment.
The impulse for a lot of sane techies would be to toss in a buck or two to be divided up by all these libraries to create funds to defend them every time this happens. Eventually the sums would be enough for people to adopt them and fix this problem. Unfortunately, I don't have a list of the relevant libraries.
Does anyone?