An interesting question that I didn't address in the article is whether the group behind xz would be putting all their eggs in one basket - i.e., are there more backdoors of this type waiting to be found?
I think the "one basket" theory is fairly likely. Running multiple similar projects in parallel paradoxically *reduces* the odds of success. Probability of discovery goes up - and once one is found, people start looking for the patterns elsewhere. If my reading of the situation is correct, we're talking about people who understand this.
This is not an argument to stop looking, especially since there were numerous (less sophisticated) attempts to backdoor OSS before, and there will be more in the future. But I wouldn't be surprised if we come up empty-handed.
If there isn't enough to do, being the maintainer of such a library would be a pretty good sinecure using the income of a small endowment.
The impulse for a lot of sane techies would be to toss in a buck or two to be divided up by all these libraries to create funds to defend them every time this happens. Eventually the sums would be enough for people to adopt them and fix this problem. Unfortunately, I don't have a list of the relevant libraries.
I used "not much to do" as a shorthand for "not much meaningful stuff to do" - you can always come up with busywork, but it's just not fun, and probably not something you want to do for a living. It's thrilling to give the world a new compression library. It's a lot less fun to deal with boneheaded compat issues on niche platforms, or mundane feature requests needed by just one or two people in the world - and do this consistently for the next 20-30 years, because these are the timelines we're talking about for a lot of this code. Mark Adler's zlib was created in 1995 and is still widely used. IJG jpeg lib was created in 1991.
I mean, I lived through this first-hand a couple of times for my projects; and here's a perspective from another developer:
For the record, I did some comment cleanup on this thread. First, there was a lot of repetitive questions and debate about one word - "foreign" - that is no longer in the article. I left one question and response for posterity, but I don't think there's much else to say.
Second, there was a flurry of whataboutist comments from people who weren't subscribed at all or joined moments before. While I normally don't curtail debate... if you're not really interested in this Substack and just want to offer snarky wisdom about Snowden or Russia, please do it on your own blog.
Perhaps an intellectual challenge from a young adult on the autism or antisocial spectrum. A question is how much damage was actually done, before the discovery. Coincidences are rare, and if this was a long game operation, then the bug that caused the delays that caused the discovery might not have happened. Is there any intelligence on the sock puppet accounts? Great post.
I'm going to take this as an opportunity to educate and assume you meant no harm. What you're describing is possibly people with sociopathy, not Autism. As someone who is Autistic and in technology a) while we may lack executive function in some areas in our life we are extremely responsible in others, like a job. B) we are not stupid and a subset are actually smarter than people with the average IQ (which also people with sociopathy could have, too). And c) we have a high degree of truthfulness and justice.
I don't want to distract from the main content, but I must take the opportunity to educate. I hope you do read up on all the amazing contributions Autistic people have contributed to society.
I am Aspie, ADHD and Tourette's (mild). I am very lucky and privileged to be high functioning and lead a normal life. I meant no harm and I appreciate your comment. I probably should not have put autism right next to ASPD (or probably any neurodivergence) because of the potential for stereotyping. Some kids and young adults on the autism spectrum (probably coexisting with ADHD and others) do seem to have a talent (actually in many cases due to a high moral stance) for black hat operations, that's all I meant. I did not mean to generalize neurodivergent people in a negative way :) have a nice day.
To much social managment and time allotted to think it is a single nerodivergent stretching thier legs. This has cross expertise team driven action written all over it.
Another real issue with foundational OSS libraries is the assumption that since the code is all available for someone to inspect, someone has carefully inspected all of it. It's almost impossible to back away from this assumption, because anyone who does so admits that they should be responsible for validating all the code they use. I don't have to point out to this audience how expensive it would be to do this, even to do it for current updates.
Inspecting code has always been done. Code in C language is easier to understand than objdump'ed asm (assembly). I think you will agree. Now let's shift to 2020 and 20230. Weve entered the so-called AI age. An AI can probably act as a decompiler and write a DLL or library archive in any language you ask of it. It will also make it prettier than earlier decompiler attempts of which I've seen screenshots (variables such as variable0, variable1, variableN, ...) . AI can rename these variables into something more human to understand it while using example code from places such as github. So perhaps we're going to see encrypted binaries soon to counter the threat of decompiling binaries, who has the master key?
What I'd like to also say is, that, yes it is expensive to code audit. We're not there that every organization can code audit. But the ultimate way, in my mind, is to have an auditing effort in every organization. When I was a young computer jock I read code, but didn't understand, now 30 years later I can understand but I really have to be selective on what I want to audit/code read. My current approach is to use as little programs as possible and for the few remaining ones they are my prime focus. Then there is always that limiting thing called time which I seem to have less of as I get older.
When I was 21 years old I wrote my first backdoor... in kernel code. The motivation was money, or lack thereof. It was 1997, a computer was expensive and I had secured a P120 with 32 MB RAM. What motivated me to backdoor an efnet IRC server was not that I wanted to hurt efnet or IRC, but I wanted to proxy FTP with this server which was directly on a 10/100 Mbit link. The backdoor was a suser() patch on FreeBSD and it wasn't until a few years later that I really felt bad about doing this. Unfortunately I hurt the whole point of Open Source software, and it took a long time for me to shake that bad karma by writing something worthy (nearly 27 years later).
Today, with cloud computers starting at around 4 EUR/mo and you get 20 TB or unlimited transfer these incentives for backdooring don't exist anymore. The eagerness and criminal energy is drained especially by the youth who are the most cash starved demographic in the chart of age groups. Also DSL and flat fees cut down on the need to "proxy" anything. We're talking the diference of billions of IPv6's available to someone and always online vs. a dynamic or (if you were lucky) static IP on a 28.8 KBps modem.
Everything shifted since the 90's as well. Back when (I later worked at an ISP too), there was abuse@ mails being answered by ISP's if they weren't aliased to /dev/null in times of information overload. Today the police services have enough staff to take care of abuse issues and contact the ISP with a subpeona instead of the ISP doing self-care of abuse. That said, the way I operated in the 90's would never work today. Law has really shook everything up.
So I tend to agree that this bad character doing this backdoor by himself, in the 2020's is probably not a single idiot's doing. More likely a cyber gang, criminal organisation or corporate or government-linked organizations. There really is no use backdooring kernel code for script kids, or even lone hackers. Unless they are severely mentally damaged and want to spend time in prison. I narrowly escaped prison, but I feel I'm still in prisonNet either way. It's a mental prison.... Take care!
I don't see how this coul be an attack on open source. Supply-chain attacks on closed source happened before (e.g., Solarwinds), and persisted for longer, so this doesn't really portray OSS as worse. And pretty much every company depends on weird OSS libraries in one way or another, so you probably don't want to rock that boat.
So what's the other motivation you're imagining? To discredit OpenSSH? It has no major competitors for accessing *nix systems, and no competitor can say that they would be resilient to an attack like that.
Speaking of Solarwinds, it's funny Nobelium was identified by Microsoft back then and liblzma backdoor has been discovered by Microsoft. In any events, this liblzma backdoor scares the shit out of me. On the positive side, thank you so much for sharing!
It was discovered by Andres Freund, not by "Microsoft". Andres has been doing amazing things long before he was employed by Microsoft and will no doubt be doing them long after (and it's for damn sure the case that MS did not corporately identify that Andres's openssh auth was running slow and instruct him to look at it, no, by his own account he did this himself).
One thing following (or doing) open source stuff tells you is that the *people* are what count. They jump from company to company but the people remain the same and generally keep doing the same things. If anything this is true too much of the time: core projects are ageing at roughly one year per year and are terribly bad at getting new younger people on board.
Companies are not people. All they can do is redirect effort: it's people that do the work.
I think you're debating a strawman here - although I can see where it's coming from, and that's on me: I closed with a provocative hypothetical, but I didn't mean this to be a jab at Andres.
I'm aware of several instances of vulnerability disclosure where some information was withheld or altered not to tip off the bad guys. It happens, it's not wrong, and it doesn't mean that the people involved are doing anything immoral. That's the point I was trying to make: as outsiders, we shouldn't assume we have all the facts - and in particular, we shouldn't assume that this threat actor isn't known to others and that they aren't watching closely. Sometimes, it's luck; other times, it's solid threat intel work that's indistinguishable from luck.
I don't think we have any reason not to believe Andres' account. The only reason I brought it up is that it felt like a good way to make a point - and it didn't cross my mind that it could be interpreted as a personal attack. I can tell you that if someone approached me and said "there's a very serious attack on the open-source community, but we can't tell the world how we found out", I'd help any way I can - and I can't see why that should cast any doubt on my ethics or skills.
FWIW, I'm sure there are critical details we're not privy to. For one, I imagine that Microsoft threat teams must've been involved at *some* point before the public announcements. And I'd be surprised if they didn't work with their peers at Google to preserve evidence, seeing how this person used @gmail.com accounts. There are several dozen people who know a lot more than we do right now.
I don't think I have a strong argument one way or the other. My gut feeling rests on a combination of three factors:
1) Indiscriminately backdooring software used by your own citizens is controversial in most democracies; in authoritarian countries, this is pretty much SOP.
2) The backdoor is not deniable and has no realistic chance of staying hidden forever, so there's no escaping the aforementioned controversy.
3) It's a backdoor that lets you target server infrastructure, not people. Western agencies generally have more selective ways in to achieve lawful goals in this space, chiefly because most of the important tech companies are residing on the agencies' home turf. In contrast, many authoritarian countries are cut off from that and need to improvise.
That said, given that it's a pretty weak hunch, and that it's tangential to the broader point, I stealth-edited the article to just say "state actor".
What I mean is they have both legal processes in place (FISA court, etc) and usually at least a somewhat sympathetic ear if they want to chat. The Snowden revelations certainly soured up that last part a bit, but it's not a patently adversarial relationship.
In contrast, if you're Amazon or Google and you're approached by the Chinese state security apparatus, I don't suspect there's gonna be much of a desire to cooperate.
This is the issue common to all dependency attacks. You're supposed to check what you use/update to. Thing is, majority of people are either unable or unwilling to do so, which makes their systems/programs vulnerable to this kind of attack vector... and I don't see this changing anytime soon.
I've made a living off of it for 25 or 30 years. I have never been able to understand what made the release configuration managers for nine major distributions drink the Kool-Aid to deploy systemd, which objectively is miserably designed code.
Who knows? Maybe a nation state was behind making that happen too.
I keep seeing people attacking systemd for this but it looks like an unthinking knee-jerk reaction to me. systemd was *not at fault* here. If systemd hadn't been picked, something else would have been -- the first step in this attack was obviously to look at shared libraries loaded into sshd's address space and pick the most obscure ubiquitous one to attack they could find. If it wasn't loaded in directly but via some library that ssh only loaded on particular distros, so much the better. That library happened to be libsystemd, but there are *no properties of libsystemd* that enabled this attack other than that it happened to load liblzma and sshd on some major distros happened to load it.
It is arguably a problem that libsystemd is such a grab-bag and loads in so much, but that may well be why this attack was launched now -- upstream libsystemd has been adapted to not load remotely as much in unless needed, and after spending so long ingratiating themselves into xz purely because libsystemd used it and sshd on some distros used that, they had to get this attack in before the next systemd release happened and propagated to the major distros, and liblzma was suddenly no longer loaded in unless journal decompression was needed (i.e., for daemons using it for launch notification, like sshd, never). So you might as well blame systemd for trying to *improve* their security and triggering this attack...
I’ve been hearing this for a long time and e.g. guys at the OpenBSD community look at systemd like a bull looks at a red flag. I can see systemd has seriously polarized *nix community. The fact is, most distros, server and desktop use it. How to live then?
Update: a more detailed analysis of the payload is here:
https://bsky.app/profile/filippo.abyssdomain.expert/post/3kowjkx2njy2b
Plus, a good summary of the obfuscation mechanism can be found here:
https://gynvael.coldwind.pl/?lang=en&id=782
An interesting question that I didn't address in the article is whether the group behind xz would be putting all their eggs in one basket - i.e., are there more backdoors of this type waiting to be found?
I think the "one basket" theory is fairly likely. Running multiple similar projects in parallel paradoxically *reduces* the odds of success. Probability of discovery goes up - and once one is found, people start looking for the patterns elsewhere. If my reading of the situation is correct, we're talking about people who understand this.
This is not an argument to stop looking, especially since there were numerous (less sophisticated) attempts to backdoor OSS before, and there will be more in the future. But I wouldn't be surprised if we come up empty-handed.
If there isn't enough to do, being the maintainer of such a library would be a pretty good sinecure using the income of a small endowment.
The impulse for a lot of sane techies would be to toss in a buck or two to be divided up by all these libraries to create funds to defend them every time this happens. Eventually the sums would be enough for people to adopt them and fix this problem. Unfortunately, I don't have a list of the relevant libraries.
Does anyone?
I used "not much to do" as a shorthand for "not much meaningful stuff to do" - you can always come up with busywork, but it's just not fun, and probably not something you want to do for a living. It's thrilling to give the world a new compression library. It's a lot less fun to deal with boneheaded compat issues on niche platforms, or mundane feature requests needed by just one or two people in the world - and do this consistently for the next 20-30 years, because these are the timelines we're talking about for a lot of this code. Mark Adler's zlib was created in 1995 and is still widely used. IJG jpeg lib was created in 1991.
I mean, I lived through this first-hand a couple of times for my projects; and here's a perspective from another developer:
https://flameeyes.blog/2022/01/11/life-of-a-core-maintainer-chase-blaze-slumb-restart/
For the record, I did some comment cleanup on this thread. First, there was a lot of repetitive questions and debate about one word - "foreign" - that is no longer in the article. I left one question and response for posterity, but I don't think there's much else to say.
Second, there was a flurry of whataboutist comments from people who weren't subscribed at all or joined moments before. While I normally don't curtail debate... if you're not really interested in this Substack and just want to offer snarky wisdom about Snowden or Russia, please do it on your own blog.
Perhaps an intellectual challenge from a young adult on the autism or antisocial spectrum. A question is how much damage was actually done, before the discovery. Coincidences are rare, and if this was a long game operation, then the bug that caused the delays that caused the discovery might not have happened. Is there any intelligence on the sock puppet accounts? Great post.
I'm going to take this as an opportunity to educate and assume you meant no harm. What you're describing is possibly people with sociopathy, not Autism. As someone who is Autistic and in technology a) while we may lack executive function in some areas in our life we are extremely responsible in others, like a job. B) we are not stupid and a subset are actually smarter than people with the average IQ (which also people with sociopathy could have, too). And c) we have a high degree of truthfulness and justice.
I don't want to distract from the main content, but I must take the opportunity to educate. I hope you do read up on all the amazing contributions Autistic people have contributed to society.
I am Aspie, ADHD and Tourette's (mild). I am very lucky and privileged to be high functioning and lead a normal life. I meant no harm and I appreciate your comment. I probably should not have put autism right next to ASPD (or probably any neurodivergence) because of the potential for stereotyping. Some kids and young adults on the autism spectrum (probably coexisting with ADHD and others) do seem to have a talent (actually in many cases due to a high moral stance) for black hat operations, that's all I meant. I did not mean to generalize neurodivergent people in a negative way :) have a nice day.
Thanks for sharing that! I totally get what you mean now! I can totally see them being hacktivists out there. Appreciate you giving me context :)
To much social managment and time allotted to think it is a single nerodivergent stretching thier legs. This has cross expertise team driven action written all over it.
Exactly. Most Neurodivergent people aren't going to want to a) take that much time and b) do something unethical.
Another real issue with foundational OSS libraries is the assumption that since the code is all available for someone to inspect, someone has carefully inspected all of it. It's almost impossible to back away from this assumption, because anyone who does so admits that they should be responsible for validating all the code they use. I don't have to point out to this audience how expensive it would be to do this, even to do it for current updates.
Inspecting code has always been done. Code in C language is easier to understand than objdump'ed asm (assembly). I think you will agree. Now let's shift to 2020 and 20230. Weve entered the so-called AI age. An AI can probably act as a decompiler and write a DLL or library archive in any language you ask of it. It will also make it prettier than earlier decompiler attempts of which I've seen screenshots (variables such as variable0, variable1, variableN, ...) . AI can rename these variables into something more human to understand it while using example code from places such as github. So perhaps we're going to see encrypted binaries soon to counter the threat of decompiling binaries, who has the master key?
This is completely irrelevant to the point I made.
What I'd like to also say is, that, yes it is expensive to code audit. We're not there that every organization can code audit. But the ultimate way, in my mind, is to have an auditing effort in every organization. When I was a young computer jock I read code, but didn't understand, now 30 years later I can understand but I really have to be selective on what I want to audit/code read. My current approach is to use as little programs as possible and for the few remaining ones they are my prime focus. Then there is always that limiting thing called time which I seem to have less of as I get older.
Great write-up!
Also, this illustration from xkcd seems fitting: https://xkcd.com/2347/
When I was 21 years old I wrote my first backdoor... in kernel code. The motivation was money, or lack thereof. It was 1997, a computer was expensive and I had secured a P120 with 32 MB RAM. What motivated me to backdoor an efnet IRC server was not that I wanted to hurt efnet or IRC, but I wanted to proxy FTP with this server which was directly on a 10/100 Mbit link. The backdoor was a suser() patch on FreeBSD and it wasn't until a few years later that I really felt bad about doing this. Unfortunately I hurt the whole point of Open Source software, and it took a long time for me to shake that bad karma by writing something worthy (nearly 27 years later).
Today, with cloud computers starting at around 4 EUR/mo and you get 20 TB or unlimited transfer these incentives for backdooring don't exist anymore. The eagerness and criminal energy is drained especially by the youth who are the most cash starved demographic in the chart of age groups. Also DSL and flat fees cut down on the need to "proxy" anything. We're talking the diference of billions of IPv6's available to someone and always online vs. a dynamic or (if you were lucky) static IP on a 28.8 KBps modem.
Everything shifted since the 90's as well. Back when (I later worked at an ISP too), there was abuse@ mails being answered by ISP's if they weren't aliased to /dev/null in times of information overload. Today the police services have enough staff to take care of abuse issues and contact the ISP with a subpeona instead of the ISP doing self-care of abuse. That said, the way I operated in the 90's would never work today. Law has really shook everything up.
So I tend to agree that this bad character doing this backdoor by himself, in the 2020's is probably not a single idiot's doing. More likely a cyber gang, criminal organisation or corporate or government-linked organizations. There really is no use backdooring kernel code for script kids, or even lone hackers. Unless they are severely mentally damaged and want to spend time in prison. I narrowly escaped prison, but I feel I'm still in prisonNet either way. It's a mental prison.... Take care!
Apart from spy agencies and governments(local/foreign), OSS has lots of enemies...
I don't see how this coul be an attack on open source. Supply-chain attacks on closed source happened before (e.g., Solarwinds), and persisted for longer, so this doesn't really portray OSS as worse. And pretty much every company depends on weird OSS libraries in one way or another, so you probably don't want to rock that boat.
So what's the other motivation you're imagining? To discredit OpenSSH? It has no major competitors for accessing *nix systems, and no competitor can say that they would be resilient to an attack like that.
Speaking of Solarwinds, it's funny Nobelium was identified by Microsoft back then and liblzma backdoor has been discovered by Microsoft. In any events, this liblzma backdoor scares the shit out of me. On the positive side, thank you so much for sharing!
It was discovered by Andres Freund, not by "Microsoft". Andres has been doing amazing things long before he was employed by Microsoft and will no doubt be doing them long after (and it's for damn sure the case that MS did not corporately identify that Andres's openssh auth was running slow and instruct him to look at it, no, by his own account he did this himself).
One thing following (or doing) open source stuff tells you is that the *people* are what count. They jump from company to company but the people remain the same and generally keep doing the same things. If anything this is true too much of the time: core projects are ageing at roughly one year per year and are terribly bad at getting new younger people on board.
Companies are not people. All they can do is redirect effort: it's people that do the work.
I think you're debating a strawman here - although I can see where it's coming from, and that's on me: I closed with a provocative hypothetical, but I didn't mean this to be a jab at Andres.
I'm aware of several instances of vulnerability disclosure where some information was withheld or altered not to tip off the bad guys. It happens, it's not wrong, and it doesn't mean that the people involved are doing anything immoral. That's the point I was trying to make: as outsiders, we shouldn't assume we have all the facts - and in particular, we shouldn't assume that this threat actor isn't known to others and that they aren't watching closely. Sometimes, it's luck; other times, it's solid threat intel work that's indistinguishable from luck.
I don't think we have any reason not to believe Andres' account. The only reason I brought it up is that it felt like a good way to make a point - and it didn't cross my mind that it could be interpreted as a personal attack. I can tell you that if someone approached me and said "there's a very serious attack on the open-source community, but we can't tell the world how we found out", I'd help any way I can - and I can't see why that should cast any doubt on my ethics or skills.
FWIW, I'm sure there are critical details we're not privy to. For one, I imagine that Microsoft threat teams must've been involved at *some* point before the public announcements. And I'd be surprised if they didn't work with their peers at Google to preserve evidence, seeing how this person used @gmail.com accounts. There are several dozen people who know a lot more than we do right now.
Great post.
Fascinating. Question: why "foreign government" and not just "government"?
I don't think I have a strong argument one way or the other. My gut feeling rests on a combination of three factors:
1) Indiscriminately backdooring software used by your own citizens is controversial in most democracies; in authoritarian countries, this is pretty much SOP.
2) The backdoor is not deniable and has no realistic chance of staying hidden forever, so there's no escaping the aforementioned controversy.
3) It's a backdoor that lets you target server infrastructure, not people. Western agencies generally have more selective ways in to achieve lawful goals in this space, chiefly because most of the important tech companies are residing on the agencies' home turf. In contrast, many authoritarian countries are cut off from that and need to improvise.
That said, given that it's a pretty weak hunch, and that it's tangential to the broader point, I stealth-edited the article to just say "state actor".
I wold dispute premise 1. The US intelligence agencies have a long and storied history of spying on and targeting their own citizens.
3) it isn’t true at all after Snowden’s leak.
They may have different motivations, but the same methods.
What I mean is they have both legal processes in place (FISA court, etc) and usually at least a somewhat sympathetic ear if they want to chat. The Snowden revelations certainly soured up that last part a bit, but it's not a patently adversarial relationship.
In contrast, if you're Amazon or Google and you're approached by the Chinese state security apparatus, I don't suspect there's gonna be much of a desire to cooperate.
FISA is a rubber stamp process and not a oversight process.
This is the issue common to all dependency attacks. You're supposed to check what you use/update to. Thing is, majority of people are either unable or unwilling to do so, which makes their systems/programs vulnerable to this kind of attack vector... and I don't see this changing anytime soon.
I have been running Linux since 0.99pl12f, SLS.
I've made a living off of it for 25 or 30 years. I have never been able to understand what made the release configuration managers for nine major distributions drink the Kool-Aid to deploy systemd, which objectively is miserably designed code.
Who knows? Maybe a nation state was behind making that happen too.
I keep seeing people attacking systemd for this but it looks like an unthinking knee-jerk reaction to me. systemd was *not at fault* here. If systemd hadn't been picked, something else would have been -- the first step in this attack was obviously to look at shared libraries loaded into sshd's address space and pick the most obscure ubiquitous one to attack they could find. If it wasn't loaded in directly but via some library that ssh only loaded on particular distros, so much the better. That library happened to be libsystemd, but there are *no properties of libsystemd* that enabled this attack other than that it happened to load liblzma and sshd on some major distros happened to load it.
It is arguably a problem that libsystemd is such a grab-bag and loads in so much, but that may well be why this attack was launched now -- upstream libsystemd has been adapted to not load remotely as much in unless needed, and after spending so long ingratiating themselves into xz purely because libsystemd used it and sshd on some distros used that, they had to get this attack in before the next systemd release happened and propagated to the major distros, and liblzma was suddenly no longer loaded in unless journal decompression was needed (i.e., for daemons using it for launch notification, like sshd, never). So you might as well blame systemd for trying to *improve* their security and triggering this attack...
I’ve been hearing this for a long time and e.g. guys at the OpenBSD community look at systemd like a bull looks at a red flag. I can see systemd has seriously polarized *nix community. The fact is, most distros, server and desktop use it. How to live then?
Obarun, Artix, and Slackware come to mind.
Gentoo also.
I thought gentoo converted to systemd as well.
Thankfully, no. It is available as an option, but not the default.
Good to know! Thanks!