An interesting question that I didn't address in the article is whether the group behind xz would be putting all their eggs in one basket - i.e., are there more backdoors of this type waiting to be found?
I think the "one basket" theory is fairly likely. Running multiple similar projects in parallel paradoxically *reduces* the odds of success. Probability of discovery goes up - and once one is found, people start looking for the patterns elsewhere. If my reading of the situation is correct, we're talking about people who understand this.
This is not an argument to stop looking, especially since there were numerous (less sophisticated) attempts to backdoor OSS before, and there will be more in the future. But I wouldn't be surprised if we come up empty-handed.
If there isn't enough to do, being the maintainer of such a library would be a pretty good sinecure using the income of a small endowment.
The impulse for a lot of sane techies would be to toss in a buck or two to be divided up by all these libraries to create funds to defend them every time this happens. Eventually the sums would be enough for people to adopt them and fix this problem. Unfortunately, I don't have a list of the relevant libraries.
For the record, I did some comment cleanup on this thread. First, there was a lot of repetitive questions and debate about one word - "foreign" - that is no longer in the article. I left one question and response for posterity, but I don't think there's much else to say.
Second, there was a flurry of whataboutist comments from people who weren't subscribed at all or joined moments before. While I normally don't curtail debate... if you're not really interested in this Substack and just want to offer snarky wisdom about Snowden or Russia, please do it on your own blog.
Perhaps an intellectual challenge from a young adult on the autism or antisocial spectrum. A question is how much damage was actually done, before the discovery. Coincidences are rare, and if this was a long game operation, then the bug that caused the delays that caused the discovery might not have happened. Is there any intelligence on the sock puppet accounts? Great post.
Another real issue with foundational OSS libraries is the assumption that since the code is all available for someone to inspect, someone has carefully inspected all of it. It's almost impossible to back away from this assumption, because anyone who does so admits that they should be responsible for validating all the code they use. I don't have to point out to this audience how expensive it would be to do this, even to do it for current updates.
When I was 21 years old I wrote my first backdoor... in kernel code. The motivation was money, or lack thereof. It was 1997, a computer was expensive and I had secured a P120 with 32 MB RAM. What motivated me to backdoor an efnet IRC server was not that I wanted to hurt efnet or IRC, but I wanted to proxy FTP with this server which was directly on a 10/100 Mbit link. The backdoor was a suser() patch on FreeBSD and it wasn't until a few years later that I really felt bad about doing this. Unfortunately I hurt the whole point of Open Source software, and it took a long time for me to shake that bad karma by writing something worthy (nearly 27 years later).
Today, with cloud computers starting at around 4 EUR/mo and you get 20 TB or unlimited transfer these incentives for backdooring don't exist anymore. The eagerness and criminal energy is drained especially by the youth who are the most cash starved demographic in the chart of age groups. Also DSL and flat fees cut down on the need to "proxy" anything. We're talking the diference of billions of IPv6's available to someone and always online vs. a dynamic or (if you were lucky) static IP on a 28.8 KBps modem.
Everything shifted since the 90's as well. Back when (I later worked at an ISP too), there was abuse@ mails being answered by ISP's if they weren't aliased to /dev/null in times of information overload. Today the police services have enough staff to take care of abuse issues and contact the ISP with a subpeona instead of the ISP doing self-care of abuse. That said, the way I operated in the 90's would never work today. Law has really shook everything up.
So I tend to agree that this bad character doing this backdoor by himself, in the 2020's is probably not a single idiot's doing. More likely a cyber gang, criminal organisation or corporate or government-linked organizations. There really is no use backdooring kernel code for script kids, or even lone hackers. Unless they are severely mentally damaged and want to spend time in prison. I narrowly escaped prison, but I feel I'm still in prisonNet either way. It's a mental prison.... Take care!
This is the issue common to all dependency attacks. You're supposed to check what you use/update to. Thing is, majority of people are either unable or unwilling to do so, which makes their systems/programs vulnerable to this kind of attack vector... and I don't see this changing anytime soon.
I've made a living off of it for 25 or 30 years. I have never been able to understand what made the release configuration managers for nine major distributions drink the Kool-Aid to deploy systemd, which objectively is miserably designed code.
Who knows? Maybe a nation state was behind making that happen too.
Update: a more detailed analysis of the payload is here:
https://bsky.app/profile/filippo.abyssdomain.expert/post/3kowjkx2njy2b
Plus, a good summary of the obfuscation mechanism can be found here:
https://gynvael.coldwind.pl/?lang=en&id=782
An interesting question that I didn't address in the article is whether the group behind xz would be putting all their eggs in one basket - i.e., are there more backdoors of this type waiting to be found?
I think the "one basket" theory is fairly likely. Running multiple similar projects in parallel paradoxically *reduces* the odds of success. Probability of discovery goes up - and once one is found, people start looking for the patterns elsewhere. If my reading of the situation is correct, we're talking about people who understand this.
This is not an argument to stop looking, especially since there were numerous (less sophisticated) attempts to backdoor OSS before, and there will be more in the future. But I wouldn't be surprised if we come up empty-handed.
If there isn't enough to do, being the maintainer of such a library would be a pretty good sinecure using the income of a small endowment.
The impulse for a lot of sane techies would be to toss in a buck or two to be divided up by all these libraries to create funds to defend them every time this happens. Eventually the sums would be enough for people to adopt them and fix this problem. Unfortunately, I don't have a list of the relevant libraries.
Does anyone?
For the record, I did some comment cleanup on this thread. First, there was a lot of repetitive questions and debate about one word - "foreign" - that is no longer in the article. I left one question and response for posterity, but I don't think there's much else to say.
Second, there was a flurry of whataboutist comments from people who weren't subscribed at all or joined moments before. While I normally don't curtail debate... if you're not really interested in this Substack and just want to offer snarky wisdom about Snowden or Russia, please do it on your own blog.
Perhaps an intellectual challenge from a young adult on the autism or antisocial spectrum. A question is how much damage was actually done, before the discovery. Coincidences are rare, and if this was a long game operation, then the bug that caused the delays that caused the discovery might not have happened. Is there any intelligence on the sock puppet accounts? Great post.
Another real issue with foundational OSS libraries is the assumption that since the code is all available for someone to inspect, someone has carefully inspected all of it. It's almost impossible to back away from this assumption, because anyone who does so admits that they should be responsible for validating all the code they use. I don't have to point out to this audience how expensive it would be to do this, even to do it for current updates.
Great write-up!
Also, this illustration from xkcd seems fitting: https://xkcd.com/2347/
When I was 21 years old I wrote my first backdoor... in kernel code. The motivation was money, or lack thereof. It was 1997, a computer was expensive and I had secured a P120 with 32 MB RAM. What motivated me to backdoor an efnet IRC server was not that I wanted to hurt efnet or IRC, but I wanted to proxy FTP with this server which was directly on a 10/100 Mbit link. The backdoor was a suser() patch on FreeBSD and it wasn't until a few years later that I really felt bad about doing this. Unfortunately I hurt the whole point of Open Source software, and it took a long time for me to shake that bad karma by writing something worthy (nearly 27 years later).
Today, with cloud computers starting at around 4 EUR/mo and you get 20 TB or unlimited transfer these incentives for backdooring don't exist anymore. The eagerness and criminal energy is drained especially by the youth who are the most cash starved demographic in the chart of age groups. Also DSL and flat fees cut down on the need to "proxy" anything. We're talking the diference of billions of IPv6's available to someone and always online vs. a dynamic or (if you were lucky) static IP on a 28.8 KBps modem.
Everything shifted since the 90's as well. Back when (I later worked at an ISP too), there was abuse@ mails being answered by ISP's if they weren't aliased to /dev/null in times of information overload. Today the police services have enough staff to take care of abuse issues and contact the ISP with a subpeona instead of the ISP doing self-care of abuse. That said, the way I operated in the 90's would never work today. Law has really shook everything up.
So I tend to agree that this bad character doing this backdoor by himself, in the 2020's is probably not a single idiot's doing. More likely a cyber gang, criminal organisation or corporate or government-linked organizations. There really is no use backdooring kernel code for script kids, or even lone hackers. Unless they are severely mentally damaged and want to spend time in prison. I narrowly escaped prison, but I feel I'm still in prisonNet either way. It's a mental prison.... Take care!
Apart from spy agencies and governments(local/foreign), OSS has lots of enemies...
Great post.
Fascinating. Question: why "foreign government" and not just "government"?
This is the issue common to all dependency attacks. You're supposed to check what you use/update to. Thing is, majority of people are either unable or unwilling to do so, which makes their systems/programs vulnerable to this kind of attack vector... and I don't see this changing anytime soon.
I have been running Linux since 0.99pl12f, SLS.
I've made a living off of it for 25 or 30 years. I have never been able to understand what made the release configuration managers for nine major distributions drink the Kool-Aid to deploy systemd, which objectively is miserably designed code.
Who knows? Maybe a nation state was behind making that happen too.