Debian Bug report logs - #1068024
revert to version that does not contain changes by bad actor

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Joey Hess <id@joeyh.name>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: revert to version that does not contain changes by bad actor

Date: Fri, 29 Mar 2024 16:25:26 -0400

[Message part 1 (text/plain, inline)]

Package: xz-utils
Version: 5.6.1+really5.4.5-1
Severity: important
Tags: security

I count a minimum of 750 commits or contributions to xz by Jia Tan, who
backdoored it.

This includes all 700 commits made after they merged a pull request in Jan 7
2023, at which point they appear to have already had direct push access, which
would have also let them push commits with forged authors. Probably a number of
other commits before that point as well.

Reverting the backdoored version to a previous version is not sufficient to
know that Jia Tan has not hidden other backdoors in it. Version 5.4.5 still
contains the majority of those commits.

Commits by them such as 18d7facd3802b55c287581405c4d49c98708c136 
and ae5c07b22a6b3766b84f409f1b6b5c100469068a show that they were deep
into analyzing the security of xz. They were well placed to insert a buffer
overflow that could allow eg, a targeted xz file to cause arbitrary code
execution. The impact of such a security hole could be much more stealthy and
bad than the known backdoor since it would allow chaining xz with other
unrelated software releases on an ongoing basis.

The package should be reverted to a version before their involvement,
which started with commit 6468f7e41a8e9c611e4ba8d34e2175c5dacdbeb4.
Or their early commits vetted and revert to a later point, but any arbitrary 
commit by a known bad and malicious actor almost certainly has less value
than the risk that a subtle change go unnoticed.

I'd suggest reverting to 5.3.1. Bearing in mind that there were security
fixes after that point for ZDI-CAN-16587 that would need to be reapplied.

-- 
see shy jo

[signature.asc (application/pgp-signature, inline)]

Message #10 received at submit@bugs.debian.org (full text, mbox, reply):

From: Aurelien Jarno <aurelien@aurel32.net>

To: Joey Hess <id@joeyh.name>, 1068024@bugs.debian.org

Cc: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Fri, 29 Mar 2024 22:32:23 +0100

[Message part 1 (text/plain, inline)]

On 2024-03-29 16:25, Joey Hess wrote:
> I'd suggest reverting to 5.3.1. Bearing in mind that there were security
> fixes after that point for ZDI-CAN-16587 that would need to be reapplied.

Note that reverted to such an old version will break packages that use
new symbols introduced since then. From a quick look, this is at least:
- dpkg
- erofs-utils
- kmod

Having dpkg in that list means that such downgrade has to be planned
carefully.

Regards
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Aurelien Jarno <aurelien@aurel32.net>, 1068024@bugs.debian.org

Cc: Joey Hess <id@joeyh.name>

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Fri, 29 Mar 2024 22:47:48 +0000 (UTC)

Aurelien Jarno dixit:

>Having dpkg in that list means that such downgrade has to be planned
>carefully.

Right. Not an argument against, though.
Instead, this is a very good idea.

What symbols are new? Can we somehow stub them, or at least where
those are used? Or change the soname, so that the old and new-older
versions are coïnstallable for during the upgrade?

bye,
//mirabilos
-- 
<igli> exceptions: a truly awful implementation of quite a nice idea.
<igli> just about the worst way you could do something like that, afaic.
<igli> it's like anti-design.  <mirabilos> that too… may I quote you on that?
<igli> sure, tho i doubt anyone will listen ;)

Message #25 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Stephan Verbücheln <verbuecheln@posteo.de>

To: 1068024@bugs.debian.org

Subject: Or remove xz altogether?

Date: Sat, 30 Mar 2024 00:48:34 +0000

Maybe the people who criticized xz back in the day for being an amateur
project implementing a defective file format were right all along?

https://www.nongnu.org/lzip/xz_inadequate.html

Regards
Stephan

Message #30 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Mark-Oliver Wolter <mow@mowny.de>

To: 1068024@bugs.debian.org, Aurelien Jarno <aurelien@aurel32.net>

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Sat, 30 Mar 2024 03:38:39 +0100

On Fri, 29 Mar 2024 22:32:23 +0100 Aurelien Jarno <aurelien@aurel32.net> 
wrote:
> Having dpkg in that list means that such downgrade has to be planned
> carefully.

Might be easier overall to spend that effort on a hard switch to zstd 
instead.

mfG mow

Message #35 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Stephan Verbücheln <verbuecheln@posteo.de>, 1068024@bugs.debian.org

Subject: Re: Bug#1068024: Or remove xz altogether?

Date: Sat, 30 Mar 2024 03:49:12 +0100

On Sat, 2024-03-30 at 00:48:34 +0000, Stephan Verbücheln wrote:
> Maybe the people who criticized xz back in the day for being an amateur
> project implementing a defective file format were right all along?
> 
> https://www.nongnu.org/lzip/xz_inadequate.html

*Sigh*, the current situation is bad enough, and has nothing to do
with the xz format or design, or the FUD and propaganda from that
link. Please drop it…

Thanks,
Guillem

Message #40 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Pierre Ynard <linkfanel@yahoo.fr>

To: 1068024@bugs.debian.org, Aurelien Jarno <aurelien@aurel32.net>, Mark-Oliver Wolter <mow@mowny.de>

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Sat, 30 Mar 2024 08:12:42 +0000 (UTC)

> Might be easier overall to spend that effort on a hard switch to zstd
> instead.

Good point. Another question that we'll probably have to ask anyway
is, what is the future of xz-utils going to be now? Regarding its
maintainership as well, both upstream and in Debian?

From what I understand, this package has been NMU'd in Debian for more
than 10 years now - thanks to Sebastian for his work - and that might
have been a point of weakness which was exploited to push a compromised
version into the Debian archive, as seen in #1067708. To quote Thorsten
from that thread:

> Very much *not* a fan of NMUs doing large changes such as
> new upstream versions.

I can't say why exactly he would not be a fan, but with hindsight that
was an interesting call.

Perhaps more importantly, it is said that the whole situation started
with the lack of resources from the upstream maintainer to maintain the
project:

https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html

which again, we can suspect at this point, was exploited with the same
modus operandi to get a compromise vector coopted in, in the form of a
new maintainer.

And now if what we suspect is true, that relief maintainer has turned
out to be a bad actor, which leaves upstream back to square one on the
maintainance issue. I understand that Lasse Collin hasn't been able to
weigh in on the situation yet.

Is xz-utils going to be maintained? Will we want to keep in the archive
an unmaintained low-level library - low-level as in, susceptible of
getting pulled as a dependency in lots of places - and rely on it for
components such as dpkg?

I'm not presuming the answers - it's still early and there is probably
more to figure out yet - but back to the original point, we might want
to consider this when sketching longer-term plans.

-- 
Pierre Ynard

Message #45 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Joey Hess <id@joeyh.name>

To: Aurelien Jarno <aurelien@aurel32.net>

Cc: 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Sat, 30 Mar 2024 08:38:19 -0400

[Message part 1 (text/plain, inline)]

Aurelien Jarno wrote:
> Note that reverted to such an old version will break packages that use
> new symbols introduced since then. From a quick look, this is at least:
> - dpkg
> - erofs-utils
> - kmod
> 
> Having dpkg in that list means that such downgrade has to be planned
> carefully.

I agree this would be a challanging downgrade. I've tried it myself
experimentally and once a downgraded liblzma5 is unpacked, dpkg-deb is broken
with missing symbol 'XZ_5.4'.

Renaming liblzma5 to something else (liblzma6?) and making dpkg-deb
depend on that seems like one way to go that would avoid messy situations.


FWIW, I rebuilt xz-utils 5.2.5-2.1~deb11u1 (from bullseye) on sid
and then got dpkg to build against that successfully after a few minor
changes to dpkg's packaging:

--- debian/libdpkg-dev.install.orig	2024-03-30 07:31:46.635365816 -0400
+++ debian/libdpkg-dev.install	2024-03-30 07:34:48.667477725 -0400
@@ -1,4 +1,5 @@
 usr/include/dpkg/*.h
-usr/lib/*/pkgconfig/libdpkg.pc
-usr/lib/*/libdpkg.a
+usr/lib/pkgconfig/libdpkg.pc
+usr/lib/libdpkg.a
 usr/share/aclocal/dpkg-*.m4
+usr/lib/libdpkg.la

(And after disabling the test suite since changes in xz message output
caused a test failure.)

-- 
see shy jo

[signature.asc (application/pgp-signature, inline)]

Message #50 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Ivan Shmakov <ivan@siamics.net>

To: 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Sat, 30 Mar 2024 12:55:21 +0000

>>>>> On 2024-03-30, Guillem Jover wrote:
>>>>> On Sat, 2024-03-30 at 00:48:34 +0000, Stephan Verbücheln wrote:

 >> Subject: Re: Bug#1068024: Or remove xz altogether?

 >> Maybe the people who criticized xz back in the day for being an amateur
 >> project implementing a defective file format were right all along?

 >> https://www.nongnu.org/lzip/xz_inadequate.html

 > *Sigh*, the current situation is bad enough, and has nothing to do
 > with the xz format or design, or the FUD and propaganda from that
 > link.  Please drop it…

	While I wouldn’t have brought it up myself, nor the arguably
	provocative Subject: change (reverted), as it’s indeed only
	tangentially related to the issue at hand (which apparently
	have resulted from the absense of good faith developers
	volunteering to maintain this project – and in a word of
	defense, the design quality of a software project /does/
	influence both the amount of maintenance work and, other
	things being equal, the desire to get involved), the comment
	above got me curious: which parts of the document linked are
	‘FUD and propaganda’?  Because I’ve just glanced it over and
	I don’t find any obvious examples.

	Granted, I’m not an expert in the field of compression and
	coding, per se, and can’t readily speak on the validity of
	most of the arguments given there, those at least appear
	plausible.  Some arguments I find obvious, such as, e. g.:

 ADD> A well-known property of CRCs is their ability to detect burst
 ADD> errors up to the size of the CRC itself.  Using a CRC larger
 ADD> than the dataword is an error because a CRC just as large as
 ADD> the dataword equally detects all errors while it produces
 ADD> a lower number of false positives.

	If there’s a reasonable rebuttal to the points raised in that
	document, I believe that a pointer to it would be appropriate
	for this discussion.  Other than that, I’m not going to go on
	this tangent any further here.  I’ll be monitoring a handful
	of Internet fora for that, though (news:alt.os.linux.debian,
	news:comp.misc, irc://irc.efnet.org/%23coders, etc.)

-- 
FSF associate member #7257  np. Moonsong — Shane Jackman

Message #57 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Jeffrey Walton <noloader@gmail.com>

To: 1068024@bugs.debian.org

Date: Sat, 30 Mar 2024 13:02:28 -0400

Lasse Collin provided a statement at <https://tukaani.org/xz-backdoor/>.

Message #62 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Joey Hess <id@joeyh.name>

To: Aurelien Jarno <aurelien@aurel32.net>

Cc: 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Sat, 30 Mar 2024 14:16:52 -0400

[Message part 1 (text/plain, inline)]

I have prepared a git repository that is a fork of xz from the point I
identified before the attacker(s) did anything to it. In my fork, I have
renamed liblzma to liblzmaunscathed. That allows it to be installed
alongside current dpkg without breaking dpkg with an old version of
liblzma. 

My git repository is here (note all my commits are gpg signed):
https://git.joeyh.name/index.cgi/xz-unscathed/

It also has a debian branch which contains a debian directory. I've
built packages of that, as well as building dpkg-1.22.6 against it.
I've attached the patch I used to build dpkg.

My build of dpkg ended up not being linked to a lzma library at all,
because liblzmaunscathed is too old to support concurrent decompression,
which the configure script detects. So dpkg-deb instead uses xz-utils
to decompress debs. I replaced xz-utils.deb with the one built from my
fork, and dpkg seems to work fine using it.

If Debian decided to go this route, you could add xz-utils-unscathed
to unstable, and at the same time update xz-utils to not build
xz-utils.deb. Then build dpkg against it. Then look into forward porting
or re-implementing concurrent decompression if that is really important
to have.

I only plan to maintain this fork minimally, eg backporting security
fixes. The goal is not to take over from xz upstream, but to get the
possibly backdoored code off of production systems ASAP. Presumably xz
upstream will come up with their own solution long-term.

-- 
see shy jo

[dpkg.patch (text/x-diff, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #67 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Jeffrey Walton <noloader@gmail.com>

To: 1068024@bugs.debian.org

Date: Sat, 30 Mar 2024 16:03:11 -0400

It looks like more analysis has revealed this is a RCE with the
payload in the modulus of a public key: "The payload is extracted from
the N value (the public key) passed to RSA_public_decrypt, checked
against a simple fingerprint, and decrypted with a fixed ChaCha20 key
before the Ed448 signature verification..." Also see
<https://www.openwall.com/lists/oss-security/2024/03/30/36>.

Message #72 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Christoph Anton Mitterer <calestyo@scientia.org>

To: 1068024@bugs.debian.org

Subject: Re: revert to version that does not contain changes by bad actor

Date: Sun, 31 Mar 2024 01:16:07 +0100

Hey.

Can we be confidently sure that going back to 5.4.5 is enough?

At least the git tag for that seems to be still signed by the
adversary:
https://git.tukaani.org/?p=xz.git;a=tag;h=9e4835399118b98954f110f76af2a0d504d2f531

The last one, still from Lasse Collin seems to be 5.4.1:
https://git.tukaani.org/?p=xz.git;a=tag;h=f52502e78bf84f516a739e8d8a1357f27eeea75f


Cheers,
Chris.

Message #77 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Alberto Garcia <berto@igalia.com>

To: Christoph Anton Mitterer <calestyo@scientia.org>, 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Sun, 31 Mar 2024 01:45:00 +0100

On Sun, Mar 31, 2024 at 01:16:07AM +0100, Christoph Anton Mitterer wrote:
> The last one, still from Lasse Collin seems to be 5.4.1:
> https://git.tukaani.org/?p=xz.git;a=tag;h=f52502e78bf84f516a739e8d8a1357f27eeea75f

There are commits from Jia before that, and some that are authored by
Lasse but thank Jia for the contribution (the oldest one is 20e7a33e).

The most recent release that does not contain any of that seems to be
v5.3.2alpha.

Berto

Message #82 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Christoph Anton Mitterer <calestyo@scientia.org>

To: 1068024@bugs.debian.org

Subject: Re: revert to version that does not contain changes by bad actor

Date: Sun, 31 Mar 2024 03:01:06 +0200

One more thing:


Is anyone on the Debian side trying to figure out how far we've been
practically affected?

I mean let's assume we're "lucky", and the backdoor is only in
5.6.0/5.6.1... and that none of the adversary's earlier commits
introduced any serious holes[0] which wouldn't be known yet.

Then many servers running Debian (which is then typically Debian
stable), would be sill safe.

However, many people (and I'd guess that includes DDs/DMs) run their
workstations/laptops with testing/unstable.
So IMO it's not enough to know that Debian stable is likely not
affected - I'd be rather relieved if I'd knew that there's a good
chance that the personal computers of most DDs/DMs (who push uploads,
etc. pp.) were (at least practically) safe.

Some guy decrypted[1] the strings from the maleware's binary payload:
https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01#file-hashes-txt-L115
where only /usr/sbin/sshd would be named.

Should(!) it turn out, that the actual binary payload actually only
affects sshd and especially does not on it's own pull in evil code, it
might at least be that all people who hadn't sshd running (or not
directly accessible because a router/firewall/etc. was in front of it),
would effectively still be safe.
No idea whether or not this is really the case - but if so, it might at
least mean that many workstations/laptops are safe, because they're not
usually directly accessible from the internet.


But even then, it would probably need to be checked whether all the
versions that Debian ever used/shipped in
testing/unstable(/experimental) were actually fully identical to the
versions that are now analysed by forensic folks.

Is the security team doing anything like that?


Cheers,
Chris.


PS: if someone from the security team reads along,
https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
which is from Sam James of Gentoo, seems to collect good information on
what is known so far.
Might be worth to add to the links in the security tracker.


[0] There are reports about an added '.' in the CMakeLists.txt
disabling some sandboxing, but AFAICS at least the current version uses
autotools for building?!

[1] He also provided the code he used to do so.
https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01?permalink_comment_id=5006546#gistcomment-5006546

Message #87 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Pierre Ynard <linkfanel@yahoo.fr>, 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Sun, 31 Mar 2024 01:36:31 +0000 (UTC)

Pierre Ynard dixit:

>version into the Debian archive, as seen in #1067708. To quote Thorsten
>from that thread:
>
>> Very much *not* a fan of NMUs doing large changes such as
>> new upstream versions.
>
>I can't say why exactly he would not be a fan, but with hindsight that
>was an interesting call.

It turned out to indeed be related, although I cannot blame Sebastian
for not spotting it, as well as it was hidden. I actually wrote about
that earlier on Fedi: (Markdown formatting lost here though)

| I was considering replying to this comment on the “please update xz
| package” bugreport earlier with that the discussion is not irrelevant
| and that it’s the maintainer’s responsibility on new upgrades to check
| for new legal issues and “other hidden gems”.
|
| I didn’t because I didn’t want to bother going in with an annoyed
| self-righteous “user”.
|
| Now it turns out all three of the involved ones were “string + number
| @ freemailer” #JiaT75 sockpuppets, so it’s probably okay I didn’t
| bother.
|
| Not that I blame Sebastian — it was very well hidden, and even my
| usual diffing between old and new version would not have found it.
|
| I do take away from this to also check the diff between VCS repo at
| the time of the release and release tarball. Perhaps also between
| branch and tag if they, like Apache Tomcat, introduce extra commits
| there.

>Is xz-utils going to be maintained? Will we want to keep in the archive
>an unmaintained low-level library - low-level as in, susceptible of
>getting pulled as a dependency in lots of places - and rely on it for
>components such as dpkg?

That scenario you describing here would actually be much less of a
problem. The problem comes when the library in that state then does
get updates that probably are not even necessary but shiny enough
people demand them.

bye,
//mirabilos (also a Debian Developer, despite the From)
-- 
When he found out that the m68k port was in a pretty bad shape, he did
not, like many before him, shrug and move on; instead, he took it upon
himself to start compiling things, just so he could compile his shell.
How's that for dedication. -- Wouter, about my Debian/m68k revival

Message #92 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Christoph Anton Mitterer <calestyo@scientia.org>, 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Sun, 31 Mar 2024 01:38:51 +0000 (UTC)

Christoph Anton Mitterer dixit:

>Can we be confidently sure that going back to 5.4.5 is enough?

No.

>The last one, still from Lasse Collin seems to be 5.4.1:

In this bugreport, we’re discussing going back to before any
contributions by the adversary. To see whether it’s an option
at all (and it sounds like a sensible step right now) which
joeyh confirmed (thanks).

bye,
//mirabilos
-- 
22:20⎜<asarch> The crazy that persists in his craziness becomes a master
22:21⎜<asarch> And the distance between the craziness and geniality is
only measured by the success 18:35⎜<asarch> "Psychotics are consistently
inconsistent. The essence of sanity is to be inconsistently inconsistent

Message #97 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Christoph Anton Mitterer <calestyo@scientia.org>, 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Sun, 31 Mar 2024 01:46:35 +0000 (UTC)

Christoph Anton Mitterer dixit:

>Is anyone on the Debian side trying to figure out how far we've been
>practically affected?

Yes, a multi-team task force is working on it and will inform users
once it is known how to proceed, inclusing how much to throw away
and rebuild.

bye,
//mirabilos
-- 
„Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund,
mksh auf jedem System zu installieren.“
	-- XTaran auf der OpenRheinRuhr, ganz begeistert
(EN: “[…]uhr.gz is a reason to install mksh on every system.”)

Message #102 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Joshua Hudson <joshudson@gmail.com>

To: 1068024@bugs.debian.org

Subject: Potential solution to your downgrade problem in dpkg

Date: Sat, 30 Mar 2024 19:55:20 -0700

[Message part 1 (text/plain, inline)]

The dpkg -> xz-utils downgrade problem has a suggestion that suggests
itself.

1) Downgrade dpkg's dependency to the last known good. It doesn't matter
how old so long as it can read the file formats. I understand this is
likely to be 5.4.1.

2) Statically link all the decompressor libraries into dpkg. Yes this means
gzip, bzip2, zstd, and xz's libraries. Today's compile and link time
optimizers are pretty good; it should be able to drop all references to the
compression side pretty much by itself. This would generally be a pretty
good change of its own as it makes the minimal system easier to understand
and the net-install CD smaller.

Then you're free to do what you need to do for all the other dependencies
of xz-utils.

[Message part 2 (text/html, inline)]

Message #107 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Joshua Hudson <joshudson@gmail.com>, 1068024@bugs.debian.org

Subject: Re: Bug#1068024: Potential solution to your downgrade problem in dpkg

Date: Sun, 31 Mar 2024 04:43:34 +0000 (UTC)

Joshua Hudson dixit:

>2) Statically link all the decompressor libraries into dpkg. Yes this means

Totally no.

Also, at this point in time, I’m pretty sure that new external
suggestions are… not very helpful. The situation is being analysed
by a cross-team taskforce, please let them do the already-stressing
job ☻

bye,
//mirabilos
-- 
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against
 ╳  HTML eMail! Also,
╱ ╲ header encryption!

Message #112 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Joey Hess <id@joeyh.name>

To: Aurelien Jarno <aurelien@aurel32.net>

Cc: 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Sun, 31 Mar 2024 16:55:11 -0400

[Message part 1 (text/plain, inline)]

Since there's been some discussion about versions, the version in my
xz-unscathed git repository is the same as xz 5.3.2alpha, with the
addition of a fix for CVE-2022-1271 that did not make it into that
version. (It was fixed in 5.2.6, but 5.3.2alpha was diverged from
5.2.5. Jia Tan was involved in 5.2.6.)

5.2.5 might be a more stable version to revert to; it also predates
Jia Tan's involvement. The CVE-2022-1271 fix would need to be included.

Note that erofs-utils apparently had a reason to need the 5.3.2alpha
release, so reverting to 5.2.5 would probably cause difficulty with that
package. That dependency versioning information is not included in the
debian sources for erofs-utils BTW. I have not checked compatability
with other packages except for dpkg.

-- 
see shy jo

[signature.asc (application/pgp-signature, inline)]

Message #117 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Joey Hess <id@joeyh.name>

To: 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Sun, 31 Mar 2024 16:57:34 -0400

[Message part 1 (text/plain, inline)]

> The situation is being analysed by a cross-team taskforce,
> please let them do the already-stressing job ☻

Sorry, didn't see that before sending my previous message.
I'll leave it to you guys.

-- 
see shy jo

[signature.asc (application/pgp-signature, inline)]

Message #122 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Jeffrey Walton <noloader@gmail.com>

To: 1068024@bugs.debian.org

Date: Sun, 31 Mar 2024 20:07:36 -0400

An important comment from oss-security mailing list message,
<https://www.openwall.com/lists/oss-security/2024/03/31/4>:

commit f9cf4c05edd14dedfe63833f8ccbe41b55823b00 (HEAD -> master,
origin/master, origin/HEAD)
Author: Lasse Collin <lasse.collin@...aani.org>
Date:   Sat Mar 30 14:36:28 2024 +0200

    CMake: Fix sabotaged Landlock sandbox check.

    It never enabled it.

Message #127 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Joey Hess <id@joeyh.name>, 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Tue, 2 Apr 2024 14:34:20 +0200

Hi!

On Sat, 2024-03-30 at 14:16:52 -0400, Joey Hess wrote:
> My git repository is here (note all my commits are gpg signed):
> https://git.joeyh.name/index.cgi/xz-unscathed/

> My build of dpkg ended up not being linked to a lzma library at all,
> because liblzmaunscathed is too old to support concurrent decompression,
> which the configure script detects. So dpkg-deb instead uses xz-utils
> to decompress debs. I replaced xz-utils.deb with the one built from my
> fork, and dpkg seems to work fine using it.

dpkg should be able to use an old liblzma w/o multi-threaded compressor
or decompressor support (both are intended to be optionally used if
available). I think the problem might be that you seem to have missed
renaming the .pc.in file, and that does not get included in the -dev
package perhaps, or not picked up then by dpkg with your attached
patch to it? I only checked the renaming commit, didn't check the
packaging nor tried to build it.

(Please do not take this mail as endorsing any specific action, just
wanted to clarify/correct the above.)

Thanks,
Guillem

Message #132 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Joey Hess <id@joeyh.name>

To: Guillem Jover <guillem@debian.org>

Cc: 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Wed, 3 Apr 2024 02:55:55 -0400

[Message part 1 (text/plain, inline)]

Guillem Jover wrote:
> dpkg should be able to use an old liblzma w/o multi-threaded compressor
> or decompressor support

Ah you're right. configure did find the library, but I'd missed updating
some ifdefs. Attached updated dpkg patch which does build with the
shared library and works.

-- 
see shy jo

[dpkg.patch (text/x-diff, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #137 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Joey Hess <id@joeyh.name>, 1068024@bugs.debian.org

Cc: Guillem Jover <guillem@debian.org>

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Wed, 3 Apr 2024 11:06:27 +0000 (UTC)

Joey Hess dixit:

>--- orig/dpkg-1.22.6/debian/control	2024-03-02 21:30:15.000000000 -0400
>+++ dpkg-1.22.6/debian/control	2024-03-30 13:14:37.746223895 -0400

> # Version needed for multi-threaded decompressor support.
>- liblzma-dev (>= 5.4.0),
>+ liblzmaunscathed-dev,

The comment probably needs adjusting as well.

> # Version needed for multi-threaded decompressor support.
>- xz-utils (>= 5.4.0) <!nocheck>,
>+ xz-utils <!nocheck>,

dito

> # Version needed for multi-threaded decompressor support.
>- liblzma-dev (>= 5.4.0),
>+ liblzmaunscathed-dev,

idem

> # Version needed for multi-threaded decompressor support.
>- xz-utils (>= 5.4.0),
>+ xz-utils,

el mismo

> # Version needed for multi-threaded decompressor support.
>- xz-utils (>= 5.4.0),
>+ xz-utils,

the same

>--- orig/dpkg-1.22.6/debian/libdpkg-dev.install	2024-02-04 22:31:16.000000000 -0400
>+++ dpkg-1.22.6/debian/libdpkg-dev.install	2024-03-30 13:25:27.043840706 -0400
>@@ -1,4 +1,5 @@
> usr/include/dpkg/*.h
>-usr/lib/*/pkgconfig/libdpkg.pc
>-usr/lib/*/libdpkg.a
>+usr/lib/pkgconfig/libdpkg.pc
>+usr/lib/libdpkg.a

Why, exactly, does the library move out of the M-A directory here?
(Probably a question for guillem though.)

>+usr/lib/libdpkg.la

IIRC we were not shipping libtool files, were we? Or am I confusing
this with some BSD ports/packages now?

bye,
//mirabilos
-- 
[...] if maybe ext3fs wasn't a better pick, or jfs, or maybe reiserfs, oh but
what about xfs, and if only i had waited until reiser4 was ready... in the be-
ginning, there was ffs, and in the middle, there was ffs, and at the end, there
was still ffs, and the sys admins knew it was good. :)  -- Ted Unangst über *fs

Message #142 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>

To: Joey Hess <id@joeyh.name>

Cc: Guillem Jover <guillem@debian.org>, 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Wed, 3 Apr 2024 21:18:57 +0200

On 2024-04-02 14:34:20 [+0200], Guillem Jover wrote:
> (Please do not take this mail as endorsing any specific action, just
> wanted to clarify/correct the above.)

Right, same here. The 5.4.x series has threaded decompression which I
would like to keep. The 5.6.x series has branchless decompressor which
improves decompressing performance.

The 5.3.x series is alpha and should not be used in production but only
for testing. I'm not sure what happens with the XZ_5.3..alpha symbols, I
hope they get ignored.

I'm going to poke upstream if there is an audit and or future plans. But
I want to stay on an official upstream release which is also used by
other distros.

> Thanks,
> Guillem

Sebastian

Message #147 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: ashim gena <ashimgena12@gmail.com>

To: 1068024@bugs.debian.org, Ashim Gena <12@gmail.com>

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Mon, 8 Apr 2024 10:36:19 +0300

[Message part 1 (text/plain, inline)]

On Fri, 29 Mar 2024 22:32:23 +0100 Aurelien Jarno <aurelien@aurel32.net>
wrote:
> On 2024-03-29 16:25, Joey Hess wrote:
> > I'd suggest reverting to 5.3.1. Bearing in mind that there were security
> > fixes after that point for ZDI-CAN-16587 that would need to be
reapplied.
>
> Note that reverted to such an old version will break packages that use
> new symbols introduced since then. From a quick look, this is at least:
> - dpkg
> - erofs-utils
> - kmod
>
> Having dpkg in that list means that such downgrade has to be planned
> carefully.
>
> Regards
> Aurelien
>
> --
> Aurelien Jarno                          GPG: 4096R/1DDD8C9B
> aurelien@aurel32.net                     http://aurel32.net


mobile

[Message part 2 (text/html, inline)]

Message #152 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Christoph Anton Mitterer <calestyo@scientia.org>

To: 1068024@bugs.debian.org

Cc: Thorsten Glaser <tg@mirbsd.de>

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Sat, 20 Apr 2024 00:54:40 +0200

Hey.


On Sun, 2024-03-31 at 01:46 +0000, Thorsten Glaser wrote:
> Yes, a multi-team task force is working on it and will inform users
> once it is known how to proceed, inclusing how much to throw away
> and rebuild.

Kindly wanted to ask whether anything has come out meanwhile of that?


I've tried to follow quite extensively what the various reverse
engineering efforts (e.g. [0], [1], [2], [3]) found out or what's
revealed on index pages like [4] or [5].

It feels as if there are still many discussions about how to prevent
such things in the future, but less so about the concrete fallout of
the particular backdoor, where it seems most people were lead to
conclude from media reports, that an attack was only possible if sshd
was actually running an reachable.

This may of course be true, which would mean that most people are
actually safe and we had quite some luck this time:
- servers, because they run stable distros that haven't had the
  backdoor
- workstations/laptops, because they typically don't run a publicly
  listending sshd

But there are still new findings about the backdoor every now and then,
like that it may read/write on IPC sockets (contained in [2]) and I've
read similar[6] without the restriction on IPC.
Also I've seen some vague statements[7] that it might "install" public
keys (didn't really grasp what was meant there - something like "in
authorized_keys"). And one report[8] talked about it collecting
usernames and IPs and passing the on to some function with unknown
purpose.

It also seems like these effort focus mostly on the 5.6.1 version and
while it's said that the 5.6.0 version is quite similar, who knows the
exact details!?


In any case and (too) long story short:

It would be nice to know whether there's still work done about finding
out whether people who had the malicious code on their systems (in any
version of the backdoor), but
- had sshd not running at all
and/or
- it was not reachable from the internet

can feel safe.

Or whether it may be possible that:
- the backdoor did call home (loaded commands from there, leaked
  private keys or so from the system)
- used completely different vectors not involving sshd
- or somehow else infested the system


Right now people might still have backups to torch their possibly
compromised systems and start over from a safe sate.


So Thorsten, in case you or someone else is aware of any [intermediate]
results from these task forces ([9[) it would be nice to hear about
them or better even in form of some "official" statement from Debian.


Thanks,
Chris.


[0] https://discord.gg/u6MzmQm5
[1] https://github.com/smx-smx/xzre
[2] https://github.com/binarly-io/binary-risk-intelligence/tree/master/xz-backdoor
[3] https://securelist.com/xz-backdoor-story-part-1/112354/
[4] https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
[5] https://github.com/przemoc/xz-backdoor-links
    https://przemoc.github.io/xz-backdoor-links/  (rendering of that)
[6] https://discord.com/channels/1223666474091020432/1223666474972090430/1230974749522530304
[7] https://discord.com/channels/1223666474091020432/1223666474972090430/1230173131746840606
[8] https://isc.sans.edu/diary/30802
[9] E.g. on d-d https://lists.debian.org/debian-devel/2024/03/msg00338.html
    Moritz Mühlenhoff has mentioned that some company was working on it
    and results were expected in some time.

Message #157 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Boud Roukema <bouddebbug@cosmo.torun.pl>

To: bug 1068024 <1068024@bugs.debian.org>

Cc: Boud Roukema <bouddebbug@cosmo.torun.pl>

Subject: latest Jia-Tan-free xz version seems to be 5.2.5

Date: Sat, 20 Apr 2024 11:49:10 +0200 (CEST)

hi all

Leaving aside the questions of dependencies of other Debian packages,
I agree that 5.2.5 looks like the most recent version that is
definitely free of any directly or indirectly authored contributions
by Jia Tan.

https://salsa.debian.org/debian/xz-utils/-/tree/v5.2.5?ref_type=tags

In the salsa full source:

$ git checkout v5.2.5
Previous HEAD position was d24a57b7 Bump version and soname for 5.2.7.
HEAD is now at 2327a461 Bump version and soname for 5.2.5.

$ git log --stat --graph |grep "Jia"

# No sign of Jia Tan.

$ git checkout v5.2.6
Previous HEAD position was 2327a461 Bump version and soname for 5.2.5.
HEAD is now at 8dfed05b Bump version and soname for 5.2.6.

$ git log --stat --graph |grep "Jia" |wc
     16     129     774

# Two commits and several 'Thanks to ... '.

Cheers
Boud


PS: For any RedHat people reading this thread: unfortunately, 5.2.5 is
before the fix 31d80c6b that Lasse Collin did to handle the "ill patch"
introduced by RHEL/CentOS7:

  https://salsa.debian.org/debian/xz-utils/-/commit/31d80c6b261b24220776dfaeb8a04f80f80e0a24

That's a RHEL problem to handle, not a Debian problem.

Message #162 received at 1068024@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Christoph Anton Mitterer <calestyo@scientia.org>, 1068024@bugs.debian.org

Subject: Re: Bug#1068024: revert to version that does not contain changes by bad actor

Date: Mon, 22 Apr 2024 23:43:21 +0000 (UTC)

Christoph Anton Mitterer dixit:

>So Thorsten, in case you or someone else is aware of any [intermediate]
>results from these task forces ([9[) it would be nice to hear about
>them or better even in form of some "official" statement from Debian.

JFTR I’m not involved in those myself.

bye,
//mirabilos
-- 
When he found out that the m68k port was in a pretty bad shape, he did
not, like many before him, shrug and move on; instead, he took it upon
himself to start compiling things, just so he could compile his shell.
How's that for dedication. -- Wouter, about my Debian/m68k revival

Send a report that this bug log contains spam.