Debian Bug report logs - #1050586
kmod does not work with XZ in-kernel module decompression

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Jon Westgate <0ryn@fsck.tv>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sat, 26 Aug 2023 16:54:54 +0100

Package: kmod
Version: 30+20230601-1
Severity: grave
Justification: renders system unusable

Dear Maintainer,

I just did a normal apt upgrade and the following packages were upgraded:
kmod 30+20230519-1 to 30-2.230601-1
libkmod2 30+20230519-1 to 30-2.230601-1

The system partially booted but systemd then prevented boot due to missing
modules,
The error message it gave was "decompresson failed with status 6"

I had to boot from a rescue usb stick, set a root password, then I was able to
interact with systemd's idea of a diagnotic console - I'm not systemd's biggest
fan
I used another system to download the older versions of kmod and libkmod2 and
as able to use dpkg to downgrade them.

I installed both on another system and it does the same.
I'm not sure if this effects other system's with sysvinit or non efi.

Regards
Jon Westgate


-- System Information:
Debian Release: trixie/sid
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 6.4.11 (SMP w/12 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_FIRMWARE_WORKAROUND, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages kmod depends on:
ii  libc6     2.37-7
ii  libkmod2  30+20230601-1
ii  liblzma5  5.4.1-0.2
ii  libssl3   3.0.10-1
ii  libzstd1  1.5.5+dfsg2-1

kmod recommends no packages.

kmod suggests no packages.

-- no debconf information

Message #14 received at 1050586@bugs.debian.org (full text, mbox, reply):

From: Marco d'Itri <md@Linux.IT>

To: Jon Westgate <0ryn@fsck.tv>, 1050586@bugs.debian.org

Subject: Re: Bug#1050586: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sat, 26 Aug 2023 19:34:47 +0200

[Message part 1 (text/plain, inline)]

On Aug 26, Jon Westgate <0ryn@fsck.tv> wrote:

> The system partially booted but systemd then prevented boot due to missing
> modules,
> The error message it gave was "decompresson failed with status 6"
Are you using compressed modules?

-- 
ciao,
Marco

[signature.asc (application/pgp-signature, inline)]

Message #19 received at 1050586@bugs.debian.org (full text, mbox, reply):

From: Marco d'Itri <md@Linux.IT>

To: Jon Westgate <0ryn@fsck.tv>, 1050586@bugs.debian.org

Subject: Re: Bug#1050586: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sat, 26 Aug 2023 19:58:09 +0200

[Message part 1 (text/plain, inline)]

On Aug 26, Jon Westgate <0ryn@fsck.tv> wrote:

> Kernel: Linux 6.4.11 (SMP w/12 CPU threads; PREEMPT)
I see that you are using a custom kernel. What is the status of the
CONFIG_MODULE_COMPRESS_* kernel configuration options?

-- 
ciao,
Marco

[signature.asc (application/pgp-signature, inline)]

Message #24 received at 1050586-quiet@bugs.debian.org (full text, mbox, reply):

From: Jon Westgate <jon@fsck.tv>

To: Marco d'Itri <md@Linux.IT>

Subject: Re: Bug#1050586: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sat, 26 Aug 2023 19:13:05 +0100

[Message part 1 (text/plain, inline)]

Yes I am using compressed modules

On 26/08/2023 18:34, Marco d'Itri wrote:
> On Aug 26, Jon Westgate <0ryn@fsck.tv> wrote:
>
>> The system partially booted but systemd then prevented boot due to missing
>> modules,
>> The error message it gave was "decompresson failed with status 6"
> Are you using compressed modules?
>

[OpenPGP_0x420652A8C309A3B2.asc (application/pgp-keys, attachment)]

[OpenPGP_signature.asc (application/pgp-signature, attachment)]

Message #29 received at 1050586-quiet@bugs.debian.org (full text, mbox, reply):

From: Jon Westgate <jon@fsck.tv>

To: Marco d'Itri <md@Linux.IT>

Subject: Re: Bug#1050586: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sat, 26 Aug 2023 19:14:13 +0100

[Message part 1 (text/plain, inline)]

CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_FORCE is not set
# CONFIG_MODULE_SIG_ALL is not set
# CONFIG_MODULE_SIG_SHA1 is not set
# CONFIG_MODULE_SIG_SHA224 is not set
CONFIG_MODULE_SIG_SHA256=y
# CONFIG_MODULE_SIG_SHA384 is not set
# CONFIG_MODULE_SIG_SHA512 is not set
CONFIG_MODULE_SIG_HASH="sha256"
# CONFIG_MODULE_COMPRESS_NONE is not set
# CONFIG_MODULE_COMPRESS_GZIP is not set
CONFIG_MODULE_COMPRESS_XZ=y
# CONFIG_MODULE_COMPRESS_ZSTD is not set
CONFIG_MODULE_DECOMPRESS=y
# CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS is not set
CONFIG_MODPROBE_PATH="/usr/sbin/modprobe"


On 26/08/2023 18:58, Marco d'Itri wrote:
> On Aug 26, Jon Westgate <0ryn@fsck.tv> wrote:
>
>> Kernel: Linux 6.4.11 (SMP w/12 CPU threads; PREEMPT)
> I see that you are using a custom kernel. What is the status of the
> CONFIG_MODULE_COMPRESS_* kernel configuration options?
>

[OpenPGP_0x420652A8C309A3B2.asc (application/pgp-keys, attachment)]

[OpenPGP_signature.asc (application/pgp-signature, attachment)]

Message #34 received at 1050586@bugs.debian.org (full text, mbox, reply):

From: Marco d'Itri <md@linux.it>

To: Jon Westgate <jon@fsck.tv>, 1050586@bugs.debian.org

Subject: Re: Bug#1050586: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sat, 26 Aug 2023 23:25:18 +0200

[Message part 1 (text/plain, inline)]

On Aug 26, Jon Westgate <jon@fsck.tv> wrote:

> Yes I am using compressed modules
And are these modules compressed with xz or something else?

This new code was introduced in the latest snapshot, and apparently it 
fails when used with kernels with compressed modules support enabled 
(which so far is not the default for Debian kenrels).

-- 
ciao,
Marco

[signature.asc (application/pgp-signature, inline)]

Message #41 received at 1050586@bugs.debian.org (full text, mbox, reply):

From: Jon Westgate <jon@fsck.tv>

To: Marco d'Itri <md@linux.it>

Subject: Re: Bug#1050586: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sat, 26 Aug 2023 23:51:27 +0100

[Message part 1 (text/plain, inline)]

Dear Marco,
I'm using XZ if I turn off compression my initrd fails it's too big.

Jon

On 26/08/2023 22:25, Marco d'Itri wrote:
> On Aug 26, Jon Westgate <jon@fsck.tv> wrote:
>
>> Yes I am using compressed modules
> And are these modules compressed with xz or something else?
>
> This new code was introduced in the latest snapshot, and apparently it
> fails when used with kernels with compressed modules support enabled
> (which so far is not the default for Debian kenrels).
>

[OpenPGP_0x420652A8C309A3B2.asc (application/pgp-keys, attachment)]

[OpenPGP_signature.asc (application/pgp-signature, attachment)]

Message #46 received at 1050586@bugs.debian.org (full text, mbox, reply):

From: Marco d'Itri <md@Linux.IT>

To: Jon Westgate <0ryn@fsck.tv>, antdev66@gmail.com, 1050586@bugs.debian.org

Subject: Re: Bug#1050586: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sun, 27 Aug 2023 03:10:09 +0200

[Message part 1 (text/plain, inline)]

On Aug 26, Jon Westgate <0ryn@fsck.tv> wrote:

> The error message it gave was "decompresson failed with status 6"
Status 6 is XZ_OPTIONS_ERROR, which means "Input was encoded with 
settings that are not supported by this XZ decoder".
So it looks like you have compressed the modules (how?) with XZ settings 
which are supported by the userspace loader but not by the kernel one.

-- 
ciao,
Marco

[signature.asc (application/pgp-signature, inline)]

Message #51 received at 1050586@bugs.debian.org (full text, mbox, reply):

From: Antonio <antdev66@gmail.com>

To: Marco d'Itri <md@Linux.IT>, Jon Westgate <0ryn@fsck.tv>, 1050586@bugs.debian.org

Subject: Re: Bug#1050586: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sun, 27 Aug 2023 10:00:29 +0200

I don't build kernel from source, I use binary packages installed from 
liquorix repositories https://liquorix.net/.

I opened a ticket to their forum, maybe a recompilation of the kernel is 
enough.

We await your reply...


Il 27/08/23 03:10, Marco d'Itri ha scritto:
> On Aug 26, Jon Westgate <0ryn@fsck.tv> wrote:
>
>> The error message it gave was "decompresson failed with status 6"
> Status 6 is XZ_OPTIONS_ERROR, which means "Input was encoded with
> settings that are not supported by this XZ decoder".
> So it looks like you have compressed the modules (how?) with XZ settings
> which are supported by the userspace loader but not by the kernel one.
>

Message #56 received at 1050586@bugs.debian.org (full text, mbox, reply):

From: Antonio <antdev66@gmail.com>

To: Marco d'Itri <md@Linux.IT>, Jon Westgate <0ryn@fsck.tv>, 1050586@bugs.debian.org

Subject: Re: Bug#1050586: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sun, 27 Aug 2023 10:14:59 +0200

ticket:

https://techpatterns.com/forums/about3040.html

Message #61 received at 1050586@bugs.debian.org (full text, mbox, reply):

From: Jon Westgate <jon@fsck.tv>

To: Marco d'Itri <md@Linux.IT>, Jon Westgate <0ryn@fsck.tv>, antdev66@gmail.com, 1050586@bugs.debian.org

Subject: Re: Bug#1050586: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sun, 27 Aug 2023 11:39:05 +0100

[Message part 1 (text/plain, inline)]

All,
To confirm that switching to ZSTD works.

I ran make menuconfig,
   [*] Enable loadable module support  --->

                Module compression mode (ZSTD) --->
        [*]   Support in-kernel module decompression
Note that I already had "Support in-kernel module decompression" 
selected when the compression method was XZ.

Would you like me to try without it?

On a side note switching module compression mode did not require a full 
recompile.
I presume it just linked in a different module decompressor, then 
compressed the modules using ZSTD on the fly when I ran make modules_install

Jon

On 27/08/2023 11:24, Jon Westgate wrote:
> Dear Marco and others,
>
> I will do a test build using ZSTD and see if that works. - I think the 
> first thing I did was to switch back to XZ when this first happened 
> Yesterday.
>
> I've been using XZ for module compression since it was available, I 
> just select it using make menuconfig I'm not using anything other than 
> a slight tweek to the debian build.
> I don't use signing as my UEFI allows me not to, the only other thing 
> I do different to Debian is to build in USB support, filesystem 
> support for ext4 and BTRFS and build in SCSI, LIBATA so that should my 
> initrd fail for any reason I'm not left high and dry with no keyboard 
> support and no access to a filesystem. I used to build in networking 
> support, but I only do that systems that I use for firewalling as 
> systemd won't allow networking unless everything else is mostly perfect.
>
> I started using module compression for net boot thin clients as it 
> made the resulting initrd way quicker to load over TFTP. Later on I 
> switched to using it on all my systems as it made the boot process 
> much quicker.
>
> Finally I found that on recent 6.x kernels it was required due to a 
> size limit that seemed to appear initrd. I'm not sure what makes this 
> limit, it's there in grub and refind.
>
> If what you say is true then kmod might have been masking a kernel bug 
> for the last year.
>
> I'm not doing anything clever here, I just get my source from 
> cdn.kernel.org, use make menuconfig or vim to edit the .config and 
> make -j8 && make -j8 modules_install && make install then reboot.
> I'm not applying any patches, I do used the NVIDIA drivers on some boxes.
>
> Jon
>
>
> On 27/08/2023 02:10, Marco d'Itri wrote:
>> On Aug 26, Jon Westgate <0ryn@fsck.tv> wrote:
>>
>>> The error message it gave was "decompresson failed with status 6"
>> Status 6 is XZ_OPTIONS_ERROR, which means "Input was encoded with
>> settings that are not supported by this XZ decoder".
>> So it looks like you have compressed the modules (how?) with XZ settings
>> which are supported by the userspace loader but not by the kernel one.
>>
>

[Message part 2 (text/html, inline)]

[OpenPGP_0x420652A8C309A3B2.asc (application/pgp-keys, attachment)]

[OpenPGP_signature.asc (application/pgp-signature, attachment)]

Message #66 received at 1050586@bugs.debian.org (full text, mbox, reply):

From: Jon Westgate <jon@fsck.tv>

To: Marco d'Itri <md@Linux.IT>, Jon Westgate <0ryn@fsck.tv>, antdev66@gmail.com, 1050586@bugs.debian.org

Subject: Re: Bug#1050586: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sun, 27 Aug 2023 11:24:28 +0100

[Message part 1 (text/plain, inline)]

Dear Marco and others,

I will do a test build using ZSTD and see if that works. - I think the 
first thing I did was to switch back to XZ when this first happened 
Yesterday.

I've been using XZ for module compression since it was available, I just 
select it using make menuconfig I'm not using anything other than a 
slight tweek to the debian build.
I don't use signing as my UEFI allows me not to, the only other thing I 
do different to Debian is to build in USB support, filesystem support 
for ext4 and BTRFS and build in SCSI, LIBATA so that should my initrd 
fail for any reason I'm not left high and dry with no keyboard support 
and no access to a filesystem. I used to build in networking support, 
but I only do that systems that I use for firewalling as systemd won't 
allow networking unless everything else is mostly perfect.

I started using module compression for net boot thin clients as it made 
the resulting initrd way quicker to load over TFTP. Later on I switched 
to using it on all my systems as it made the boot process much quicker.

Finally I found that on recent 6.x kernels it was required due to a size 
limit that seemed to appear initrd. I'm not sure what makes this limit, 
it's there in grub and refind.

If what you say is true then kmod might have been masking a kernel bug 
for the last year.

I'm not doing anything clever here, I just get my source from 
cdn.kernel.org, use make menuconfig or vim to edit the .config and make 
-j8 && make -j8 modules_install && make install then reboot.
I'm not applying any patches, I do used the NVIDIA drivers on some boxes.

Jon


On 27/08/2023 02:10, Marco d'Itri wrote:
> On Aug 26, Jon Westgate <0ryn@fsck.tv> wrote:
>
>> The error message it gave was "decompresson failed with status 6"
> Status 6 is XZ_OPTIONS_ERROR, which means "Input was encoded with
> settings that are not supported by this XZ decoder".
> So it looks like you have compressed the modules (how?) with XZ settings
> which are supported by the userspace loader but not by the kernel one.
>

[OpenPGP_0x420652A8C309A3B2.asc (application/pgp-keys, attachment)]

[OpenPGP_signature.asc (application/pgp-signature, attachment)]

Message #71 received at 1050586@bugs.debian.org (full text, mbox, reply):

From: Marco d'Itri <md@linux.it>

To: Jon Westgate <jon@fsck.tv>

Cc: Jon Westgate <0ryn@fsck.tv>, antdev66@gmail.com, 1050586@bugs.debian.org

Subject: Re: Bug#1050586: kmod: Updating to kmod to 30+20230601-1 results in a non booting system modules cannot be decompressed

Date: Sun, 27 Aug 2023 12:42:33 +0200

[Message part 1 (text/plain, inline)]

Control: retitle -1 kmod does not work with XZ in-kernel module decompression

On Aug 27, Jon Westgate <jon@fsck.tv> wrote:

> Note that I already had "Support in-kernel module decompression" selected
> when the compression method was XZ.
> 
> Would you like me to try without it?
No need to: we know that everything works fine without in-kernel 
decompression, because this is how the Debian kernel is configured.

-- 
ciao,
Marco

[signature.asc (application/pgp-signature, inline)]

Message #74 received at 1050582-done@bugs.debian.org (full text, mbox, reply):

From: Marco d'Itri <md@Linux.IT>

To: Martin Nybo Andersen <tweek@tweek.dk>, 1050582-done@bugs.debian.org

Subject: Re: Bug#1050582: kmod update corrupts systemd uefi boot

Date: Sun, 24 Sep 2023 23:37:56 +0200

[Message part 1 (text/plain, inline)]

On Sep 15, Martin Nybo Andersen <tweek@tweek.dk> wrote:

> The problem is, that kmod has switched to using the kernel decompressor when
> available, which is XZ Embedded. This version doesn't handle CRC64 and
> dictionaries larger than 1 MiB.
Thank you a lot for the detailed explanation.
Since this is not a kmod bug and Debian does not ship compressed modules 
I will close the bug.

-- 
ciao,
Marco

[signature.asc (application/pgp-signature, inline)]

Send a report that this bug log contains spam.