Debian Bug report logs - #919207
squashfs-tools: frag_deflator_thread removal patch breaks lzo compression

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: submit@bugs.debian.org

Subject: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Sun, 13 Jan 2019 19:31:34 +0000

[Message part 1 (text/plain, inline)]

Package: squashfs-tools
Version: 1:4.3-9
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: toolchain randomness
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org, Alexander Couzens <lynxis@fe80.eu>

Hi Laszlo,

Re. <https://bugs.debian.org/918480#32> and similar, please find
attached the following patch that I have rebased against
squashfs-tools 4.3-9:

    From: Alexander Couzens <lynxis@fe80.eu>
    Date: Tue, 8 Jan 2019 10:57:00 +0100
    Subject: remove frag_deflator_thread

    frag_deflator_thread compress fragments.
    Replace the deflator_thread with a function and
    use the function instead of the to_frag queue.
    ---
     squashfs-tools/info.c       |  5 ---
     squashfs-tools/mksquashfs.c | 76 ++++++++++++++++-----------------------------
     squashfs-tools/mksquashfs.h |  2 +-
     squashfs-tools/restore.c    | 15 ++-------
     4 files changed, 30 insertions(+), 68 deletions(-)

This, I believe, is the final patch required for reproducible squashfs
images at least for Tails [0] and would mean we could drop our custom
package in that derivative distribution.

 [0] https://tails.boum.org/


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

[0016-remove-frag_deflator_thread.patch (text/x-patch, attachment)]

Message #10 received at 919207-close@bugs.debian.org (full text, mbox, reply):

From: Laszlo Boszormenyi (GCS) <gcs@debian.org>

To: 919207-close@bugs.debian.org

Subject: Bug#919207: fixed in squashfs-tools 1:4.3-10

Date: Tue, 15 Jan 2019 20:47:33 +0000

Source: squashfs-tools
Source-Version: 1:4.3-10

We believe that the bug you reported is fixed in the latest version of
squashfs-tools, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 919207@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Laszlo Boszormenyi (GCS) <gcs@debian.org> (supplier of updated squashfs-tools package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Tue, 15 Jan 2019 18:12:19 +0000
Source: squashfs-tools
Binary: squashfs-tools
Architecture: source
Version: 1:4.3-10
Distribution: unstable
Urgency: medium
Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Changed-By: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Description:
 squashfs-tools - Tool to create and append to squashfs filesystems
Closes: 916400 919207
Changes:
 squashfs-tools (1:4.3-10) unstable; urgency=medium
 .
   * Backport upstream Zstandard support (closes: #916400).
 .
   [ Alexander Couzens <lynxis@fe80.eu> ]
   * Remove frag_deflator_thread (closes: #919207).
Checksums-Sha1:
 6a22fe71dadcfc8c3b1e3ff2081d6501ba350457 1879 squashfs-tools_4.3-10.dsc
 a48caff1ef195cb1380626f7c7ece59d7e60d464 27036 squashfs-tools_4.3-10.debian.tar.xz
 a8f6f768ba656b2291c55477d9f788cc10df27b0 6941 squashfs-tools_4.3-10_amd64.buildinfo
Checksums-Sha256:
 b8451847553bafeab3b145474a1e61268ceae333f87c1f2a127cf7647a1c9fd2 1879 squashfs-tools_4.3-10.dsc
 a7496805599a2a461282354896873061681e79213e3396e9bc44832932634f38 27036 squashfs-tools_4.3-10.debian.tar.xz
 00c3382790baa6710665aa0e22a3b95061d21c2b5013a77084631271837a4227 6941 squashfs-tools_4.3-10_amd64.buildinfo
Files:
 b134b21eed5fc107a5917c181bc8b41b 1879 kernel optional squashfs-tools_4.3-10.dsc
 e38939964b76ee39337ec2766c60a1ec 27036 kernel optional squashfs-tools_4.3-10.debian.tar.xz
 ff3860dc5c4d0318da3bdfba6603b5ac 6941 kernel optional squashfs-tools_4.3-10_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEfYh9yLp7u6e4NeO63OMQ54ZMyL8FAlw+OdMACgkQ3OMQ54ZM
yL/bOhAAih+KVF2ITvnMxw9I45J/4KLguj2aAwQuCVcpUrpjY7qlAASLLrmw64L2
qQCWdOhTMBDj7utzUJE1AK8WbyLUNAox1dYKKuW6V5jVuT/1uwtIcKg0YYfdRxul
c7PU68QaxZQ0OiAyikNhrEBW33qWybzqj8okjIvNo3buKJ166LRSkCAjeaH+DsqN
5mu0q7YAvAVx0rBh2o9WOZmaCWkwK3gVCVe18Y8uvWHZmC1YfehKIpnZy9hduIGn
uIR4aBtO+zVc2Cr5J9ptBX3mzDqyJxdFPVkC/w1+0U1+WnTKZAfOtgrqfu9m+fPo
Y/aF/Ojqs4DZeKl0EYvvQu2T/AF6/bTJELs+HDM1r0Edqej65SxvL4PktyXopzjn
yvl/g7Hg799Dgoy0+yv1yNnxDX30jT8CGj2jiXbcXXPSNX0OWIyBhZN8fBbZ6ODL
1qwvQOvg2wXanIWBtU2Xh3fzQWbYFFQzQHN6QFHtXy1LxSFN5Ug5LixAWFZfi5vq
F1B2a5drsCfijQ5THkIh9dZM0tJk4vM1pzqEnPg/VdGIsUiQ4TNbzBu10Wwa4JyE
RFOHpYQfxaKfA13vhWRQoWtWciEG9XTWdc8xYLCnd8xpP4oeM2MYkUlTGtawjEu/
GftnZas61NeXHtpxQmq3hIhBF/Uqgayho72d18bcziZrg5wYya0=
=N5a3
-----END PGP SIGNATURE-----

Message #15 received at 919207@bugs.debian.org (full text, mbox, reply):

From: László Böszörményi (GCS) <gcs@debian.org>

To: Chris Lamb <lamby@debian.org>, Alexander Couzens <lynxis@fe80.eu>

Cc: 919207@bugs.debian.org

Subject: Re: Bug#919207: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Fri, 18 Jan 2019 08:23:26 +0100

Control: reopen -1

Hi Chris and Alexander,

On Sun, Jan 13, 2019 at 8:33 PM Chris Lamb <lamby@debian.org> wrote:
> Re. <https://bugs.debian.org/918480#32> and similar, please find
> attached the following patch that I have rebased against
> squashfs-tools 4.3-9:
>
>     From: Alexander Couzens <lynxis@fe80.eu>
>     Date: Tue, 8 Jan 2019 10:57:00 +0100
>     Subject: remove frag_deflator_thread
[...]
 I've applied this and uploaded to -10, but it breaks lzo compression.
You can test it with the following:
$ mksquashfs dir/with/some/files/ files.shfs -noappend -no-recovery
-no-xattrs -no-exports -comp lzo
It segfaults or print 'double free or corrupt' but always fail soon.
May you check this patch please?

Thanks,
Laszlo/GCS

Message #24 received at 919207@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: László Böszörményi (GCS) <gcs@debian.org>, Alexander Couzens <lynxis@fe80.eu>

Cc: 919207@bugs.debian.org

Subject: Re: Bug#919207: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Fri, 18 Jan 2019 19:28:06 +0000

[Message part 1 (text/plain, inline)]

László Böszörményi wrote:

> >     From: Alexander Couzens <lynxis@fe80.eu>
> >     Date: Tue, 8 Jan 2019 10:57:00 +0100
> >     Subject: remove frag_deflator_thread
> [...]
>  I've applied this and uploaded to -10, but it breaks lzo compression.

Oh dear, I'm really sorry about that. Updated patch attached, with
the interdiff being:

     frag_deflator_thread compress fragments.
     Replace the deflator_thread with a function and
     use the function instead of the to_frag queue.
    +
    +Updated Fri, 18 Jan 2019 19:22:58 +0000 by Chris Lamb <lamby@debian.org> to
    +fix issue with lack of compressor initialisation affecting (at least) LZO
    +compression.
     ---
      
    @@ -71,11 +75,11 @@ index 62888ec..465f9fe 100644
     +void frag_deflator(struct file_buffer *file_buffer)
      {
     -	void *stream = NULL;
    --	int res;
    --
    --	res = compressor_init(comp, &stream, block_size, 1);
    --	if(res)
    --		BAD_ERROR("frag_deflator:: compressor_init failed\n");
    + 	int res;
    + 
    + 	res = compressor_init(comp, &stream, block_size, 1);
    + 	if(res)
    + 		BAD_ERROR("frag_deflator:: compressor_init failed\n");


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

[0016-remove-frag_deflator_thread.patch (text/x-patch, attachment)]

Message #33 received at 919207-close@bugs.debian.org (full text, mbox, reply):

From: Laszlo Boszormenyi (GCS) <gcs@debian.org>

To: 919207-close@bugs.debian.org

Subject: Bug#919207: fixed in squashfs-tools 1:4.3-11

Date: Sat, 19 Jan 2019 10:09:06 +0000

Source: squashfs-tools
Source-Version: 1:4.3-11

We believe that the bug you reported is fixed in the latest version of
squashfs-tools, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 919207@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Laszlo Boszormenyi (GCS) <gcs@debian.org> (supplier of updated squashfs-tools package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Fri, 18 Jan 2019 20:45:11 +0000
Source: squashfs-tools
Binary: squashfs-tools
Architecture: source
Version: 1:4.3-11
Distribution: unstable
Urgency: medium
Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Changed-By: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Description:
 squashfs-tools - Tool to create and append to squashfs filesystems
Closes: 919207
Changes:
 squashfs-tools (1:4.3-11) unstable; urgency=medium
 .
   [ Chris Lamb <lamby@debian.org> ]
   * Fix issue with lack of compressor initialisation affecting (at least) LZO
     compression (closes: #919207).
Checksums-Sha1:
 6dfaabb7923643fb51b7827178fc79ad9a85818c 1879 squashfs-tools_4.3-11.dsc
 f0c17496980efba2c3141f2a831db1e6ec960ed9 27108 squashfs-tools_4.3-11.debian.tar.xz
 7f519c763c907931afdada4ded07681f4fb39873 6941 squashfs-tools_4.3-11_amd64.buildinfo
Checksums-Sha256:
 bcbfea967a3555c6e7b08050ef612f03f356c75c6ab1749f29e61f12bb9a6277 1879 squashfs-tools_4.3-11.dsc
 959147fe13543f1ead5907dcf4c21da9115c17e6d5ca5e4f0a51f9017d76b400 27108 squashfs-tools_4.3-11.debian.tar.xz
 e6a718c14e8a1cced8f4bf2ffb9b9a0726b71abb9887485d0d28274f8d264341 6941 squashfs-tools_4.3-11_amd64.buildinfo
Files:
 b17bdd9cb29ce674803d48762dfb7fbb 1879 kernel optional squashfs-tools_4.3-11.dsc
 740e3d363d2d46b80570e9ca2d9c0d0d 27108 kernel optional squashfs-tools_4.3-11.debian.tar.xz
 23ea02094e9435249be538eca328b705 6941 kernel optional squashfs-tools_4.3-11_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEfYh9yLp7u6e4NeO63OMQ54ZMyL8FAlxC67MACgkQ3OMQ54ZM
yL9Y+RAAmRWqv1kU3d2X8o8qVAKCubCzCKm8rbOrAfXOfPzh0awJ0IXo5V4x/oSM
UYHwtb3l7F2iP6bYRokqLhde/Qh3n5ny56GXrU3sivEno1IVzaOXfWcdf6OsCX7Q
kF2F+EvHF6FhgdOAlkTHj4KacHtw+aCXJm3c6HIYUqCQGSyCodvFDse/0OGmH+/G
vHIcr5QIabgd4F49UzLUDKVzQ8RzGAek0As2DHwhWBtj7YC+m9jZOEuAOkSgk0NO
Bt9glxg/KbAjG+pSoeRXmKDHeOCLHkz+HEPnJ/7oXLX2ML3RrqLpQPud3NGQslLT
by8q4MOpfYKgiu3LT46gUWlpiSTyyX4Pee4rukWXk/4wwKDbNeh/OOqy8IUQsF1z
27+C04Xm0z1W9qT5a0Fe7f0V4D6eH7MamftypJxGZ/u7dUKzRbmF2M06U19lJh/U
ISaP1QPi7oyeUmKvv6HQAuwcyJuJuN/ReHqNgsRFDOW9TuXyjX21MEFbFe74n3TZ
vD7P64DpWa8ldZ/5kyG3iR+WemYt6Hp+XmgMkdegxVm4ZdYa5lVZoS14H3ar2crD
NHBIIpAtIcmFbhsZU3J58oeBfMeiRiJ+xAt36vuOARzKIoB9GZTFCoyDEhofbc5l
mfOr6+qOdJiTPQxlR3xPthmLX1m+6sRuERQQ9bA2hnPbHsFARZI=
=oWiw
-----END PGP SIGNATURE-----

Message #38 received at 919207@bugs.debian.org (full text, mbox, reply):

From: Dominik George <natureshadow@debian.org>

To: Chris Lamb <lamby@debian.org>

Cc: 919207@bugs.debian.org, 923711@bugs.debian.org

Subject: Re: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Mon, 18 Mar 2019 19:20:51 +0100

[Message part 1 (text/plain, inline)]

Hi Chris,

On Sun, Jan 13, 2019 at 07:31:34PM +0000, Chris Lamb wrote:
> Re. <https://bugs.debian.org/918480#32> and similar, please find
> attached the following patch that I have rebased against
> squashfs-tools 4.3-9:
> 
> […]

Please be aware that this patch caused another breakage in mksquashfs:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923711

As this breaks quite a few processes for me, I would be glad if you
could look into what is missing in the patch to fix the memory leak.

Cheers,
Nik

[signature.asc (application/pgp-signature, inline)]

Message #43 received at 919207@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: "Dominik George" <natureshadow@debian.org>

Cc: 919207@bugs.debian.org, 923711@bugs.debian.org, "Alexander Couzens" <lynxis@fe80.eu>

Subject: Re: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Mon, 18 Mar 2019 14:24:29 -0400

[Adding lynxis@fe80.eu to CC]

Hi Dominik,

> > Re. <https://bugs.debian.org/918480#32> and similar, please find
> > attached the following patch that I have rebased against
> > squashfs-tools 4.3-9:
> > 
> > […]
> 
> Please be aware that this patch caused another breakage in mksquashfs:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923711
> 
> As this breaks quite a few processes for me, I would be glad if you
> could look into what is missing in the patch to fix the memory leak.

Thanks, and ACK. (Hey, lyxis!)


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Message #48 received at 919207@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: "Dominik George" <natureshadow@debian.org>

Cc: 919207@bugs.debian.org, 923711@bugs.debian.org, "Alexander Couzens" <lynxis@fe80.eu>

Subject: Re: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Mon, 18 Mar 2019 14:28:20 -0400

Chris Lamb wrote:

> Thanks, and ACK.

Looks like a few other folks have hit this issue:

  https://redmine.tails.boum.org/code/issues/16457


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Message #53 received at 919207@bugs.debian.org (full text, mbox, reply):

From: László Böszörményi (GCS) <gcs@debian.org>

To: Chris Lamb <lamby@debian.org>

Cc: Dominik George <natureshadow@debian.org>, 919207@bugs.debian.org, 923711@bugs.debian.org, Alexander Couzens <lynxis@fe80.eu>, 921146@bugs.debian.org

Subject: Re: Bug#923711: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Sun, 24 Mar 2019 11:26:25 +0100

[Message part 1 (text/plain, inline)]

Hi all,

On Mon, Mar 18, 2019 at 7:33 PM Chris Lamb <lamby@debian.org> wrote:

> Looks like a few other folks have hit this issue:
>   https://redmine.tails.boum.org/code/issues/16457
>
 This is also the case why not all CPU cores are used  during the run of
mksquashfs.
Am I right that this isn't going to be fixed soon? I will remove this patch
on Monday in the afternoon as the release of Buster is getting close.

Regards,
Laszlo/GCS

[Message part 2 (text/html, inline)]

Message #58 received at 919207@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: László Böszörményi <gcs@debian.org>

Cc: "Dominik George" <natureshadow@debian.org>, 919207@bugs.debian.org, 923711@bugs.debian.org, "Alexander Couzens" <lynxis@fe80.eu>, 921146@bugs.debian.org

Subject: Re: Bug#923711: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Mon, 25 Mar 2019 10:14:43 -0400

Hi László,

> Am I right that this isn't going to be fixed soon? I will remove this 
> patch on Monday in the afternoon as the release of Buster is getting 
> close.

Alas, I don't think I will be able to get to writing a patch/fix today
as I don't fully understand the leak (or, actually, the original patch!)
just yet.

Lynxis, do you have any quick pointers, however?


Best wishes,,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Message #63 received at 919207@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: "Alexander Couzens" <lynxis@fe80.eu>

Cc: László Böszörményi <gcs@debian.org>, "Dominik George" <natureshadow@debian.org>, 919207@bugs.debian.org, 923711@bugs.debian.org, 921146@bugs.debian.org

Subject: Re: Bug#923711: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Mon, 25 Mar 2019 10:31:55 -0400

Dear lynxis,

> original, my patch had the problem, that it use the 
> same stream (compressor context) as the main thread. This seems to be
> ok, as long the compression is not lzo (I tested with lzma).
> 
> After your patch to fix lzo, the compressor will be initialized with a
> new context on every fragment instead only once for all fragments.
> Also important, all compressor_init's will leak, but since they are
> only called once, it's not a big problem to leak some kbs.

Ah!

Laszlo, would you like any assistance with a debdiff or is lynxis'
patch sufficient asis? You will have to patch the existing patch, if
you see what I mean...


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Message #68 received at 919207@bugs.debian.org (full text, mbox, reply):

From: Alexander Couzens <lynxis@fe80.eu>

To: "Chris Lamb" <lamby@debian.org>

Cc: László Böszörményi <gcs@debian.org>, "Dominik George" <natureshadow@debian.org>, 919207@bugs.debian.org, 923711@bugs.debian.org, 921146@bugs.debian.org

Subject: Re: Bug#923711: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Mon, 25 Mar 2019 15:49:42 +0100

[Message part 1 (text/plain, inline)]

> Laszlo, would you like any assistance with a debdiff or is lynxis'
> patch sufficient asis? You will have to patch the existing patch, if
> you see what I mean...

Here is an updated patch to match debian's state of squashfs-tools.

Best,
lynxis

[0018-mksquashfs-fix-compressor-initialisation-in-frag_def.patch (text/x-patch, attachment)]

[Message part 3 (application/pgp-signature, inline)]

Message #73 received at 919207@bugs.debian.org (full text, mbox, reply):

From: Alexander Couzens <lynxis@fe80.eu>

To: "Chris Lamb" <lamby@debian.org>

Cc: László Böszörményi <gcs@debian.org>, "Dominik George" <natureshadow@debian.org>, 919207@bugs.debian.org, 923711@bugs.debian.org, 921146@bugs.debian.org

Subject: Re: Bug#923711: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Mon, 25 Mar 2019 15:27:40 +0100

[Message part 1 (text/plain, inline)]

Hi lamby,

> Alas, I don't think I will be able to get to writing a patch/fix today
> as I don't fully understand the leak (or, actually, the original
> patch!) just yet.
> 
> Lynxis, do you have any quick pointers, however?

original, my patch had the problem, that it use the 
same stream (compressor context) as the main thread. This seems to be
ok, as long the compression is not lzo (I tested with lzma).

After your patch to fix lzo, the compressor will be initialized with a
new context on every fragment instead only once for all fragments.
Also important, all compressor_init's will leak, but since they are
only called once, it's not a big problem to leak some kbs.

I attached a patch to fix this to the bug #923711
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923711
It would be nice if you can test it.

Best,
lynxis

PS: If you like to write a small test script, I'm happy to add it to the
CI ;)

[Message part 2 (application/pgp-signature, inline)]

Message #78 received at 919207@bugs.debian.org (full text, mbox, reply):

From: László Böszörményi (GCS) <gcs@debian.org>

To: Alexander Couzens <lynxis@fe80.eu>

Cc: Chris Lamb <lamby@debian.org>, Dominik George <natureshadow@debian.org>, 919207@bugs.debian.org, 923711@bugs.debian.org, 921146@bugs.debian.org

Subject: Re: Bug#923711: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Mon, 25 Mar 2019 21:11:59 +0100

On Mon, Mar 25, 2019 at 3:50 PM Alexander Couzens <lynxis@fe80.eu> wrote:
> Here is an updated patch to match debian's state of squashfs-tools.
 I'm playing with the different patched versions for hours now.
Indeed, it seems with your latest patch the memory leak is bearable.
I've never tested CPU usage before. I've realized it's rhapsodical,
sometimes all my CPU cores utilized on 100%, sometimes one on ~15%.
Should I wait for other fix(es)?

Thanks,
Laszlo/GCS

Message #83 received at 919207@bugs.debian.org (full text, mbox, reply):

From: Alexander Couzens <lynxis@fe80.eu>

To: "László Böszörményi (GCS)" <gcs@debian.org>

Cc: Chris Lamb <lamby@debian.org>, Dominik George <natureshadow@debian.org>, 919207@bugs.debian.org, 923711@bugs.debian.org, 921146@bugs.debian.org

Subject: Re: Bug#923711: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Mon, 25 Mar 2019 23:03:58 +0100

[Message part 1 (text/plain, inline)]

Hi Laszlo,

>  I'm playing with the different patched versions for hours now.
> Indeed, it seems with your latest patch the memory leak is bearable.

Nice to hear. I compressed a 8GB tree of toolchains and rootfs with
valgrind (took > 10h ;) without any noticable leaks.

> I've never tested CPU usage before. I've realized it's rhapsodical,
> sometimes all my CPU cores utilized on 100%, sometimes one on ~15%.
> Should I wait for other fix(es)?

No. I'ven't looked into it. But this is a long standing issue.

First of all, as squashfs-tools wasn't written in the last years, when
reproducible builds became more famous. So it's not written
with reproducible building in mind.
For me is reproducible builds more important than using all cpu cores.
But I don't use it with gigabytes images.

It got a bit worse with removing the frag_deflator thread, as there
is one workload thread less. The old frag_deflator thread has been
spawnd $cpus times. fragments are small files.

The cpu usage will also depend on the compression algorithm.
If you like to profile mksquashfs-tools, I'm happy to help to understand
the code base.

Maybe there is a different approach to do it reproducible. Create
first an index over all files, ensure a proper order through multiple
queues via an index. But I'm not sure, if this would be really faster
than it's now. Btw: even with `-processors 1` it will spawn multiple
threads which should use more than 1 core.

Best,
lynxis

[Message part 2 (application/pgp-signature, inline)]

Message #88 received at 919207@bugs.debian.org (full text, mbox, reply):

From: László Böszörményi (GCS) <gcs@debian.org>

To: Alexander Couzens <lynxis@fe80.eu>

Cc: Chris Lamb <lamby@debian.org>, Dominik George <natureshadow@debian.org>, 919207@bugs.debian.org, 923711@bugs.debian.org, 921146@bugs.debian.org

Subject: Re: Bug#923711: squashfs-tools: Please apply frag_deflator_thread removal patch

Date: Tue, 26 Mar 2019 11:32:05 +0100

Hi lynxis,

On Mon, Mar 25, 2019 at 11:04 PM Alexander Couzens <lynxis@fe80.eu> wrote:
> >  I'm playing with the different patched versions for hours now.
> > Indeed, it seems with your latest patch the memory leak is bearable.
>
> Nice to hear. I compressed a 8GB tree of toolchains and rootfs with
> valgrind (took > 10h ;) without any noticable leaks.
 I used a smaller directory for rootfs but run several times with the
different patched squashfs-tools. The -11 package version used ten
times more memory it was limited to due to the leak. With your fix, it
still used more but max ten percent more.

> > I've never tested CPU usage before. I've realized it's rhapsodical,
> > sometimes all my CPU cores utilized on 100%, sometimes one on ~15%.
> > Should I wait for other fix(es)?
>
> No. I'ven't looked into it. But this is a long standing issue.
 But is it caused by your patch or upstream version has the same
problem as well?

> First of all, as squashfs-tools wasn't written in the last years, when
> reproducible builds became more famous. So it's not written
> with reproducible building in mind.
> For me is reproducible builds more important than using all cpu cores.
> But I don't use it with gigabytes images.
 Yeah, it's quite an old software without real development in the recent years.
I do agree that reproducibility is important, but most users might not
want it. Everyone would like their squashfs generated as quick as
possible especially if they have the HW (CPU with many cores) for
that. I don't want them to experience degraded performance.

> It got a bit worse with removing the frag_deflator thread, as there
> is one workload thread less. The old frag_deflator thread has been
> spawnd $cpus times. fragments are small files.
 Then it's strange. Even with the frag_deflator thread removed,
sometimes all my CPU cores are used. Sometimes not, but as I
understand you, I should try different compression algorithms.

> Maybe there is a different approach to do it reproducible. Create
> first an index over all files, ensure a proper order through multiple
> queues via an index. But I'm not sure, if this would be really faster
> than it's now. Btw: even with `-processors 1` it will spawn multiple
> threads which should use more than 1 core.
 This sounds more complex work than it can be achieved in this week.
Maybe a complete rewrite would be better then on the long run.

Cheers,
Laszlo/GCS

Send a report that this bug log contains spam.