Debian Bug report logs - #876416
libzstd FTBFS on mips(el): Allocation error : not enough memory

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Graham Inggs <ginggs@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: libzstd FTBFS on mips(el): Allocation error : not enough memory

Date: Fri, 22 Sep 2017 00:42:48 +0200

Source: libzstd
Version: 1.3.1+dfsg-1
Severity: serious

Hi Maintainer

Builds of libzstd/1.3.1+dfsg-1 FTBFS on mips and mipsel where they
were successful in the past.


/*stdin*\            :  5.90%   (99000000 => 5842431 bytes, /*stdout*\)
roundTripTest: ./datagen -g6000000000 -P99 |  ../programs/zstd -v1 -T2
|  ../programs/zstd -d

*** zstd command line interface 32-bits v1.3.1, by Yann Collet ***

Read : 6 MB
Read : 12 MB
Read : 16 MB
...
Read : 5711 MB
Read : 5716 MB
Read : 5720 MB


/*stdin*\            :  9.40%   (6000000000 => 563851359 bytes, /*stdout*\)
roundTripTest: ./datagen -g1500000000 -P97 |  ../programs/zstd -v1
-T999 |  ../programs/zstd -d

*** zstd command line interface 32-bits v1.3.1, by Yann Collet ***
zstd: error 11 : Allocation error : not enough memory
zstd: /*stdin*\: unexpected end of file
Files tmp1 and tmp2 differ
Makefile:260: recipe for target 'zstd-playTests' failed


Regards
Graham

Message #10 received at 876416@bugs.debian.org (full text, mbox, reply):

From: Andreas Tille <tille@debian.org>

To: Graham Inggs <ginggs@debian.org>, 876416@bugs.debian.org

Cc: debian-mips@lists.debian.org

Subject: Re: Bug#876416: libzstd FTBFS on mips(el): Allocation error : not enough memory

Date: Fri, 22 Sep 2017 08:36:45 +0200

control: tags -1 help

Hi Graham,

since I have no idea how to deal with this I'm forwarding the issue
to debian-mips list asking desperately for help.

Kind regards

      Andreas.

On Fri, Sep 22, 2017 at 12:42:48AM +0200, Graham Inggs wrote:
> Source: libzstd
> Version: 1.3.1+dfsg-1
> Severity: serious
> 
> Hi Maintainer
> 
> Builds of libzstd/1.3.1+dfsg-1 FTBFS on mips and mipsel where they
> were successful in the past.
> 
> 
> /*stdin*\            :  5.90%   (99000000 => 5842431 bytes, /*stdout*\)
> roundTripTest: ./datagen -g6000000000 -P99 |  ../programs/zstd -v1 -T2
> |  ../programs/zstd -d
> 
> *** zstd command line interface 32-bits v1.3.1, by Yann Collet ***
> 
> Read : 6 MB
> Read : 12 MB
> Read : 16 MB
> ...
> Read : 5711 MB
> Read : 5716 MB
> Read : 5720 MB
> 
> 
> /*stdin*\            :  9.40%   (6000000000 => 563851359 bytes, /*stdout*\)
> roundTripTest: ./datagen -g1500000000 -P97 |  ../programs/zstd -v1
> -T999 |  ../programs/zstd -d
> 
> *** zstd command line interface 32-bits v1.3.1, by Yann Collet ***
> zstd: error 11 : Allocation error : not enough memory
> zstd: /*stdin*\: unexpected end of file
> Files tmp1 and tmp2 differ
> Makefile:260: recipe for target 'zstd-playTests' failed
> 
> 
> Regards
> Graham
> 
> _______________________________________________
> Debian-med-packaging mailing list
> Debian-med-packaging@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/debian-med-packaging
> 

-- 
http://fam-tille.de

Message #17 received at 876416@bugs.debian.org (full text, mbox, reply):

From: James Cowgill <jcowgill@jcowgill.uk>

To: Andreas Tille <tille@debian.org>

Cc: Graham Inggs <ginggs@debian.org>, 876416@bugs.debian.org, debian-mips@lists.debian.org

Subject: Re: Bug#876416: libzstd FTBFS on mips(el): Allocation error : not enough memory

Date: Fri, 22 Sep 2017 11:16:52 +0100

[Message part 1 (text/plain, inline)]

Hi,

On 22/09/17 07:36, Andreas Tille wrote:
> control: tags -1 help
> 
> Hi Graham,
> 
> since I have no idea how to deal with this I'm forwarding the issue
> to debian-mips list asking desperately for help.
[...]
>> /*stdin*\            :  9.40%   (6000000000 => 563851359 bytes, /*stdout*\)
>> roundTripTest: ./datagen -g1500000000 -P97 |  ../programs/zstd -v1
>> -T999 |  ../programs/zstd -d
>>
>> *** zstd command line interface 32-bits v1.3.1, by Yann Collet ***
>> zstd: error 11 : Allocation error : not enough memory

I think the problem is using too many threads with big stacks. When you
use "zstd -T999", zstd will try to create 256 threads (the maximum
amount). Since the default stack size is 8M, this will try to allocate
2G of virtual memory. On most 32-bit architectures which have 3G address
space boundaries, there is just enough memory to do this. On mips and
mipsel, the boundary is at 2G so there will not be enough space to
allocate all these stacks along with any existing memory.

To fix this you could:

Use "ulimit -s 4096" to half the default stack size. This is only a
workaround for the Debian packaging though - you can't expect users to
do this.

Reduce the value of ZSTDMT_NBTHREADS_MAX. In my testing, using 253
threads worked so you may not need to reduce it much (but maybe to a
more rounder number). You could limit it to mips if you want, but please
only limit it to 32-bit mips:
 #if defined(__mips__) && !defined(__mips64)

Thanks,
James

[signature.asc (application/pgp-signature, attachment)]

Message #22 received at 876416-close@bugs.debian.org (full text, mbox, reply):

From: Andreas Tille <tille@debian.org>

To: 876416-close@bugs.debian.org

Subject: Bug#876416: fixed in libzstd 1.3.1+dfsg-2

Date: Fri, 22 Sep 2017 15:06:21 +0000

Source: libzstd
Source-Version: 1.3.1+dfsg-2

We believe that the bug you reported is fixed in the latest version of
libzstd, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 876416@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Andreas Tille <tille@debian.org> (supplier of updated libzstd package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Fri, 22 Sep 2017 15:29:35 +0200
Source: libzstd
Binary: libzstd-dev libzstd1 zstd
Architecture: source
Version: 1.3.1+dfsg-2
Distribution: unstable
Urgency: medium
Maintainer: Debian Med Packaging Team <debian-med-packaging@lists.alioth.debian.org>
Changed-By: Andreas Tille <tille@debian.org>
Description:
 libzstd-dev - fast lossless compression algorithm -- development files
 libzstd1   - fast lossless compression algorithm
 zstd       - fast lossless compression algorithm -- CLI tool
Closes: 876416
Changes:
 libzstd (1.3.1+dfsg-2) unstable; urgency=medium
 .
   * Team upload
   * Follow hint given by James Cowgill to recude number of threads
     on 32Bit MIPS architecture
     Closes: #876416
Checksums-Sha1:
 81304e3c8eba63526ac4f266c3815fb3d3b392ed 2117 libzstd_1.3.1+dfsg-2.dsc
 669acc0319607e0c742fc50f6d70fd7915ad59d5 10040 libzstd_1.3.1+dfsg-2.debian.tar.xz
 2d295802ffdbfb0baf6eacf531cf7a76e526b584 13396 libzstd_1.3.1+dfsg-2_source.buildinfo
Checksums-Sha256:
 d184f99aa5d8537f68128fcf6df2d1e4a1e30282d682dcd2d52fefe395f0345c 2117 libzstd_1.3.1+dfsg-2.dsc
 2dbe165469da27dd31897ea5afeb99a334a6644fb614c4742cb8f7cf0b127cb5 10040 libzstd_1.3.1+dfsg-2.debian.tar.xz
 cac1471b9fad60857a4d47c3291aff323d27d6cf3c85898439648d7a638484e0 13396 libzstd_1.3.1+dfsg-2_source.buildinfo
Files:
 81a024885202b856bb6cc2dfa3ccedbb 2117 libs optional libzstd_1.3.1+dfsg-2.dsc
 bc82821fc748c0deac3d2a2b2319d677 10040 libs optional libzstd_1.3.1+dfsg-2.debian.tar.xz
 478f7f7fb558b3165ede741bfea2dcd1 13396 libs optional libzstd_1.3.1+dfsg-2_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQJCBAEBCAAsFiEE8fAHMgoDVUHwpmPKV4oElNHGRtEFAlnFE/UOHHRpbGxlYUBy
a2kuZGUACgkQV4oElNHGRtEccw/7B/CLAJ11fyrhwxPKe/geKY50grVGYd/03uto
qSz7Q+qiC4ZcoKgwxBNJW9J9U6hzb42K/K+q32wr5IVNACDsIK1YiNdfi8yB4VFz
YBLyQbDTE53lvlxXjxTqGMJeT1FdeODgC5OBFecAvsyh7HFtJbKUgekspk4QgYn2
08xT4w+g6o3bQWVHAZxzEoL7VgnUxB66MSPXeMyg1MfsP6raEJscLbq946OtHPEg
aE6g+CdGAfXY0ynC7+plYDnzDb1XQrdCwrmdWP2V4JSmblst7USeffBbprquG19E
HwRzdWXVfzvRwLnON68lXnnVtUCEGaWewjW0QfAz8X5o9wyUofD2TOxs/iBFqsxs
k66kxNZlOpxM9g3CI4C5rGXssOWqpK+SPJyoIi7RlOMPSaWTTfI3JLUesBvmjbSL
3fktR1RYi/fRAaqouVGPuq8/MxMUsVgW91sokUqNy/VN5PqSPcWm0w80YnQSKLRj
NydJQNSqdzhFspIvKwCu1jdiExWkkq/W2rjIJtj3WuxAVEScYht1utki9kmfmGmd
6xG5uxRmRZep0bOPApsKcfukKKzyXVOxSufBQOYnjdtub+mVQsSnO2SjP9NYAVON
d4XQ4xiTnTClqN3SCOju5aHA76R2zXWxG96Z6+ovU7rjj0yshFh3wFPDBrcZoNjS
+KY5N2U=
=zBCE
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.