Debian Bug report logs - #899060
FTBFS: even if tests pass, dh_auto_test fails

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Adam Borowski <kilobyte@angband.pl>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: FTBFS: even if tests pass, dh_auto_test fails

Date: Fri, 18 May 2018 20:40:56 +0200

[Message part 1 (text/plain, inline)]

Source: monkeysphere
Version: 0.41-1
Severity: serious
Justification: fails to build from source (but built successfully in the past)

Hi!
I'm afraid that during an armhf archive rebuild, new _reliable_ build
failure popped up.  For extra fun, there's also a new quite likely random
failure (other than #861457).

On amd64, out of 100 runs, 43 fail during the testsuite, usually with the
"Connection refused" problem.  On armhf the ratio looks similar.

If no random problem happens, the build still fails with:

.----
##################################################
 Monkeysphere basic tests completed successfully!
##################################################
### removing temp dir...
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
dh_auto_test: make -j2 -Oline test returned exit code 2
debian/rules:6: recipe for target 'build' failed
make: *** [build] Error 25
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
`----

To exclude a peculiarity of my test setup (btrfs, new non-Debian kernel,
etc), official buildds of second-class archs that tried monkeysphere
recently concur.

Build logs of two types (failed and passed testsuite) attached.


Meow!
-- System Information:
Debian Release: buster/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: armhf (armv7l)

Kernel: Linux 4.16.5-00026-gc10e6da81c74 (SMP w/4 CPU cores; PREEMPT)
Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE=C.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: sysvinit (via /sbin/init)

[monkeysphere_armhf-2018-05-18T14:22:07Z.build (text/plain, attachment)]

[monkeysphere_armhf.build (inode/symlink, attachment)]

Message #12 received at 899060@bugs.debian.org (full text, mbox, reply):

From: Jameson Graef Rollins <jrollins@finestructure.net>

To: Adam Borowski <kilobyte@angband.pl>, 899060@bugs.debian.org

Subject: Re: Bug#899060: FTBFS: even if tests pass, dh_auto_test fails

Date: Sun, 17 Jun 2018 23:47:17 -0700

[Message part 1 (text/plain, inline)]

I am unclear how to test this right now as I don't have direct access to
an armhf system.  It builds fine on the systems I have access to.

Given that it seems to build fine on other systems, I'm tempted to solve
this problem in the short term by just excluding armhf.

If anyone thinks that's a bad idea please let me know.  Also suggestions
for how to reproduce are welcome.

jamie.

[signature.asc (application/pgp-signature, inline)]

Message #17 received at 899060@bugs.debian.org (full text, mbox, reply):

From: Adam Borowski <kilobyte@angband.pl>

To: 899060@bugs.debian.org

Subject: Re: Bug#899060: FTBFS: even if tests pass, dh_auto_test fails

Date: Mon, 18 Jun 2018 19:56:10 +0200

On Sun, Jun 17, 2018 at 11:47:17PM -0700, Jameson Graef Rollins wrote:
> I am unclear how to test this right now as I don't have direct access to
> an armhf system.  It builds fine on the systems I have access to.
> 
> Given that it seems to build fine on other systems, I'm tempted to solve
> this problem in the short term by just excluding armhf.

Huh?  The failure is in no way arch-specific.  I just reproduced it again
on multiple machines:
* amd64 AMD Phenom II 6x:
  amd64 i386 x32 arm64/qemu armhf/qemu
* amd64 Intel Braswell 4x:
  amd64 i386 x32
* arm64 Pine64 (Allwinner A64) 4x:
  arm64
* i386 Pentium 4 (non-nx) 1x2(HT):
  i386

With not a single of these succeeding.

They use sbuild on different filesystems, and vanilla vs Debian kernels,
with obvious common parts being the same network and up-to-date kernel
(oldest is 4.16.something).

But, official buildds use older kernels, and they still fail (although archs
that have succeeded in the past are not rebuilt unless binNMUed).

So does reproducible:
https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/monkeysphere.html
(fails on all 4 tested: amd64 i386 armhf arm64)


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ There's an easy way to tell toy operating systems from real ones.
⣾⠁⢰⠒⠀⣿⡁ Just look at how their shipped fonts display U+1F52B, this makes
⢿⡄⠘⠷⠚⠋⠀ the intended audience obvious.  It's also interesting to see OSes
⠈⠳⣄⠀⠀⠀⠀ go back and forth wrt their intended target.

Message #22 received at 899060@bugs.debian.org (full text, mbox, reply):

From: Clint Adams <clint@debian.org>

To: Adam Borowski <kilobyte@angband.pl>, 899060@bugs.debian.org

Subject: Re: Bug#899060: FTBFS: even if tests pass, dh_auto_test fails

Date: Mon, 25 Jun 2018 00:55:29 +0000

On Fri, May 18, 2018 at 08:40:56PM +0200, Adam Borowski wrote:
> ##################################################
>  Monkeysphere basic tests completed successfully!
> ##################################################
> ### removing temp dir...
> make[1]: Leaving directory '/<<PKGBUILDDIR>>'
> dh_auto_test: make -j2 -Oline test returned exit code 2

Does it happen without -j2?

Message #27 received at 899060@bugs.debian.org (full text, mbox, reply):

From: Adam Borowski <kilobyte@angband.pl>

To: Clint Adams <clint@debian.org>

Cc: 899060@bugs.debian.org

Subject: Re: Bug#899060: FTBFS: even if tests pass, dh_auto_test fails

Date: Mon, 25 Jun 2018 03:11:25 +0200

On Mon, Jun 25, 2018 at 12:55:29AM +0000, Clint Adams wrote:
> On Fri, May 18, 2018 at 08:40:56PM +0200, Adam Borowski wrote:
> > ##################################################
> >  Monkeysphere basic tests completed successfully!
> > ##################################################
> > ### removing temp dir...
> > make[1]: Leaving directory '/<<PKGBUILDDIR>>'
> > dh_auto_test: make -j2 -Oline test returned exit code 2
> 
> Does it happen without -j2?

You can't just remove that, as in absence of any parallelism settings,
modern dh level take the number of available CPUs.  So you need explicit
DEB_BUILD_OPTIONS=parallel=1 to get -j1.

With such a single-threaded build, the test suite fails much sooner; in
every case it dies on the first test it does.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ There's an easy way to tell toy operating systems from real ones.
⣾⠁⢰⠒⠀⣿⡁ Just look at how their shipped fonts display U+1F52B, this makes
⢿⡄⠘⠷⠚⠋⠀ the intended audience obvious.  It's also interesting to see OSes
⠈⠳⣄⠀⠀⠀⠀ go back and forth wrt their intended target.

Message #32 received at submit@bugs.debian.org (full text, mbox, reply):

From: Antoine Beaupre <anarcat@debian.org>

To: Adam Borowski <kilobyte@angband.pl>, 899060@bugs.debian.org

Cc: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Re: Bug#899060: FTBFS: even if tests pass, dh_auto_test fails

Date: Wed, 26 Sep 2018 19:52:39 -0400

[Message part 1 (text/plain, inline)]

On Fri, May 18, 2018 at 08:40:56PM +0200, Adam Borowski wrote:
> If no random problem happens, the build still fails with:
> 
> .----
> ##################################################
>  Monkeysphere basic tests completed successfully!
> ##################################################
> ### removing temp dir...
> make[1]: Leaving directory '/<<PKGBUILDDIR>>'
> dh_auto_test: make -j2 -Oline test returned exit code 2
> debian/rules:6: recipe for target 'build' failed
> make: *** [build] Error 25
> dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
> `----
> 
> To exclude a peculiarity of my test setup (btrfs, new non-Debian kernel,
> etc), official buildds of second-class archs that tried monkeysphere
> recently concur.
> 
> Build logs of two types (failed and passed testsuite) attached.

For thre record, the above failed testsuite log fails on:

2018/05/18 14:25:50 socat[27609] E connect(5, AF=1 "/<<PKGBUILDDIR>>/tests/tmp/ms.cco/ssh-socket", 74): Connection refused
ssh_exchange_identification: Connection closed by remote host

./tests/basic: line 81: kill: (27487) - No such process
##### ssh connection test FAILED. returned: 255, expected: 0
FAILED!
### removing temp dir...

That is strange in itself, but the "passed" one is much stranger, as
mentioned.

On Mon, Jun 18, 2018 at 07:56:10PM +0200, Adam Borowski wrote:
[...] 
> So does reproducible:
> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/monkeysphere.html
> (fails on all 4 tested: amd64 i386 armhf arm64)

That, however, seems to be a different problem, at least now. The amd64
failed with:

ms: importing key from file '/build/1st/monkeysphere-0.41/tests/tmp/ms.9XY/ssh_host_rsa_key'...
RSA.xs:198: OpenSSL error: no start line at /build/1st/monkeysphere-0.41/tests/../src/pem2openpgp line 1104, <STDIN> line 1.
gpg: no valid OpenPGP data found.
FAILED!

https://tests.reproducible-builds.org/debian/rbuild/unstable/amd64/monkeysphere_0.41-1.rbuild.log.gz

I wonder if that shouldn't be treated separately: I suspect this is
related to the recent changes in OpenSSH's key storage format.

So I'd propose we keep this bug for "kilobyte's weird heisenbug FTBFS"
and open a separate issue for the RSA key parser.. :/

A.

[signature.asc (application/pgp-signature, inline)]

Message #42 received at 899060@bugs.debian.org (full text, mbox, reply):

From: Antoine Beaupré <anarcat@debian.org>

To: Adam Borowski <kilobyte@angband.pl>, 899060@bugs.debian.org

Subject: Re: Bug#899060: FTBFS: even if tests pass, dh_auto_test fails

Date: Wed, 26 Sep 2018 22:20:38 -0400

[Message part 1 (text/plain, inline)]

On 2018-09-26 19:52:39, Antoine Beaupre wrote:

[...]

> For thre record, the above failed testsuite log fails on:
>
> 2018/05/18 14:25:50 socat[27609] E connect(5, AF=1 "/<<PKGBUILDDIR>>/tests/tmp/ms.cco/ssh-socket", 74): Connection refused
> ssh_exchange_identification: Connection closed by remote host
>
> ./tests/basic: line 81: kill: (27487) - No such process
> ##### ssh connection test FAILED. returned: 255, expected: 0
> FAILED!
> ### removing temp dir...
>
> That is strange in itself, but the "passed" one is much stranger, as
> mentioned.

I can reproduce the latter (the "failed" test) reliably. It fails at
different places, however. Here are two logfiles, for example, which
both fail, but in different tests, with the same error message:

[log (text/plain, attachment)]

[log2 (text/plain, attachment)]

[Message part 4 (text/plain, inline)]

Now unfortunately, I have zero idea where to go from here. Note that
there is at least one other report of Monkeysphere's test suite failing
randomly, although this report seems to differ from #861457.

[...]

> I wonder if that shouldn't be treated separately: I suspect this is
> related to the recent changes in OpenSSH's key storage format.
>
> So I'd propose we keep this bug for "kilobyte's weird heisenbug FTBFS"
> and open a separate issue for the RSA key parser.. :/

I reported that bug and fixed it, in #909700. I also pushed commits to
fix #902318 and #902367 but we're unfortunately still stuck here. :/

A.

-- 
To understand how any society functions you must understand the
relationship between the men and the women
                        - Angela Davis

Message #51 received at 899060@bugs.debian.org (full text, mbox, reply):

From: Sunil Mohan Adapa <sunil@medhas.org>

To: 899060@bugs.debian.org, control@bugs.debian.org

Subject: Re: Bug#899060: FTBFS: even if tests pass, dh_auto_test fails

Date: Fri, 12 Oct 2018 13:46:40 -0700

[Message part 1 (text/plain, inline)]

tags 899060 +patch
thanks

Hello,

The attached patch seems to fix the problem.

It appears that if a socket that socat wants to listen on exists, socat
fails (socat(1)). When socat finishes by itself, ssh-socket will be
unlinked but if it is killed for some reason than socket file is not
unlinked. The attached patch tells socat to remove the socket before
starting its normal activity.

I got the hint towards this when I stopped cleaning the temp home
directory and read the sshd.log after a failure. I was to reproduce the
problem with high success rate before the patch and after the applying
the patch, I can't reproduce it building 10 times. All of the logs on
this bug report (that are not related to OpenSSL pem2openpgp, fixed in
#909700) are similar and should get fixed with this patch. I didn't
investigate why this behavior was not seen earlier (socat changes?
something else?).

I believe this is the last of the unfixed serious issues. I think this
package is set to cause removal issues for other packages in testing.
Please consider uploading this fix, along with other unreleased fixes, soon.

Thank you,

-- 
Sunil

[0001-tests-Ensure-that-stale-sockets-don-t-fail-socat-Clo.patch (text/x-patch, attachment)]

[signature.asc (application/pgp-signature, attachment)]

Message #56 received at 899060@bugs.debian.org (full text, mbox, reply):

From: Sunil Mohan Adapa <sunil@medhas.org>

To: 899060@bugs.debian.org

Subject: Re: Bug#899060: FTBFS: even if tests pass, dh_auto_test fails

Date: Mon, 15 Oct 2018 10:02:51 -0700

[Message part 1 (text/plain, inline)]

Hello,

FreedomBox is set to be autoremoved from testing on 2018/10/21 (in about
5 days) due to mod-gnutls which won't transition into testing which in
turn due won't migrate to testing due to its dependency on moneysphere.
This effects all FreedomBox users.

Please consider uploading the monkeysphere package immediately.

Thanks,

-- 
Sunil

[signature.asc (application/pgp-signature, attachment)]

Message #61 received at 899060-close@bugs.debian.org (full text, mbox, reply):

From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>

To: 899060-close@bugs.debian.org

Subject: Bug#899060: fixed in monkeysphere 0.42-1

Date: Tue, 16 Oct 2018 18:06:22 +0000

Source: monkeysphere
Source-Version: 0.42-1

We believe that the bug you reported is fixed in the latest version of
monkeysphere, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 899060@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Daniel Kahn Gillmor <dkg@fifthhorseman.net> (supplier of updated monkeysphere package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Tue, 16 Oct 2018 12:32:30 -0400
Source: monkeysphere
Binary: monkeysphere agent-transfer
Architecture: source
Version: 0.42-1
Distribution: unstable
Urgency: medium
Maintainer: Debian Privacy Tools Maintainers <pkg-privacy-maintainers@lists.alioth.debian.org>
Changed-By: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Description:
 agent-transfer - copy a secret key from GnuPG's gpg-agent to OpenSSH's ssh-agent
 monkeysphere - leverage the OpenPGP web of trust for SSH and TLS authentication
Closes: 883015 899060 902318 902320 902367 908228 909700
Changes:
 monkeysphere (0.42-1) unstable; urgency=medium
 .
   * new upstream bugfix release.
 .
   [ Daniel Kahn Gillmor ]
   * make print_date_from_seconds_since_the_epoch deal better with bad input
 .
   [ Helmut Grohne ]
   * use generic compiler (closes: #883015)
 .
   [ Daniel Kahn Gillmor ]
   * clean up test suite failures when built against newer GnuPG
 .
   [ Clint Adams ]
   * Remove RSAAuthentication from test ssh config (Closes: #902318)
 .
   [ Antoine Beaupré ]
   * fix more gnupg2 colons changes (Closes: #902367)
   * yet more colon fixes that escaped previous inspections
   * write old-style PEM files to unbreak test suite (Closes: #909700)
 .
   [ Clint Adams ]
   * Remove deprecated option from test sshd config (Closes: #902320)
 .
   [ Sunil Mohan Adapa ]
   * tests: Ensure that stale sockets don't fail socat (Closes: #899060)
 .
   [ Daniel Kahn Gillmor ]
   * use --send-keys instead of --send (closes: #908228)
   * prepare new release
   * enable tests to operate on system installation
   * d/copyright: use https for Format:
   * use priority optional instead of extra
   * added autopkgtest
   * override package-contains-documentation-outside-usr-share-doc for
     transitions README.txt
   * install examples in monkeysphere package
   * d/watch: use https
   * packaging handled by pkg-privacy-team (updated uploaders)
Checksums-Sha1:
 03190e3a46d0f4305e0c70b8e29fdc7c55f12483 1879 monkeysphere_0.42-1.dsc
 546a30e9d003422ee296961417e10ebe9875d3cb 110415 monkeysphere_0.42.orig.tar.gz
 f320e4b5f9d0dece78ed5482ab8d27597bb20c34 6784 monkeysphere_0.42-1.debian.tar.xz
 35224fa0bdad8347c5233d325a7fe7fff541b685 12714 monkeysphere_0.42-1_amd64.buildinfo
Checksums-Sha256:
 d43f26ef40cbbc39b6ce64f113eea8d6e195c53d993d328da0af865172e0c9a3 1879 monkeysphere_0.42-1.dsc
 c1c956b1c86aaa44134fc1a9d75f5aef61266e3a9d8a6218b45d6c54bb7c58c1 110415 monkeysphere_0.42.orig.tar.gz
 dcc0abfeedca392aa32b65994a97f87e6c0d05b78acdca643fb416ea1d2e5446 6784 monkeysphere_0.42-1.debian.tar.xz
 469f78ede8160e3eb4a33d7113ec3b09d1aaedeabcdfd62ac7eadfd97b7d25a6 12714 monkeysphere_0.42-1_amd64.buildinfo
Files:
 6ac2d8ae0a47520504fa6c79b49b87bc 1879 net optional monkeysphere_0.42-1.dsc
 56b5f9f66481eec8716a1726366239b4 110415 net optional monkeysphere_0.42.orig.tar.gz
 02180d3667e081a2bbb71f96e00060b9 6784 net optional monkeysphere_0.42-1.debian.tar.xz
 6d315b1e8b6a8deeed38b57c6f5b6282 12714 net optional monkeysphere_0.42-1_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iHUEARYKAB0WIQTTaP514aqS9uSbmdJsHx7ezFD6UwUCW8YmMgAKCRBsHx7ezFD6
UxIeAP0SC+hD17dMdlVzB8wI8fyKEtbcZoheV034ES5HAwBc7wD8Dm0EreKCx7pG
WwC6BeaHKz7uUeOdIzHHed3k5eqUmQA=
=jgbh
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.