Debian Bug report logs - #976037
ms-gsl's autopkg tests are broken by design

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Matthias Klose <doko@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: ms-gsl's autopkg tests are broken by design

Date: Sat, 28 Nov 2020 17:21:05 +0100

Source: src:ms-gsl
Version: 3.1.0-3
Severity: serious

$ cat debian/tests/control
Tests: with-gcc8-std14 with-gcc8-std17
Depends: cmake, g++-8, libgtest-dev, pkg-config
Restrictions: allow-stderr, skip-not-installable

Tests: with-gcc9-std14 with-gcc9-std17
Depends: cmake, g++-9, libgtest-dev, pkg-config
Restrictions: allow-stderr, skip-not-installable

Tests: with-gcc10-std17 with-gcc10-std20
Depends: cmake, g++-10, libgtest-dev, pkg-config
Restrictions: allow-stderr, skip-not-installable

Tests: with-clang8-std14 with-clang8-std17
Depends: cmake, clang-8, libc++-8-dev, libc++abi-8-dev, libgtest-dev, pkg-config
Restrictions: allow-stderr, skip-not-installable

Tests: with-clang9-std14 with-clang9-std17
Depends: cmake, clang-9, libc++-9-dev, libc++abi-9-dev, libgtest-dev, pkg-config
Restrictions: allow-stderr, skip-not-installable

Tests: with-clang10-std14 with-clang10-std17
Depends: cmake, clang-10, libc++-10-dev, libc++abi-10-dev, libgtest-dev, pkg-config
Restrictions: allow-stderr, skip-not-installable

Tests: with-clang11-std17 with-clang11-std20
Depends: cmake, clang-11, libc++-11-dev, libc++abi-11-dev, libgtest-dev, pkg-config
Restrictions: allow-stderr, skip-not-installable

Tests: with-clang12-std17 with-clang12-std20
Depends: cmake, clang-12, libc++-12-dev, libc++abi-12-dev, libgtest-dev, pkg-config
Restrictions: allow-stderr, skip-not-installable

Now, there's at least one test (the clang-12 one), which never will succeed. So
what's the value of these tests when you always have failures due to
non-existing versions?  Please just test for the default gcc and clang.

Message #8 received at 976037@bugs.debian.org (full text, mbox, reply):

From: Andrei POPESCU <andreimpopescu@gmail.com>

To: Matthias Klose <doko@debian.org>, 976037@bugs.debian.org

Cc: ms-gsl@packages.debian.org

Subject: Re: Bug#976037: ms-gsl's autopkg tests are broken by design

Date: Sun, 29 Nov 2020 07:32:52 +0200

[Message part 1 (text/plain, inline)]

Control: reassign -1 src:ms-gsl 3.1.0-3

On Sb, 28 nov 20, 17:21:05, Matthias Klose wrote:
> Source: src:ms-gsl
> Version: 3.1.0-3
> Severity: serious
> 
> $ cat debian/tests/control
> Tests: with-gcc8-std14 with-gcc8-std17
> Depends: cmake, g++-8, libgtest-dev, pkg-config
> Restrictions: allow-stderr, skip-not-installable
> 
> Tests: with-gcc9-std14 with-gcc9-std17
> Depends: cmake, g++-9, libgtest-dev, pkg-config
> Restrictions: allow-stderr, skip-not-installable
> 
> Tests: with-gcc10-std17 with-gcc10-std20
> Depends: cmake, g++-10, libgtest-dev, pkg-config
> Restrictions: allow-stderr, skip-not-installable
> 
> Tests: with-clang8-std14 with-clang8-std17
> Depends: cmake, clang-8, libc++-8-dev, libc++abi-8-dev, libgtest-dev, pkg-config
> Restrictions: allow-stderr, skip-not-installable
> 
> Tests: with-clang9-std14 with-clang9-std17
> Depends: cmake, clang-9, libc++-9-dev, libc++abi-9-dev, libgtest-dev, pkg-config
> Restrictions: allow-stderr, skip-not-installable
> 
> Tests: with-clang10-std14 with-clang10-std17
> Depends: cmake, clang-10, libc++-10-dev, libc++abi-10-dev, libgtest-dev, pkg-config
> Restrictions: allow-stderr, skip-not-installable
> 
> Tests: with-clang11-std17 with-clang11-std20
> Depends: cmake, clang-11, libc++-11-dev, libc++abi-11-dev, libgtest-dev, pkg-config
> Restrictions: allow-stderr, skip-not-installable
> 
> Tests: with-clang12-std17 with-clang12-std20
> Depends: cmake, clang-12, libc++-12-dev, libc++abi-12-dev, libgtest-dev, pkg-config
> Restrictions: allow-stderr, skip-not-installable
> 
> Now, there's at least one test (the clang-12 one), which never will succeed. So
> what's the value of these tests when you always have failures due to
> non-existing versions?  Please just test for the default gcc and clang.

Reassigning to correct package.

Kind regards,
Andrei
-- 
Looking after bugs assigned to unknown or inexistent packages

[signature.asc (application/pgp-signature, inline)]

Message #19 received at 976037@bugs.debian.org (full text, mbox, reply):

From: Nicholas Guriev <guriev-ns@ya.ru>

To: 976037@bugs.debian.org

Cc: Matthias Klose <doko@debian.org>

Subject: Re: ms-gsl's autopkg tests are broken by design

Date: Tue, 01 Dec 2020 12:55:03 +0300

Hello!
On Sat, 28 Nov 2020 17:21:05 +0100 Matthias Klose <doko@debian.org>
wrote:
> Now, there's at least one test (the clang-12 one), which never will
succeed. So
> what's the value of these tests when you always have failures due to
> non-existing versions?  Please just test for the default gcc and
clang.

The main idea is to let me discover issues as early as possible, before
new toolchains become default. When a new compiler goes into archive,
this CI test is triggered, although it does not block migrating when the
compiler is not available yet due to "skip-not-installable" restriction.

Testing against default GCC and Clang is already done on buildd trough
the d/rules file and thanks to debhelper(7) which calls to `make test`
during package build. Does it make sense to duplicate the work and do
the same trough autopkgtest(1)?

Message #24 received at 976037@bugs.debian.org (full text, mbox, reply):

From: Matthias Klose <doko@debian.org>

To: Nicholas Guriev <guriev-ns@ya.ru>, 976037@bugs.debian.org

Cc: Matthias Klose <doko@debian.org>

Subject: Re: ms-gsl's autopkg tests are broken by design

Date: Tue, 1 Dec 2020 11:38:37 +0100

On 12/1/20 10:55 AM, Nicholas Guriev wrote:
> Hello!
> On Sat, 28 Nov 2020 17:21:05 +0100 Matthias Klose <doko@debian.org>
> wrote:
>> Now, there's at least one test (the clang-12 one), which never will
> succeed. So
>> what's the value of these tests when you always have failures due to
>> non-existing versions?  Please just test for the default gcc and
> clang.
> 
> The main idea is to let me discover issues as early as possible, before
> new toolchains become default. When a new compiler goes into archive,
> this CI test is triggered, although it does not block migrating when the
> compiler is not available yet due to "skip-not-installable" restriction.
> 
> Testing against default GCC and Clang is already done on buildd trough
> the d/rules file and thanks to debhelper(7) which calls to `make test`
> during package build. Does it make sense to duplicate the work and do
> the same trough autopkgtest(1)?

but the problem is that you don't get any valid test result for testing.

The test will always fail in testing, and the migration to testing is not
blocked because this is not seen as a regression. It doesn't matter if the tests
will fail in unstable or not.

So no, autopkg tests should not depend on packages which are not available in
testing.

Message #27 received at 976037-submitter@bugs.debian.org (full text, mbox, reply):

From: Nicholas Guriev <guriev-ns@ya.ru>

To: 976037-submitter@bugs.debian.org

Subject: Bug#976037 marked as pending in ms-gsl

Date: Sat, 26 Dec 2020 13:43:43 +0000

Control: tag -1 pending

Hello,

Bug #976037 in ms-gsl reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below and you can check the diff of the fix at:

https://salsa.debian.org/debian/ms-gsl/-/commit/66a9aa1dea09f4619e2ced31c8f277fda0eaa9b9

------------------------------------------------------------------------
Run autopkgtests only against default GCC and Clang

 * At Matthias Klose's request, removed auto-tests with fixed versions of GCC
   or Clang. Instead, use default compilers.
 * Rewritten the tests so that they include system-wide installed GSL headers.

Closes: #976037
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/976037

Message #34 received at 976037-quiet@bugs.debian.org (full text, mbox, reply):

From: Nicholas Guriev <nicholas@guriev.su>

To: "Steve M. Robbins" <smr@debian.org>, 976037-quiet@bugs.debian.org

Cc: Matthias Klose <doko@debian.org>

Subject: Re: Bug#976037: marked as pending in ms-gsl

Date: Sat, 26 Dec 2020 17:35:08 +0300

[Message part 1 (text/plain, inline)]

I have rewritten auto-tests, so they do not longer require non-default
versions of compilers. Since the tests with GCC rely on the same version
of the compiler that built Google Test framework, I think preconditions
of Bug#972944 lose its relevance, and there is no need to rebuild the
googletest from sources (in /usr/src/googletest).

Steve Robbins, am I right in my judgments?


On Sat, 2020-12-26 at 13:43 +0000, Nicholas Guriev wrote:
> Control: tag -1 pending
> 
> Hello,
> 
> Bug #976037 in ms-gsl reported by you has been fixed in the
> Git repository and is awaiting an upload. You can see the commit
> message below and you can check the diff of the fix at:
> 
> https://salsa.debian.org/debian/ms-gsl/-/commit/66a9aa1dea09f4619e2ced31c8f277fda0eaa9b9
> 
> ------------------------------------------------------------------------
> Run autopkgtests only against default GCC and Clang
> 
>  * At Matthias Klose's request, removed auto-tests with fixed versions of GCC
>    or Clang. Instead, use default compilers.
>  * Rewritten the tests so that they include system-wide installed GSL headers.
> 
> Closes: #976037
> ------------------------------------------------------------------------
> 
> (this message was generated automatically)

[signature.asc (application/pgp-signature, inline)]

Message #39 received at 976037@bugs.debian.org (full text, mbox, reply):

From: Gianfranco Costamagna <locutusofborg@debian.org>

To: 976037@bugs.debian.org, nicholas@guriev.su

Subject: Re: Bug#976037: marked as pending in ms-gsl

Date: Sat, 26 Dec 2020 15:59:00 +0100

Hello Nicholas,

if you want to test newer compilers, you are free to do it, but maybe you can mark tests as flaky and
exit 77 if they fail, so at least you can see failures by manually looking at logs...

This way you won't block migration to testing, you won't be RC buggy, but you will have the possibility to
have test logs.

what do you think?


G.

On Sat, 26 Dec 2020 17:35:08 +0300 Nicholas Guriev <nicholas@guriev.su> wrote:
> I have rewritten auto-tests, so they do not longer require non-default
> versions of compilers. Since the tests with GCC rely on the same version
> of the compiler that built Google Test framework, I think preconditions
> of Bug#972944 lose its relevance, and there is no need to rebuild the
> googletest from sources (in /usr/src/googletest).
> 
> Steve Robbins, am I right in my judgments?
> 
> 
> On Sat, 2020-12-26 at 13:43 +0000, Nicholas Guriev wrote:
> > Control: tag -1 pending
> > 
> > Hello,
> > 
> > Bug #976037 in ms-gsl reported by you has been fixed in the
> > Git repository and is awaiting an upload. You can see the commit
> > message below and you can check the diff of the fix at:
> > 
> > https://salsa.debian.org/debian/ms-gsl/-/commit/66a9aa1dea09f4619e2ced31c8f277fda0eaa9b9
> > 
> > ------------------------------------------------------------------------
> > Run autopkgtests only against default GCC and Clang
> > 
> >  * At Matthias Klose's request, removed auto-tests with fixed versions of GCC
> >    or Clang. Instead, use default compilers.
> >  * Rewritten the tests so that they include system-wide installed GSL headers.
> > 
> > Closes: #976037
> > ------------------------------------------------------------------------
> > 
> > (this message was generated automatically)
> 

Message #44 received at 976037-quiet@bugs.debian.org (full text, mbox, reply):

From: Steven Robbins <steve@sumost.ca>

To: 976037-quiet@bugs.debian.org, Nicholas Guriev <nicholas@guriev.su>

Subject: Re: Bug#976037: marked as pending in ms-gsl

Date: Sat, 26 Dec 2020 11:02:16 -0600

[Message part 1 (text/plain, inline)]

On Saturday, December 26, 2020 8:35:08 A.M. CST Nicholas Guriev wrote:
> I have rewritten auto-tests, so they do not longer require non-default
> versions of compilers. Since the tests with GCC rely on the same version
> of the compiler that built Google Test framework, I think preconditions
> of Bug#972944 lose its relevance, and there is no need to rebuild the
> googletest from sources (in /usr/src/googletest).
> 
> Steve Robbins, am I right in my judgments?

I believe there remain some corner cases where the build options of the 
software under test may not precisely match those used in building googletest 
-- leading to compile or test failures.  One example is described in bug 
#789267.

That said, many projects do indeed successfully use the compiled library.  So 
you are probably fine.  If you encounter odd failures, you can always revert 
back to building gtest.

Regards,
-Steve

[signature.asc (application/pgp-signature, inline)]

Message #49 received at 976037@bugs.debian.org (full text, mbox, reply):

From: Nicholas Guriev <nicholas@guriev.su>

To: Gianfranco Costamagna <locutusofborg@debian.org>, 976037@bugs.debian.org

Subject: Re: Bug#976037: marked as pending in ms-gsl

Date: Sun, 27 Dec 2020 15:30:45 +0300

[Message part 1 (text/plain, inline)]

On Sat, 2020-12-26 at 15:59 +0100, Gianfranco Costamagna wrote:
> if you want to test newer compilers, you are free to do it, but maybe you can mark tests as flaky and
> exit 77 if they fail, so at least you can see failures by manually looking at logs...
> 
> This way you won't block migration to testing, you won't be RC buggy, but you will have the possibility to
> have test logs.
> 
> what do you think?

In such case, I will not receive notifications about failed tests and so
I can miss new errors. But I concluded that testing with GitLab CI on
salsa.d.o would be better option. A test script for this can be written
without need for release the package and it would not disturb anyone.


On Sat, 2020-12-26 at 11:02 -0600, Steven Robbins wrote:
> I believe there remain some corner cases where the build options of the 
> software under test may not precisely match those used in building googletest 
> -- leading to compile or test failures.  One example is described in bug 
> #789267.
> 
> That said, many projects do indeed successfully use the compiled library.  So 
> you are probably fine.  If you encounter odd failures, you can always revert 
> back to building gtest.

Okay, thank you for comments. If something goes wrong, I will bear in
mind possible differences in compilation flags.

[signature.asc (application/pgp-signature, inline)]

Message #54 received at 976037-close@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 976037-close@bugs.debian.org

Subject: Bug#976037: fixed in ms-gsl 3.1.0-4

Date: Mon, 28 Dec 2020 21:18:27 +0000

Source: ms-gsl
Source-Version: 3.1.0-4
Done: Nicholas Guriev <guriev-ns@ya.ru>

We believe that the bug you reported is fixed in the latest version of
ms-gsl, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 976037@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Nicholas Guriev <guriev-ns@ya.ru> (supplier of updated ms-gsl package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Sun, 27 Dec 2020 17:13:40 +0300
Source: ms-gsl
Architecture: source
Version: 3.1.0-4
Distribution: unstable
Urgency: medium
Maintainer: Nicholas Guriev <guriev-ns@ya.ru>
Changed-By: Nicholas Guriev <guriev-ns@ya.ru>
Closes: 976037
Changes:
 ms-gsl (3.1.0-4) unstable; urgency=medium
 .
   * Run autopkgtests only against default GCC and Clang (closes: #976037).
   * Bump Standards-Versions to 4.5.1, no related changes for this.
Checksums-Sha1:
 30f8e7d7732749b1d14bab539795846b53461079 2015 ms-gsl_3.1.0-4.dsc
 f46059fefa2188e603863109862696796af7d112 6480 ms-gsl_3.1.0-4.debian.tar.xz
 08e411ffe6919f187ab516014c60fe64546b8bd5 5240 ms-gsl_3.1.0-4_source.buildinfo
Checksums-Sha256:
 a4349ab6b8d36642defa1567dcb384a202a31c7c41983736766e28fc04e18d04 2015 ms-gsl_3.1.0-4.dsc
 4eddccb6ae03a907435da926f9e1af9c09ee323caa06e41b126b8ca1d971c30f 6480 ms-gsl_3.1.0-4.debian.tar.xz
 5ad0d9046cf405ac27b84005473ea471154fe48c3b656d7c60978520cfabf8c4 5240 ms-gsl_3.1.0-4_source.buildinfo
Files:
 048bdf52334820951e3052c5b6d3b170 2015 libdevel optional ms-gsl_3.1.0-4.dsc
 ce0ba0c0ebfca42b093a9a4f99db4077 6480 libdevel optional ms-gsl_3.1.0-4.debian.tar.xz
 51f328c544c66cb0ca6eab3ca996682d 5240 libdevel optional ms-gsl_3.1.0-4_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEERoDLeOit93I/iGLK2bXpN3piwCsFAl/qRvsACgkQ2bXpN3pi
wCuubQ//WNwNjoOAxohgp82M6FJP03UyCsZ4JYyfHWq5K6k1D7VjQLoR+Nkfooyt
UyIRZxcXx3+26yWy+zJRfiDZOxXmqsgbAuhQS38QmaGURlqk4dHYobWKUkSAIwIG
2LJH8/C9rIlDRv96q5JgqLUUZm/S40E9EUSVDZejz5NDDcUKXF5YPIewtpEmVbMH
l07pudmDcQb6+ed9a0DikK/7HWhK/CbQ3pUhIbf3mJ9iNLBp578lSpd2KYeMDpTr
VUvWnWAtM7/wWAefaCMj4E0P4vqThjwHt+hdxIiXuE7dQ+z4+fWJvDb49TtBh7J7
H62GdH21oWE+/fsqiV6wUdt8xMY7dAsHxZPXUysJ4DTxPeQAlmCz1GCEra/FINFI
kPXYx1X5GI021OfscYAfMeTYwGBgLE0yLhkB3l0CulAQzenw4Bhiyy9N8zzHPRgt
41DJgkDicm1duE+nbL8KoC6yoS5nGE39B4peJ9LW3pmdtXF857iFfqXxTmOSGoai
9v2Jcyc0A3V32DOXTLOTuzHAtmbReeg8o08/pZa66Meo3SvGFj/dzI+rBse2hwcZ
mZTXg0ffvEHpIGFFNjrKH5Gx6vIRueFyrNYeiHtMRbq3x9VKEcSGI/SoL7qOrkP6
PzYNvGNMtriP0REBA0DhHjDypD6OUzPijGRJPVOA0dPNVPfZWkI=
=0KVC
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.