Debian Bug report logs - #1024971
pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Louis-Philippe Véronneau <pollo@debian.org>

To: submit@bugs.debian.org

Subject: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Date: Sun, 27 Nov 2022 18:46:58 -0500

[Message part 1 (text/plain, inline)]

Package: dh-python
Version: 5.20221122
Severity: wishlist

Dear maintainers,

Too often, a mistake or a misconfiguration leads to no tests being 
detected when trying to run the upstream testsuite.

When this happens, the result of the test command typically looks like 
"Ran 0 tests in 0.000s".

I thought we could catch this via Lintian and warn people, but I just 
realised Lintian does not have access to the build log.

This means if we want people to be aware of what, in my opinion, is a 
build failure, it should be done via pybuild.

As such, it would be nice if pybuild considered this case as a failure 
and exited if it happens. We probably will need to do a MBF beforehand 
though, as I'm sure it happens in tons of packages.

If there are no tests for real, I think it's OK to ask people to disable 
them altogether :)

Cheers,

-- 
  ⢀⣴⠾⠻⢶⣦⠀
  ⣾⠁⢠⠒⠀⣿⡁  Louis-Philippe Véronneau
  ⢿⡄⠘⠷⠚⠋   pollo@debian.org / veronneau.org
  ⠈⠳⣄

[OpenPGP_0xE1E5457C8BAD4113.asc (application/pgp-keys, attachment)]

[OpenPGP_signature (application/pgp-signature, attachment)]

Message #8 received at 1024971-submitter@bugs.debian.org (full text, mbox, reply):

From: Scott Kitterman <debian@kitterman.com>

To: 1024971-submitter@bugs.debian.org

Subject: Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Date: Mon, 28 Nov 2022 00:37:21 -0500

[Message part 1 (text/plain, inline)]

On Sunday, November 27, 2022 6:46:58 PM EST Louis-Philippe Véronneau wrote:
> Package: dh-python
> Version: 5.20221122
> Severity: wishlist
> 
> Dear maintainers,
> 
> Too often, a mistake or a misconfiguration leads to no tests being 
> detected when trying to run the upstream testsuite.
> 
> When this happens, the result of the test command typically looks like 
> "Ran 0 tests in 0.000s".
> 
> I thought we could catch this via Lintian and warn people, but I just 
> realised Lintian does not have access to the build log.
> 
> This means if we want people to be aware of what, in my opinion, is a 
> build failure, it should be done via pybuild.
> 
> As such, it would be nice if pybuild considered this case as a failure 
> and exited if it happens. We probably will need to do a MBF beforehand 
> though, as I'm sure it happens in tons of packages.
> 
> If there are no tests for real, I think it's OK to ask people to disable 
> them altogether :)

I think the first step would to do a rebuild and see how many packages this 
breaks.  It's probably too late for bookworm as I expect it'll be a bunch.

Scott K

[signature.asc (application/pgp-signature, inline)]

Message #13 received at 1024971@bugs.debian.org (full text, mbox, reply):

From: Stefano Rivera <stefanor@debian.org>

To: Louis-Philippe Véronneau <pollo@debian.org>, 1024971@bugs.debian.org

Subject: Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Date: Mon, 28 Nov 2022 18:44:36 +0000

Hi Louis-Philippe (2022.11.27_23:46:58_+0000)
> When this happens, the result of the test command typically looks like "Ran
> 0 tests in 0.000s".

I don't think unittest provides an interface to achieve this.

We're probably stuck parsing logs if we want it.

Pytest makes it easy: https://github.com/pytest-dev/pytest/pull/817

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

Message #20 received at 1024971@bugs.debian.org (full text, mbox, reply):

From: Stefano Rivera <stefanor@debian.org>

To: 1024971@bugs.debian.org

Cc: Louis-Philippe Véronneau <pollo@debian.org>

Subject: Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Date: Thu, 27 Apr 2023 07:53:59 -0400

Hi Louis-Philippe (2022.11.28_14:44:36_-0400)
> I don't think unittest provides an interface to achieve this.

Now implemented! https://github.com/python/cpython/pull/102051

This will cause all the packages that don't have tests to fail their
empty test suite. We'll have to decide what to do about that...

Stefano

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

Message #23 received at 1024971-submitter@bugs.debian.org (full text, mbox, reply):

From: Stefano Rivera <stefanor@debian.org>

To: 1024971-submitter@bugs.debian.org

Subject: Bug#1024971 marked as pending in dh-python

Date: Thu, 05 Sep 2024 16:54:50 +0000

Control: tag -1 pending

Hello,

Bug #1024971 in dh-python reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below and you can check the diff of the fix at:

https://salsa.debian.org/python-team/tools/dh-python/-/commit/048976a62124123ef1edc66c075b589605a13c57

------------------------------------------------------------------------
pybuild: Stop ignoring an exit code of 5, for no tests found. (Closes: #1024971)
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/1024971

Message #30 received at 1024971-close@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 1024971-close@bugs.debian.org

Subject: Bug#1024971: fixed in dh-python 6.20240905+exp1

Date: Thu, 05 Sep 2024 17:04:03 +0000

[Message part 1 (text/plain, inline)]

Source: dh-python
Source-Version: 6.20240905+exp1
Done: Stefano Rivera <stefanor@debian.org>

We believe that the bug you reported is fixed in the latest version of
dh-python, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1024971@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Stefano Rivera <stefanor@debian.org> (supplier of updated dh-python package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Thu, 05 Sep 2024 18:51:19 +0200
Source: dh-python
Architecture: source
Version: 6.20240905+exp1
Distribution: experimental
Urgency: medium
Maintainer: Piotr Ożarowski <piotr@debian.org>
Changed-By: Stefano Rivera <stefanor@debian.org>
Closes: 1024971
Changes:
 dh-python (6.20240905+exp1) experimental; urgency=medium
 .
   * dh_python3: Suppress generated dependencies that would be satisfied by
     python3 >= 3.9.
   * Run pybuild with --verbose in autopkgtests.
   * Upload to experimental with deprecated features disabled:
     - Drop the setuptools dependency.
     - pybuild: Stop ignoring an exit code of 5, for no tests found.
       (Closes: #1024971)
Checksums-Sha1:
 e767c58a7d93eb3aba2001dcb218ca22261df86f 1708 dh-python_6.20240905+exp1.dsc
 e3f1b371e7b757f2122b854075b163c172ee3ab3 125120 dh-python_6.20240905+exp1.tar.xz
 8a59eaf4c7863045428f87a3d547f9de2e3ba11e 8597 dh-python_6.20240905+exp1_source.buildinfo
Checksums-Sha256:
 1b5950dcbb1da2282dca80e01d949e2815da8b88f651147708cd8115b6943bfd 1708 dh-python_6.20240905+exp1.dsc
 2f25db6266a49ded7179695a8e2e7d924291c05698125ce6362f0c51a9ef4538 125120 dh-python_6.20240905+exp1.tar.xz
 e40386430c54e62e8e7bd1645da11e0686576ee0427b4c28033a09cf15057aa1 8597 dh-python_6.20240905+exp1_source.buildinfo
Files:
 501cfe57433d27c3bc68f8a9110e34cd 1708 python optional dh-python_6.20240905+exp1.dsc
 96b79287e4655b4e65ab4107d9cdeb8c 125120 python optional dh-python_6.20240905+exp1.tar.xz
 ada6467e5da5a22171b6f02921814464 8597 python optional dh-python_6.20240905+exp1_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iIoEARYKADIWIQTumtb5BSD6EfafSCRHew2wJjpU2AUCZtniSRQcc3RlZmFub3JA
ZGViaWFuLm9yZwAKCRBHew2wJjpU2N3PAP0fzxWYLDpcFaOTjrOAlsl3dtWaxcOs
quemHxtFNX/NowEA4QWIhdUjdFTCJNdSo9+Re9g8rpeI7SNfWCUqXcI1qw8=
=RhVe
-----END PGP SIGNATURE-----

[Message part 2 (application/pgp-signature, inline)]

Message #35 received at 1024971@bugs.debian.org (full text, mbox, reply):

From: Stefano Rivera <stefanor@debian.org>

To: debian-python@lists.debian.org

Cc: Louis-Philippe Véronneau <pollo@debian.org>, 1024971@bugs.debian.org

Subject: Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Date: Sun, 8 Sep 2024 15:43:33 +0200

[Message part 1 (text/plain, inline)]

Hi Louis-Philippe (2022.11.28_01:46:58_+0200)
> Too often, a mistake or a misconfiguration leads to no tests being detected
> when trying to run the upstream testsuite.
> 
> When this happens, the result of the test command typically looks like "Ran
> 0 tests in 0.000s".

A couple of years later:

We've implemented the feature in Python 3.12, unittest's runner now
exits with return value 5 if no tests were discovered, like pytest does.
https://github.com/python/cpython/issues/113661

We initially ignored this exit status in dh-python, to allow us to
transition to Python 3.12, without too much pain.

I've just done a rebuild test without ignoring it to see how much effort
it would take to be able to use this feature:

I built 6440 packages build-depending on dh-python in one way or
another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.

To fix these build failures, package maintainers would have these
options:

1. Get the build to run some unit tests (assuming they exist),
2. override_dh_auto_test with something noop,
3. export PYBUILD_DISABLE=test,
4. We could make this failure opt-in in dh-python. Maybe via an explicit
   --test-unittest option that selects the unittest runner. If you don't
   explicitly select this runner, you'd get an attempt to run tests by
   with unittest, and no failure if no tests are found.

The downside of options 2 and 3 is that if the package does start
gaining an upstream test suite, nothing ever will attempt to run it. I
think that's OK?

If you want to experiment yourself, there's a version of dh-python in
experimental that will treat no tests as an error.

Should we file bugs for these packages? Or implement option 4?

Attached is a current dd-list.

Stefano

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[dh-python-no-unittests.packages.txt.xz (application/x-xz, attachment)]

[dh-python-no-unittests.ddlist.txt.xz (application/x-xz, attachment)]

Message #40 received at 1024971@bugs.debian.org (full text, mbox, reply):

From: Julian Gilbey <julian@d-and-j.net>

To: debian-python@lists.debian.org, Louis-Philippe Véronneau <pollo@debian.org>, 1024971@bugs.debian.org

Subject: Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Date: Sun, 8 Sep 2024 22:33:49 +0100

On Sun, Sep 08, 2024 at 03:43:33PM +0200, Stefano Rivera wrote:
> [...]
> We've implemented the feature in Python 3.12, unittest's runner now
> exits with return value 5 if no tests were discovered, like pytest does.
> https://github.com/python/cpython/issues/113661
> [...]
> 
> I built 6440 packages build-depending on dh-python in one way or
> another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.

My guess is that most of these 1124 have no tests at all, rather than
having a misconfigured setup.  A unittest is the pybuild default test
framework, unittest is used and fails to find any tests, hence all of
these failures.

> To fix these build failures, package maintainers would have these
> options:
> 
> 1. Get the build to run some unit tests (assuming they exist),
> 2. override_dh_auto_test with something noop,
> 3. export PYBUILD_DISABLE=test,
> 4. We could make this failure opt-in in dh-python. Maybe via an explicit
>    --test-unittest option that selects the unittest runner. If you don't
>    explicitly select this runner, you'd get an attempt to run tests by
>    with unittest, and no failure if no tests are found.

I like option 4 for the above reason.  But implementing this would
mean that all of the packages that currently *do* use unittest
(intentionally, but without having to code it explicitly as it's the
default) would suddenly not have any tests running until they
proactively add --test-unittest or set PYBUILD_TEST_UNITTEST = 1 or
similar.  This seems like an unfortunate consequence.

Is there a way of looking at the logs of the packages that passed the
build to identify which ones successfully passed tests using unittest?
There is also the issue of packages that use unittest (as the default)
via autopkgtest-pkg-pybuild but override dh_auto_test during the
build, though that will be much rarer.

Best wishes,

   Julian

Message #45 received at 1024971@bugs.debian.org (full text, mbox, reply):

From: Stefano Rivera <stefanor@debian.org>

To: debian-python@lists.debian.org, Louis-Philippe Véronneau <pollo@debian.org>, 1024971@bugs.debian.org

Subject: Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Date: Mon, 9 Sep 2024 08:31:52 +0000

Re-sent, with the bug in CC

Hi Julian (2024.09.08_21:33:49_+0000)
> > I built 6440 packages build-depending on dh-python in one way or
> > another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.
> 
> My guess is that most of these 1124 have no tests at all, rather than
> having a misconfigured setup.  A unittest is the pybuild default test
> framework, unittest is used and fails to find any tests, hence all of
> these failures.

Yes, almost certainly.

> > 4. We could make this failure opt-in in dh-python. Maybe via an explicit
> >    --test-unittest option that selects the unittest runner. If you don't
> >    explicitly select this runner, you'd get an attempt to run tests by
> >    with unittest, and no failure if no tests are found.
> 
> I like option 4 for the above reason.  But implementing this would
> mean that all of the packages that currently *do* use unittest
> (intentionally, but without having to code it explicitly as it's the
> default) would suddenly not have any tests running until they
> proactively add --test-unittest or set PYBUILD_TEST_UNITTEST = 1 or
> similar.  This seems like an unfortunate consequence.

I would leave unittest as the default runner, but without missing test
detection.

That's a slightly unexpected behaviour, but it makes the default case
work.

Downside is that you have to opt-in to missing test detection. Maybe we
can have a lintian tag for that?

Stefano

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

Message #50 received at 1024971@bugs.debian.org (full text, mbox, reply):

From: Julian Gilbey <julian@d-and-j.net>

To: debian-python@lists.debian.org, Louis-Philippe Véronneau <pollo@debian.org>, 1024971@bugs.debian.org

Subject: Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Date: Mon, 9 Sep 2024 16:19:51 +0100

On Mon, Sep 09, 2024 at 08:31:52AM +0000, Stefano Rivera wrote:
> > Hi Julian (2024.09.08_21:33:49_+0000)
> > > I built 6440 packages build-depending on dh-python in one way or
> > > another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.
> > 
> > My guess is that most of these 1124 have no tests at all, rather than
> > having a misconfigured setup.  A unittest is the pybuild default test
> > framework, unittest is used and fails to find any tests, hence all of
> > these failures.
> 
> Yes, almost certainly.
> [...]
> 
> I would leave unittest as the default runner, but without missing test
> detection.
> 
> That's a slightly unexpected behaviour, but it makes the default case
> work.
> 
> Downside is that you have to opt-in to missing test detection. Maybe we
> can have a lintian tag for that?

That seems a bit heavy to ask for.

Is there any way of identifying those packages that do genuinely use
unittest?  If there are not that many of them, then implementing a
--test-unittest option would be a good way to go.  I would imagine the
following timeline:

(1) --test-unittest is introduced as an option to explicitly select
unittest as the test framework.  When --test-unittest is specified,
the test will fail if no tests are found.  unittest is still used as a
fallback test framework; in this case, the dh_auto_test call will
succeed if no tests are run.

(2) Add some sort of warning for pybuild-using packages that run
dh_auto_test but haven't specified a test framework and for which
autodetection of the test framework fails.  If there aren't any tests
to run, an empty override_dh_auto_test target should be specified.

(3) Stop using unittest as the default test framework, and fail if no
test framework has been specified or autodetected.

But that might be overkill for something which may not actually be
much of a problem.

Best wishes,

   Julian

Message #55 received at 1024971@bugs.debian.org (full text, mbox, reply):

From: Stefano Rivera <stefanor@debian.org>

To: Julian Gilbey <julian@d-and-j.net>

Cc: debian-python@lists.debian.org, Louis-Philippe Véronneau <pollo@debian.org>, 1024971@bugs.debian.org

Subject: Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Date: Tue, 10 Sep 2024 15:05:47 +0000

Hi Julian (2024.09.09_15:19:51_+0000)
> That seems a bit heavy to ask for.
> 
> Is there any way of identifying those packages that do genuinely use
> unittest?

From 6438 build logs:
- 651 don't call dh_auto_test
- 2180 do something custom
- 1989 use pytest
- 25 use nose
- 18 use nose2
- 23 use tox
- 3 use stestr
- 1561 packages use pybuild's unittest runner
  * 391 pass
  * 1170 fail
    + 1139 NO TESTS RAN
    + 33 the test suite failed

(numbers don't quite add up, because this was a lot of grep | wc -l)

> If there are not that many of them, then implementing a
> --test-unittest option would be a good way to go.  I would imagine the
> following timeline:
> 
> (1) --test-unittest is introduced as an option to explicitly select
> unittest as the test framework.  When --test-unittest is specified,
> the test will fail if no tests are found.  unittest is still used as a
> fallback test framework; in this case, the dh_auto_test call will
> succeed if no tests are run.
> 
> (2) Add some sort of warning for pybuild-using packages that run
> dh_auto_test but haven't specified a test framework and for which
> autodetection of the test framework fails.  If there aren't any tests
> to run, an empty override_dh_auto_test target should be specified.
> 
> (3) Stop using unittest as the default test framework, and fail if no
> test framework has been specified or autodetected.
> 
> But that might be overkill for something which may not actually be
> much of a problem.

Yeah, that can work. We can also just abort after step 2.

Stefano

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

Message #60 received at 1024971@bugs.debian.org (full text, mbox, reply):

From: Timo Röhling <roehling@debian.org>

To: debian-python@lists.debian.org, 1024971@bugs.debian.org

Subject: Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Date: Fri, 13 Sep 2024 10:56:25 +0200

[Message part 1 (text/plain, inline)]

Hi Stefano,

* Stefano Rivera <stefanor@debian.org> [2024-09-08 15:43]:
>To fix these build failures, package maintainers would have these
>options:
>
>1. Get the build to run some unit tests (assuming they exist),
>2. override_dh_auto_test with something noop,
>3. export PYBUILD_DISABLE=test,
>4. We could make this failure opt-in in dh-python. Maybe via an explicit
>   --test-unittest option that selects the unittest runner. If you don't
>   explicitly select this runner, you'd get an attempt to run tests by
>   with unittest, and no failure if no tests are found.
Maybe we can start by making missing tests in autopkgtest fail? 
As you need to opt-in via "Testsuite: autopkgtest-pkg-pybuild", we 
can be quite certain that missing tests are not intentional.

I think that would work very nicely together with option 4 for the 
build itself. You might even use the "Testsuite" setting as opt-in 
flag, but that is probably taking it too far.


Cheers
Timo

-- 
⢀⣴⠾⠻⢶⣦⠀   ╭────────────────────────────────────────────────────╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling                                       │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄⠀⠀⠀⠀   ╰────────────────────────────────────────────────────╯

[signature.asc (application/pgp-signature, inline)]

Send a report that this bug log contains spam.