Debian Bug report logs - #851558
autopkgtest: define Restrictions for tests that aren't suitable for gating CI

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: autopkgtest: define Restrictions for tests that aren't suitable for gating CI

Date: Mon, 16 Jan 2017 09:06:19 +0000

Package: autopkgtest
Version: 4.3
Severity: wishlist

As mentioned in recent discussion on debian-devel, I don't think all
autopkgtests are sufficiently reliable, stable or important to be used to
gate CI or block testing migration. (In Debian terms: every test failure
is a bug, but not every bug is release-critical.) However, I don't want
to have to disable unimportant or known-unreliable tests altogether, and
lose the ability to see how their results change over time: I would prefer
to keep running them for the logs, and just ignore the side-effects.

Specifically, if a maintainer is unable to fix a particular unreliable
or broken test, or the non-RC bug that it exposes, I think it's still
correct for that test to continue to run on CI infrastructure, with
its result logged but the usual side-effects of that result ignored.
This can give the maintainer valuable feedback to send to upstream
(for example I can now tell ostree's author that
ostree/test-local-pull.sh.test fails around half the time, but the
other pull tests that sporadically failed have stopped failing since
a thread-safety issue in libsoup was fixed).

With the ability to ignore selected restrictions (#850494), this could
be addressed by a restriction, perhaps something like one of these:

    Restrictions: experimental
    Restrictions: intermittently-fails
    Restrictions: may-fail
    Restrictions: unimportant

possibly accompanied by:

    Features: known-bug=http://bugs.debian.org/1234

Tests with that restriction could perhaps be run normally, but
skipped when determining whether a batch of changes would break testing.

This is similar in concept to the TODO directive in TAP
<https://testanything.org/tap-specification.html>, which marks a
failure as not invalidating the overall success of the test.

    S

Message #10 received at 851558@bugs.debian.org (full text, mbox, reply):

From: Martin Pitt <mpitt@debian.org>

To: Simon McVittie <smcv@debian.org>, 851558@bugs.debian.org

Subject: Re: Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI

Date: Mon, 16 Jan 2017 12:34:50 +0100

[Message part 1 (text/plain, inline)]

control: tag -1 confirmed

Hello Simon,

Simon McVittie [2017-01-16  9:06 +0000]:
> Specifically, if a maintainer is unable to fix a particular unreliable
> or broken test, or the non-RC bug that it exposes, I think it's still
> correct for that test to continue to run on CI infrastructure, with
> its result logged but the usual side-effects of that result ignored.

FTR, in Ubuntu (same as in the discussion going on for adopting this into
Debian) we use britney overrides for this, like in

  http://bazaar.launchpad.net/~ubuntu-release/britney/hints-ubuntu/view/head:/pitti

i. e. the tests still run, but britney won't block the package on its failure.
The advantage is that it's much easier to control by the release team (it
doesn't require package uploads), and that you can keep more state: the hints
usually apply to a (maximum) version so that you can say "1.2-3 is known-broken
because of unrelated changes in $foo, but I expect the next upload to fix this
again", so that you can't cheat from the side of the developer.

As long as Debian doesn't actually use these tests for gating, there is no
problem in making them appear as failed. Particular in the tracker you actually
do want to point out failing tests. Or what other side effects do you mean?

> With the ability to ignore selected restrictions (#850494), this could
> be addressed by a restriction, perhaps something like one of these:
> 
>     Restrictions: experimental
>     Restrictions: intermittently-fails
>     Restrictions: may-fail
>     Restrictions: unimportant

I'm fine with adding this, as from the maintainer's perspective she could just
as well do an upload with commenting out that test, and both would appear in a
diff either way. This is for a different use case than chasing down regressing
tests (what I mentioned above), and thus seems useful for introducing new tests
which still need some work to fully work on the distro infrastructure.

Maybe "expected-failure" which is the term used by other test runners? (The
bikeshedding season is on! ☺ )

> possibly accompanied by:
> 
>     Features: known-bug=http://bugs.debian.org/1234

This won't modify the behaviour of the test, so a comment above Restrictions:
would just do as well -- but as unknown features are ignored, this is fine of
course.

> Tests with that restriction could perhaps be run normally, but
> skipped when determining whether a batch of changes would break testing.

I think autopkgtest would show their result as "EXPECTED-FAIL" or so and exit
with 0 (at least for that particular test case)?

Thanks for the suggestion!

Martin

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

[signature.asc (application/pgp-signature, inline)]

Message #17 received at 851558@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Martin Pitt <mpitt@debian.org>

Cc: 851558@bugs.debian.org

Subject: Re: Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI

Date: Mon, 16 Jan 2017 15:29:32 +0000

On Mon, 16 Jan 2017 at 12:34:50 +0100, Martin Pitt wrote:
> Maybe "expected-failure" which is the term used by other test runners? (The
> bikeshedding season is on! ☺ )

The bikeshedding season appears to be 1st January to 31st December,
inclusive. (No bikeshedding allowed during the leap second? :-)

I avoided suggesting "expected-failure" because it is actually not fully
consistent. In some test frameworks (Automake XFAIL_TESTS), success where
a failure was expected makes the overall test *fail*; in others (TAP tests
that report "not ok [n] # TODO"), it leaves the overall test succeeding,
possibly with a warning (because in principle now you can switch it to be
a normal test, if you trust it to be reliably passing, which of course you
don't necessarily for a while).

Because in the real world tests are not always 100% reliable, my priority
is to define something that I can correctly put on a test that
intermittently fails. If you want to have "must fail" *as well*, that
would be fine (but I'm probably not going to use that one).

It would also be nice if we had an exit status that marked a test as
skipped, perhaps 77 like in Automake - at the moment, a test that
probes its environment and determines that it can't run here has to
exit 0 and show up as passed. Would you be interested in a patch
for this?

> >     Features: known-bug=http://bugs.debian.org/1234
> 
> This won't modify the behaviour of the test, so a comment above Restrictions:
> would just do as well -- but as unknown features are ignored, this is fine of
> course.

My thought was that frontends (ci.debian.net) could maybe extract this
information and use it to present autopkgtest's raw results a bit more
helpfully.

> I think autopkgtest would show their result as "EXPECTED-FAIL" or so and exit
> with 0 (at least for that particular test case)?

Something like that, yes; either that, or report EXPECTED-FAIL or similar
in text, but pretend the test case was skipped when deciding what
exit-status autopkgtest should have? (So in practice it would exit 2)

    S

Message #22 received at 851558@bugs.debian.org (full text, mbox, reply):

From: Paul Gevers <elbrus@debian.org>

To: 851558@bugs.debian.org, Simon McVittie <smcv@debian.org>

Subject: Re: Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI

Date: Mon, 16 Jan 2017 21:53:22 +0100

[Message part 1 (text/plain, inline)]

Hi Martin,

On 16-01-17 12:34, Martin Pitt wrote:
> Simon McVittie [2017-01-16  9:06 +0000]:
>> Specifically, if a maintainer is unable to fix a particular unreliable
>> or broken test, or the non-RC bug that it exposes, I think it's still
>> correct for that test to continue to run on CI infrastructure, with
>> its result logged but the usual side-effects of that result ignored.
> 
> FTR, in Ubuntu (same as in the discussion going on for adopting this into
> Debian) we use britney overrides for this, like in
> 
>   http://bazaar.launchpad.net/~ubuntu-release/britney/hints-ubuntu/view/head:/pitti
> 
> i. e. the tests still run, but britney won't block the package on its failure.

Can this be done on a per test basis, or only on a per package basis? I
think there is need for this to (also) work on the former.

Paul

[signature.asc (application/pgp-signature, attachment)]

Message #27 received at 851558@bugs.debian.org (full text, mbox, reply):

From: Martin Pitt <mpitt@debian.org>

To: Simon McVittie <smcv@debian.org>

Cc: 851558@bugs.debian.org

Subject: Re: Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI

Date: Sun, 22 Jan 2017 16:55:09 +0100

Hello Simon,

Simon McVittie [2017-01-16 15:29 +0000]:
> I avoided suggesting "expected-failure" because it is actually not fully
> consistent. In some test frameworks (Automake XFAIL_TESTS), success where
> a failure was expected makes the overall test *fail*; in others (TAP tests
> that report "not ok [n] # TODO"), it leaves the overall test succeeding,
> possibly with a warning (because in principle now you can switch it to be
> a normal test, if you trust it to be reliably passing, which of course you
> don't necessarily for a while).

I'm leaning towards the second semantics -- i. e. a passing exfail test is
still a pass. If we need the first semantics we could add a "mustfail" instead
later on -- but this is just syntactic sugar, because if you actually expect
that you can just invert the exit code in your test yourself. (Of course you
can implement an exfail behaviour the same way, but this is the more important
variant IMHO).

> It would also be nice if we had an exit status that marked a test as
> skipped, perhaps 77 like in Automake - at the moment, a test that
> probes its environment and determines that it can't run here has to
> exit 0 and show up as passed. Would you be interested in a patch
> for this?

autopkgtest already defines exit code (bit mask) 2 for that, so it should use
that one. Returning 2 instead of 0 when encountering an exfail test sounds
again like a bikeshedding exercise to me, but I do agree that it's good to
mechanically determine if there were any exfailing tests; and as "0 or 2 is
success" has been a long-standing API now and being relied on in several CI
systems (at least debci, ubuntu's britney, and autopkgtest.ubuntu.com) I
wouldn't like to introduce a new code either. So I suppose "2" it is.

> > >     Features: known-bug=http://bugs.debian.org/1234
> > 
> > This won't modify the behaviour of the test, so a comment above Restrictions:
> > would just do as well -- but as unknown features are ignored, this is fine of
> > course.
> 
> My thought was that frontends (ci.debian.net) could maybe extract this
> information and use it to present autopkgtest's raw results a bit more
> helpfully.

Fine for me in principle. This would be a mere exercise in convention and
documentation. However, I don't like that it's independent from the proposed
restriction. I. e. that feature makes no sense without "exfail". How about
merging it:

  Features: exfail=http://bugs.debian.org/1234

and if a failing test has an "exfail" or "exfail=..." feature it gets treated as
"skipped" instead.

> Something like that, yes; either that, or report EXPECTED-FAIL or similar
> in text, but pretend the test case was skipped when deciding what
> exit-status autopkgtest should have? (So in practice it would exit 2)

Agreed.

Martin
-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

Message #32 received at 851558@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Martin Pitt <mpitt@debian.org>

Cc: 851558@bugs.debian.org

Subject: Re: Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI

Date: Sun, 22 Jan 2017 18:07:49 +0000

On Sun, 22 Jan 2017 at 16:55:09 +0100, Martin Pitt wrote:
> > It would also be nice if we had an exit status that marked a test as
> > skipped, perhaps 77 like in Automake - at the moment, a test that
> > probes its environment and determines that it can't run here has to
> > exit 0 and show up as passed. Would you be interested in a patch
> > for this?
> 
> autopkgtest already defines exit code (bit mask) 2 for that, so it should use
> that one. Returning 2 instead of 0 when encountering an exfail test sounds
> again like a bikeshedding exercise to me

Sorry, I didn't make myself clear. What I mean is: suppose I want to
test that some tool (ostree or tar or whatever) can back up and restore
xattrs. I can't test that unless I have some scratch area that lets the
current user set xattrs. So I might write a test like this:

    #!/bin/sh

    touch "$ADTTMP/file"

    if ! setfattr -n does-this-work -v value "$ADTTMP/file" 2>/dev/null; then
        echo "Can't run this test, $ADTTMP doesn't support extended attributes"
        exit 77
    fi

    # ... actual test elided

(Or in a real example I might probe $ADTTMP, /tmp and /var/tmp, and use the
first one that does support xattrs, if any - but you get the idea.)

Or, imagine I want to test a non-http network protocol, perhaps testing
whether my XMPP client can connect to an XMPP server. As we discussed
on #851556, we would expect this to work on ci.debian.net, but not on
Ubuntu's equivalent. I could probe for this with ": | nc xmpp.debian.net 5222"
or something.

With Automake, I would be able to exit 77 to get the test recorded as
having been skipped, but with current autopkgtest I would have to exit 0.

This would hopefully affect autopkgtest's exit status the same way as if
the test had been skipped due to "Restrictions: something-you-dont-support" -
so if one or more tests exit 77, autopkgtest would exit 2.

> How about merging it:
> 
>   Features: exfail=http://bugs.debian.org/1234
> 
> and if a failing test has an "exfail" or "exfail=..." feature it gets
> treated as "skipped" instead.

The reason I suggested a Restriction was that until the test-runner has
been upgraded to an autopkgtest version that knows the new syntax, the test
would be skipped (not run) because of the unknown restriction. But I can
see your point, and your proposed syntax seems good.

    S

Message #37 received at 851558@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Martin Pitt <mpitt@debian.org>, 851558@bugs.debian.org

Subject: Re: Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI

Date: Thu, 14 Jun 2018 17:51:44 +0100

Control: forwarded -1 https://salsa.debian.org/ci-team/autopkgtest/merge_requests/19

On Mon, 16 Jan 2017 at 12:34:50 +0100, Martin Pitt wrote:
> Simon McVittie [2017-01-16  9:06 +0000]:
> > Specifically, if a maintainer is unable to fix a particular unreliable
> > or broken test, or the non-RC bug that it exposes, I think it's still
> > correct for that test to continue to run on CI infrastructure, with
> > its result logged but the usual side-effects of that result ignored.

I've implemented this (provisionally named "flaky", but other names are
available for bikeshedding).

If the flaky test passes, it is recorded as PASS as usual, and
autopkgtest's exit status is the same as if a normal test had passed.
If it fails, it is recorded as FLAKY, and autopkgtest's exit status is
the same as if it had been skipped due to Restrictions: isolation-machine
or similar.

    smcv

Message #44 received at 851558@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Martin Pitt <mpitt@debian.org>

Cc: 851558@bugs.debian.org

Subject: Re: Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI

Date: Thu, 14 Jun 2018 17:54:41 +0100

On Sun, 22 Jan 2017 at 18:07:49 +0000, Simon McVittie wrote:
> Suppose I want to
> test that some tool (ostree or tar or whatever) can back up and restore
> xattrs. I can't test that unless I have some scratch area that lets the
> current user set xattrs.
...
> Or, imagine I want to test a non-http network protocol, perhaps testing
> whether my XMPP client can connect to an XMPP server. As we discussed
> on #851556, we would expect this to work on ci.debian.net, but not on
> Ubuntu's equivalent. I could probe for this with ": | nc xmpp.debian.net 5222"
> or something.
> 
> With Automake, I would be able to exit 77 to get the test recorded as
> having been skipped, but with current autopkgtest I would have to exit 0.
> 
> This would hopefully affect autopkgtest's exit status the same way as if
> the test had been skipped due to "Restrictions: something-you-dont-support" -
> so if one or more tests exit 77, autopkgtest would exit 2.

I've implemented this at
<https://salsa.debian.org/ci-team/autopkgtest/merge_requests/20>
since an implementation is probably easier to comment on than an idea.
It's really orthogonal to what I first asked for in #851558.

    smcv

Message #47 received at 851558-submitter@bugs.debian.org (full text, mbox, reply):

From: smcv@debian.org

To: 851558-submitter@bugs.debian.org

Subject: Bug #851558 in autopkgtest marked as pending

Date: Fri, 15 Jun 2018 02:07:25 +0000

Control: tag -1 pending

Hello,

Bug #851558 in autopkgtest reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below, and you can check the diff of the fix at:

https://salsa.debian.org/ci-team/autopkgtest/commit/fb23dd0aee83b16473672c3616614a146ed291ed

------------------------------------------------------------------------
Add support for flaky tests

This restriction is useful if a package contains tests that are not yet
reliable enough for gating CI, but can usefully be run to assess how
often they fail, with the intention of promoting the test to non-flaky
when it is reliable.

Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #851558

------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/851558

Message #54 received at 851558-close@bugs.debian.org (full text, mbox, reply):

From: Paul Gevers <elbrus@debian.org>

To: 851558-close@bugs.debian.org

Subject: Bug#851558: fixed in autopkgtest 5.4

Date: Mon, 02 Jul 2018 11:19:01 +0000

Source: autopkgtest
Source-Version: 5.4

We believe that the bug you reported is fixed in the latest version of
autopkgtest, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 851558@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Paul Gevers <elbrus@debian.org> (supplier of updated autopkgtest package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Mon, 02 Jul 2018 11:50:21 +0200
Source: autopkgtest
Binary: autopkgtest
Architecture: source
Version: 5.4
Distribution: unstable
Urgency: medium
Maintainer: Debian CI team <team+ci@tracker.debian.org>
Changed-By: Paul Gevers <elbrus@debian.org>
Description:
 autopkgtest - automatic as-installed testing for Debian packages
Closes: 832751 851558 867081 897170 901643
Changes:
 autopkgtest (5.4) unstable; urgency=medium
 .
   [ Simon McVittie ]
   * README.package-tests.rst: Document AUTOPKGTEST_NORMAL_USER
   * d/tests/lxd: Don't assume all test-runners set AUTOPKGTEST_NORMAL_USER
   * qemu: Only set up base image device if requested
   * qemu: Document --baseimage
   * qemu: Update test for --baseimage no longer being the default
   * qemu, lxc, lxd: Look for a user account in the 1000-59999 range
     (Closes: #897170)
   * qemu: Add a shortcut for running tests on an EFI-booted image
   * doc: Describe how to parse Features, Restrictions, Classes
   * Add support for flaky tests (Closes: #851558)
   * Add support for tests declaring themselves to have been skipped
   * autopkgtest(1): Document FLAKY as a possible summary status
 .
   [ Balint Reczey ]
   * Fix bashism in retrying apt update
 .
   [ Paul Gevers ]
   * Enable testing to continue after badpkg (Closes: #832751)
   * adt_testbed.py fix missed piece of regular expression in commit a7e1dad
   * d/tests/autopkgtest Update SchrootRunner
   * runner/autopkgtest: Drop Ubuntu 12.04 fallback
   * manpage: make ordering consistent with --help (Closes: #901643)
 .
   [ Julian Andres Klode ]
   * ssh-setup/nova: Add support for keystone v3 auth (LP: #1767433)
 .
   [ Rafael Laboissiere ]
   * Set Maintainer email address to team+ci@tracker.debian.org
 .
   [ Niko Tyni ]
   * Add a couple of testcases for versioned provides support
   * Ensure synthesized test dependencies are not satisfied by versioned Provides
     (Closes: #867081)
   * Remove the old '(>= 0)' hack for ensuring '@' pulls in real packages
Checksums-Sha1:
 1720155af7c18ae32212b2e6a0bde1a73b79e98b 1582 autopkgtest_5.4.dsc
 f9b0c5ab38b22484d0176eec1f115bb0b369116b 173636 autopkgtest_5.4.tar.xz
Checksums-Sha256:
 5ccc59ec2985b5557763764309867a6068f7139de3e9d06f1e43e240894b4aa4 1582 autopkgtest_5.4.dsc
 c8c5c14298105d1798ce9a20a4f7d2b837279939a94da5fdc761111517575030 173636 autopkgtest_5.4.tar.xz
Files:
 b12bae17fc3461b1d46d93bf4863c768 1582 devel optional autopkgtest_5.4.dsc
 7abe82af7806ef1c477f0fb1be65b7c7 173636 devel optional autopkgtest_5.4.tar.xz

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEWLZtSHNr6TsFLeZynFyZ6wW9dQoFAls59cMACgkQnFyZ6wW9
dQo6kQf/XkKDoSKtvBNmtwkmV62jefiKqvdZlJVVxCyfnrQ5V/3ObsZAkKNCIL8e
MIlVOXrbGWSFj4qsnVH1l6+1pZkKjYboL+65xqazt2K8bWwTe1zi6/TDFnU/dcg5
HXS6s0/5U2eO5UvrDObxaKZFwvbex/DrMzTGR3xluvsYheVIA89+qD5MfJ/rUSP8
fMBb8GcbGN2hTIB1PufXhg0wnj99j3tE30Ziycpe87+14Ol9aTpu8b+p3+XU1Uwa
LJ4F0+iDnfkG0433GXEGVV4dhq0IpaQySp77DPRC9GQ/OeXod406mU/CtUZkQxnK
ZhMfnGZ81Ez8qmrpFDEJoLzTn+pWUw==
=joYo
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.