Debian Bug report logs - #624716
[britney2] Does not properly undo failed migrations

Package: release.debian.org; Maintainer for release.debian.org is Debian Release Team <debian-release@lists.debian.org>;

Reported by: "Adam D. Barratt" <adam@adam-barratt.org.uk>

Date: Sat, 30 Apr 2011 20:42:02 UTC

Severity: important

Tags: patch

Done: "Adam D. Barratt" <adam@adam-barratt.org.uk>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian Release Team <debian-release@lists.debian.org>:
Bug#624716; Package release.debian.org. (Sat, 30 Apr 2011 20:42:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Adam D. Barratt" <adam@adam-barratt.org.uk>:
New Bug report received and forwarded. Copy sent to Debian Release Team <debian-release@lists.debian.org>. (Sat, 30 Apr 2011 20:42:04 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: "Adam D. Barratt" <adam@adam-barratt.org.uk>
To: submit@bugs.debian.org
Subject: [britney2] Does not properly undo failed migrations
Date: Sat, 30 Apr 2011 21:39:30 +0100
Package: release.debian.org
Severity: important
User: release.debian.org@packages.debian.org
Usertag: britney

Filing as a bug so it's documented somewhere more formal and doesn't get
forgotten.

On Sun, 2011-04-10 at 10:29 +0000, Cron Daemon wrote:
> Traceback (most recent call last):
>   File "/srv/release.debian.org/britney/code/b2/britney.py", line 2841, in <module>
>     Britney().main()
>   File "/srv/release.debian.org/britney/code/b2/britney.py", line 2836, in main
>     self.upgrade_testing()
>   File "/srv/release.debian.org/britney/code/b2/britney.py", line 2548, in upgrade_testing
>     self.do_all()
>   File "/srv/release.debian.org/britney/code/b2/britney.py", line 2447, in do_all
>     (nuninst_end, extra, tundo) = self.iter_packages(upgrade_me, selected, nuninst=nuninst_end)
>   File "/srv/release.debian.org/britney/code/b2/britney.py", line 2243, in iter_packages
>     pkg_name, suite, affected, undo = self.doop_source(pkg)
>   File "/srv/release.debian.org/britney/code/b2/britney.py", line 2039, in doop_source
>     if binaries[parch][0][binary][SOURCE] != pkg_name: continue
> KeyError: 'fixincludes'

(and the docvert-openoffice.org and openoffice.org-dmaths crashes from
mid-March, which I suspect are other occurrences of the same issue;
possibly also related to #618805)

From a bit of debugging, this appears to be an issue with cases where a
particular binary package has either moved source packages (as in the
"fixincludes" case, from gcc-4.5 to gcc-4.6) or moved from being
arch-dependent to arch:all, and the package is considered during
multiple phases of the b2 run (either as part of an "easy" and then
during the main run, or in the main run and then in a "hint").

When deciding whether to migrate a package, britney removes all of the
current binary packages - making an "undo" list - and then adds the
binary packages from the new version.  If the new version is not
accepted then the "undo" list is used to revert things to the previous
state.  However, in the "fixincludes" case at least, that reversion
seems not to be applied correctly.  The run in which the above error
occurred included an "easy" for gcc-4.5, gcc-4.6 and a few other
packages, which failed; when the main run came to consider gcc-4.5 again
the sources dict still listed "fixincludes" as being built from gcc-4.5
but the binaries dict included the information for gcc-4.5/unstable,
meaning that it was missing the "fixincludes" package.

This appears to be "fixable" by disabling the use of the pre_process
dict, but at the expense of making the run slower.

Regards,

Adam





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Release Team <debian-release@lists.debian.org>:
Bug#624716; Package release.debian.org. (Mon, 02 May 2011 15:18:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Adam D. Barratt" <adam@adam-barratt.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Release Team <debian-release@lists.debian.org>. (Mon, 02 May 2011 15:18:06 GMT) Full text and rfc822 format available.

Message #10 received at 624716@bugs.debian.org (full text, mbox):

From: "Adam D. Barratt" <adam@adam-barratt.org.uk>
To: 624716@bugs.debian.org
Subject: Re: Cron <release@franck> ~release/britney/code/b1/britney urgencies bugs pkglists sync_b2 run run_b2 save stats summary >~release/britney/britney.log && ~release/britney/code/b1/britney udeb_report
Date: Mon, 02 May 2011 16:15:33 +0100
On Mon, 2011-05-02 at 10:14 +0000, Cron Daemon wrote:
[...]
>   File "/srv/release.debian.org/britney/code/b2/britney.py", line 2243, in iter_packages
>     pkg_name, suite, affected, undo = self.doop_source(pkg)
>   File "/srv/release.debian.org/britney/code/b2/britney.py", line 2039, in doop_source
>     if binaries[parch][0][binary][SOURCE] != pkg_name: continue
> KeyError: 'libthunar-vfs-1-2'

This looks like another case of #624716; the binary package moved from
thunar to thunar-vfs.

Regards,

Adam





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Release Team <debian-release@lists.debian.org>:
Bug#624716; Package release.debian.org. (Wed, 04 May 2011 19:03:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Adam D. Barratt" <adam@adam-barratt.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Release Team <debian-release@lists.debian.org>. (Wed, 04 May 2011 19:03:03 GMT) Full text and rfc822 format available.

Message #15 received at 624716@bugs.debian.org (full text, mbox):

From: "Adam D. Barratt" <adam@adam-barratt.org.uk>
To: 624716@bugs.debian.org
Subject: Re: Bug#624716: [britney2] Does not properly undo failed migrations
Date: Wed, 04 May 2011 19:58:18 +0100
[Message part 1 (text/plain, inline)]
tag 624716 + patch
thanks

On Sat, 2011-04-30 at 21:39 +0100, Adam D. Barratt wrote:
> On Sun, 2011-04-10 at 10:29 +0000, Cron Daemon wrote:
> >   File "/srv/release.debian.org/britney/code/b2/britney.py", line 2039, in doop_source
> >     if binaries[parch][0][binary][SOURCE] != pkg_name: continue
> > KeyError: 'fixincludes'
[...]
> >From a bit of debugging, this appears to be an issue with cases where a
> particular binary package has either moved source packages (as in the
> "fixincludes" case, from gcc-4.5 to gcc-4.6) or moved from being
> arch-dependent to arch:all, and the package is considered during
> multiple phases of the b2 run (either as part of an "easy" and then
> during the main run, or in the main run and then in a "hint").

Or, indeed, in multiple hints.

> When deciding whether to migrate a package, britney removes all of the
> current binary packages - making an "undo" list - and then adds the
> binary packages from the new version.  If the new version is not
> accepted then the "undo" list is used to revert things to the previous
> state.  However, in the "fixincludes" case at least, that reversion
> seems not to be applied correctly.

I believe I've worked out what's happening here.  The undo process is
essentially:

  for each block of undo data
    undo source package changes
    remove newly added binary packages
    re-add removed binary packages
    revert changes to virtual packages

In each of the cases where a binary B has moved from source S1 to S2,
what ends up happening is that the undo data for S1 is processed first,
giving us:

  re-add binary "B", removed from source S1
  remove binary "B", newly added by source S2

with the net result that the binary list no longer contains B but the
sources dict lists it as being built by S1.  Restructuring the undo loop
so that all removals are processed before any re-introductions fixes the
crash and seems to produce the same logical result afaics.

I intend to apply the attached patch within the next few days unless
someone raises a problem with it.

Regards,

Adam
[b2-624716.diff (text/x-patch, attachment)]

Added tag(s) patch. Request was from "Adam D. Barratt" <adam@adam-barratt.org.uk> to control@bugs.debian.org. (Wed, 04 May 2011 19:03:04 GMT) Full text and rfc822 format available.

Reply sent to "Adam D. Barratt" <adam@adam-barratt.org.uk>:
You have taken responsibility. (Mon, 09 May 2011 18:15:05 GMT) Full text and rfc822 format available.

Notification sent to "Adam D. Barratt" <adam@adam-barratt.org.uk>:
Bug acknowledged by developer. (Mon, 09 May 2011 18:15:05 GMT) Full text and rfc822 format available.

Message #22 received at 624716-done@bugs.debian.org (full text, mbox):

From: "Adam D. Barratt" <adam@adam-barratt.org.uk>
To: "Adam D. Barratt" <adam@adam-barratt.org.uk>, 624716-done@bugs.debian.org
Subject: Re: Bug#624716: [britney2] Does not properly undo failed migrations
Date: Mon, 09 May 2011 19:12:46 +0100
On Wed, 2011-05-04 at 19:58 +0100, Adam D. Barratt wrote:
> In each of the cases where a binary B has moved from source S1 to S2,
> what ends up happening is that the undo data for S1 is processed first,
> giving us:
> 
>   re-add binary "B", removed from source S1
>   remove binary "B", newly added by source S2
> 
> with the net result that the binary list no longer contains B but the
> sources dict lists it as being built by S1.  Restructuring the undo loop
> so that all removals are processed before any re-introductions fixes the
> crash and seems to produce the same logical result afaics.
> 
> I intend to apply the attached patch within the next few days unless
> someone raises a problem with it.

No-one did, so I pushed the patch earlier today.

Regards,

Adam





Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Tue, 07 Jun 2011 07:35:04 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Thu Apr 24 07:15:40 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.