Debian Bug report logs - #660409
Rewrite Britney's installability tester in python

Package: release.debian.org; Maintainer for release.debian.org is Debian Release Team <debian-release@lists.debian.org>;

Reported by: Niels Thykier <niels@thykier.net>

Date: Sat, 18 Feb 2012 22:48:02 UTC

Severity: wishlist

Tags: patch

Done: Niels Thykier <niels@thykier.net>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, niels@thykier.net, Debian Release Team <debian-release@lists.debian.org>:
Bug#660409; Package release.debian.org. (Sat, 18 Feb 2012 22:48:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Niels Thykier <niels@thykier.net>:
New Bug report received and forwarded. Copy sent to niels@thykier.net, Debian Release Team <debian-release@lists.debian.org>. (Sat, 18 Feb 2012 22:48:05 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Niels Thykier <niels@thykier.net>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: Rewrite Britney's installability tester in python
Date: Sat, 18 Feb 2012 23:28:08 +0100
Package: release.debian.org
Severity: wishlist
User: release.debian.org@packages.debian.org
Usertags: britney


Hi,

I took a look at replacing Britney's installability tester[1].  In my
follow up mail I will attach my patches[2].  Part of my focus has been
to implement this as a(n almost) compatible replacement for the old
module.

Result-wise it gives identical results to our current implementation
in the test suite.  Speed-wise, I believe it is "fast enough" to be
used.  I have some ideas for further optimization, though these will
most likely imply structural changes to other parts of Britney.

My testing suggests that using Python2.7 will greatly affect the
runtime with this module.  The master branch correctly only see a
similar improvement in one of the live-data samples[3].

Please note that the britney-tests repository is currently not setup
to use the live-data as regression test.  If you want to verify my
results, consider using "test-branch" to pit it against the master
branch.

~Niels

[1] The "dark corner" written in C that no ones ever looks at
if they can avoid it.

[2] I do not trust myself with reportbug to get it right in this
email.  :)

[3] Runtimes the master branch vs my branch.  The "-py2.6" marker means
Python2.6 was used as interpreter.  Same for "-py2.7" and Python2.7.

+------------+--------------+--------------+--------------+--------------+
| live-data  | master-py2.6 | master-py2.7 | branch-py2.6 | branch-py2.7 |
+------------+--------------+--------------+--------------+--------------+
| 2011-12-13 |   201.363s   |   186.369s   |   173.224s   |   135.778s   |
| 2011-12-20 |    97.947s   |    93.213s   |   142.128s   |   111.234s   |
| 2012-01-04 |    74.255s   |    70.503s   |   117.565s   |    95.287s   |
+------------+--------------+--------------+--------------+--------------+

NB: 2011-12-13 triggers an AIEEE in the master branch.





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Release Team <debian-release@lists.debian.org>:
Bug#660409; Package release.debian.org. (Sat, 18 Feb 2012 22:57:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Niels Thykier <niels@thykier.net>:
Extra info received and forwarded to list. Copy sent to Debian Release Team <debian-release@lists.debian.org>. (Sat, 18 Feb 2012 22:57:06 GMT) Full text and rfc822 format available.

Message #10 received at 660409@bugs.debian.org (full text, mbox):

From: Niels Thykier <niels@thykier.net>
To: 660409@bugs.debian.org
Subject: Re: Bug#660409: Rewrite Britney's installability tester in python
Date: Sat, 18 Feb 2012 23:52:08 +0100
[Message part 1 (text/plain, inline)]
tags 660409 + patch
thanks


On 2012-02-18 23:28, Niels Thykier wrote:
> Package: release.debian.org
> Severity: wishlist
> User: release.debian.org@packages.debian.org
> Usertags: britney
> 
> 
> Hi,
> 
> I took a look at replacing Britney's installability tester[1].  In my
> follow up mail I will attach my patches[2].  Part of my focus has been
> to implement this as a(n almost) compatible replacement for the old
> module.
> 
> [...]
> 
> ~Niels
> 
> [...]
> 
> 

See attached patches.

~Niels

[0001-Move-installability-testing-to-a-separate-module.patch (text/x-patch, attachment)]
[0002-Defer-building-the-nun-inst-cache.patch (text/x-patch, attachment)]
[0003-Rewrite-installability-tester.patch (text/x-patch, attachment)]

Added tag(s) patch. Request was from Niels Thykier <niels@thykier.net> to control@bugs.debian.org. (Sat, 18 Feb 2012 22:57:08 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Release Team <debian-release@lists.debian.org>:
Bug#660409; Package release.debian.org. (Sun, 19 Feb 2012 13:09:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Niels Thykier <niels@thykier.net>:
Extra info received and forwarded to list. Copy sent to Debian Release Team <debian-release@lists.debian.org>. (Sun, 19 Feb 2012 13:09:08 GMT) Full text and rfc822 format available.

Message #17 received at 660409@bugs.debian.org (full text, mbox):

From: Niels Thykier <niels@thykier.net>
To: 660409@bugs.debian.org
Subject: Re: Bug#660409: Rewrite Britney's installability tester in python
Date: Sun, 19 Feb 2012 14:06:16 +0100
[Message part 1 (text/plain, inline)]
Fixup patch for 0003.  I discovered that the live-data samples could run
for 10 minutes without completing the InstallabilityTester._optimize
call, when enabling all architectures.

With patches 0001..0004 applied, I get the following runtimes for my
branch with all architectures enabled in live-data.

Python2.7:

  Running live-2011-12-13... done (462.214s)
  Running live-2011-12-20... done (493.302s)
  Running live-2012-01-04... done (335.115s)

Python2.6

  Running live-2011-12-13... done (853.107s)
  Running live-2011-12-20... done (945.488s)
  Running live-2012-01-04... done (591.630s)

As a comparision, here is the master branch with Python2.6:

  Running live-2011-12-13... done (533.754s)
  Running live-2011-12-20... done (518.489s)
  Running live-2012-01-04... done (301.428s)

... and Python2.7:

  Running live-2011-12-13... done (450.644s)
  Running live-2011-12-20... done (427.680s)
  Running live-2012-01-04... done (261.718s)

~Niels

[0004-Fixup-of-broken-loop-in-_optimize.patch (text/x-patch, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Release Team <debian-release@lists.debian.org>:
Bug#660409; Package release.debian.org. (Wed, 18 Sep 2013 20:15:09 GMT) Full text and rfc822 format available.

Acknowledgement sent to Niels Thykier <niels@thykier.net>:
Extra info received and forwarded to list. Copy sent to Debian Release Team <debian-release@lists.debian.org>. (Wed, 18 Sep 2013 20:15:09 GMT) Full text and rfc822 format available.

Message #22 received at 660409@bugs.debian.org (full text, mbox):

From: Niels Thykier <niels@thykier.net>
To: 660409@bugs.debian.org
Subject: Re: Bug#660409: Rewrite Britney's installability tester in python
Date: Wed, 18 Sep 2013 22:10:24 +0200
[Message part 1 (text/plain, inline)]
Hi,

I am back again, with my branch rewritten and rebased.

The rewritten installability tester uses significantly less memory than
the current C-module.  In her current state, she tends to exceed 2.5GB
in the live-data tests.  With these patches she stops at 1.2-1.3GB as
far as I can trace with random sampling via top(1).

Review welcome,
~Niels


[0001-Defer-building-the-nun-inst-cache.patch (text/x-diff, attachment)]
[0002-Rewrite-installability-tester.patch (text/x-diff, attachment)]

Reply sent to Niels Thykier <niels@thykier.net>:
You have taken responsibility. (Mon, 02 Dec 2013 20:09:10 GMT) Full text and rfc822 format available.

Notification sent to Niels Thykier <niels@thykier.net>:
Bug acknowledged by developer. (Mon, 02 Dec 2013 20:09:10 GMT) Full text and rfc822 format available.

Message #27 received at 660409-done@bugs.debian.org (full text, mbox):

From: Niels Thykier <niels@thykier.net>
To: 660409-done@bugs.debian.org
Subject: Re: Bug#660409: Rewrite Britney's installability tester in python
Date: Mon, 02 Dec 2013 21:05:13 +0100
On 2013-09-18 22:10, Niels Thykier wrote:
> Hi,
> 
> I am back again, with my branch rewritten and rebased.
> 
> The rewritten installability tester uses significantly less memory than
> the current C-module.  In her current state, she tends to exceed 2.5GB
> in the live-data tests.  With these patches she stops at 1.2-1.3GB as
> far as I can trace with random sampling via top(1).
> 
> Review welcome,
> ~Niels
> 
> 

Merged into master on the 27th of November.

~Niels





Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Tue, 31 Dec 2013 07:27:10 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Fri Apr 18 14:23:23 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.