Debian Bug report logs - #658937
testing.pl: needs rewriting

Package: release.debian.org; Maintainer for release.debian.org is Debian Release Team <debian-release@lists.debian.org>;

Reported by: "Adam D. Barratt" <adam@adam-barratt.org.uk>

Date: Mon, 6 Feb 2012 21:03:01 UTC

Severity: important

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian Release Team <debian-release@lists.debian.org>:
Bug#658937; Package release.debian.org. (Mon, 06 Feb 2012 21:03:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Adam D. Barratt" <adam@adam-barratt.org.uk>:
New Bug report received and forwarded. Copy sent to Debian Release Team <debian-release@lists.debian.org>. (Mon, 06 Feb 2012 21:03:04 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: "Adam D. Barratt" <adam@adam-barratt.org.uk>
To: submit@bugs.debian.org
Subject: testing.pl: needs rewriting
Date: Mon, 06 Feb 2012 20:58:01 +0000
Package: release.debian.org
Severity: important
User: release.debian.org@packages.debian.org
Usertag: tools

migration/testing.pl needs to be updated to make it react better when
called repeatedly, or to avoid the need to do so.  Right now it's
sufficiently resource hungry to make a 16-core machine unhappy.

An initial idea might be to statically generate package pages, although
this would need to be benchmarked for feasibility.  testing.pl could
then simply become a redirect to /package/blah or similar.





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Release Team <debian-release@lists.debian.org>:
Bug#658937; Package release.debian.org. (Mon, 13 Feb 2012 13:27:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Adam D. Barratt" <adam@adam-barratt.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Release Team <debian-release@lists.debian.org>. (Mon, 13 Feb 2012 13:27:04 GMT) Full text and rfc822 format available.

Message #10 received at 658937@bugs.debian.org (full text, mbox):

From: "Adam D. Barratt" <adam@adam-barratt.org.uk>
To: <658937@bugs.debian.org>
Subject: Re: Bug#658937: testing.pl: needs rewriting
Date: Mon, 13 Feb 2012 13:23:49 +0000
On 06.02.2012 20:58, Adam D. Barratt wrote:
> migration/testing.pl needs to be updated to make it react better when
> called repeatedly, or to avoid the need to do so.  Right now it's
> sufficiently resource hungry to make a 16-core machine unhappy.
>
> An initial idea might be to statically generate package pages, 
> although
> this would need to be benchmarked for feasibility.  testing.pl could
> then simply become a redirect to /package/blah or similar.

It turns out that the cron job already generated cache files for the 
packages pages, which were then returned by the CGI (except presumably 
for the brief window while they were being re-generated).  
testing.pl?package=$foo now serves up the cache page via mod_rewrite.  
This does mean that there's a brief window once an hour where the pages 
aren't available, but that could be resolved by having the cron job use 
a temporary data store which is then swapped to live.

The extra query string parameters aren't currently supported.  We 
should work out which are actually in use, and whether we can 
re-introduce support for them in a sane way (without either killing 
franck or exploding the file system use).

Adam




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Release Team <debian-release@lists.debian.org>:
Bug#658937; Package release.debian.org. (Thu, 10 May 2012 16:45:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Julian Gilbey <jdg@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Release Team <debian-release@lists.debian.org>. (Thu, 10 May 2012 16:45:03 GMT) Full text and rfc822 format available.

Message #15 received at 658937@bugs.debian.org (full text, mbox):

From: Julian Gilbey <jdg@debian.org>
To: "Adam D. Barratt" <adam@adam-barratt.org.uk>, 658937@bugs.debian.org
Subject: Re: Bug#658937: testing.pl: needs rewriting
Date: Thu, 10 May 2012 17:42:24 +0100
On Mon, Feb 13, 2012 at 01:23:49PM +0000, Adam D. Barratt wrote:
> On 06.02.2012 20:58, Adam D. Barratt wrote:
> >migration/testing.pl needs to be updated to make it react better when
> >called repeatedly, or to avoid the need to do so.  Right now it's
> >sufficiently resource hungry to make a 16-core machine unhappy.
> >
> >An initial idea might be to statically generate package pages,
> >although
> >this would need to be benchmarked for feasibility.  testing.pl could
> >then simply become a redirect to /package/blah or similar.

Stupid question: is it running under mod_perl?  If so, surely there's
some way to cache the database between invocations so it doesn't have
to be regenerated on each call?

Please ignore me if I'm talking nonsense....

   Julian




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Release Team <debian-release@lists.debian.org>:
Bug#658937; Package release.debian.org. (Thu, 10 May 2012 17:06:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Adam D. Barratt" <adam@adam-barratt.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Release Team <debian-release@lists.debian.org>. (Thu, 10 May 2012 17:06:07 GMT) Full text and rfc822 format available.

Message #20 received at 658937@bugs.debian.org (full text, mbox):

From: "Adam D. Barratt" <adam@adam-barratt.org.uk>
To: Julian Gilbey <jdg@debian.org>, <658937@bugs.debian.org>
Subject: Re: Bug#658937: testing.pl: needs rewriting
Date: Thu, 10 May 2012 17:56:36 +0100
On 10.05.2012 17:42, Julian Gilbey wrote:
> On Mon, Feb 13, 2012 at 01:23:49PM +0000, Adam D. Barratt wrote:
>> On 06.02.2012 20:58, Adam D. Barratt wrote:
>> >migration/testing.pl needs to be updated to make it react better 
>> when
>> >called repeatedly, or to avoid the need to do so.  Right now it's
>> >sufficiently resource hungry to make a 16-core machine unhappy.
>> >
>> >An initial idea might be to statically generate package pages,
>> >although
>> >this would need to be benchmarked for feasibility.  testing.pl 
>> could
>> >then simply become a redirect to /package/blah or similar.
>
> Stupid question: is it running under mod_perl?

The server in question doesn't appear to have libapache2-mod-perl2 
installed, so I'm going to say "no". ;-)

> If so, surely there's
> some way to cache the database between invocations so it doesn't have
> to be regenerated on each call?

The per-package pages were already cached and it's that cache that 
we're currently using, via mod_rewrite, to serve the pages that are 
linked from services such as the PTS.  There's no support for any of the 
other querystring parameters right now, because we'd end up with 
n-combinatorial versions of each file around.

In theory, yes, it could probably be rewritten to store the interesting 
data somewhere and cross-reference that, rather than re-reading the 
packages files all the time.  As I mentioned in my mail opening this 
report though, whatever is available needs to be sufficiently sane not 
to make a machine of the size of ftp-master/release.d.o fall over when 
comment spammers spider several thousand of the pages in a short time 
and thus make DSA frown a good deal.

Regards,

Adam




Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Thu Apr 17 07:14:20 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.