Debian Bug report logs - #977743
packages.debian.org: "list of files" fails on sid/testing arch:all packages - not processing Contents-all.gz ?

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: "Rebecca N. Palmer" <rebecca_palmer@zoho.com>

To: submit@bugs.debian.org

Subject: packages.debian.org: "list of files" fails on sid/testing arch:all packages - not processing Contents-all.gz ?

Date: Sat, 19 Dec 2020 22:40:09 +0000

Package: www.debian.org

On the packages.d.o pages of arch:all packages from sid, bullseye or 
experimental, the "list of files" link gives the error message "No such 
package in this suite on this architecture."

This does not affect arch:any packages, or packages from stable.

e.g.
fails - https://packages.debian.org/sid/all/python3-pandas/filelist
OK - https://packages.debian.org/buster/all/python3-pandas/filelist
OK - https://packages.debian.org/sid/amd64/python3-pandas-lib/filelist

Possible cause: these suites have a separate Contents-all.gz file, but 
https://salsa.debian.org/webmaster-team/packages/-/blob/master/bin/parse-contents#L178 
appears to assume there isn't and try to use 
Contents-${last_arch_checked}.gz.

Message #16 received at 977743@bugs.debian.org (full text, mbox, reply):

From: Felix Lechner <felix.lechner@lease-up.com>

To: 977006@bugs.debian.org, 977743@bugs.debian.org

Cc: Louis-Philippe Véronneau <pollo@debian.org>, "Rebecca N. Palmer" <rebecca_palmer@zoho.com>, Adam D Barratt <adam@adam-barratt.org.uk>

Subject: Re: packages.debian.org: file list broken for some packages

Date: Wed, 11 Aug 2021 20:10:06 -0700

[Message part 1 (text/plain, inline)]

Control: tags -1 + patch

Hi Rebecca,
Hi Louis-Philippe,

> On the packages.d.o pages of arch:all packages from sid, bullseye or
> experimental, the "list of files" link gives the error message "No such
> package in this suite on this architecture."

First of all, thank you both for the excellent detective work! The
issue was caused by commit 81824d23 in daklib [1] in which the archive
started to provide—on Oct 25, 2020 7:08am PDT and for releases past
buster—separate Contents files containing the file paths in Arch:all
packages.

From what I can tell, the code generating the web pages for
packages.d.o did not read those files for releases post buster.

I filed a merge request that I believe solves the issue. [2] It was
tested on the Debian node that generates the web pages
(picconi.debian.org). It comes with two caveats:

(1) Due to insufficient permissions I created an improvised
environment, described further below, that may not fully mimic
production runs.

(2) The second commit addresses a condition that should have prevented
the code from performing at all, although apparently it didn't. That
opens up the possibility that I misunderstood the existing code and,
for my tests, created a runtime environment that differed appreciably
from production.

To test the MR, I cloned my feature branch into my home directory on
picconi.debian.org. I then applied the local patch below this message.

Next I ran the command './bin/setup-site /home/lechner/packages
packages.debian.org' as suggested in ./INSTALL and started the test
with '/home/lechner/packages/cron.d/200process_archive'. (I also
created the folders './files/db' and './tmp' in the base directory of
the Git repo, which was my working directory.) The run finished
without errors and produced the attached log.

Now the databases are more even in size across architectures. Here is
a partial listing of the relevant folder ./files/db: (The full listing
for *.db is attached.)

   0 filelists_sid_all.db
129M filelists_sid_alpha.db
132M filelists_sid_amd64.db
208M filelists_sid_arm64.db
126M filelists_sid_armel.db
128M filelists_sid_armhf.db
128M filelists_sid_hppa.db
131M filelists_sid_i386.db
128M filelists_sid_m68k.db
127M filelists_sid_mips64el.db
127M filelists_sid_mipsel.db
123M filelists_sid_powerpcspe.db
132M filelists_sid_ppc64.db
129M filelists_sid_ppc64el.db
130M filelists_sid_riscv64.db
127M filelists_sid_s390x.db
125M filelists_sid_sh4.db
129M filelists_sid_sparc64.db
130M filelists_sid_x32.db

All packages for Arch:all are symbolic links (human size zero). I am
not sure why arm64 is so large.

Perhaps someone with the appropriate user privileges could pull my
feature branch from the merge request [2] into
/srv/packages.debian.org and test it on the live system. The cron run
can be triggered by hand.

A better long-term solution would be to produce separate transfer
files for Arch:all, but that may not work until buster is being
dropped from the archive. Thank you both for your hard work!

Kind regards
Felix Lechner

[1] https://salsa.debian.org/ftp-team/dak/-/commit/81824d2326f5cc50fdcb95c81f9f26864aebaa15
[2] https://salsa.debian.org/webmaster-team/packages/-/merge_requests/20

* * *

[local patch]

lechner@picconi:~/packages$ git diff
diff --git a/bin/parse-contents b/bin/parse-contents
index a1bfc35..7c5f166 100755
--- a/bin/parse-contents
+++ b/bin/parse-contents
@@ -51,6 +51,9 @@ my @sections = @SECTIONS;
 # Add empty section, need to search Contents directly at dist root,
for debports compat
 push(@sections, "");

+$DBDIR = "/home/lechner/packages/files/db";
+my $TMPDIR = "/home/lechner/packages/tmp";
+
 my %debports_hash;
 # copy from config.sh ${arch_debports}
 @debports_hash{qw( alpha hppa ia64 m68k powerpcspe ppc64 riscv64 sh4
sparc64 x32 )} = ();
@@ -166,9 +169,9 @@ for my $suite (@suites) {

     # Piping from sort's output doesn't really scale with 16 GB worth
     # of input, so let's store in a temporary file:
-    my $rev_path_file = "$TOPDIR/tmp/${suite}.sorted";
+    my $rev_path_file = "$TMPDIR/${suite}.sorted";
     print "Merging reverse path lists for ${suite}...\n";
-    system("sort -T $TOPDIR/tmp -m $DBDIR/reverse_${suite}_*.txt -o
${rev_path_file}") == 0
+    system("sort -T $TMPDIR -m $DBDIR/reverse_${suite}_*.txt -o
${rev_path_file}") == 0
        or die "Failed to build merged list";
     my $rev_path_size = stat($rev_path_file)->size;

diff --git a/cron.d/200process_archive b/cron.d/200process_archive
index 29a7385..eecd412 100755
--- a/cron.d/200process_archive
+++ b/cron.d/200process_archive
@@ -5,13 +5,13 @@
 cd "$topdir"

 date
-./bin/parse-translations --english-only
-date
-./bin/parse-packages
-date
-./bin/parse-sources
-date
-./bin/parse-translations
-date
-./bin/parse-contents
+#./bin/parse-translations --english-only
+#date
+#./bin/parse-packages
+#date
+#./bin/parse-sources
+#date
+#./bin/parse-translations
+#date
+/home/lechner/packages/bin/parse-contents
 date

* * *

[parse-contents.log.xz (application/octet-stream, attachment)]

[db-listing.txt.xz (application/x-xz, attachment)]

Message #19 received at 977006-done@bugs.debian.org (full text, mbox, reply):

From: Rhonda D'Vine <noreply@salsa.debian.org>

To: 977006-done@bugs.debian.org

Subject: Bug#977006 fixed in www.debian.org

Date: Mon, 23 Aug 2021 22:00:16 +0000

Hello,

Bug #977006 in www.debian.org reported by you has been fixed in the Git repository.
You can see the commit message below and you can check the diff of the fix at:

https://salsa.debian.org/webmaster-team/packages/-/commit/5718bf8656446e68766616ad1ba25f44f1b0b9d6

------------------------------------------------------------------------
Merge branch 'file-list-arch-all' into 'master'

Fix file lists on packages.d.o. (Closes: #977006, #977743, #980888, #923974)

See merge request webmaster-team/packages!20
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/977006

Message #26 received at 977006-done@bugs.debian.org (full text, mbox, reply):

From: Rhonda D'Vine <noreply@salsa.debian.org>

To: 977006-done@bugs.debian.org

Subject: Bug#977006 fixed in www.debian.org

Date: Tue, 25 Jan 2022 06:07:59 +0000

Hello,

Bug #977006 in www.debian.org reported by you has been fixed in the Git repository.
You can see the commit message below and you can check the diff of the fix at:

https://salsa.debian.org/webmaster-team/packages/-/commit/bf85d3091224ee751753c1f43fb5a625313fb31b

------------------------------------------------------------------------
Merge branch 'fix_division_by_zero' into 'master'

Fix division by zero in parse-contents (Closes: #977006)

See merge request webmaster-team/packages!24
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/977006

Send a report that this bug log contains spam.