Debian Bug report logs - #977006
packages.debian.org: filelist is currently broken for some packages

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Louis-Philippe Véronneau <pollo@debian.org>

To: submit@bugs.debian.org

Subject: packages.debian.org: filelist is currently broken for some packages

Date: Wed, 9 Dec 2020 18:43:08 -0500

[Message part 1 (text/plain, inline)]

Package: www.debian.org
Severity: important
User: www.debian.org@packages.debian.org
Usertag: packages

Dear maintainers,

The "filelist" feature on packages.debian.org has been broken for a
number for days on some packages. I sadly cannot find a common
denominator. It doesn't seem to be upload-date specific,
language-specific nor architecture-specific.

Only packages in bullseye and sid seem to be affected.

The error message I get is:

-----------------------------------------------------
Error

No such package in this suite on this architecture.
-----------------------------------------------------

Here are some examples:

https://packages.debian.org/sid/all/supysonic/filelist
https://packages.debian.org/bullseye/all/isbg/filelist
https://packages.debian.org/bullseye/all/libprismatic-schema-clojure/filelist

At first I though the problem might be upload-date specific, but firefox
was uploaded to unstable on 2020/11/18 (after isbg) and filelist still
works:

https://packages.debian.org/sid/amd64/firefox/filelist

It's not a question of architecture either. For example, xorg-server
works, was uploaded to buster on 2020/12/01 and is an arch "all" package:

https://packages.debian.org/buster/all/xorg-server-source/filelist

Without access to server logs, I don't think I can really debug this
further :(

-- 
  ⢀⣴⠾⠻⢶⣦⠀
  ⣾⠁⢠⠒⠀⣿⡁  Louis-Philippe Véronneau
  ⢿⡄⠘⠷⠚⠋   pollo@debian.org / veronneau.org
  ⠈⠳⣄

[OpenPGP_signature (application/pgp-signature, attachment)]

Message #10 received at 977006@bugs.debian.org (full text, mbox, reply):

From: Louis-Philippe Véronneau <pollo@debian.org>

To: 977006@bugs.debian.org

Subject: Re: packages.debian.org: filelist is currently broken for some packages

Date: Thu, 10 Dec 2020 00:32:53 -0500

[Message part 1 (text/plain, inline)]

Ok, so more debug info, as apparently as a DD I do have access to
picconi.debian.org

1. The file list data is stored in Berkeley DBs in
srv/packages.debian.org/files/db/contents/filelists_${suite}_${arch}.db

2. The packages I referenced before are missing, but only in certain
archs. For example, both "supysonic" and "isbg" are arch "all" packages,
but only end up in:

* filelists_sid_alpha.db
* filelists_sid_riscv64.db
* filelists_sid_ppc64.db
* filelists_sid_sh4.db
* filelists_sid_x32.db
* filelists_sid_hppa.db
* filelists_sid_sparc64.db
* filelists_sid_m68k.db

libprismatic-schema-clojure (also arch "all) ends up in:

* filelists_sid_alpha.db
* filelists_sid_riscv64.db
* filelists_sid_ppc64.db
* filelists_sid_sh4.db
* filelists_sid_x32.db
* filelists_sid_hppa.db
* filelists_sid_sparc64.db
* filelists_sid_m68k.db
* filelists_sid_mips.db
* filelists_sid_hurd-i386.db
* filelists_sid_powerpcspe.db
* filelists_sid_kfreebsd-amd64.db
* filelists_sid_kfreebsd-i386.db

So it's not a fixed list of archs being processed correctly.

3. _Something_ clearly is wrong with the filelists.db files generated,
as some are _much_ smaller that other. For example, one would expect
amd64 to be the largest file...

31M	../filelists_sid_all.db
31M	../filelists_sid_armel.db
31M	../filelists_sid_s390x.db
32M	../filelists_sid_mips64el.db
32M	../filelists_sid_mipsel.db
33M	../filelists_sid_armhf.db
34M	../filelists_sid_ppc64el.db
35M	../filelists_sid_i386.db
36M	../filelists_sid_amd64.db
81M	../filelists_sid_s390.db
84M	../filelists_sid_ia64.db
89M	../filelists_sid_sparc.db
107M	../filelists_sid_powerpc.db
110M	../filelists_sid_hurd-i386.db
111M	../filelists_sid_kfreebsd-amd64.db
111M	../filelists_sid_kfreebsd-i386.db
117M	../filelists_sid_arm64.db
121M	../filelists_sid_mips.db
122M	../filelists_sid_sh4.db
123M	../filelists_sid_powerpcspe.db
127M	../filelists_sid_hppa.db
127M	../filelists_sid_m68k.db
128M	../filelists_sid_alpha.db
128M	../filelists_sid_riscv64.db
128M	../filelists_sid_sparc64.db
129M	../filelists_sid_x32.db
131M	../filelists_sid_ppc64.db

4. /srv/packages.debian.org/bin/parse-contents L101-181 is the code
responsible for generating the filelists.db files.

5. /srv/packages.debian.org/bin/daily is the script calling
parse-contents. My guess is it's called by a cron job or triggers on an
archive refresh, but I don't have the permissions to find out.

6. looking at the cron logs in /srv/packages.debian.org/files/logs,
parse-contents seems to run correctly and update the filelists on a
frequent basis without outputting errors in the log.

7. the local debian mirror seems to be up to date, so the error doesn't
come from there.

As to why those packages aren't listed in the filelists, no clue!

Hopefully that'll help someone if I don't get around fixing this bug
myself (chances I won't...)

-- 
  ⢀⣴⠾⠻⢶⣦⠀
  ⣾⠁⢠⠒⠀⣿⡁  Louis-Philippe Véronneau
  ⢿⡄⠘⠷⠚⠋   pollo@debian.org / veronneau.org
  ⠈⠳⣄

[OpenPGP_signature (application/pgp-signature, attachment)]

Message #17 received at 977006@bugs.debian.org (full text, mbox, reply):

From: Felix Lechner <felix.lechner@lease-up.com>

To: 977006@bugs.debian.org, 977743@bugs.debian.org

Cc: Louis-Philippe Véronneau <pollo@debian.org>, "Rebecca N. Palmer" <rebecca_palmer@zoho.com>, Adam D Barratt <adam@adam-barratt.org.uk>

Subject: Re: packages.debian.org: file list broken for some packages

Date: Wed, 11 Aug 2021 20:10:06 -0700

[Message part 1 (text/plain, inline)]

Control: tags -1 + patch

Hi Rebecca,
Hi Louis-Philippe,

> On the packages.d.o pages of arch:all packages from sid, bullseye or
> experimental, the "list of files" link gives the error message "No such
> package in this suite on this architecture."

First of all, thank you both for the excellent detective work! The
issue was caused by commit 81824d23 in daklib [1] in which the archive
started to provide—on Oct 25, 2020 7:08am PDT and for releases past
buster—separate Contents files containing the file paths in Arch:all
packages.

From what I can tell, the code generating the web pages for
packages.d.o did not read those files for releases post buster.

I filed a merge request that I believe solves the issue. [2] It was
tested on the Debian node that generates the web pages
(picconi.debian.org). It comes with two caveats:

(1) Due to insufficient permissions I created an improvised
environment, described further below, that may not fully mimic
production runs.

(2) The second commit addresses a condition that should have prevented
the code from performing at all, although apparently it didn't. That
opens up the possibility that I misunderstood the existing code and,
for my tests, created a runtime environment that differed appreciably
from production.

To test the MR, I cloned my feature branch into my home directory on
picconi.debian.org. I then applied the local patch below this message.

Next I ran the command './bin/setup-site /home/lechner/packages
packages.debian.org' as suggested in ./INSTALL and started the test
with '/home/lechner/packages/cron.d/200process_archive'. (I also
created the folders './files/db' and './tmp' in the base directory of
the Git repo, which was my working directory.) The run finished
without errors and produced the attached log.

Now the databases are more even in size across architectures. Here is
a partial listing of the relevant folder ./files/db: (The full listing
for *.db is attached.)

   0 filelists_sid_all.db
129M filelists_sid_alpha.db
132M filelists_sid_amd64.db
208M filelists_sid_arm64.db
126M filelists_sid_armel.db
128M filelists_sid_armhf.db
128M filelists_sid_hppa.db
131M filelists_sid_i386.db
128M filelists_sid_m68k.db
127M filelists_sid_mips64el.db
127M filelists_sid_mipsel.db
123M filelists_sid_powerpcspe.db
132M filelists_sid_ppc64.db
129M filelists_sid_ppc64el.db
130M filelists_sid_riscv64.db
127M filelists_sid_s390x.db
125M filelists_sid_sh4.db
129M filelists_sid_sparc64.db
130M filelists_sid_x32.db

All packages for Arch:all are symbolic links (human size zero). I am
not sure why arm64 is so large.

Perhaps someone with the appropriate user privileges could pull my
feature branch from the merge request [2] into
/srv/packages.debian.org and test it on the live system. The cron run
can be triggered by hand.

A better long-term solution would be to produce separate transfer
files for Arch:all, but that may not work until buster is being
dropped from the archive. Thank you both for your hard work!

Kind regards
Felix Lechner

[1] https://salsa.debian.org/ftp-team/dak/-/commit/81824d2326f5cc50fdcb95c81f9f26864aebaa15
[2] https://salsa.debian.org/webmaster-team/packages/-/merge_requests/20

* * *

[local patch]

lechner@picconi:~/packages$ git diff
diff --git a/bin/parse-contents b/bin/parse-contents
index a1bfc35..7c5f166 100755
--- a/bin/parse-contents
+++ b/bin/parse-contents
@@ -51,6 +51,9 @@ my @sections = @SECTIONS;
 # Add empty section, need to search Contents directly at dist root,
for debports compat
 push(@sections, "");

+$DBDIR = "/home/lechner/packages/files/db";
+my $TMPDIR = "/home/lechner/packages/tmp";
+
 my %debports_hash;
 # copy from config.sh ${arch_debports}
 @debports_hash{qw( alpha hppa ia64 m68k powerpcspe ppc64 riscv64 sh4
sparc64 x32 )} = ();
@@ -166,9 +169,9 @@ for my $suite (@suites) {

     # Piping from sort's output doesn't really scale with 16 GB worth
     # of input, so let's store in a temporary file:
-    my $rev_path_file = "$TOPDIR/tmp/${suite}.sorted";
+    my $rev_path_file = "$TMPDIR/${suite}.sorted";
     print "Merging reverse path lists for ${suite}...\n";
-    system("sort -T $TOPDIR/tmp -m $DBDIR/reverse_${suite}_*.txt -o
${rev_path_file}") == 0
+    system("sort -T $TMPDIR -m $DBDIR/reverse_${suite}_*.txt -o
${rev_path_file}") == 0
        or die "Failed to build merged list";
     my $rev_path_size = stat($rev_path_file)->size;

diff --git a/cron.d/200process_archive b/cron.d/200process_archive
index 29a7385..eecd412 100755
--- a/cron.d/200process_archive
+++ b/cron.d/200process_archive
@@ -5,13 +5,13 @@
 cd "$topdir"

 date
-./bin/parse-translations --english-only
-date
-./bin/parse-packages
-date
-./bin/parse-sources
-date
-./bin/parse-translations
-date
-./bin/parse-contents
+#./bin/parse-translations --english-only
+#date
+#./bin/parse-packages
+#date
+#./bin/parse-sources
+#date
+#./bin/parse-translations
+#date
+/home/lechner/packages/bin/parse-contents
 date

* * *

[parse-contents.log.xz (application/octet-stream, attachment)]

[db-listing.txt.xz (application/x-xz, attachment)]

Message #26 received at 977006-done@bugs.debian.org (full text, mbox, reply):

From: Rhonda D'Vine <noreply@salsa.debian.org>

To: 977006-done@bugs.debian.org

Subject: Bug#977006 fixed in www.debian.org

Date: Mon, 23 Aug 2021 22:00:16 +0000

Hello,

Bug #977006 in www.debian.org reported by you has been fixed in the Git repository.
You can see the commit message below and you can check the diff of the fix at:

https://salsa.debian.org/webmaster-team/packages/-/commit/5718bf8656446e68766616ad1ba25f44f1b0b9d6

------------------------------------------------------------------------
Merge branch 'file-list-arch-all' into 'master'

Fix file lists on packages.d.o. (Closes: #977006, #977743, #980888, #923974)

See merge request webmaster-team/packages!20
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/977006

Message #35 received at 977006-done@bugs.debian.org (full text, mbox, reply):

From: Rhonda D'Vine <noreply@salsa.debian.org>

To: 977006-done@bugs.debian.org

Subject: Bug#977006 fixed in www.debian.org

Date: Mon, 23 Aug 2021 22:00:16 +0000

Hello,

Bug #977006 in www.debian.org reported by you has been fixed in the Git repository.
You can see the commit message below and you can check the diff of the fix at:

https://salsa.debian.org/webmaster-team/packages/-/commit/4b9e2a852096626f61b281f89f57fa9a49968f84

------------------------------------------------------------------------
Concatenate results from Contents-${arch}.gz and Contents-all.gz when the latter is present. (Closes: #977006, #977743, #980888)

Since commit 81824d23 in daklib, the archive provides separate Contents files
with packages from Arch:all for some suites. [1] (All suites post-buster appear to
be affected.) Those file lists were not scanned for the web pages powering
packages.d.o, and the file lists were unavailable online.

This commit attempts to resolve the issue by parsing the file for Arch:all
immediately afterwards, if it is present. The results are effectively
concatenated.

A better long-term solution might be to produce separate transfer files for
Arch:all, but that may not work until buster is dropped from the archive.

[1] https://salsa.debian.org/ftp-team/dak/-/commit/81824d2326f5cc50fdcb95c81f9f26864aebaa15
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/977006

Message #44 received at 977006@bugs.debian.org (full text, mbox, reply):

From: Vipul Kumar <kumar@onenetbeyond.org>

To: 977006@bugs.debian.org

Subject: Re: packages.debian.org: file list broken for some Sid packages

Date: Wed, 10 Nov 2021 11:12:50 +0000

Control: reopen -1 !

Hi,

The "filelist" feature on packages.debian.org is still broken for some 
Sid's packages. Patch indeed worked for Bullseye and Bookworm (neither 
of which worked before), but somehow not for Sid. For example, 
https://packages.debian.org/bookworm/all/golang-github-oklog-ulid-dev/filelist 
is working, but 
https://packages.debian.org/sid/all/golang-github-oklog-ulid-dev/filelist is 
not.


Cheers,
Vipul

Message #53 received at 977006@bugs.debian.org (full text, mbox, reply):

From: Jochen Sprickerhof <jspricke@debian.org>

To: Vipul Kumar <kumar@onenetbeyond.org>, 977006@bugs.debian.org

Subject: Re: Bug#977006: packages.debian.org: file list broken for some Sid packages

Date: Tue, 11 Jan 2022 10:15:40 +0100

[Message part 1 (text/plain, inline)]

* Vipul Kumar <kumar@onenetbeyond.org> [2021-11-10 11:12]:
>The "filelist" feature on packages.debian.org is still broken for some 
>Sid's packages. Patch indeed worked for Bullseye and Bookworm (neither 
>of which worked before), but somehow not for Sid. For example, https://packages.debian.org/bookworm/all/golang-github-oklog-ulid-dev/filelist 
>is working, but https://packages.debian.org/sid/all/golang-github-oklog-ulid-dev/filelist 
>is not.

This seems due to a division by zero in the script, I've opened a MR 
with a patch:

https://salsa.debian.org/webmaster-team/packages/-/merge_requests/24

Cheers Jochen

[signature.asc (application/pgp-signature, inline)]

Message #58 received at 977006-done@bugs.debian.org (full text, mbox, reply):

From: Rhonda D'Vine <noreply@salsa.debian.org>

To: 977006-done@bugs.debian.org

Subject: Bug#977006 fixed in www.debian.org

Date: Tue, 25 Jan 2022 06:07:59 +0000

Hello,

Bug #977006 in www.debian.org reported by you has been fixed in the Git repository.
You can see the commit message below and you can check the diff of the fix at:

https://salsa.debian.org/webmaster-team/packages/-/commit/bf85d3091224ee751753c1f43fb5a625313fb31b

------------------------------------------------------------------------
Merge branch 'fix_division_by_zero' into 'master'

Fix division by zero in parse-contents (Closes: #977006)

See merge request webmaster-team/packages!24
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/977006

Message #67 received at 977006-done@bugs.debian.org (full text, mbox, reply):

From: Rhonda D'Vine <noreply@salsa.debian.org>

To: 977006-done@bugs.debian.org

Subject: Bug#977006 fixed in www.debian.org

Date: Tue, 25 Jan 2022 06:07:59 +0000

Hello,

Bug #977006 in www.debian.org reported by you has been fixed in the Git repository.
You can see the commit message below and you can check the diff of the fix at:

https://salsa.debian.org/webmaster-team/packages/-/commit/6d0e431318631f83964fdb08d557490558f46514

------------------------------------------------------------------------
Fix division by zero in parse-contents (Closes: #977006)

Illegal division by zero at ./bin/parse-contents line 64, <CONT> line 250000.

gzip -l archive/debports/sid/Contents-alpha.gz
         compressed        uncompressed  ratio uncompressed_name
           10690116                   0   0.0% archive/debports/sid/Contents-alpha
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/977006

Message #68 received at 977006-done@bugs.debian.org (full text, mbox, reply):

From: Rhonda D'Vine <noreply@salsa.debian.org>

To: 977006-done@bugs.debian.org

Subject: Bug#977006 fixed in www.debian.org

Date: Tue, 25 Jan 2022 06:17:14 +0000

Hello,

Bug #977006 in www.debian.org reported by you has been fixed in the Git repository.
You can see the commit message below and you can check the diff of the fix at:

https://salsa.debian.org/webmaster-team/packages/-/commit/bf85d3091224ee751753c1f43fb5a625313fb31b

------------------------------------------------------------------------
Merge branch 'fix_division_by_zero' into 'master'

Fix division by zero in parse-contents (Closes: #977006)

See merge request webmaster-team/packages!24
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/977006

Message #69 received at 977006-done@bugs.debian.org (full text, mbox, reply):

From: Rhonda D'Vine <noreply@salsa.debian.org>

To: 977006-done@bugs.debian.org

Subject: Bug#977006 fixed in www.debian.org

Date: Tue, 25 Jan 2022 06:17:14 +0000

Hello,

Bug #977006 in www.debian.org reported by you has been fixed in the Git repository.
You can see the commit message below and you can check the diff of the fix at:

https://salsa.debian.org/webmaster-team/packages/-/commit/6d0e431318631f83964fdb08d557490558f46514

------------------------------------------------------------------------
Fix division by zero in parse-contents (Closes: #977006)

Illegal division by zero at ./bin/parse-contents line 64, <CONT> line 250000.

gzip -l archive/debports/sid/Contents-alpha.gz
         compressed        uncompressed  ratio uncompressed_name
           10690116                   0   0.0% archive/debports/sid/Contents-alpha
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/977006

Send a report that this bug log contains spam.