Debian Bug report logs - #513663
[general] need infrastructure to check related packages

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: [general] need infrastructure to check related packages

Date: Fri, 30 Jan 2009 21:39:31 -0800

Package: lintian
Version: 2.2.0
Severity: wishlist

Lintian currently checks source packages and binary packages in isolation,
except for a few fragile hacks.  However, overwhelmingly Lintian is run in
one of two ways: on a *.changes file, from which the source package and all
binary packages built from it are available, or across the entire archive.

There have been multiple requests that require cross-package checks within
the packages built from the same source package, either between binary
packages or between the binary package and the source package.  Examples
include:

#120323 -- avoid false positives on man pages provided by dependencies
#217023 -- check for dangling symlinks
#513544 -- avoid false positive if upstream ships an empty changelog

When processing a *.changes file, I think it would be reasonable to set up
the lab for all of the packages being processed before running any check
scripts.  The check scripts could then rely on being able to peek at the
labs for all packages generated from the same source package.

lintian.d.o is harder.  Currently, all source packages are checked first
and then left unpacked to level one.  Then, all binary packages are
checked.  This means the source package is always available while checking
the binary packages, but all the binary packages may not be available.

One option would be to unpack all binary packages to level one before
doing any checks, but this requires redoing the unpack work again to get
a level two unpack when actually checking it, making an archive-wide run
much slower.

The best option is probably to add additional smarts to Lintian's
processing and chase references between the packages so that, before
processing a binary package, Lintian ensures that all binary packages
from the same source package are also unpacked in the lab.  I think the
best way to do that would be to more closely simulate the processing
order Lintian uses when processing *.changes files.  Rather than processing
all binary packages one at a time, process them in groups by source
package, unpacking them all, running all checks, and then resetting the
lab to level one.

This bug will track the infrastructure work required to implement this
so that other bugs can block on this bug.

I don't currently have time to do this implementation.  If anyone else
wants to tackle it, it would be a great project.

-- System Information:
Debian Release: 5.0
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable'), (1, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.26-1-686 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages lintian depends on:
ii  binutils            2.18.1~cvs20080103-7 The GNU assembler, linker and bina
ii  diffstat            1.45-2               produces graph of changes introduc
ii  dpkg-dev            1.14.24              Debian package development tools
ii  file                4.26-1               Determines file type using "magic"
ii  gettext             0.17-4               GNU Internationalization utilities
ii  intltool-debian     0.35.0+20060710.1    Help i18n of RFC822 compliant conf
ii  libdigest-sha-perl  5.47-1               Perl extension for SHA-1/224/256/3
ii  libipc-run-perl     0.80-2               Perl module for running processes
ii  libparse-debianchan 1.1.1-2              parse Debian changelogs and output
ii  libtimedate-perl    1.1600-9             Time and date functions for Perl
ii  liburi-perl         1.35.dfsg.1-1        Manipulates and accesses URI strin
ii  man-db              2.5.2-3              on-line manual pager
ii  perl [libdigest-sha 5.10.0-19            Larry Wall's Practical Extraction 

lintian recommends no packages.

Versions of packages lintian suggests:
pn  binutils-multiarch            <none>     (no description available)
ii  libtext-template-perl         1.44-1.2   Text::Template perl module
ii  man-db                        2.5.2-3    on-line manual pager

-- no debconf information

Message #16 received at 513663@bugs.debian.org (full text, mbox, reply):

From: Niels Thykier <niels@thykier.net>

To: 513663@bugs.debian.org

Subject: Re: [general] need infrastructure to check related packages

Date: Mon, 03 Jan 2011 20:34:09 +0100

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hey

I have had a look at this nearly 2 year old bug and I think it would be
great if we could find a solution for cross-package checks. To make it
even better I am interested in working on this. That being said I have
not work on Lintian for very long so ...

Is it correctly asserted that we always check source packages first
because this is the only way we currently have to guarantee that the
information from the source package unpacked when checking a binary
(assuming the source is present at all)? Or is there also another reason
for this ordering (e.g. some undocumented assumption on this)?

~Niels
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJNIiSwAAoJEAVLu599gGRCn+8P/0HD8OGmwZGgLTNlyCq3yPD1
d35H5jz0eVUFJHoRDDx8nMCAt3IxN6PeP556PzWtLdWW3/JF6GGDj11RsaePj9LT
XRicpLDCG8c9lb4k4RFIVeBiuu/IMAa8qlSU17T8i4z5EjyUR1vvIkAeuzAc+AW1
5cqt1w8U0lxosfi2UxRRKAvPY3o+/4xoLGJwDqG5G3A8Zrc8HhKR3qGpPwK2LFYd
CLx7xooVm0MkOjTUv85bKYWfAHb/zts7vFsLoCa7GcoM80oLJ/4yo+IKzVYo0Ezc
offebCQKabmsUYeMeQ0nO4y1GbxIoSL7JYKWDJfQEaj9c3gh13A1ltlxsO/RR9+d
7FZH1Ik8FHR6+++zgaEHJ5+NH6XvrNhcH31Azcujg3rQEvmBw1npUDxuFsaie3pq
GsovMfstVh8+9RYXUk0GLREg8ToA6C/COuE6WVykvjpJVyYWHHYsUORS/zHq6Jhh
z4n1XlCnrxxaLkDTrf794lWsCZPvZpcryG1NzNb0c0R8drW9lJVrBHdnvmWCzXMd
n6yrVGUgbJLQ9aLRTEz1mGIoDeoXrfym0T/eyD6gNg/XSMIb1cvA8uxxqNZ/WNzx
IqGnXpMdq5Etu8xpcSn09lSaOZz8HH+POG/C5XUoNWOtur/8whOKVqqrxCs4o1Q+
s7L19IhLPdLTafroUv3I
=KUd6
-----END PGP SIGNATURE-----

Message #21 received at 513663@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: Niels Thykier <niels@thykier.net>

Cc: 513663@bugs.debian.org

Subject: Re: Bug#513663: [general] need infrastructure to check related packages

Date: Wed, 05 Jan 2011 16:15:39 -0800

Niels Thykier <niels@thykier.net> writes:

> I have had a look at this nearly 2 year old bug and I think it would be
> great if we could find a solution for cross-package checks. To make it
> even better I am interested in working on this. That being said I have
> not work on Lintian for very long so ...

> Is it correctly asserted that we always check source packages first
> because this is the only way we currently have to guarantee that the
> information from the source package unpacked when checking a binary
> (assuming the source is present at all)? Or is there also another reason
> for this ordering (e.g. some undocumented assumption on this)?

It looks like we currently check source packages first in the normal mode
of running lintian on a *.changes file as an artifact of the fact that
dpkg-genchanges always lists the source package first.  The code (in
Lintian::Schedule) doesn't do anything particularly intelligent.

It is definitely the case that we need to check the source package before
the binary packages, or at least unpack the source package before checking
the binary packages, if we want to use metadata about the source package
in the binary package checks.  I think all the cross-package checks we've
run into so far have wanted things in that order: binary checks wanting
things from the source package, but not the source checks wanting things
from binary packages.

If we start explicitly taking advantage of this by doing more
cross-package testing, we should modify Lintian::Schedule to ensure that
all source packages are checked before all binary packages even if people
are doing things like lintian *.deb *.dsc.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>

Message #26 received at 513663@bugs.debian.org (full text, mbox, reply):

From: Niels Thykier <niels@thykier.net>

To: 513663@bugs.debian.org

Subject: Re: Bug#513663: [general] need infrastructure to check related packages

Date: Thu, 06 Jan 2011 01:52:20 +0100

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 2011-01-06 01:15, Russ Allbery wrote:
> Niels Thykier <niels@thykier.net> writes:
> 
>> I have had a look at this nearly 2 year old bug and I think it would be
>> great if we could find a solution for cross-package checks. To make it
>> even better I am interested in working on this. That being said I have
>> not work on Lintian for very long so ...
> 
>> Is it correctly asserted that we always check source packages first
>> because this is the only way we currently have to guarantee that the
>> information from the source package unpacked when checking a binary
>> (assuming the source is present at all)? Or is there also another reason
>> for this ordering (e.g. some undocumented assumption on this)?
> 
> It looks like we currently check source packages first in the normal mode
> of running lintian on a *.changes file as an artifact of the fact that
> dpkg-genchanges always lists the source package first.  The code (in
> Lintian::Schedule) doesn't do anything particularly intelligent.
> 

Really? Judging from Lintian::Schedule I would say it is partly
enforcing this regardless of the output order of dpkg-genchanges

# for each package (the sort is to make sure that source packages are
# before the corresponding binary packages--this has the advantage that
binary
# can use information from the source packages if these are unpacked)
my %type_sort = ('b' => 1, 'u' => 1, 's' => 2, 'c' => 3 );
sub get_all {
    return sort({$type_sort{$b->{type}} <=> $type_sort{$a->{type}}}
		@{$_[0]->{schedule}});
}

> It is definitely the case that we need to check the source package before
> the binary packages, or at least unpack the source package before checking
> the binary packages, if we want to use metadata about the source package
> in the binary package checks.  I think all the cross-package checks we've
> run into so far have wanted things in that order: binary checks wanting
> things from the source package, but not the source checks wanting things
> from binary packages.
> 



> If we start explicitly taking advantage of this by doing more
> cross-package testing, we should modify Lintian::Schedule to ensure that
> all source packages are checked before all binary packages even if people
> are doing things like lintian *.deb *.dsc.
> 

As I understand "cross-package", we are looking at trying to ensure that
all binary/udeb packages produced from a single source package is
unpacked before any of binaries/udebs and even the source itself is
checked. That is:

 lintian A.changes B.changes

then when lintian checks any package from A.changes, then all packages
from A.changes are unpacked to the minimum required level required by
the check being run on the current package.

  If so, I do not see an issue with processing one source package
immediately followed by its binaries/udebs before continuing to the next
source package[1].  Or were you simply mentioning that lintian must
produce the same results for lintian *.dsc *.deb vs. lintian *.deb *.dsc?

About the ordering itself: As long as the source package or the changes
file is passed, we got all the information needed to generate a correct
ordering for these packages.
  But what about multiple binaries without a source? If a user passes
*.deb, should lintian then try to order the packages such that binary
packages built from the same source (as defined in the deb's control
file) are made available before the checks (just as if they had been run
with their source)?

That is, if I run:

 lintian A.deb B.deb A-data.deb B-data.deb

should lintian then do A + A-data and then B + B-data or just in
"whatever" order it feels like?

~Niels

[1] I.e. how is A.dsc B.dsc A.deb B.deb a better order than A.dsc A.deb
B.dsc B.deb?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJNJRJCAAoJEAVLu599gGRCFigP/jfJYkzHrhJ9YTH+UwdcSDqd
MPutWhrm4RR6chUxWgOm8XgRLqBCWDsncC9Xe0U5hdpE3mAEXIWkxpXsyhGsx6G1
gp7OQHKwmP4IhfP9ej9fFcOVMx51XKjSAaGTJIk4yYhgPWU/7WlOzec0AzqWZXIu
uDEcBVLCNqvrI46I40+eizk71AbELQBbU8IAHHaCC0GVcyF/f1h41KL29E/wi7gJ
dtw98p5zCD2fqsMVRxdQREj+hFvEN+UygmMgVybjjQxiMk1jCteOFiBGkR6qEIBF
kj9MDl5DFUNrVUSliz3xy9AsRnyIEEOs9if8hBMsHzNA+f4AvAJt68F4skVm29eZ
TBRZZqDKyvpA7zn5y0qZBKekpM7A0XEl74aV33FnSrl5pEXaqO1tLkGVlimLg3Ue
EABTWuPmi+OSipbQ0T0HHtI9kFa5L4RI2ujs4ErMVhsy0zaHFudi433WV5CaqPrr
rQNq/Nc+wUFmwp3w/MtkaK7PSWLrx6SHeJt9Oz+eqU7+z0iVgPMzgvSr6nwN+KIb
goVmApMQGGUaEUQD6o2M18WG9TxyG1LGe0K59zC986GybMQrVXXs/dY9+TGj/s6S
6+PpMK9PSGD/ZA1mgttLik4CGYSqiGD2Pq66KH4XJUrAuNhTYet2DwSAX8+8kP48
lr6LGtqswWHvjw2wsAcx
=CrtA
-----END PGP SIGNATURE-----

Message #31 received at 513663@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: Niels Thykier <niels@thykier.net>

Cc: 513663@bugs.debian.org

Subject: Re: Bug#513663: [general] need infrastructure to check related packages

Date: Thu, 06 Jan 2011 09:27:12 -0800

Niels Thykier <niels@thykier.net> writes:

> Really? Judging from Lintian::Schedule I would say it is partly
> enforcing this regardless of the output order of dpkg-genchanges

> # for each package (the sort is to make sure that source packages are
> # before the corresponding binary packages--this has the advantage that
> binary
> # can use information from the source packages if these are unpacked)
> my %type_sort = ('b' => 1, 'u' => 1, 's' => 2, 'c' => 3 );
> sub get_all {
>     return sort({$type_sort{$b->{type}} <=> $type_sort{$a->{type}}}
> 		@{$_[0]->{schedule}});
> }

Oh!  I missed that.  I was only looking at how stuff was put into the
schedule, not looking at how it was pulled out, and made the erroneous
assumption last night that it just returned the array.

Excellent.  So it's already coping properly with it.

> As I understand "cross-package", we are looking at trying to ensure that
> all binary/udeb packages produced from a single source package is
> unpacked before any of binaries/udebs and even the source itself is
> checked. That is:

>  lintian A.changes B.changes

> then when lintian checks any package from A.changes, then all packages
> from A.changes are unpacked to the minimum required level required by
> the check being run on the current package.

Yup, correctly.

>   If so, I do not see an issue with processing one source package
> immediately followed by its binaries/udebs before continuing to the next
> source package[1].

Right.

> Or were you simply mentioning that lintian must produce the same results
> for lintian *.dsc *.deb vs. lintian *.deb *.dsc?

Yeah, that's all I was saying, and it looks like we already handle that.

> About the ordering itself: As long as the source package or the changes
> file is passed, we got all the information needed to generate a correct
> ordering for these packages.
>   But what about multiple binaries without a source? If a user passes
> *.deb, should lintian then try to order the packages such that binary
> packages built from the same source (as defined in the deb's control
> file) are made available before the checks (just as if they had been run
> with their source)?

> That is, if I run:

>  lintian A.deb B.deb A-data.deb B-data.deb

> should lintian then do A + A-data and then B + B-data or just in
> "whatever" order it feels like?

I think what we'd need to do is unpack all the packages we're going to
check before we start checking them (provided that we're not running on
the whole archive), since we otherwise won't know the correct sequence
that will let the inter-binary checks work properly.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>

Message #36 received at 513663@bugs.debian.org (full text, mbox, reply):

From: Niels Thykier <niels@thykier.net>

To: 513663@bugs.debian.org

Subject: Re: Bug#513663: [general] need infrastructure to check related packages

Date: Thu, 06 Jan 2011 18:59:12 +0100

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 2011-01-06 18:27, Russ Allbery wrote:
> Niels Thykier <niels@thykier.net> writes:
> [...]
>> That is, if I run:
> 
>>  lintian A.deb B.deb A-data.deb B-data.deb
> 
>> should lintian then do A + A-data and then B + B-data or just in
>> "whatever" order it feels like?
> 
> I think what we'd need to do is unpack all the packages we're going to
> check before we start checking them (provided that we're not running on
> the whole archive), since we otherwise won't know the correct sequence
> that will let the inter-binary checks work properly.
> 

That sounds like an approach we could take (at least if we cannot
trivial determine how the packages are related).  I will probably get
back to this when I am done refactoring/reducing the size of
frontend/lintian.

Speaking of unpacking; as I understand it we once had an "unpack-level
2" which has now been removed in favour of collection.  I am considering
to remove the "unpack-level 1" as well and move everything to collection
as a part of this.
  I feel the unpack code makes the lintian code harder to read and
understand.  If we migrate the last of the unpack stuff to collection I
think we can reduce the complexity of the "PACKAGE: foreach" loop
considerably.
  I.e. It took me a while to figure out that the $unpack_level is either
1 (if $action is "check" or "unpack") or 0 if $action is "remove" in the
"PACKAGE: foreach". No other $action reaches that far and the user
cannot influence $unpack_level (beyond changing $action).

~Niels
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJNJgLwAAoJEAVLu599gGRCsyUQAIDkADGQDPMV4CNBwyE+eQyq
UZV7Z6dJ4v0BeqanzjlAtP/VUG94OKJIaCPNH6M5KL1Bxl+PVYGGgujvEPS+PNQ7
Xx96Sb3CFQHtpBcgDaeWZxqG3qA5ORZXT6qpdEa41TGFEgHu8xXcMjoEoqq6vm2w
Zx39vU+4wMdbW03Cphgfa+DCMANJWGDHYc32vDQM4VJLinshKvIYm8MyNqKKDHYJ
OIImI0PfU5wg8ekT7JxR3FzBm6/h3tSAi+dZoJCoZhN+Bh38xSfnMvDLU4/Ok9tx
BWva3RSKfdS5GBHint4HGh7YqSs4KlhaMg4l3QKDwCJaE9G277eYNiAJQ8gSpaU7
PCXjjMVhjMVfM66+vjjpRUaFjTsz8RD27YtgLtw0az+6cWX3LyCvsNceb4OMkMYT
62tZ9xNG5a9KzquTshfeM9GGAIA0ARrIQnLaMErvf3UQ+wRokHVCwqrr66iZAxFA
dab/g+Mw1FLUJGHC9pm9rGxupoz5moq9SoomHCbZXpcqT+uMKfWZuNoE7Cy9WFZp
SWDDlWK+JLIvcIPULV2K0RuVL9sEEexr6AG9oIPNLOlm7MWOYOWaE4CG9D/P/aTX
7R1nQCVU06l9NwtYfjBYCv1xM5KmSLm8h4LSPt9gCzAyx8pOae4+Tlzs8JeFOFby
TXSIH9Na9HeW2VR0+72W
=00mE
-----END PGP SIGNATURE-----

Message #41 received at 513663@bugs.debian.org (full text, mbox, reply):

From: Raphael Geissert <geissert@debian.org>

To: 513663@bugs.debian.org

Subject: Re: Bug#513663: [general] need infrastructure to check related packages

Date: Thu, 06 Jan 2011 13:01:32 -0600

Niels Thykier wrote:

> Speaking of unpacking; as I understand it we once had an "unpack-level
> 2" which has now been removed in favour of collection.  I am considering
> to remove the "unpack-level 1" as well and move everything to collection
> as a part of this.

The frontend needs a couple of tweaks in order to do that. The rest should 
be straight-forward, I think (but I might be wrong, it's been some months 
since I last looked at that item of my ToDo list.)
The frontend basically assumes that the package directory is already when it 
starts calling the collection scripts, however, the directory is created at 
unpack level 1.

The same tweak would be needed if we want the unpack level 1 replacement 
collection script to take care of removing the directory. The Auto-Remove 
flag should of course not be specified. The collection script needs to be 
called with 'remove-<package type>'

Deciding what collection scripts need to depend on the unpack level 1 
replacement might be tricky, though. Basically they all depend on it.

Cheers,
-- 
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net

Message #46 received at 513663@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: Niels Thykier <niels@thykier.net>

Cc: 513663@bugs.debian.org

Subject: Re: Bug#513663: [general] need infrastructure to check related packages

Date: Thu, 06 Jan 2011 11:18:55 -0800

Niels Thykier <niels@thykier.net> writes:

> Speaking of unpacking; as I understand it we once had an "unpack-level
> 2" which has now been removed in favour of collection.  I am considering
> to remove the "unpack-level 1" as well and move everything to collection
> as a part of this.

>   I feel the unpack code makes the lintian code harder to read and
> understand.  If we migrate the last of the unpack stuff to collection I
> think we can reduce the complexity of the "PACKAGE: foreach" loop
> considerably.

As long as we have some way of marking some collections as transient and
some collections to be retained, and then provide some way of overriding
that, that sounds fine.  I much prefer talking about everything as
collections.  We just need to retain the capability of generating a
Lintian lab for the whole archive without including in the lab all the
unpacked binary and source packages.

>   I.e. It took me a while to figure out that the $unpack_level is either
> 1 (if $action is "check" or "unpack") or 0 if $action is "remove" in the
> "PACKAGE: foreach". No other $action reaches that far and the user
> cannot influence $unpack_level (beyond changing $action).

Yeah, there's a lot of code that does stuff like that in frontend/lintian
that will benefit greatly from refactoring into clearer-defined modules.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>

Message #51 received at 513663@bugs.debian.org (full text, mbox, reply):

From: Niels Thykier <niels@thykier.net>

To: 513663@bugs.debian.org

Subject: Re: Bug#513663: [general] need infrastructure to check related packages

Date: Thu, 06 Jan 2011 23:49:24 +0100

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 2011-01-06 20:18, Russ Allbery wrote:
> Niels Thykier <niels@thykier.net> writes:
> 
>> Speaking of unpacking; as I understand it we once had an "unpack-level
>> 2" which has now been removed in favour of collection.  I am considering
>> to remove the "unpack-level 1" as well and move everything to collection
>> as a part of this.
> 
>>   I feel the unpack code makes the lintian code harder to read and
>> understand.  If we migrate the last of the unpack stuff to collection I
>> think we can reduce the complexity of the "PACKAGE: foreach" loop
>> considerably.
> 
> As long as we have some way of marking some collections as transient and
> some collections to be retained, and then provide some way of overriding
> that, that sounds fine.  I much prefer talking about everything as
> collections.  We just need to retain the capability of generating a
> Lintian lab for the whole archive without including in the lab all the
> unpacked binary and source packages.
> 

Currently the "repack" is never triggered as far as I can tell.  The
reason why lintian properly cleans up is that collections like unpacked
contains the "Auto-removed: yes" line.  This particular line is
currently overruled by the --keep-lab switch (or if the package lab
disappears, but that only happens on a remove action).
  Unless I am mistaken the old unpack layer only exists to run
unpack/unpack-<type>-l1 and remove the entry on a remove action.
Starting with the latter $action eq 'remove' can easily replace the
$unpack_level variable here.
  At least the bin and the changes appears to be "easy" to convert to
collection.  I have not had a full look at the source script, but I
doubt it will be really hard.

There is the one unhandled case - namely if all collection scripts
contains "auto-remove: yes", then the directory itself will not be
removed... but then again we do not currently handle that anyway so it
will not be a regression.

>>   I.e. It took me a while to figure out that the $unpack_level is either
>> 1 (if $action is "check" or "unpack") or 0 if $action is "remove" in the
>> "PACKAGE: foreach". No other $action reaches that far and the user
>> cannot influence $unpack_level (beyond changing $action).
> 
> Yeah, there's a lot of code that does stuff like that in frontend/lintian
> that will benefit greatly from refactoring into clearer-defined modules.
> 

My current effort in the infra-513663 branch has been to encapsulate
packages inside the lab and reduce the complexity of the "PACKAGE: foreach".
  I intend to let these Lab::Package instances take care of the
book-keeping inside its defined little space; my next goal is probably
to make collection book-keeping go in there as well (that is creation
and removal of the ".<script>-<version>" files).
  I think once that is done the "PACKAGE: foreach" will be much simpler
(at least a lot shorter).

~Niels
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJNJkb0AAoJEAVLu599gGRCK2EQAJRLMrWyZqIw/hpENbNIxj/W
sVPXnXYloBcrw/eKhTHOX2Kj1S848d9pvEgBnyCLGiUhLzOAjYzuI9bXmkqrINFJ
EKMy1xcpyQQ2UNtRSBTg76cYG8n39xGQ6hU9MdWedX6lHm6qGy1r7ok/3iVVUvpP
bzAn+AuDbAMGBCGmKuWKFCMjstBUpS/k8CGdYJ9AZgRxcbVbCpknZ742MRRBfwsk
uFgkueLqfTxt/HJtrJNHXmwrt70BuqZw6fxIzczrvlSShPvn8gHcrDMs+WE8G8TQ
ZphJlJPMoUTgZ0jKh/FJcTi4BnSjpiGu1VUisWxo85Vs93e+goXXbUJvaw2CqIIS
Gp2lp3xD1Oc2uNgvXceU5NHONovZT2QsP+zo2rsfI3NkS22ZFC494NKTTKuiMCLz
lHulpq7ARqVwNmCmlLwfeT+rroU+JqLl+yO1NYk2trb8OLV+2Prna6/WkkvnIHhc
PAVo7PlIvpqBRDBJq04eB8BxQ2JFQBLKf0+0Caifs7SX/5c/+Fo4mpxWjehhPzQY
YCMEporaEk4gEjmaUwQhAQaW6MMoxpOHKsTkC74Pxm7tUeAbbVMwzk4nqcyAywck
GSOYIdSIrAQkIESLmDbDRuZYjx/L4MneE1a2H2pnq7Gjlb33E1+xFRuHohscmScS
XCADG5jyjTEzNMlL3mpa
=ESFb
-----END PGP SIGNATURE-----

Message #56 received at 513663@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: 513663@bugs.debian.org

Subject: Re: Bug#513663: [general] need infrastructure to check related packages

Date: Thu, 06 Jan 2011 19:38:29 -0800

Niels Thykier <niels@thykier.net> writes:
> On 2011-01-06 20:18, Russ Allbery wrote:

>> As long as we have some way of marking some collections as transient
>> and some collections to be retained, and then provide some way of
>> overriding that, that sounds fine.  I much prefer talking about
>> everything as collections.  We just need to retain the capability of
>> generating a Lintian lab for the whole archive without including in the
>> lab all the unpacked binary and source packages.

> Currently the "repack" is never triggered as far as I can tell.  The
> reason why lintian properly cleans up is that collections like unpacked
> contains the "Auto-removed: yes" line.  This particular line is
> currently overruled by the --keep-lab switch (or if the package lab
> disappears, but that only happens on a remove action).

Oh, right, I forgot Raphael had already taken care of that.  Okay, that's
all set, then.

> There is the one unhandled case - namely if all collection scripts
> contains "auto-remove: yes", then the directory itself will not be
> removed... but then again we do not currently handle that anyway so it
> will not be a regression.

Yeah, and I don't think we'd ever do that anyway.

> My current effort in the infra-513663 branch has been to encapsulate
> packages inside the lab and reduce the complexity of the "PACKAGE: foreach".
>   I intend to let these Lab::Package instances take care of the
> book-keeping inside its defined little space; my next goal is probably
> to make collection book-keeping go in there as well (that is creation
> and removal of the ".<script>-<version>" files).
>   I think once that is done the "PACKAGE: foreach" will be much simpler
> (at least a lot shorter).

Excellent!

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>

Message #65 received at 513663-close@bugs.debian.org (full text, mbox, reply):

From: Niels Thykier <niels@thykier.net>

To: 513663-close@bugs.debian.org

Subject: Bug#513663: fixed in lintian 2.5.0~rc3

Date: Thu, 21 Apr 2011 12:02:10 +0000

Source: lintian
Source-Version: 2.5.0~rc3

We believe that the bug you reported is fixed in the latest version of
lintian, which is due to be installed in the Debian FTP archive:

lintian_2.5.0~rc3.dsc
  to main/l/lintian/lintian_2.5.0~rc3.dsc
lintian_2.5.0~rc3.tar.gz
  to main/l/lintian/lintian_2.5.0~rc3.tar.gz
lintian_2.5.0~rc3_all.deb
  to main/l/lintian/lintian_2.5.0~rc3_all.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 513663@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Niels Thykier <niels@thykier.net> (supplier of updated lintian package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Thu, 21 Apr 2011 12:29:45 +0200
Source: lintian
Binary: lintian
Architecture: source all
Version: 2.5.0~rc3
Distribution: unstable
Urgency: low
Maintainer: Debian Lintian Maintainers <lintian-maint@debian.org>
Changed-By: Niels Thykier <niels@thykier.net>
Description: 
 lintian    - Debian package checker
Closes: 120323 316283 513663 575447 587925 614879 618587 619075 620120 620829 621658 621667 621681 622124 622396 622974 623031
Changes: 
 lintian (2.5.0~rc3) unstable; urgency=low
 .
   * Summary of tag changes:
     + Added:
       - dir-or-file-in-run
       - intra-source-package-circular-dependency
       - package-contains-broken-symlink
       - classpath-contains-relative-path
       - jar-not-in-usr-share
       - executable-jar-without-main-class
       - missing-dep-on-jarwrapper
       - missing-classpath
       - javalib-but-no-public-jars
       - missing-manifest
       - codeless-jar
       - missing-pre-dependency-on-multiarch-support
 .
   * checks/*.desc:
     + [NT] Updated the Needs-Info field to include the new
       collections where needed.
   * checks/binaries:
     + [NT] Accepted patch from Loïc Minier to support the armhf
       architecture.  (Closes: #618587)
     + [NT] Drop wrong checks for multiarch directories.  Multiarch
       directories are only allowed in packages of the given architecture.
       Thanks to Steve R. Langasek for the patch.
   * checks/circular-deps{,.desc}:
     + [NT] Added to check for circular dependencies between
       binaries from the same source.  It requires all binaries
       packages to be present as well as the source package to be
       effective.  Thanks to Bill Allombert for the suggestion.
       (Closes: #316283)
   * checks/debhelper:
     + [NT] Use new alt-dh_commands data file to fetch alternative
       dependencies for dh_commands, which are sometimes provided
       indirectly by meta or API packages.
   * checks/fields:
     + [NT] Do not emit needless-dependency-on-jre for libX-gcj
       packages and only emit the tag at most once per package.
       Thanks to Rene Engelhard for the report.
       (Closes: #622396)
   * checks/files{,.desc}:
     + [NT] Added dir-or-file-in-run tag.  (Closes: #623031)
     + [NT] New tag missing-pre-dependency-on-multiarch-support,
       Severity: serious, to warn when a package installs libraries to the
       multiarch directory without taking care of upgrades.  Thanks to
       Steve R. Langasek for the patch.
 .
   * checks/java{,.desc}:
     + [NT] Added file based on patches submitted by Vicent Fourmond.
       This new file checks for jar files in weird locations.
       (Closes: #620829, #575447)
   * checks/manpages:
     + [NT] Fixed false-positive binary-without-manpage when the
       manpage is in a direct dependency of the package and the
       package is checked together with its dependency.
       (Closes: #120323)
   * checks/scripts:
     + [NT] Fixed false positive missing-dep-for-interpreter, if
       the interpreter was dash, since dash is now essential.
       (Closes: #620120)
   * checks/symlinks{,.desc}:
     + [NT] New file that checks for broken symlinks.  A symlink
       is considered broken if it does not exist in the package
       itself or in its direct dependecies.  Since only absolute
       symlinks are checked at the moment, this only partly
       fixes #217023.
 .
   * collection/*.desc:
     + [NT] Updated the Needs-Info field to include the new
       collections where needed.
   * collection/{bin-pkg-control,fields,index}{,.desc}:
     + [NT] Added to replace various unpack scripts.
   * collection/java-info{,.desc}:
     + [NT] Accepted patch from Vincent Fourmond to implement
       Java related data collection.
 .
   * data/binaries/multiarch:
     + [NT] Removed by patch from Steve R. Langasek.
   * data/debhelper/alt-dh_commands:
     + [NT] New file; contains alternative dependencies for
       dh_python2 and dh_python3.  (Closes: #614879)
   * data/fields/architectures:
     + [NT] Updated to include armhf.
   * data/files/triplets:
     + [NT] Updated to include armhf triplet.
     + [NT] Refresh with the final approved multiarch paths by patch
       from Steve R. Langasek.
   * data/output/manual-references:
     + [NT] Accepted patch from Vincent Fourmond to add the links
       to the Java Policy.
   * data/spelling/corrections:
     + [NT] Added a lot of spelling mistakes with corrections.
       Kudos to Kevin Ryde for these.  (Closes: #619075)
   * data/spelling/corrections-multiword:
     + [NT] Removed "helps to" as a spelling mistake.  Thanks to
       Nicholas Bamber for the report.  (Closes: #622124)
   * data/standards-version/release-dates:
     + [NT] Added 3.9.2 as the newest Standards-Version.  Thanks to
       Sven Joachim for the report.  (Closes: #621667)
 .
   * debian/control:
     + [NT] Bumped Standards-Version to 3.9.2.
     + [NT] Updated Build-Depends for debiandoc -> docbook change of
       the manual.
     + [NT] Added missing Build-Depends on libhtml-parser-perl.  Also
       added it to suggests, since it is used for XML output.
   * debian/{docs,rules}:
     + [NT] Updated to use/install docbook instead of debiandoc.
 .
   * doc/lintianrc.example:
     + [NT] Removed reference to LINTIAN_UNPACK_LEVEL.
   * doc/lintian.sgml:
     + [RG] Removed file - replaced by doc/lintian.xml.
   * doc/lintian.xml:
     + [RG] Added to migrate away from debiandoc.  (Closes: #587925)
 .
   * frontend/lintian:
     + [NT] Removed the deprecated --unpack-level argument.  Only
       two unpack levels were available and they were equal to
       the --remove and --unpack options.
     + [NT] Stopped accepting the environment/config variables
       LINTIAN_UNPACK_LEVEL and LINTIAN_SECTION.  The former is
       redundant as explained above and the latter was deprecated
       in favor of LINTIAN_AREA.
     + [NT] Refactored the frontend to group packages together based
       on their source package.  This allows for cross-package checks.
       (Closes: #513663)
 .
   * lib/{Lab/Package,Lintian/Processable*}.pm:
     + [NT] Added files to assist in package grouping.
   * lib/Lintian/Command/Simple.pm:
     + [NT] Added support for chanding directory before running the
       command.
   * lib/Lintian/Collect.pm,lib/Lintian/Collect/*.pm:
     + [NT] Removed assumption that all the information handled by
       these are available in the current directory.
   *  lib/Lintian/Schedule.pm:
     + [NT] Removed file, replaced by Lintian::ProcessablePool.
 .
   * private/refresh-archs:
     + [SRL] update to output directory mappings based on DEB_HOST_MULTIARCH
       instead of DEB_HOST_GNU_TYPE.
   * private/runtests:
     + [NT] Added support for dumping build logs if a test fails.
       (Closes: #621658)
     + [NT] Disabled pkgbinarymangler during tests.  (Closes: #621681)
 .
   * t/tests/binaries-from-other-arch:
     + [NT] Accepted patch from Benjamin Drung to accept i686-linux-gnu
       as a valid triplet.  This fixes test failure in Ubuntu.
       (Closes: #622974)
 .
   * unpack/unpack-{binpkg,changes}-l1:
     + [NT] Removed unpack scripts for binpkg and changes as these have
       been replaced by collections.
Checksums-Sha1: 
 6f88eb54d5f8e30ae8a31b1a16dd5d05bc4e686b 2418 lintian_2.5.0~rc3.dsc
 4fa7b3037ab3d1a4edb4088d8043a28e01ed7d2a 923754 lintian_2.5.0~rc3.tar.gz
 3a3f6ff2e475f6c2661eb8b50331fbc099adba4b 580936 lintian_2.5.0~rc3_all.deb
Checksums-Sha256: 
 6c9cfd453337a069ba5ee89999c4fdd13e8217d08f518d4ce16e364e3e26d22e 2418 lintian_2.5.0~rc3.dsc
 b53bd93ba636176c791f3fe6cd0bec1cdc9bd9f5fc4031156a22e0c26501a799 923754 lintian_2.5.0~rc3.tar.gz
 592dbc6bed763bceaef6a6f97d6aad5b211e1e547b00c5a4e1a54a0fce3a0f10 580936 lintian_2.5.0~rc3_all.deb
Files: 
 1bf8adb924eb4af90c844dc3c9d36f03 2418 devel optional lintian_2.5.0~rc3.dsc
 2bb07c53f51974bad1cd18bd34979a2e 923754 devel optional lintian_2.5.0~rc3.tar.gz
 9eea149b15445d0545acb71e18a1e2f2 580936 devel optional lintian_2.5.0~rc3_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBCAAGBQJNsBo2AAoJEAVLu599gGRC6Y0P/0V5DIsqZtaHVr5OCUBKhekf
gBMzqs+4gYwygGfTHMm1NhARD3jrggAW4QPB7yTo0q0PXCObNT43KySwdm1l+5Mt
+k9TUVper+o4uQ2yXHu9QyC1I5ZZ/eHnyhVylJez8SwytA72QldsbnCDQN0xalbe
9tLje+3oUTPjFuax3Z6IbxHMTfPtdGcsdTd8e++NyC8qFeojWtXQ8kqwj8oqFUki
IvmK0KO3BR2jMZDAPFGeHLptfRVpHCDDeK2hajaOdSNCE61nrkBLI5Lrbtaxe9o+
gJVPzJm+o/jz38eO/8PqnIMpzw/AqdX3HSnNcqI3jOhs3JTTZ8ocEtZ/ntytb3NN
pn8cJKAyrMe5Hqp8S2l3QYrGiRavOmJwqJY29/StvbZEXX+I0P4Duia54BvpwxSb
fd/mMl0K903wLIsVZrw/fFjkrDZtNiGhN2SrbsZYJ1XithYFbi0KJGnYg0dVUaPd
T2NVQ1LznxAQ75abszrETa0R9awwAjyVlJaRo2+ufyG5s6VKTmG+sSha5+lLig3p
2GPjT15RhyFTTs60EyQ8FHaKFZIB5O6LWVddujnmWzkVDcC8pw7IBbIRi7k7btkP
UqVLM0ctWdRyZZegLal04mZl54bMRnXhIgPtvseWYZLd6FfM82V8OsrvzRZKl29/
CdSFde2g1ASiz8p7BVlZ
=Uc2S
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.