Debian Bug report logs - #987017
recommends 3 different ways to find obsolete packages, pick one

Reply or subscribe to this bug.

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Antoine Beaupre <anarcat@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: recommends 3 different ways to find obsolete packages, pick one

Date: Thu, 15 Apr 2021 13:24:29 -0400

Package: release-notes
Severity: minor

The release notes, in sections 4.2.2 and 4.8, actually suggest *three*
*different* ways of finding what are essential orphaned packages:

    aptitude search '~o'
    aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
    apt-forktracer | sort

Then I also know of those:

    apt-show-versions | grep -v /bullseye
    aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
    aptitude search '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'

I frankly don't quite know where I stand with all this anymore, but I
am getting the strong feeling we're sending an incoherent message
here. :)

In my personal documentation, I've settled on `apt-forktracer`, but I
suspect we might want to stick with `aptitude search '~obsolete'`
because that matches other documentation in the release notes (and
allows for easy purging).

Is there any reason why we have all that diversity?

What's the right way to do what we actually want here?

-- System Information:
Debian Release: 10.9
  APT prefers stable-debug
  APT policy: (500, 'stable-debug'), (500, 'stable'), (1, 'experimental'), (1, 'unstable'), (1, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.19.0-16-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_CA.UTF-8, LC_CTYPE=fr_CA.UTF-8 (charmap=UTF-8), LANGUAGE=fr_CA.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Message #10 received at 987017@bugs.debian.org (full text, mbox, reply):

From: Antoine Beaupré <anarcat@debian.org>

To: Debian Bug Tracking System <987017@bugs.debian.org>

Subject: Re: recommends 3 different ways to find obsolete packages, pick one

Date: Thu, 15 Apr 2021 13:39:55 -0400

I actually forgot that bullseye itself introduces yet another one:

    apt list ~obsolete

Apparently, those are also a thing:

     comm -23 <(dpkg-query -W -f '${db:Status-Abbrev}\t${Package}\n' | grep '^.[^nc]' | cut -f2 | sort) <(apt-cache dumpavail | sed -rn 's/^Package: (.*)/\1/p' | sort -u)
     apt list --installed | awk -F/ '/\[installed,local\]/{print $1}'

-- 
I've got to design so you can put it together out of garbage cans. In
part because that's what I started from, but mostly because I don’t
trust the industrial structure—they might decide to suppress us
weirdos and try to deny us the parts we need.
                       - Lee Felsenstein

Message #15 received at 987017@bugs.debian.org (full text, mbox, reply):

From: Justin B Rye <justin.byam.rye@gmail.com>

To: 987017@bugs.debian.org

Cc: Antoine Beaupre <anarcat@debian.org>

Subject: Re: Bug#987017: recommends 3 different ways to find obsolete packages, pick one

Date: Thu, 15 Apr 2021 21:08:27 +0100

Antoine Beaupre wrote:
> The release notes, in sections 4.2.2 and 4.8, actually suggest *three*
> *different* ways of finding what are essential orphaned packages:

I don't think you mean "orphan" in either of the senses known to
"https://wiki.debian.org/Glossary#orphan".
 
>     aptitude search '~o'

This is nice and simple, but frankly as an aptitude user I wouldn't
bother.  Instead I'd do what the text just above mentions - launch
aptitude, notice that there was a category for "Obsolete and Locally
Created Packages" (which would tell me I'd somehow lost track of my
kernel packages), and purge everything in that category from the
full-screen interface, without going back to the commandline.

>     aptitude search '?narrow(?installed, ?not(?origin(Debian)))'

This one (apparently an improvement on the '~i(!~ODebian)' search that
was recommended for buster, though the logic is too subtle for me to
remember) is looking for a slightly different thing at a different
stage in the upgrade process: it's part of the section about getting
rid of "non-Debian packages" *before* the upgrade.  The '~o' and
'!~ODebian' searches find different kinds of "unwanted" package.

>     apt-forktracer | sort

I've never quite been able to understand how it is that anybody
could get themselves into the situation of *needing* this specialised
package installed to work around the weirdness of their setup, but
still need to be told what it is that's unusual about their system.
 
> Then I also know of those:
> 
>     apt-show-versions | grep -v /bullseye

This is another package I've never needed to install on my stable
desktop precisely because it's my stable desktop.  If I had a reason
to install it, presumably I'd already know about the reason, and step
one in the bullseye upgrade process should be to get rid of that.

>     aptitude search '?narrow(?installed, ?not(?origin(Debian)))'

Yes, that's the one you listed above.

>     aptitude search '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'

Does that do something similar to the above, but less intelligibly, or
something different?
 
> I frankly don't quite know where I stand with all this anymore, but I
> am getting the strong feeling we're sending an incoherent message
> here. :)

It's two different messages in two different parts of the release
notes.  The first message is roughly "before the upgrade, look at what
you've got installed.  If it's a mix of complex pins and PPA packages
and nonsense like that, start by getting rid of all the crazy stuff".
Unfortunately, this relies on the reader to apply some common sense,
so we've fallen into the trap of replacing "subjective" advice with
"objective" diagnostic commandlines.

The second message is "after the upgrade, throw out all the stuff
that isn't supported any longer".  This really is trivially easy to
automate, as long as you don't confuse it with the previous task.
 
> In my personal documentation, I've settled on `apt-forktracer`,

I'd be interested to know what you find it useful for.

> but I
> suspect we might want to stick with `aptitude search '~obsolete'`
> because that matches other documentation in the release notes (and
> allows for easy purging).

But it isn't looking for the same thing.
 
> Is there any reason why we have all that diversity?
> 
> What's the right way to do what we actually want here?

 [---suture---]
> I actually forgot that bullseye itself introduces yet another one:
>
>     apt list ~obsolete

And indeed for section 4.8, which is dealing with tidying up *after*
the upgrade, it might make sense to recommend the use of apt instead
of aptitude here.

> Apparently, those are also a thing:
>
>      comm -23 <(dpkg-query -W -f '${db:Status-Abbrev}\t${Package}\n' | grep '^.[^nc]' | cut -f2 | sort) <(apt-cache dumpavail | sed -rn 's/^Package: (.*)/\1/p' | sort -u)
>      apt list --installed | awk -F/ '/\[installed,local\]/{print $1}'

If they're not getting shorter, you're going the wrong way.
-- 
JBR	with qualifications in linguistics, experience as a Debian
	sysadmin, and probably no clue about this particular package

Message #20 received at 987017@bugs.debian.org (full text, mbox, reply):

From: Antoine Beaupré <anarcat@debian.org>

To: Justin B Rye <justin.byam.rye@gmail.com>, 987017@bugs.debian.org

Subject: Re: Bug#987017: recommends 3 different ways to find obsolete packages, pick one

Date: Thu, 15 Apr 2021 16:23:12 -0400

On 2021-04-15 21:08:27, Justin B Rye wrote:
> Antoine Beaupre wrote:
>> The release notes, in sections 4.2.2 and 4.8, actually suggest *three*
>> *different* ways of finding what are essential orphaned packages:
>
> I don't think you mean "orphan" in either of the senses known to
> "https://wiki.debian.org/Glossary#orphan".

I should have said "obsolete", I think.

>>     aptitude search '~o'
>
> This is nice and simple, but frankly as an aptitude user I wouldn't
> bother.  Instead I'd do what the text just above mentions - launch
> aptitude, notice that there was a category for "Obsolete and Locally
> Created Packages" (which would tell me I'd somehow lost track of my
> kernel packages), and purge everything in that category from the
> full-screen interface, without going back to the commandline.

I think the point here is that you can do:

    aptitude purge '~o'

... to avoid loading the GUI.

>>     aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
>
> This one (apparently an improvement on the '~i(!~ODebian)' search that
> was recommended for buster, though the logic is too subtle for me to
> remember) is looking for a slightly different thing at a different
> stage in the upgrade process: it's part of the section about getting
> rid of "non-Debian packages" *before* the upgrade.  The '~o' and
> '!~ODebian' searches find different kinds of "unwanted" package.

Maybe it would be worth clarifying that distinction then?

It might help if the former `~o` is expanded to `?obsolete` in the text,
to make it clearer how it compares with the latter.

>>     apt-forktracer | sort
>
> I've never quite been able to understand how it is that anybody
> could get themselves into the situation of *needing* this specialised
> package installed to work around the weirdness of their setup, but
> still need to be told what it is that's unusual about their system.

I actually find the output of apt-forktracer to be quite handy.

>> Then I also know of those:
>> 
>>     apt-show-versions | grep -v /bullseye
>
> This is another package I've never needed to install on my stable
> desktop precisely because it's my stable desktop.  If I had a reason
> to install it, presumably I'd already know about the reason, and step
> one in the bullseye upgrade process should be to get rid of that.

Yeah, although I guess that's the point of those commands: figure out if
there's a mess in there.

I find apt-show-versions to be comparable to apt-forktracer, but a bit
more flexible.

>>     aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
>
> Yes, that's the one you listed above.

Ah yes, sorry for the dupe.

>>     aptitude search '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'
>
> Does that do something similar to the above, but less intelligibly, or
> something different?

I frankly have no idea anymore.

>> I frankly don't quite know where I stand with all this anymore, but I
>> am getting the strong feeling we're sending an incoherent message
>> here. :)
>
> It's two different messages in two different parts of the release
> notes.  The first message is roughly "before the upgrade, look at what
> you've got installed.  If it's a mix of complex pins and PPA packages
> and nonsense like that, start by getting rid of all the crazy stuff".
> Unfortunately, this relies on the reader to apply some common sense,
> so we've fallen into the trap of replacing "subjective" advice with
> "objective" diagnostic commandlines.

I see. Maybe a quick oneliner explanation before the command could help
alleviate that confusion then...

> The second message is "after the upgrade, throw out all the stuff
> that isn't supported any longer".  This really is trivially easy to
> automate, as long as you don't confuse it with the previous task.

Same here, I guess.

>> In my personal documentation, I've settled on `apt-forktracer`,
>
> I'd be interested to know what you find it useful for.

I like that it shows the related versions in the archive, and that it
has very terse output.

>> but I
>> suspect we might want to stick with `aptitude search '~obsolete'`
>> because that matches other documentation in the release notes (and
>> allows for easy purging).
>
> But it isn't looking for the same thing.

Okay, so just stick to aptitude in both cases then, and don't introduce
apt-forktracer. :)

>> Is there any reason why we have all that diversity?
>> 
>> What's the right way to do what we actually want here?
>
>  [---suture---]
>> I actually forgot that bullseye itself introduces yet another one:
>>
>>     apt list ~obsolete
>
> And indeed for section 4.8, which is dealing with tidying up *after*
> the upgrade, it might make sense to recommend the use of apt instead
> of aptitude here.

Yeah, and then we get rid of aptitude in the docs in bullseye +1 :)

So, TL;DR: we should have this before, to find and cleanup foreign
packages:

    aptitude search '?narrow(?installed, ?not(?origin(Debian)))'

Presumably `apt list` might support that as well in bullseye?

Then after the upgrade to bullseye, we use this to find unsupported
/ removed packages:

   apt list ~obsolete

And the release notes could use their own TL;DR: before those commands. :)

>> Apparently, those are also a thing:
>>
>>      comm -23 <(dpkg-query -W -f '${db:Status-Abbrev}\t${Package}\n' | grep '^.[^nc]' | cut -f2 | sort) <(apt-cache dumpavail | sed -rn 's/^Package: (.*)/\1/p' | sort -u)
>>      apt list --installed | awk -F/ '/\[installed,local\]/{print $1}'
>
> If they're not getting shorter, you're going the wrong way.

Definitely. That was mostly meant as a joke.

-- 
Choose a job you love and you will never have to work a day in your life.
                         - Confucius

Message #25 received at 987017@bugs.debian.org (full text, mbox, reply):

From: Justin B Rye <justin.byam.rye@gmail.com>

To: 987017@bugs.debian.org

Cc: Antoine Beaupré <anarcat@debian.org>

Subject: Re: Bug#987017: recommends 3 different ways to find obsolete packages, pick one

Date: Fri, 16 Apr 2021 09:19:35 +0100

Antoine Beaupré wrote
>>>     aptitude search '~o'
>>
>> This is nice and simple, but frankly as an aptitude user I wouldn't
>> bother.  Instead I'd do what the text just above mentions - launch
>> aptitude, notice that there was a category for "Obsolete and Locally
>> Created Packages" (which would tell me I'd somehow lost track of my
>> kernel packages), and purge everything in that category from the
>> full-screen interface, without going back to the commandline.
> 
> I think the point here is that you can do:
> 
>     aptitude purge '~o'
> 
> ... to avoid loading the GUI.

I'd imagine it's to avoid having to deal with that interface - it's
much more powerful (and indeed faster) if you're used to it, but we
don't want novice users to need to guess it's "↓↓↓_gg"!
 
>>>     aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
>>
>> This one (apparently an improvement on the '~i(!~ODebian)' search that
>> was recommended for buster, though the logic is too subtle for me to
>> remember) is looking for a slightly different thing at a different
>> stage in the upgrade process: it's part of the section about getting
>> rid of "non-Debian packages" *before* the upgrade.  The '~o' and
>> '!~ODebian' searches find different kinds of "unwanted" package.
> 
> Maybe it would be worth clarifying that distinction then?

I don't know how we make texts more separate than printing them
in two different places with explanatory paragraphs that talk
about different things!  But let me have a look.

# 4.2.2. Remove non-Debian packages
#
# Below there are two methods for finding installed packages that did
# not come from Debian, using either aptitude or apt-forktracer.
# Please note that neither of them are 100% accurate (e.g. the
# aptitude example will list packages that were once provided by
# Debian but no longer are, such as old kernel packages).

Now that you come to mention it I've always thought that was a bad
example, since after all it isn't exactly a false-positive - old
linux-images really are no-longer-Debian packages, and if you've got
some lying around even before the upgrade, this would be an
appropriate time to get rid of them, as we go on to say a little
later.  Wait, why *is* 4.2.2 before 4.2.3?  shouldn't it be
 * sort out the package database (pending actions or whatever)
 * make sure you're at the latest point release
 * remove any non-Debian packages
 * similarly but separately, remove any obsolete packages
 * tidy any relic configs
 [...]
Maybe this is turning into the sort of disruptive low-priority change
that should wait for next time.
 
> It might help if the former `~o` is expanded to `?obsolete` in the text,
> to make it clearer how it compares with the latter.

Unfortunately people also use "obsolete" to mean "autoremovable"!
And this also makes the search less simple and easier to confuse with
the one in 4.2...
 
[...]
>>> In my personal documentation, I've settled on `apt-forktracer`,
>>
>> I'd be interested to know what you find it useful for.
> 
> I like that it shows the related versions in the archive, and that it
> has very terse output.

Yes, but how do you come to be running a system with non-Debian
repositories in your sources and installing packages to inspect the
gory details without already realising you've done that?

Now that we've got "https://deb.debian.org/debian/", we're close to
being able to say that standard procedure is "for the duration of the
upgrade, comment out any lines that don't match that URL".
 
>>  [---suture---]
>>> I actually forgot that bullseye itself introduces yet another one:
>>>
>>>     apt list ~obsolete
>>
>> And indeed for section 4.8, which is dealing with tidying up *after*
>> the upgrade, it might make sense to recommend the use of apt instead
>> of aptitude here.
> 
> Yeah, and then we get rid of aptitude in the docs in bullseye +1 :)

Aptitude may no longer be recommended for dist-upgrades, but there are
still reasons to prefer it for day-to-day admin on stable systems.
For bullseye-to-bookworm we'll probably want to use apt recipes
wherever possible, with at most a mention that aptitude can also do
this from the fullscreen mode.  But since we also point at 4.8 from
4.2, we can't do that yet; we don't want the extra complication of
having to say "if you haven't upgraded yet then use aptitude".
 
> So, TL;DR: we should have this before, to find and cleanup foreign
> packages:
> 
>     aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
> 
> Presumably `apt list` might support that as well in bullseye?

Afraid not - "E: Regex compilation error".
-- 
JBR	with qualifications in linguistics, experience as a Debian
	sysadmin, and probably no clue about this particular package

Message #30 received at 987017@bugs.debian.org (full text, mbox, reply):

From: Antoine Beaupré <anarcat@debian.org>

To: Justin B Rye <justin.byam.rye@gmail.com>, 987017@bugs.debian.org

Subject: Re: Bug#987017: recommends 3 different ways to find obsolete packages, pick one

Date: Tue, 20 Apr 2021 11:36:30 -0400

On 2021-04-16 09:19:35, Justin B Rye wrote:
> Antoine Beaupré wrote

[...]


>>>>     aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
>>>
>>> This one (apparently an improvement on the '~i(!~ODebian)' search that
>>> was recommended for buster, though the logic is too subtle for me to
>>> remember) is looking for a slightly different thing at a different
>>> stage in the upgrade process: it's part of the section about getting
>>> rid of "non-Debian packages" *before* the upgrade.  The '~o' and
>>> '!~ODebian' searches find different kinds of "unwanted" package.
>> 
>> Maybe it would be worth clarifying that distinction then?
>
> I don't know how we make texts more separate than printing them
> in two different places with explanatory paragraphs that talk
> about different things!  But let me have a look.
>
> # 4.2.2. Remove non-Debian packages
> #
> # Below there are two methods for finding installed packages that did
> # not come from Debian, using either aptitude or apt-forktracer.
> # Please note that neither of them are 100% accurate (e.g. the
> # aptitude example will list packages that were once provided by
> # Debian but no longer are, such as old kernel packages).
>
> Now that you come to mention it I've always thought that was a bad
> example, since after all it isn't exactly a false-positive - old
> linux-images really are no-longer-Debian packages, and if you've got
> some lying around even before the upgrade, this would be an
> appropriate time to get rid of them, as we go on to say a little
> later.  Wait, why *is* 4.2.2 before 4.2.3?  shouldn't it be
>  * sort out the package database (pending actions or whatever)
>  * make sure you're at the latest point release
>  * remove any non-Debian packages
>  * similarly but separately, remove any obsolete packages
>  * tidy any relic configs
>  [...]
> Maybe this is turning into the sort of disruptive low-priority change
> that should wait for next time.

Indeed. It does seem like 4.2.3 should be before 4.2.2.

I am not sure we should tell people to "remove any non-Debian package"
before the upgrade. They might have legitimate reasons to have
third-party package repositories...?

>> It might help if the former `~o` is expanded to `?obsolete` in the text,
>> to make it clearer how it compares with the latter.
>
> Unfortunately people also use "obsolete" to mean "autoremovable"!
> And this also makes the search less simple and easier to confuse with
> the one in 4.2...

I just meant to avoid using the shortcut, obscuring it won't help with
this sort of confusiong.

> [...]
>>>> In my personal documentation, I've settled on `apt-forktracer`,
>>>
>>> I'd be interested to know what you find it useful for.
>> 
>> I like that it shows the related versions in the archive, and that it
>> has very terse output.
>
> Yes, but how do you come to be running a system with non-Debian
> repositories in your sources and installing packages to inspect the
> gory details without already realising you've done that?

You have forgotten, it's been years since you messed around with your
sources.list.

> Now that we've got "https://deb.debian.org/debian/", we're close to
> being able to say that standard procedure is "for the duration of the
> upgrade, comment out any lines that don't match that URL".

I am not sure we want the release notes to standardize on that as
well... But I have considered writing a parser that checks sources.list
Origin headers and fixes things up correctly. Because my checklist is
getting ridiculous:

    : Check for pinned, on hold, packages, and possibly disable &&
    rm -f /etc/apt/preferences /etc/apt/preferences.d/* &&
    rm -f /etc/apt/sources.list.d/backports.debian.org.list &&
    rm -f /etc/apt/sources.list.d/backports.list &&
    rm -f /etc/apt/sources.list.d/bullseye.list &&
    rm -f /etc/apt/sources.list.d/buster-backports.list &&
    rm -f /etc/apt/sources.list.d/experimental.list &&
    rm -f /etc/apt/sources.list.d/incoming.list &&
    rm -f /etc/apt/sources.list.d/proposed-updates.list &&
    rm -f /etc/apt/sources.list.d/sid.list &&
    rm -f /etc/apt/sources.list.d/testing.list &&
    apt update && apt -y upgrade &&

>>>  [---suture---]
>>>> I actually forgot that bullseye itself introduces yet another one:
>>>>
>>>>     apt list ~obsolete
>>>
>>> And indeed for section 4.8, which is dealing with tidying up *after*
>>> the upgrade, it might make sense to recommend the use of apt instead
>>> of aptitude here.
>> 
>> Yeah, and then we get rid of aptitude in the docs in bullseye +1 :)
>
> Aptitude may no longer be recommended for dist-upgrades, but there are
> still reasons to prefer it for day-to-day admin on stable systems.

I'm not arguing for deprecating aptitude altogether, but it would seem
to me that using less tools in the release notes would be better.

> For bullseye-to-bookworm we'll probably want to use apt recipes
> wherever possible, with at most a mention that aptitude can also do
> this from the fullscreen mode.  But since we also point at 4.8 from
> 4.2, we can't do that yet; we don't want the extra complication of
> having to say "if you haven't upgraded yet then use aptitude".

Agreed.

>> So, TL;DR: we should have this before, to find and cleanup foreign
>> packages:
>> 
>>     aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
>> 
>> Presumably `apt list` might support that as well in bullseye?
>
> Afraid not - "E: Regex compilation error".

That's really too bad! ;)

a.
-- 
To be naive and easily deceived is impermissible, today more than
ever, when the prevailing untruths may lead to a catastrophe because
they blind people to real dangers and real possibilities.
                        - Erich Fromm

Message #33 received at 987017@bugs.debian.org (full text, mbox, reply):

From: Andrei POPESCU <andreimpopescu@gmail.com>

To: Antoine Beaupré <anarcat@debian.org>, 987017@bugs.debian.org

Cc: Justin B Rye <justin.byam.rye@gmail.com>

Subject: Re: Bug#987017: recommends 3 different ways to find obsolete packages, pick one

Date: Wed, 21 Apr 2021 10:19:51 +0300

[Message part 1 (text/plain, inline)]

On Ma, 20 apr 21, 11:36:30, Antoine Beaupré wrote:
> 
> I am not sure we should tell people to "remove any non-Debian package"
> before the upgrade. They might have legitimate reasons to have
> third-party package repositories...?

A dist-upgrade is difficult enough, reducing complexity introduced by 
non-Debian packages and/or repositories (of questionable quality) seems 
obvious to me.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser

[signature.asc (application/pgp-signature, inline)]

Message #38 received at 987017@bugs.debian.org (full text, mbox, reply):

From: Paul Gevers <elbrus@debian.org>

To: Antoine Beaupré <anarcat@debian.org>, 987017@bugs.debian.org

Subject: Re: Bug#987017: recommends 3 different ways to find obsolete packages, pick one

Date: Sun, 18 Dec 2022 22:10:15 +0100

[Message part 1 (text/plain, inline)]

Hi

On 15-04-2021 19:39, Antoine Beaupré wrote:
> I actually forgot that bullseye itself introduces yet another one:
> 
>      apt list ~obsolete

But (I'm running bookworm):

paul@mulciber ~/release-team/release-notes $ apt list ~obsolete
Listing... Error!
E: input:2-9: error: Expected end of file
   ~obsolete
     ^^^^^^^

Did this work in the past?

Paul

[OpenPGP_signature (application/pgp-signature, attachment)]

Message #43 received at 987017@bugs.debian.org (full text, mbox, reply):

From: Stuart Prescott <stuart@debian.org>

To: 987017@bugs.debian.org

Subject: Re: Bug#987017: recommends 3 different ways to find obsolete packages, pick one

Date: Mon, 19 Dec 2022 18:46:40 +1100

On 19/12/2022 08:10, Paul Gevers wrote:
> Hi
> 
> On 15-04-2021 19:39, Antoine Beaupré wrote:
>> I actually forgot that bullseye itself introduces yet another one:
>>
>>      apt list ~obsolete
> 
> But (I'm running bookworm):
> 
> paul@mulciber ~/release-team/release-notes $ apt list ~obsolete
> Listing... Error!
> E: input:2-9: error: Expected end of file
>     ~obsolete
>       ^^^^^^^
> 
> Did this work in the past?

Doubt that ever worked as written — that should be « apt list '~o' ». 
(The ~ patterns are single letters followed by arguments.)

To tail back through this bug, there's a mixture of different recipes 
considered here. They do different things and will produce very 
different results depending on whether they are run before or after 
editing the sources.list+apt update step.

"obsolete" packages are packages that are currently installed but that 
package name is not available in any current apt source. The version of 
the package installed can be higher or lower than what is currently 
available.
- If you s/bullseye/bookworm/+apt update, packages that are upgradable 
are not obsolete; the package names that have disappeared completely 
are, obsolete.
- If you installed packages from bullseye-backports and then removed 
bullseye-backports from your sources.list, those packages aren't 
available any more, but are also not obsolete.

Obsolete packages are matched by both the aptitude and apt searches ~o. 
Locally built/manually downloaded packages that are in no apt source 
will also match ~o; locally built/manually downloaded packages where the 
package name is also in an apt repo will _not_ be included.

There's a second collection of packages where the currently installed 
version cannot be located in any available apt repository. While 
'obsolete' packages was only about the package name, this is about the 
(name, version, architecture) tuple. This set of "not available" 
packages includes the yet-to-be upgraded packages, and the ones that are 
somehow at a higher version than what is currently available.


So... the actual commands discussed:


aptitude search '~o'
apt list '~o'
 Both of these look for obsolete packages as defined above. On this 
machine it lists gcc-8-base linux-image-5.16.0-2-amd64-unsigned (left 
over from a previous release and churn in sid) but also "dpkg -i" 
installed packages like crossover and and draw.io (where those package 
names do not exist in Debian). My own backports are not included because 
the package names exist. Likewise packages like rclone I've grabbed the 
upstream .deb and installed it on this machine are not included.

aptitude search 
'?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'
apt list | grep 'installed,local'
 This is a monster (it may well have been reinvented multiple times, 
but I will own up to responsibility for creating it at least one time 
c2009). It's filtering out apt's magic repo of 'now', which accompanies 
all installed packages (a bit like /var/lib/dpkg/status in the output of 
apt-cache policy foo), to find packages that are currently installed but 
for which (name, version, architecture) doesn't exist in a currently 
configured repo. On this machine, it matches some 3rd party packages 
that are installed via dpkg -i and for which there is no apt repo, like 
crossover (just like 'obsolete') as well as local backports that I've 
made that are not sitting in any repo, and those upstream .debs like 
rclone. Backports that I poked into my own local repo are not included 
because they are available. Packages that I installed directly out of 
sid onto this bullseye machine and have never upgraded are also on the 
list because the version part of the (name, version, arch) tuple doesn't 
match. (The treatment of local repositories is slightly different for 
the two versions of this command given above)

aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
apt list '?narrow(?installed, ?not(?origin(Debian)))'
 This looks for packages not currently installable from a repo with 
'Debian' as the name in the Release file, which includes the case where 
(name, version, arch) doesn't match. Because it's matching on the 
version string of the package, it spots that upstream .deb for rclone, 
the local backports, etc. It also adds in 3rd party packages that are in 
apt repositories, so on this machine signal-desktop and slack-desktop. 
Note that lots of 3rd party repositories are misconfigured which is 
unhelpful.


Which of the above is useful depends on the stage of the upgrade. 
Looking for obsolete packages after the upgrade is an important part of 
the cleanup process. Looking for 3rd party packages before the upgrade 
is an important part of the preparation.

Importantly, the output of any of these commands will depend on whether 
the sources.list has been edited; we see that quite frequently in 
#debian, where someone has edited the sources.list for the upgrade and 
is then freaked out by absolutely every single package suddenly being 
'not available'.

cheers
Stuart


-- 
Stuart Prescott   http://www.nanonanonano.net/ stuart@nanonanonano.net
Debian Developer  http://www.debian.org/       stuart@debian.org
GPG fingerprint   90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7

Message #48 received at 987017@bugs.debian.org (full text, mbox, reply):

From: Richard Lewis <richard.lewis.debian@googlemail.com>

To: 987017@bugs.debian.org

Subject: Re: recommends 3 different ways to find obsolete packages, pick one

Date: Mon, 8 May 2023 10:08:16 +0100

On Thu, 15 Apr 2021 13:24:29 -0400 Antoine Beaupre <anarcat@debian.org> wrote:

> The release notes, in sections 4.2.2 and 4.8, actually suggest *three*
> *different* ways of finding what are essential orphaned packages:
>
>     aptitude search '~o'
>     aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
>     apt-forktracer | sort
>

I think this bug has been fixed.

Current text is pretty clear there are two separate issues:
- "non-debian packages" (4.2.2, which has the second two commands):
remove before upgrading
   (and (perhaps no-longer?) mentions orphaned packages at all)
- "removed from debian" (4.8, which has the first command): can only
be done after upgrading

It is perhaps worth considering more clearly splitting "before the release" and
"after the release" actions into separate chapters? (post bullseye?) -
eg 4.7 and 4.8 could be put at the start of chapter 5
as they seem like 'issues to be aware of for the new release' as much
as the are about 'upgrading'.

Message #53 received at 987017@bugs.debian.org (full text, mbox, reply):

From: Manny <debbug.987017@sideload.33mail.com>

To: Debian Bug Tracking System <987017@bugs.debian.org>

Subject: release-notes: Giving many ways to do something *is* useful

Date: Sat, 27 Apr 2024 23:48:50 +0200

Package: release-notes
Followup-For: Bug #987017
X-Debbugs-Cc: debbug.release-notes@sideload.33mail.com

@ Antoine Beaupre

> Is there any reason why we have all that diversity?
> …
> I'm not arguing for deprecating aptitude altogether, but it would seem
> to me that using less tools in the release notes would be better.

As an aptitude user, I was bothered by the lack of aptitude ways of
doing things in the upgrade guide. I had to research and use guesswork
to work out what is the aptitude-equivalent command, which complicated
my effort. For the minimal system upgrade, users were directed to run
“apt upgrade --without-new-pkgs”. I thought why is the aptitude
approach missing?  Is aptitude incapable of this?  I then went to the
man page and found “aptitude safe-upgrade”. I am still not sure if
that is the equivalent command. And if so, is it exacly the same or
are there differences?

Whether someone wants to know a bit of many tools or be a master of
few tools is a matter of preference, but the docs would ideally
accommodate both kinds of users (though not necessarily in the same
doc… that’s another matter - but if the different methods are
side-by-side in the same doc it helps users learn about the
equivalences and makes it easier for them to settle on a preferred
method). But certainly it’s sensible to drop methods that have no
advantage of any kind.

The advice I was given early in my Debian years was that apt-get was a
more primitive command and aptitude was more complete/comprehensive,
that it logs or tracks more things and generally the better tool
according to folks giving IRC support. I think aptitude calls apt
IIRC, which makes it a higher level tool.

> I am not sure we should tell people to "remove any non-Debian package"
> before the upgrade. They might have legitimate reasons to have
> third-party package repositories...?

Concur. I’m not sure what the past release notes said, but the
Bookworm release notes simply bluntly direct users to “Remove
non-Debian packages”. This was frustrating for me. **Why?** I want to
know why I am doing something. The list of non-Debian packages
contained pkgs *I need*. Users need to know why they are directed to
destroy something they need.

Is there a real likeliness that a non-Debian pkg will make a mess or
disaster of the upgrade?  Or is this step a generic “we only
officially support our officially distributed software” scenario?

I decided to go against the guidance. There was one non-Debian pkg
that I no longer used, so removal was a trivial choice for that
one. But I left the other non-Debian pkgs alone. Some of them broke
and some survived.

The guide should probably suggest removing any non-Debian pkgs that
are not needed to mitigate dependency complications, but simply warn
that non-Debian pkgs allowed to persist might not run correctly and
should be also treated with low priority if conflicts arise.

@ Justin B Rye

>>     aptitude search '~o'
>
> This is nice and simple, but frankly as an aptitude user I wouldn't
> bother.  Instead I'd do what the text just above mentions - launch
> aptitude, notice that there was a category for "Obsolete and Locally
> Created Packages"

If the guide is intended to help train the user and advance their
Debian skills, then the CLI advice is probably favorable because it’s
more likely to improve the user’s knowledge than a UI that needs no
manual.

If the guide is intended to just execute steps to do a job (git’r
done), then the CLI is still the winner because it’s just one line for
a user to copy-paste and get instant results with minimal text and
minimal reading.

Briefly mentioning the UI option probably doesn’t hurt though in
addition to the CLI.

>>     apt-forktracer | sort
> 
> I've never quite been able to understand how it is that anybody
> could get themselves into the situation of *needing* this specialised
> package installed to work around the weirdness of their setup, but
> still need to be told what it is that's unusual about their system.
> …
>> In my personal documentation, I've settled on `apt-forktracer`,
>
> I'd be interested to know what you find it useful for.

For me, apt-forktracer gives 1 pkg additional output:

$ apt-forktracer | sort
linux-image-5.10.0-16-amd64 (5.10.127-2)
linux-image-5.10.0-19-amd64 (5.10.149-2)
linux-image-5.10.0-6-amd64 (5.10.28-1)
linux-image-5.10.0-7-amd64 (5.10.40-1)
linux-image-5.10.0-8-amd64 (5.10.46-5)
mastodon-archive (1.4.4-izzy1)
openjdk-11-jre (11.0.23+9-1~deb11u1) [Debian: 11.0.22+7-1~deb11u1]
openjdk-11-jre-headless (11.0.23+9-1~deb11u1) [Debian: 11.0.22+7-1~deb11u1]
ungoogled-chromium (90.0.4430.212-1.sid1)
ungoogled-chromium-common (90.0.4430.212-1.sid1)
ungoogled-chromium-sandbox (90.0.4430.212-1.sid1)
ungoogled-chromium-shell (90.0.4430.212-1.sid1)
wire-desktop (3.35.3348-3348)
xdiskusage (1.60-4~bpo12+1) [Debian Backports: 1.60-4~bpo12+1] [Debian: 1.48-10.1+b1]

The “apt list '?narrow(?installed, ?not(?origin(Debian)))'” command
excludes the last line (xdiskusage). If those two commands are
expected to have identical results, then maybe we have a bug.

>> Apparently, those are also a thing:
>>
>>      comm -23 <(dpkg-query -W -f '${db:Status-Abbrev}\t${Package}\n' | grep '^.[^nc]' | cut -f2 | sort) <(apt-cache dumpavail | sed -rn 's/^Package: (.*)/\1/p' | sort -u)
>>      apt list --installed | awk -F/ '/\[installed,local\]/{print $1}'
>
> If they're not getting shorter, you're going the wrong way.

Yes, assuming the intent of the guide is purely to get a job
done. OTOH, the long versions are a good learning opportunity.

If this were a PDF-formatted textbook, it would be tempting to put the
lengthy but academically rich versions in a distinctly colored
sideline box maybe with a graduation hat or some nerdy symbol in the
corner to make it easy for people in git’r done mode to ignore and
simultaneously easy for curious minds to study.

> Aptitude may no longer be recommended for dist-upgrades, but there are
> still reasons to prefer it for day-to-day admin on stable systems.
> For bullseye-to-bookworm we'll probably want to use apt recipes
> wherever possible, with at most a mention that aptitude can also do
> this from the fullscreen mode.  But since we also point at 4.8 from
> 4.2, we can't do that yet; we don't want the extra complication of
> having to say "if you haven't upgraded yet then use aptitude".

Can you elaborate?  Why would aptitude lose ground on dist upgrades?
Is it deficient in some way?

As an aptitude user, my temptation is to look for the aptitude
approach. So merely omitting aptitude from the guide only encumbers
aptitude users. If there is a good reason for omitting an aptitude
approach, the guide should state why. Otherwise users might question
the quality and comprehensiveness of the guide.

Message #58 received at 987017@bugs.debian.org (full text, mbox, reply):

From: Paul Gevers <elbrus@debian.org>

To: Manny <debbug.987017@sideload.33mail.com>, 987017@bugs.debian.org

Subject: Re: Bug#987017: release-notes: Giving many ways to do something *is* useful

Date: Sun, 28 Apr 2024 08:29:25 +0200

[Message part 1 (text/plain, inline)]

Hi,

[Release Team member hat on]

On 27-04-2024 11:48 p.m., Manny wrote:
> As an aptitude user, I was bothered by the lack of aptitude ways of
> doing things in the upgrade guide.

I anything, I prefer the Release Notes to move to using one tool in the 
instructions, without insinuating that it's the only way. I think that 
tool should be apt nowadays. We've made steps in that direction during 
the last release cycle, i.e. we moved away from aptitude.

> Whether someone wants to know a bit of many tools or be a master of
> few tools is a matter of preference, but the docs would ideally
> accommodate both kinds of users (though not necessarily in the same
> doc… that’s another matter - but if the different methods are
> side-by-side in the same doc it helps users learn about the
> equivalences and makes it easier for them to settle on a preferred
> method). But certainly it’s sensible to drop methods that have no
> advantage of any kind.

I don't think it's the role of the Release Notes to do this. We should 
show a consistent document and use examples that work. Which also means 
we should ensure they remain working. Doing that work for multiple tools 
is extra work I don't want to spend.

> The advice I was given early in my Debian years was that apt-get was a
> more primitive command and aptitude was more complete/comprehensive,

I understood that that used to be the case in the past. apt is now much 
more apt.

> that it logs or tracks more things and generally the better tool
> according to folks giving IRC support. I think aptitude calls apt
> IIRC, which makes it a higher level tool.

Both call libapt-pkg6.0(|t64), both are higher level tools using the 
same library, just like packagekit and synaptic and more.

>> I am not sure we should tell people to "remove any non-Debian package"
>> before the upgrade. They might have legitimate reasons to have
>> third-party package repositories...?
> 
> Concur. I’m not sure what the past release notes said, but the
> Bookworm release notes simply bluntly direct users to “Remove
> non-Debian packages”. This was frustrating for me. **Why?** I want to
> know why I am doing something. The list of non-Debian packages
> contained pkgs *I need*. Users need to know why they are directed to
> destroy something they need.

Ok, so we should give more reason, see below and patches welcome.

> Is there a real likeliness that a non-Debian pkg will make a mess or
> disaster of the upgrade?  Or is this step a generic “we only
> officially support our officially distributed software” scenario?

non-debian package can create situations where apt (the library) can't 
resolve the situation anymore or has more difficulty. And then yes, it 
should be clear you're on your own to solve the situation. I *think* 
that apt can opt for removal of packages in some situations. Normally 
for a pure Debian system, you would expect that to happen for things you 
still need (and a bug would be fair), while if it happens because on 
non-Debian packages, it's unfair to call that a Debian bug.

> I decided to go against the guidance. There was one non-Debian pkg
> that I no longer used, so removal was a trivial choice for that
> one. But I left the other non-Debian pkgs alone. Some of them broke
> and some survived.

Ack.

> The guide should probably suggest removing any non-Debian pkgs that
> are not needed to mitigate dependency complications, but simply warn
> that non-Debian pkgs allowed to persist might not run correctly and
> should be also treated with low priority if conflicts arise.

Agreed. Or might cause other packages to be removed, so extra care and 
attention during the upgrade is needed.

> If the guide is intended to help train the user and advance their
> Debian skills, then the CLI advice is probably favorable because it’s
> more likely to improve the user’s knowledge than a UI that needs no
> manual.

That's not the purpose of the Release Notes.

> As an aptitude user, my temptation is to look for the aptitude
> approach. So merely omitting aptitude from the guide only encumbers
> aptitude users. If there is a good reason for omitting an aptitude
> approach, the guide should state why. Otherwise users might question
> the quality and comprehensiveness of the guide.

We could add a statement that while more tools exist. All automated 
testing of upgrades that I know of use apt-get, so that's the obvious 
choice. aptitude doesn't get as much testing.

Paul

[OpenPGP_signature.asc (application/pgp-signature, attachment)]

Send a report that this bug log contains spam.