Debian Bug report logs - #809441
apt: provide a global resource of upgradeable packages

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Thilo Six <debian@Xk2c.de>

To: Debian submit bugs <submit@bugs.debian.org>

Subject: apt: provide a global resource for upgradeable packages

Date: Wed, 30 Dec 2015 19:23:55 +0100

[Message part 1 (text/plain, inline)]

Package: apt
Version: 1.0.9.8.2
Severity: wishlist

Dear Maintainer,

in Debian there are several tools which calculate the number of
available pkgs updates and the specific data
<pkg> <version_old> <version_new>
on their own.

This results in several tools here that all do the same thing over and over again.
(Update notifier e.g. apper, byobu GNU screen wrapper, to include it in status
line, and so on...)

Local i have a script which does this once everytime 'update' is run.
I attach it.

I would like to ask APT maintainers if you would be interested to include/create
such a global resource which could be easyly reused by other clients then.


PS.
I follow your work on apt 1.1 with great interest! Thank you for that!



kind regards,

     Thilo



-- System Information:
Debian Release: 8.2
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'proposed-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16.0-4-amd64 (SMP w/2 CPU cores)
Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages apt depends on:
ii  debian-archive-keyring  2014.3
ii  gnupg                   1.4.18-7
ii  libapt-pkg4.12          1.0.9.8.2
ii  libc6                   2.19-18+deb8u1
ii  libgcc1                 1:4.9.2-10
ii  libstdc++6              4.9.2-10

apt recommends no packages.

Versions of packages apt suggests:
pn  apt-doc     <none>
ii  aptitude    0.6.11-1+b1
ii  dpkg-dev    1.17.26
ii  python-apt  0.9.3.12
ii  synaptic    0.81.3
ii  wajig       2.17

-- Configuration Files:
/etc/cron.daily/apt changed [not included]

-- no debconf information

[apt-toupgrade (text/plain, attachment)]

Message #12 received at 809441@bugs.debian.org (full text, mbox, reply):

From: Thilo Six <debian@Xk2c.de>

To: 809441@bugs.debian.org

Subject: Re: apt: provide a global resource for upgradeable packages

Date: Wed, 30 Dec 2015 22:30:46 +0100

Thilo Six schrieb/wrote:
> Package: apt
> Version: 1.0.9.8.2
> Severity: wishlist
> 
> Dear Maintainer,
> 
> in Debian there are several tools which calculate the number of
> available pkgs updates and the specific data
> <pkg> <version_old> <version_new>
> on their own.
> 
> This results in several tools here that all do the same thing over and over again.
> (Update notifier e.g. apper, byobu GNU screen wrapper, to include it in status
> line, and so on...)
> 
> Local i have a script which does this once everytime 'update' is run.
> I attach it.

I just wanted to add:
The idea behind it is to do the expensive calculating only once when needed,
e.g. after update, install, remove, purge ...

The additional files created by this script "$DEB_AVAIL_FULL_FILE" and
"$DEB_INSTALLED_FILE" are there to easily reuse that information elsewhere e.g.
completion.

Zsh deb-package completion uses 'command apt-cache --generate pkgnames' to
regenerate the list of packages everytime on the fly (when needed). Although
'command apt-cache --generate pkgnames' is quite fast, simply 'mapfile'ing an
already compiled list is faster:

% time command apt-cache --generate pkgnames
command apt-cache --generate pkgnames  0,09s user 0,20s system 63% cpu 0,459 total

% time ( list2=("${(f@)${mapfile[/var/lib/apt/meta-data/deb-avail-all]%$'\n'}}") )
( list2=("${(f@)${mapfile[/var/lib/apt/meta-data/deb-avail-all]%$'\n'}}") ; )
0,17s user 0,00s system 99% cpu 0,169 total


'mapfile'ing is also possible with Bash.



kind regards,

     Thilo

Message #17 received at 809441@bugs.debian.org (full text, mbox, reply):

From: Julian Andres Klode <jak@debian.org>

To: Thilo Six <debian@Xk2c.de>, 809441@bugs.debian.org

Subject: Re: Bug#809441: apt: provide a global resource for upgradeable packages

Date: Thu, 31 Dec 2015 14:54:32 +0100

On Wed, Dec 30, 2015 at 07:23:55PM +0100, Thilo Six wrote:
> Package: apt
> Version: 1.0.9.8.2
> Severity: wishlist
> 
> Dear Maintainer,
> 
> in Debian there are several tools which calculate the number of
> available pkgs updates and the specific data
> <pkg> <version_old> <version_new>
> on their own.

What does define an available package update for you? The easiest definition
is:

(1) installed package where candidate is newer than installed version

But there is also:

(2) installed package where a newer version exists

and:

(3) installed package that can be upgraded to a newer version.


> This results in several tools here that all do the same thing over and over again.
> (Update notifier e.g. apper, byobu GNU screen wrapper, to include it in status
> line, and so on...)
> 
> Local i have a script which does this once everytime 'update' is run.
> I attach it.
> 
> I would like to ask APT maintainers if you would be interested to include/create
> such a global resource which could be easyly reused by other clients then.

We have an API. An API is there for people to use it. It's not that fast
though, it takes about 400ms to complete for stupid reasons, unless you
are clever -- The one below takes less than 80 ms on my system.

We need to optimise our other API the pkgDepCache, as that incurs a 400ms
overhead or more, as its startup time is linear to the number of dependencies.

-- Source code:

#include <apt-pkg/cachefile.h>
#include <apt-pkg/init.h>
#include <apt-pkg/version.h>
#include <iostream>

int main(void)
{
    struct error_dump {
        ~error_dump() {
            _error->DumpErrors();
        }
    } dumper;
    pkgInitConfig(*_config);
    pkgInitSystem(*_config, _system);

    pkgCacheFile file;

    pkgCache *cache = file.GetPkgCache();
    pkgPolicy *policy = file.GetPolicy();
    int count = 0;
    if (cache == nullptr || policy == nullptr)
        return 1;

    for (auto P = cache->PkgBegin(); !P.end(); ++P) {
        auto I = P.CurrentVer();
        if (I.end())
            continue;

        auto C =policy->GetCandidateVer(P);
        if (C.end())
            continue;
        if (_system->VS->CmpVersion(C.VerStr(), I.VerStr()) > 0)
            count++;
    }

    std::cerr << count << std::endl;
    return 0;
}


-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.

When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to (`inline'). Thank you.

Message #22 received at 809441@bugs.debian.org (full text, mbox, reply):

From: Thilo Six <debian@Xk2c.de>

To: Julian Andres Klode <jak@debian.org>, 809441@bugs.debian.org

Subject: Re: Bug#809441: apt: provide a global resource for upgradeable packages

Date: Fri, 1 Jan 2016 13:11:07 +0100

Hello Julian,

Happy New Year!


>> in Debian there are several tools which calculate the number of
>> available pkgs updates and the specific data
>> <pkg> <version_old> <version_new>
>> on their own.
> 
> What does define an available package update for you? The easiest definition
> is:
> 
> (1) installed package where candidate is newer than installed version
> 
> But there is also:
> 
> (2) installed package where a newer version exists
> 
> and:
> 
> (3) installed package that can be upgraded to a newer version.

Thanks for your answer.
I am a bit overwhelmed by this question actually.

Well my personal use case for this is, is to have a "preview" of what come
around the corner when i next do "full-upgrade".
The sample script i gave has served me quite well in this regard for several
months now. It is actually modeled after:
% wajig toupgrade

Please note with my solution there is one catch though.
As "full-upgrade" can also install additional pkgs or remove them, depending on
that one of the colums <version_old> or <version_new> is empty.

If i have to choose one option from the above i would go for 3.
Expecting it would also respect pinning then?

>> This results in several tools here that all do the same thing over and over again.
>> (Update notifier e.g. apper, byobu GNU screen wrapper, to include it in status
>> line, and so on...)
>>
>> Local i have a script which does this once everytime 'update' is run.
>> I attach it.
>>
>> I would like to ask APT maintainers if you would be interested to include/create
>> such a global resource which could be easyly reused by other clients then.
> 
> We have an API. An API is there for people to use it. It's not that fast
> though, it takes about 400ms to complete for stupid reasons, unless you
> are clever -- The one below takes less than 80 ms on my system.

Well, my opinion here is, also a precompiled file with all the information
needed is also an API.
In the end personally i do not care much about how this API is technical
realised, as long it provides the needed features and that includes speed.
The below code (which i simply do not understand as i do not speak C) probably
is nice, but is difficult to reuse in shell scripts (on its own).
And shell is my #1 tool in this regard.

The idea was to have one simple resource which could straight forward and fast
be used to inform about upcoming updates, e.g. here is the list, formate it to
your will and be done.
Instead of recompiling this list in the first place over and over again, which is:
a) very expensive
b) therefore is quite difficult to handle (even file locking comes into play
   which is a rather challenging task to do in shell).
c) wastes time

A precomiled file with all the informations needed even has the advantage that
inotify could be used, instead of repolling the file over again.

> We need to optimise our other API the pkgDepCache, as that incurs a 400ms
> overhead or more, as its startup time is linear to the number of dependencies.

Yes, that is quite a challenge. Doing a completion in shell on all avail pkgs
without any prefix takes several seconds here (actually feels like minutes).
With prefix it is much faster (at least when prefix != 'lib'  ;)

I realise you have to take care of a lot of things to not break.
I am a bit sad to not being able to help much with the design of this.

Thanks for your answer.


kind regards,

     Thilo


> -- Source code:
> 
> #include <apt-pkg/cachefile.h>
> #include <apt-pkg/init.h>
> #include <apt-pkg/version.h>
> #include <iostream>
> 
> int main(void)
> {
>     struct error_dump {
>         ~error_dump() {
>             _error->DumpErrors();
>         }
>     } dumper;
>     pkgInitConfig(*_config);
>     pkgInitSystem(*_config, _system);
> 
>     pkgCacheFile file;
> 
>     pkgCache *cache = file.GetPkgCache();
>     pkgPolicy *policy = file.GetPolicy();
>     int count = 0;
>     if (cache == nullptr || policy == nullptr)
>         return 1;
> 
>     for (auto P = cache->PkgBegin(); !P.end(); ++P) {
>         auto I = P.CurrentVer();
>         if (I.end())
>             continue;
> 
>         auto C =policy->GetCandidateVer(P);
>         if (C.end())
>             continue;
>         if (_system->VS->CmpVersion(C.VerStr(), I.VerStr()) > 0)
>             count++;
>     }
> 
>     std::cerr << count << std::endl;
>     return 0;
> }

Send a report that this bug log contains spam.