Debian Bug report logs - #678227
[apt] Indices updates download descriptions in all languages by default

version graph

Package: apt; Maintainer for apt is APT Development Team <deity@lists.debian.org>; Source for apt is src:apt.

Reported by: Filipus Klutiero <chealer@gmail.com>

Date: Wed, 20 Jun 2012 03:06:01 UTC

Severity: important

Found in version apt/0.9.6

Fixed in version apt/0.9.7.5

Done: Michael Vogt <mvo@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Wed, 20 Jun 2012 03:06:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Filipus Klutiero <chealer@gmail.com>:
New Bug report received and forwarded. Copy sent to APT Development Team <deity@lists.debian.org>. (Wed, 20 Jun 2012 03:06:04 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Filipus Klutiero <chealer@gmail.com>
To: submit@bugs.debian.org
Subject: [apt] Indices updates download descriptions in all languages by default
Date: Tue, 19 Jun 2012 23:02:15 -0400
Package: apt
Version: 0.9.6
Severity: normal

With the modularization of package descriptions in distinct index files, 
there is now one file for each language which has some descriptions. By 
default, APT fetches all of these. Since the protocol is not optimized 
for this, indices update are now pretty slow, even when nothing changed.

The Acquire::Languages configuration item controls which languages are 
downloaded. By default, "environment" and English are (explicitly) 
downloaded. For example, in a French system, Acquire::Languages would 
normally request (explicitly) French and English description files to be 
downloaded. But as explained in apt.conf(5), files for more languages 
may be downloaded:
>  Note: To prevent problems resulting from APT being executed in 
> different environments (e.g. by different users or by other programs) 
> all Translation files which are found in /var/lib/apt/lists/ will be 
> added to the end of the list (after an implicit "none").

This is hacky, but not a particularly bad hack in practice, since most 
systems do not use more than 2 languages.

But since /var/lib/apt/lists/ initially contains the files for all 
languages, files for all languages are downloaded/updated, by default. 
This happens even if a single language is hardcoded as 
Acquire::Language's value.

The workaround is to remove unwanted descriptions files from 
/var/lib/apt/lists/:

# rm /var/lib/apt/lists/*_Translation-*


I imagine the fix is to only have files for the requested languages in 
/var/lib/apt/lists/ after installation.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=641967 has information 
about a similar bug where description files would slow indices updates.





Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Mon, 27 Aug 2012 17:45:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Samuel Thibault <sthibault@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Mon, 27 Aug 2012 17:45:06 GMT) Full text and rfc822 format available.

Message #10 received at 678227@bugs.debian.org (full text, mbox):

From: Samuel Thibault <sthibault@debian.org>
To: 641967@bugs.debian.org, 678227@bugs.debian.org
Cc: debian-release@lists.debian.org
Subject: Apt downloads all description translations
Date: Mon, 27 Aug 2012 19:43:23 +0200
Control: tags 641967 + important
Control: tags 678227 + important

Hello,

The issue also happens at debian-installer time, thus downloading way
more than necessary, and taking a lot of disk space (seen 82M here) for
no reason, thus moving the "embedded" barrier yet higher for Debian.

Samuel



Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Mon, 27 Aug 2012 21:45:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Filipus Klutiero <chealer@gmail.com>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Mon, 27 Aug 2012 21:45:03 GMT) Full text and rfc822 format available.

Message #15 received at 678227@bugs.debian.org (full text, mbox):

From: Filipus Klutiero <chealer@gmail.com>
To: Samuel Thibault <sthibault@debian.org>
Cc: 678227@bugs.debian.org
Subject: Re: Bug#641967: Apt downloads all description translations
Date: Mon, 27 Aug 2012 17:42:05 -0400
On 2012-08-27 13:43, Samuel Thibault wrote:
> Control: tags 641967 + important
> Control: tags 678227 + important
>
> Hello,
>
> The issue also happens at debian-installer time, thus downloading way
> more than necessary, and taking a lot of disk space (seen 82M here) for
> no reason, thus moving the "embedded" barrier yet higher for Debian.
>
> Samuel
>
Thanks Samuel, but I suppose you meant "severity foo important".



Severity set to 'important' from 'normal' Request was from Samuel Thibault <sthibault@debian.org> to control@bugs.debian.org. (Mon, 27 Aug 2012 21:57:05 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Mon, 27 Aug 2012 22:00:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Samuel Thibault <sthibault@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Mon, 27 Aug 2012 22:00:03 GMT) Full text and rfc822 format available.

Message #22 received at 678227@bugs.debian.org (full text, mbox):

From: Samuel Thibault <sthibault@debian.org>
To: Filipus Klutiero <chealer@gmail.com>
Cc: 678227@bugs.debian.org
Subject: Re: Bug#641967: Apt downloads all description translations
Date: Mon, 27 Aug 2012 23:52:30 +0200
Filipus Klutiero, le Mon 27 Aug 2012 17:42:05 -0400, a écrit :
> On 2012-08-27 13:43, Samuel Thibault wrote:
> >Control: tags 641967 + important
> >Control: tags 678227 + important
> 
> Thanks Samuel, but I suppose you meant "severity foo important".

Right, thanks :)

Samuel



Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Tue, 28 Aug 2012 08:27:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Cyril Brulebois <kibi@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Tue, 28 Aug 2012 08:27:05 GMT) Full text and rfc822 format available.

Message #27 received at 678227@bugs.debian.org (full text, mbox):

From: Cyril Brulebois <kibi@debian.org>
To: Samuel Thibault <sthibault@debian.org>, 641967@bugs.debian.org, 678227@bugs.debian.org, debian-release@lists.debian.org
Subject: Re: Apt downloads all description translations
Date: Tue, 28 Aug 2012 10:23:28 +0200
[Message part 1 (text/plain, inline)]
Samuel Thibault <sthibault@debian.org> (27/08/2012):
> The issue also happens at debian-installer time, thus downloading way
> more than necessary, and taking a lot of disk space (seen 82M here)
> for no reason, thus moving the "embedded" barrier yet higher for
> Debian.

This is additionally annoyingly slow, it would be nice to have a fix, at
least as far as d-i is concerned.

Mraw,
KiBi.
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Tue, 28 Aug 2012 08:33:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to David Kalnischkies <kalnischkies+debian@gmail.com>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Tue, 28 Aug 2012 08:33:03 GMT) Full text and rfc822 format available.

Message #32 received at 678227@bugs.debian.org (full text, mbox):

From: David Kalnischkies <kalnischkies+debian@gmail.com>
To: Samuel Thibault <sthibault@debian.org>, 678227@bugs.debian.org, debian-release@lists.debian.org
Subject: Re: Bug#641967: Apt downloads all description translations
Date: Tue, 28 Aug 2012 10:31:23 +0200
On Mon, Aug 27, 2012 at 7:43 PM, Samuel Thibault <sthibault@debian.org> wrote:
> Control: tags 641967 + important

Is closed for a long time now and unrelated in so far as nobody
is setting this variable - otherwise the auto-detection would not
play safe here. I am dropping the cc to it therefore.
(And the other thing mentioned in it: That the download of 10 files is
 slower than the download of 5 files is not a bug, but common sense.)


> Control: tags 678227 + important

(Happily went under my radar as his reporter manages every time
 he got out of my kill-file to rejoin it in a few days, if not hours as
 I am probably on his as he continues his tirade against InRelease
 or now Translation-* without giving a damn what I or even others say)


If you could add a message to this bug telling a casual reader what
you think is a (or the) bug here, please do so. I don't see one.
I only see one which quotes the documentation explaining exactly
why this behavior is present and yet vouches to break it without a
reason beside that an update is now slower than before.
(With the hidden message that data is download which isn't needed by
 him, but failing to understand that APT has no way of knowing what
 all users on the system might need and therefore does the most sensible
 thing: keeping all data up-to-date it currently uses)


> The issue also happens at debian-installer time, thus downloading way

(I don't know why an "also" is in that sentence …)

apt-cdrom copies every Translation-* file from the disk to /var/lib/apt/lists
and libapt applications nowadays will proceed with updating them all
rather than dropping all not in the current environment (which nowadays
is usually LANG=C as run by cron or an APT front-end).

This is NOT new. Translation files exist for a long time and apt-cdrom
copies them for a long time. The difference is that we now have
a) a Translation-en file containing the English long descriptions
b) libapt supporting users who can understand more than one language
c) libapt supporting systems with more than one admin

So these files were already copied to /var/lib/apt/lists, it's just that
apt-get and everything else using libapt will not drop Translation-* files
in there on the first run as there is a good chance that this run will be
done by cron or a front-end in LANG=C loosing all this valuable data.


> more than necessary, and taking a lot of disk space (seen 82M here) for
> no reason, thus moving the "embedded" barrier yet higher for Debian.

No reason indeed if your only target audience is a crowd of people who
perfectly speak English (en_US) and can choose packages just by looking
at a short description. If you want to support more people - and the effort
many people put into creating these Translations files suggests that at least
some want to support them - we need to ship these files to them.
Not to mention the poor souls who like to read English long descriptions …


Embedded systems aren't in any way less supported than before by this.
The very same config option used since ever to disable the download of
Translation files is still supported. In a way, it is even better supported
as you have the option of getting right of -en, too, saving even more MB.


Best regards

David Kalnischkies



Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Tue, 28 Aug 2012 08:51:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Samuel Thibault <sthibault@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Tue, 28 Aug 2012 08:51:03 GMT) Full text and rfc822 format available.

Message #37 received at 678227@bugs.debian.org (full text, mbox):

From: Samuel Thibault <sthibault@debian.org>
To: David Kalnischkies <kalnischkies+debian@gmail.com>
Cc: 678227@bugs.debian.org, debian-release@lists.debian.org
Subject: Re: Bug#641967: Apt downloads all description translations
Date: Tue, 28 Aug 2012 10:47:43 +0200
David Kalnischkies, le Tue 28 Aug 2012 10:31:23 +0200, a écrit :
> > The issue also happens at debian-installer time, thus downloading way
> 
> (I don't know why an "also" is in that sentence …)

That's actually because 678227 is what made debian-boot consider the
issue quoted above "already reported" (see #684954).

> apt-cdrom copies every Translation-* file from the disk to /var/lib/apt/lists

Why?  Why not just for the current locale?

> and libapt applications nowadays will proceed with updating them all
> rather than dropping all not in the current environment (which nowadays
> is usually LANG=C as run by cron or an APT front-end).
> 
> This is NOT new.

I haven't seen that with squeeze: when I run apt-get update, only the
Translation file for the current locale gets downloaded.

> Translation files exist for a long time and apt-cdrom
> copies them for a long time. The difference is that we now have
> a) a Translation-en file containing the English long descriptions
> b) libapt supporting users who can understand more than one language
> c) libapt supporting systems with more than one admin

I don't see how that should make apt-get download all languages.

> So these files were already copied to /var/lib/apt/lists, it's just that
> apt-get and everything else using libapt will not drop Translation-* files
> in there on the first run as there is a good chance that this run will be
> done by cron or a front-end in LANG=C loosing all this valuable data.

Then don't drop it in the LANG=C case, but do drop it in the LANG!=C
case.

> > more than necessary, and taking a lot of disk space (seen 82M here) for
> > no reason, thus moving the "embedded" barrier yet higher for Debian.
> 
> No reason indeed if your only target audience is a crowd of people who
> perfectly speak English (en_US) and can choose packages just by looking
> at a short description.

<sarcastic too>There is no reason to download all languages unless you
target the very few people who can speak all languages the descriptions
have been translated to.</sarcastic too>

> Embedded systems aren't in any way less supported than before by this.

82M clearly hurts. Really. That's like 15% of the installation size!

Samuel



Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Tue, 28 Aug 2012 10:51:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Julian Andres Klode <jak@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Tue, 28 Aug 2012 10:51:03 GMT) Full text and rfc822 format available.

Message #42 received at 678227@bugs.debian.org (full text, mbox):

From: Julian Andres Klode <jak@debian.org>
To: Samuel Thibault <sthibault@debian.org>, David Kalnischkies <kalnischkies+debian@gmail.com>, 678227@bugs.debian.org, debian-release@lists.debian.org
Subject: Re: Bug#678227: Bug#641967: Apt downloads all description translations
Date: Tue, 28 Aug 2012 12:47:09 +0200
On Tue, Aug 28, 2012 at 10:47:43AM +0200, Samuel Thibault wrote:
> David Kalnischkies, le Tue 28 Aug 2012 10:31:23 +0200, a écrit :
> > > The issue also happens at debian-installer time, thus downloading way
> > 
> > (I don't know why an "also" is in that sentence …)
> 
> That's actually because 678227 is what made debian-boot consider the
> issue quoted above "already reported" (see #684954).
> 
> > apt-cdrom copies every Translation-* file from the disk to /var/lib/apt/lists
> 
> Why?  Why not just for the current locale?

How about copying files for all configured locales on the system? Unless
someone installes locales-all, this would be the best solution in my
opinion (and the user can still overwrite it with APT configuration
options anyway).

> > So these files were already copied to /var/lib/apt/lists, it's just that
> > apt-get and everything else using libapt will not drop Translation-* files
> > in there on the first run as there is a good chance that this run will be
> > done by cron or a front-end in LANG=C loosing all this valuable data.
> 
> Then don't drop it in the LANG=C case, but do drop it in the LANG!=C
> case.

And let us not forget C.UTF-8 -- it seems that C.utf8 and similar is not
recognized, contrary to normal languages, so we'd need to check for C
and C.UTF-8.

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.



Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Tue, 28 Aug 2012 11:09:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Julien Cristau <jcristau@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Tue, 28 Aug 2012 11:09:05 GMT) Full text and rfc822 format available.

Message #47 received at 678227@bugs.debian.org (full text, mbox):

From: Julien Cristau <jcristau@debian.org>
To: Julian Andres Klode <jak@debian.org>, Samuel Thibault <sthibault@debian.org>, David Kalnischkies <kalnischkies+debian@gmail.com>, 678227@bugs.debian.org, debian-release@lists.debian.org
Subject: Re: Bug#678227: Bug#641967: Apt downloads all description translations
Date: Tue, 28 Aug 2012 13:05:33 +0200
[Message part 1 (text/plain, inline)]
On Tue, Aug 28, 2012 at 12:47:09 +0200, Julian Andres Klode wrote:

> On Tue, Aug 28, 2012 at 10:47:43AM +0200, Samuel Thibault wrote:
> > David Kalnischkies, le Tue 28 Aug 2012 10:31:23 +0200, a écrit :
> > > > The issue also happens at debian-installer time, thus downloading way
> > > 
> > > (I don't know why an "also" is in that sentence …)
> > 
> > That's actually because 678227 is what made debian-boot consider the
> > issue quoted above "already reported" (see #684954).
> > 
> > > apt-cdrom copies every Translation-* file from the disk to /var/lib/apt/lists
> > 
> > Why?  Why not just for the current locale?
> 
> How about copying files for all configured locales on the system? Unless
> someone installes locales-all, this would be the best solution in my
> opinion (and the user can still overwrite it with APT configuration
> options anyway).
> 
Just because I have locales-all installed doesn't mean I want every
single translation downloaded.  Getting just the english ones is painful
enough thankyouverymuch.

Cheers,
Julien
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Tue, 28 Aug 2012 11:18:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Julian Andres Klode <jak@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Tue, 28 Aug 2012 11:18:05 GMT) Full text and rfc822 format available.

Message #52 received at 678227@bugs.debian.org (full text, mbox):

From: Julian Andres Klode <jak@debian.org>
To: Julien Cristau <jcristau@debian.org>
Cc: Samuel Thibault <sthibault@debian.org>, David Kalnischkies <kalnischkies+debian@gmail.com>, 678227@bugs.debian.org, debian-release@lists.debian.org
Subject: Re: Bug#678227: Bug#641967: Apt downloads all description translations
Date: Tue, 28 Aug 2012 13:14:18 +0200
On Tue, Aug 28, 2012 at 01:05:33PM +0200, Julien Cristau wrote:
> On Tue, Aug 28, 2012 at 12:47:09 +0200, Julian Andres Klode wrote:
> 
> > On Tue, Aug 28, 2012 at 10:47:43AM +0200, Samuel Thibault wrote:
> > > David Kalnischkies, le Tue 28 Aug 2012 10:31:23 +0200, a écrit :
> > > > > The issue also happens at debian-installer time, thus downloading way
> > > > 
> > > > (I don't know why an "also" is in that sentence …)
> > > 
> > > That's actually because 678227 is what made debian-boot consider the
> > > issue quoted above "already reported" (see #684954).
> > > 
> > > > apt-cdrom copies every Translation-* file from the disk to /var/lib/apt/lists
> > > 
> > > Why?  Why not just for the current locale?
> > 
> > How about copying files for all configured locales on the system? Unless
> > someone installes locales-all, this would be the best solution in my
> > opinion (and the user can still overwrite it with APT configuration
> > options anyway).
> > 
> Just because I have locales-all installed doesn't mean I want every
> single translation downloaded.  Getting just the english ones is painful
> enough thankyouverymuch.

Right, but the installer does not install locales-all (right?). So by
default, only the files for the configured locales will be copied. 

We could add a switch to apt-cdrom to copy all configured locales for
d-i purposes, and then once we have translation files in /var/lib/apt/lists
only recognize those languages and the ones configured via APT
configuration. We don't even need a switch, we could just look at this
list if no languages are explicitely configured for APT or translations
found in /var/lib/apt/lists.

This solves the problem with initial setup copying all translation
files and should work for everyone.


-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.



Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Tue, 28 Aug 2012 11:51:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Neil Williams <codehelp@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Tue, 28 Aug 2012 11:51:06 GMT) Full text and rfc822 format available.

Message #57 received at 678227@bugs.debian.org (full text, mbox):

From: Neil Williams <codehelp@debian.org>
To: Samuel Thibault <sthibault@debian.org>
Cc: David Kalnischkies <kalnischkies+debian@gmail.com>, 678227@bugs.debian.org
Subject: Re: Bug#641967: Apt downloads all description translations
Date: Tue, 28 Aug 2012 12:46:59 +0100
[Message part 1 (text/plain, inline)]
On Tue, 28 Aug 2012 10:47:43 +0200
Samuel Thibault <sthibault@debian.org> wrote:

> > > more than necessary, and taking a lot of disk space (seen 82M here) for
> > > no reason, thus moving the "embedded" barrier yet higher for Debian.

I would say that this is at best misdirection - there are numerous ways
to sort out download size issues for embedded systems, none of which
require changes to apt.

There can be other problems with apt on embedded, but those have nothing
to do with translated descriptions and I'm waiting until after the
release before setting aside time for reporting/discussing those.

> > No reason indeed if your only target audience is a crowd of people who
> > perfectly speak English (en_US) and can choose packages just by looking
> > at a short description.

(Many embedded systems do not allow any part of the base OS to be
visible, let alone translated. The only translations are in the top
level UI and no users get to see the base system, ever, so there is no
problem here. This is why manpages and translations can be trivially
removed in Emdebian - the "user" is never allowed to login. Any
translations are constrained only to the single UI.)

Embedded devices do not typically expose a package manager via the UI -
typically because the embedded device has no direct connectivity - so
this is a complete red herring. Developer boards do not determine what
is useful for embedded systems in general.
 
> <sarcastic too>There is no reason to download all languages unless you
> target the very few people who can speak all languages the descriptions
> have been translated to.</sarcastic too>
> 
> > Embedded systems aren't in any way less supported than before by this.
> 
> 82M clearly hurts. Really. That's like 15% of the installation size!

Embedded systems can just turn off the downloads entirely or use
packages from Emdebian which do not provide the translations *and* give
you a 40% smaller download in the first place, etc.etc.

Whatever other reasons there may be for this bug, I would recommend
that the "embedded barrier" argument is entirely false and should be
ignored for this issue.

If 82Mb is 15% of your installation size, you should be considering
Emdebian which can get your entire installation below 100Mb or ~200Mb
with an X server.

That is not a problem with apt, it's a problem of selecting a different
apt source and/or existing apt config options.

-- 


Neil Williams
=============
http://www.linux.codehelp.co.uk/

[Message part 2 (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Tue, 28 Aug 2012 12:09:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Samuel Thibault <sthibault@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Tue, 28 Aug 2012 12:09:06 GMT) Full text and rfc822 format available.

Message #62 received at 678227@bugs.debian.org (full text, mbox):

From: Samuel Thibault <sthibault@debian.org>
To: Neil Williams <codehelp@debian.org>
Cc: David Kalnischkies <kalnischkies+debian@gmail.com>, 678227@bugs.debian.org
Subject: Re: Bug#641967: Apt downloads all description translations
Date: Tue, 28 Aug 2012 14:07:32 +0200
Note that I have put quotes around "embedded". I don't mean embedded as
in user-oriented devices, but anything that people use Debian to tinker
with: virtual machines, small routers, etc. We have already crossed the
512M barrier, if we could avoid the 1G barrier as long as easily
possible, that'd be good.

> If 82Mb is 15% of your installation size, you should be considering
> Emdebian which can get your entire installation below 100Mb or ~200Mb
> with an X server.

It's not "my" installation size. It's the standard Debian base
installation size.  Having to use embedian just because Debian got a 15%
increase for no apparent good reason looks wrong.

Samuel



Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Thu, 30 Aug 2012 17:45:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to David Kalnischkies <kalnischkies+debian@gmail.com>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Thu, 30 Aug 2012 17:45:05 GMT) Full text and rfc822 format available.

Message #67 received at 678227@bugs.debian.org (full text, mbox):

From: David Kalnischkies <kalnischkies+debian@gmail.com>
To: 678227@bugs.debian.org, debian-release@lists.debian.org
Cc: Julian Andres Klode <jak@debian.org>, Samuel Thibault <sthibault@debian.org>
Subject: Re: Bug#678227: Bug#641967: Apt downloads all description translations
Date: Thu, 30 Aug 2012 19:43:21 +0200
On Tue, Aug 28, 2012 at 1:14 PM, Julian Andres Klode <jak@debian.org> wrote:
> We could add a switch to apt-cdrom to copy all configured locales for
> d-i purposes,

The question is how apt-cdrom gets to know which locales are configured.
If I read the d-i syslog correctly apt-cdrom is run before 'locales'
is installed and localechooser sets the chroot up, so to have apt-cdrom in
the correct environment localechooser setup either needs to be part of the
bootstrap or we make it so that apt-cdrom doesn't copy any Translation files
and localechooser calls after setting up 'locales' apt-get update to copy
the Translation files over [0]. Or we tell localechooser (or some later d-i
 stage) that it should remove 'excess' Translation files.

The later would have the advantage of working better in multi-user setups
(with different l10n environments), but I am not sure if you can actually
do that with d-i currently - the first two don't, the second additional
changes behavior of apt-cdrom in a way breaking older d-i versions and/or
any other installer/user.

> and then once we have translation files in /var/lib/apt/lists
> only recognize those languages and the ones configured via APT
> configuration. We don't even need a switch, we could just look at this
> list if no languages are explicitely configured for APT or translations
> found in /var/lib/apt/lists.

Mhh. Yeah, we really need to do this to 'preserve' configuration or
otherwise we just get all Translation files again on each update
with a cdrom source (but kinda orthogonal to the "problem" at hand).

So something along the lines of maybe:
if acquire::languages set OR files in /var/lib/apt/lists:
	copy as configured
else
	copy everything

"acquire::languages set" is a bit flaky through, so maybe just
if LANG != "C" OR files in /var/lib/apt/lists
(and "C.UTF-8" and "POSIX", right?)


Best regards

David Kalnischkies



Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Thu, 30 Aug 2012 22:09:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Samuel Thibault <sthibault@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Thu, 30 Aug 2012 22:09:02 GMT) Full text and rfc822 format available.

Message #72 received at 678227@bugs.debian.org (full text, mbox):

From: Samuel Thibault <sthibault@debian.org>
To: David Kalnischkies <kalnischkies+debian@gmail.com>, 678227@bugs.debian.org
Cc: debian-release@lists.debian.org, Julian Andres Klode <jak@debian.org>
Subject: Re: Bug#678227: Bug#641967: Apt downloads all description translations
Date: Fri, 31 Aug 2012 00:07:41 +0200
David Kalnischkies, le Thu 30 Aug 2012 19:43:21 +0200, a écrit :
> On Tue, Aug 28, 2012 at 1:14 PM, Julian Andres Klode <jak@debian.org> wrote:
> > We could add a switch to apt-cdrom to copy all configured locales for
> > d-i purposes,
> 
> The question is how apt-cdrom gets to know which locales are configured.
> If I read the d-i syslog correctly apt-cdrom is run before 'locales'

No, locales is part of the base system, and thus installed and
configured before CD-ROM inspection by apt-cdrom. apt-cdrom-setup is
not a step in d-i, it's apt-setup which is one, done after bootstrap,
and invoking apt-cdrom-setup. apt-cdrom is actually run in the locale
chosen during installation, so it can simply follow that (I wonder why
it doesn't seem to be doing it).

Samuel



Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Thu, 06 Sep 2012 14:51:08 GMT) Full text and rfc822 format available.

Acknowledgement sent to David Kalnischkies <kalnischkies+debian@gmail.com>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Thu, 06 Sep 2012 14:51:08 GMT) Full text and rfc822 format available.

Message #77 received at 678227@bugs.debian.org (full text, mbox):

From: David Kalnischkies <kalnischkies+debian@gmail.com>
To: 678227@bugs.debian.org, debian-release@lists.debian.org
Cc: Samuel Thibault <sthibault@debian.org>
Subject: Re: Bug#678227: Bug#641967: Apt downloads all description translations
Date: Thu, 6 Sep 2012 16:49:55 +0200
On Fri, Aug 31, 2012 at 12:07 AM, Samuel Thibault <sthibault@debian.org> wrote:
> David Kalnischkies, le Thu 30 Aug 2012 19:43:21 +0200, a écrit :
>> On Tue, Aug 28, 2012 at 1:14 PM, Julian Andres Klode <jak@debian.org> wrote:
>> > We could add a switch to apt-cdrom to copy all configured locales for
>> > d-i purposes,
>>
>> The question is how apt-cdrom gets to know which locales are configured.
>> If I read the d-i syslog correctly apt-cdrom is run before 'locales'
>
> No, locales is part of the base system, and thus installed and
> configured before CD-ROM inspection by apt-cdrom. apt-cdrom-setup is
> not a step in d-i, it's apt-setup which is one, done after bootstrap,

I take your "No" and re-raise it to "No, yes, but twice".

apt-cdrom is run twice, first as I described it. Feel free to check it
yourself, the first package installed (aka the first entry in
 /var/log/apt/history.log) is 'locales' because everything not part of the
bootstrap is installed with apt-get which uses as source the cdrom
(what else, as mirrors aren't configured yet).

After some more packages are installed the installation reaches the apt-
setup stage in which a mirror will be chosen - but as its first task it
runs apt-cdrom again as you said it - in the locale chosen and installed
earlier. This makes this problem actually solvable, at least for installers
with that behavior …


> and invoking apt-cdrom-setup. apt-cdrom is actually run in the locale
> chosen during installation, so it can simply follow that (I wonder why
> it doesn't seem to be doing it).

As said, apt-cdrom never did and nobody changed it until now - likely also
because nobody of the few people who take care of APT use apt-cdrom on
a daily basis currently - or at least in my case at all.
(which explains a few more needed bugfixes in upcoming 0.9.7.5 …)


I wonder how many people actually depend on that behavior (meaning especially
other distro-installers and multi-language systems), but ignoring my fear to
be once again be flamed to death for an "easy and only logical" behavior
change I just changed apt-cdrom to drop all not configured Translation-*
files from the copy list.

Meaning that on the first run above you will get only Translation-en, while
you get Translation-en and Translation-$LANG (and possibly some more friends
defined by the various environment variables and configs) in the second.
Installers allowing to create multiple users while installing (with different
$LANG) need to re-run apt-cdrom in all of these $LANGs to get the optimal
behavior.


I tried it with a modified multi-arch beta1 cd [0], so at least for debian
it seems to work, but I only run it until a mirror is chosen for the lack
of network, so more testers are welcomed of course …
(You find it only the bzr branch for now, as we haven't made our minds
 yet how to solve #686346 without too much work and breakage risk -
 life would be so easy if dpkg and APT^Wmyself would agree just once …)


If someone feels the urge to kill me for this change now: Add
APT::CDROM::DropTranslation false;
to your configuration file/commandline invocation and you are back
to your "normal" and beloved previous behavior.


Best regards

David Kalnischkies

[0] some hints at http://wiki.debian.org/DebianInstaller/Modify/CD



Reply sent to Michael Vogt <mvo@debian.org>:
You have taken responsibility. (Tue, 11 Sep 2012 15:36:06 GMT) Full text and rfc822 format available.

Notification sent to Filipus Klutiero <chealer@gmail.com>:
Bug acknowledged by developer. (Tue, 11 Sep 2012 15:36:06 GMT) Full text and rfc822 format available.

Message #82 received at 678227-close@bugs.debian.org (full text, mbox):

From: Michael Vogt <mvo@debian.org>
To: 678227-close@bugs.debian.org
Subject: Bug#678227: fixed in apt 0.9.7.5
Date: Tue, 11 Sep 2012 15:32:49 +0000
Source: apt
Source-Version: 0.9.7.5

We believe that the bug you reported is fixed in the latest version of
apt, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 678227@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Michael Vogt <mvo@debian.org> (supplier of updated apt package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Tue, 11 Sep 2012 15:56:44 +0200
Source: apt
Binary: apt libapt-pkg4.12 libapt-inst1.5 apt-doc libapt-pkg-dev libapt-pkg-doc apt-utils apt-transport-https
Architecture: source all amd64
Version: 0.9.7.5
Distribution: unstable
Urgency: low
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Michael Vogt <mvo@debian.org>
Description: 
 apt        - commandline package manager
 apt-doc    - documentation for APT
 apt-transport-https - https download transport for APT
 apt-utils  - package managment related utility programs
 libapt-inst1.5 - deb package format runtime library
 libapt-pkg-dev - development files for APT's libapt-pkg and libapt-inst
 libapt-pkg-doc - documentation for APT development
 libapt-pkg4.12 - package managment runtime library
Closes: 670900 678227 684435 685192 685989 686346 686975
Changes: 
 apt (0.9.7.5) unstable; urgency=low
 .
   [ Manpages translation updates ]
   * Japanese (KURASAWA Nozomu) (Closes: #684435)
   * Portuguese (Américo Monteiro) (Closes: #686975)
 .
   [ David Kalnischkies ]
   * handle packages without a mandatory architecture (debian-policy §5.3)
     by introducing a pseudo-architecture 'none' so that the small group of
     users with these packages can get right of them without introducing too
     much hassle for other users (Closes: #686346)
   * apt-pkg/cdrom.cc:
     - copy only configured translation files from a CD-ROM and not all
       available translation files preventing new installs with d-i from
       being initialized with all translations (Closes: #678227)
     - handle Components in the reduction for the source.list as multi-arch CDs
       otherwise create duplicated source entries (e.g. "wheezy main main")
   * apt-pkg/packagemanager.cc:
     - unpack versions only in case a different version from the package
       is currently in unpack state to recover from broken system states
       (like different file in M-A:same package and other dpkg errors)
       and avoid re-unpack otherwise (Closes: #670900)
   * debian/control:
     - let libapt-pkg break apt < 0.9.4 to ensure that the installed http-
       method supports the new redirection-style, thanks to Raphael Geissert
       for reporting & testing (Closes: #685192)
   * doc/apt_preferences.5.xml:
     - use the correct interval (x <= P < y) for pin value documentation as
       these are the intervals used by the code (Closes: #685989)
   * apt-pkg/indexcopy.cc:
     - do not create duplicated flat-archive CD-ROM sources for foreign
       architectures on multi-arch CD-ROMs
     - do not warn about files which have a record in the Release file, but
       are not present on the CD to mirror the behavior of the other methods
       and to allow uncompressed indexes to be dropped without scaring users
   * apt-pkg/pkgcachegen.cc:
     - do not create 'native' (or now 'none') package structures as a side
       effect of description translation parsing as it pollutes the cache
Checksums-Sha1: 
 7e666c8c7e7c7ab27c858d54a19485232229ac98 1689 apt_0.9.7.5.dsc
 f87aebbc6b9c413821bc5f19c91c87224f016357 3387864 apt_0.9.7.5.tar.gz
 6e54a50f3f23ca461e2525df41a856f98beab286 260874 apt-doc_0.9.7.5_all.deb
 f13303f91b27168ae497e1de8ec2af407a104a43 960408 libapt-pkg-doc_0.9.7.5_all.deb
 d74697ed798b7e089bf40a3e05fd80532590ae33 882860 libapt-pkg4.12_0.9.7.5_amd64.deb
 1afa928bbd988ce3a910bc6ecd93c27fb673d84d 164498 libapt-inst1.5_0.9.7.5_amd64.deb
 73f443563ef51198d12678c0611d9af678a1a8f7 1234400 apt_0.9.7.5_amd64.deb
 f35d3c5999a8fbe336b98ff7bb439a8f45b54c51 185406 libapt-pkg-dev_0.9.7.5_amd64.deb
 195ed4819d3fddf1f3cf7b9770d47ab4e8c4b592 373924 apt-utils_0.9.7.5_amd64.deb
 e4fbf830b93d5cd83b1202005b544de2b3741c08 107078 apt-transport-https_0.9.7.5_amd64.deb
Checksums-Sha256: 
 0c04c31d986810d2b52de501a45fad989b0ec55d8a96f73544d89126c58de45e 1689 apt_0.9.7.5.dsc
 82ff902cc33dab89e875c5827f625a64b4b872c34f1a16a8f04e07a7a1c30ced 3387864 apt_0.9.7.5.tar.gz
 ca64252135bae07fa54a534488e8a234f282113b1353ddfc0e39072834ee7c5a 260874 apt-doc_0.9.7.5_all.deb
 6dbe1b97ad10fe96a90f58d7a378346545ca58f6c2629331c802aae53ba490b6 960408 libapt-pkg-doc_0.9.7.5_all.deb
 913b5a7f26ed9de17f4e4257189f2136f2d00af783da80654a1b7f57e514beb6 882860 libapt-pkg4.12_0.9.7.5_amd64.deb
 cb40f8f8788e71b034de387c78c61e0be9b2b09cacfe8c7b995394f4b2c5cf06 164498 libapt-inst1.5_0.9.7.5_amd64.deb
 3c0d3d4b72d5de132560a26e3c86b9d8d759354e6a349b46b1b203b18b0c5563 1234400 apt_0.9.7.5_amd64.deb
 11ca12b5f43266108df9845bf5040f71d0ddddccd9760dad7a1e4efb8e4f92da 185406 libapt-pkg-dev_0.9.7.5_amd64.deb
 78e2f8b98489dde3d297ca855c3da72e6b583b97352993de7e3c697190fa0590 373924 apt-utils_0.9.7.5_amd64.deb
 442c083c20401c30dbbfcd9f71af85451bd81dec4dc847885ef222dede4cbf54 107078 apt-transport-https_0.9.7.5_amd64.deb
Files: 
 83737a282f81be3d24be43a60ef5aa11 1689 admin important apt_0.9.7.5.dsc
 51bed58178d73c21e240cb4e137afa93 3387864 admin important apt_0.9.7.5.tar.gz
 2ef4da2a1cece4afc1688c5d3b9e3aeb 260874 doc optional apt-doc_0.9.7.5_all.deb
 a983ad6026b1fd187988a21220298740 960408 doc optional libapt-pkg-doc_0.9.7.5_all.deb
 aa21106040013490cfe64f26020e0340 882860 libs important libapt-pkg4.12_0.9.7.5_amd64.deb
 6851100d5db7997c2f4e58b179908a15 164498 libs important libapt-inst1.5_0.9.7.5_amd64.deb
 073cfd3f4781fe1978ca00ea6853e78f 1234400 admin important apt_0.9.7.5_amd64.deb
 acaaa33ff73372b684ffa75e652c1cec 185406 libdevel optional libapt-pkg-dev_0.9.7.5_amd64.deb
 7da608a838d4a58ce26c62498c2d70c2 373924 admin important apt-utils_0.9.7.5_amd64.deb
 06997202bccda2f2cb72fb162b17b03a 107078 admin optional apt-transport-https_0.9.7.5_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlBPV4AACgkQliSD4VZixzTcvwCePyBv2X3kGh0Ba/ATKszd2zxr
/H4AoIK/zrO+im4bX7XaRgX/KqvAMmh4
=Oqfs
-----END PGP SIGNATURE-----




Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#678227; Package apt. (Mon, 24 Sep 2012 16:03:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Samuel Thibault <sthibault@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Mon, 24 Sep 2012 16:03:03 GMT) Full text and rfc822 format available.

Message #87 received at 678227@bugs.debian.org (full text, mbox):

From: Samuel Thibault <sthibault@debian.org>
To: David Kalnischkies <kalnischkies+debian@gmail.com>
Cc: 678227@bugs.debian.org, debian-release@lists.debian.org
Subject: Re: Bug#678227: Bug#641967: Apt downloads all description translations
Date: Mon, 24 Sep 2012 18:01:35 +0200
David Kalnischkies, le Thu 06 Sep 2012 16:49:55 +0200, a écrit :
> On Fri, Aug 31, 2012 at 12:07 AM, Samuel Thibault <sthibault@debian.org> wrote:
> > David Kalnischkies, le Thu 30 Aug 2012 19:43:21 +0200, a écrit :
> >> On Tue, Aug 28, 2012 at 1:14 PM, Julian Andres Klode <jak@debian.org> wrote:
> >> > We could add a switch to apt-cdrom to copy all configured locales for
> >> > d-i purposes,
> >>
> >> The question is how apt-cdrom gets to know which locales are configured.
> >> If I read the d-i syslog correctly apt-cdrom is run before 'locales'
> >
> > No, locales is part of the base system, and thus installed and
> > configured before CD-ROM inspection by apt-cdrom. apt-cdrom-setup is
> > not a step in d-i, it's apt-setup which is one, done after bootstrap,
> 
> I take your "No" and re-raise it to "No, yes, but twice".
> 
> apt-cdrom is run twice, first as I described it.

Ah, right, I didn't know about that one.

> Meaning that on the first run above you will get only Translation-en, while
> you get Translation-en and Translation-$LANG (and possibly some more friends
> defined by the various environment variables and configs) in the second.

Confirmed with a daily image, thanks!

Samuel



Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Sun, 28 Oct 2012 07:27:47 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Thu Apr 17 15:46:16 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.