Debian Bug report logs - #479681
perl: Locale::Gettext should be part of the core modules

version graph

Package: perl; Maintainer for perl is Niko Tyni <ntyni@debian.org>; Source for perl is src:perl.

Reported by: Raphael Hertzog <hertzog@debian.org>

Date: Tue, 6 May 2008 06:27:01 UTC

Severity: wishlist

Found in version perl/5.10.0-9

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Brendan O'Dea <bod@debian.org>:
Bug#479681; Package perl. Full text and rfc822 format available.

Acknowledgement sent to Raphael Hertzog <hertzog@debian.org>:
New Bug report received and forwarded. Copy sent to Brendan O'Dea <bod@debian.org>. Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Raphael Hertzog <hertzog@debian.org>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: perl: Locale::Gettext should be part of the core modules
Date: Tue, 06 May 2008 08:18:02 +0200
Package: perl
Version: 5.10.0-9
Severity: wishlist

Locale::Gettext is used by many scripts/modules to provide localized
strings to the user. Those scripts are used in package configuration
scripts (postinst, preinst, postrm, prerm, config) and they should
always work when possible... even when perl is being upgraded to
a new perl with an incompatible XS ABI.

To offer such a guaranty, the nicest solution is to provide the module in
the same package than the perl binary itself. Hence it should become part
of perl-base in Debian and should be part of core modules at the upstream
level.

Please discuss this on p5p with upstream and if they agree to do it for
the next upstream release, please consider doing it right now as a
temporary fork so that we can see if it helps with the upgrade problems
described in http://bugs.debian.org/479220

-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.24-1-686 (SMP w/1 CPU core)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages perl depends on:
ii  libc6                         2.7-10     GNU C Library: Shared libraries
ii  libdb4.6                      4.6.21-8   Berkeley v4.6 Database Libraries [
ii  libgdbm3                      1.8.3-3    GNU dbm database routines (runtime
ii  perl-base                     5.10.0-9   The Pathologically Eclectic Rubbis
ii  perl-modules                  5.10.0-9   Core Perl modules

Versions of packages perl recommends:
ii  netbase                       4.32       Basic TCP/IP networking system
ii  perl-doc                      5.10.0-9   Perl documentation

-- no debconf information




Information forwarded to debian-bugs-dist@lists.debian.org, Brendan O'Dea <bod@debian.org>:
Bug#479681; Package perl. Full text and rfc822 format available.

Acknowledgement sent to "Brendan O'Dea" <bod@debian.org>:
Extra info received and forwarded to list. Copy sent to Brendan O'Dea <bod@debian.org>. Full text and rfc822 format available.

Message #10 received at 479681@bugs.debian.org (full text, mbox):

From: "Brendan O'Dea" <bod@debian.org>
To: "Ian Jackson" <ian@davenant.greenend.org.uk>, debian-devel@lists.debian.org, perl@packages.debian.org, 479681@bugs.debian.org
Subject: Re: Perl symbol problem - release critical (Re: Bug#489132)
Date: Fri, 4 Jul 2008 11:21:03 +1000
On Fri, Jul 4, 2008 at 6:17 AM, Raphael Hertzog <hertzog@debian.org> wrote:
> This is why I suggested to integrate liblocale-gettext-perl in perl-base
> itself. This would be the simplest/nicest solution IMO. It would always be
> synchronized with the current perl.
>
> See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=479681

I'm inclined at this point to follow this suggestion.

--bod




Information forwarded to debian-bugs-dist@lists.debian.org, Brendan O'Dea <bod@debian.org>:
Bug#479681; Package perl. Full text and rfc822 format available.

Acknowledgement sent to Niko Tyni <ntyni@debian.org>:
Extra info received and forwarded to list. Copy sent to Brendan O'Dea <bod@debian.org>. Full text and rfc822 format available.

Message #15 received at 479681@bugs.debian.org (full text, mbox):

From: Niko Tyni <ntyni@debian.org>
To: debian-devel@lists.debian.org
Cc: Ian Jackson <ian@davenant.greenend.org.uk>, 479681@bugs.debian.org, 489132@bugs.debian.org
Subject: Re: Perl symbol problem - release critical (Re: Bug#489132)
Date: Sun, 3 Aug 2008 21:31:07 +0300
On Thu, Jul 03, 2008 at 10:17:47PM +0200, Raphael Hertzog wrote:
> On Thu, 03 Jul 2008, Ian Jackson wrote:

> > * This problem is clearly release critical.  I don't think documenting
> >   a release critical bug of this severity in the release notes is
> >   acceptable.  Furthermore, the proposed workaround is very cumbersome
> >   due to the necessary installation ordering.
> 
> It's clearly release critical but doesn't necessarily happen on all
> upgrades. It depends if update-alternatives/dpkg-divert is called
> between the liblocale-gettext-perl and perl-base unpack.

Sorry to join in this late, I've been on a family vacation.  Although my
name is on the perl package, I'm really a bit out of my depth here.
Help is welcome.

For the record, at least #489132 (Cc'd), #479220, #479711, #488300, and
#479681 (Cc'd) are related.

As far as I understand, in the case of Etch->Lenny upgrades and
Locale::gettext (which is the most pressing issue here):

- apt will always upgrade both liblocale-gettext-perl and perl-base in
  the same go because of their dependencies
- dpkg will always unpack (and configure) perl-base before unpacking
  liblocale-gettext-perl because the latter Pre-Depends on the former

Furthermore, in my test upgrades with apt the new perl-base is unpacked
(and configured) right before liblocale-gettext-perl gets unpacked, so
no maintainer scripts from other packages get run in between. I don't
claim that this behaviour is guaranteed, only that I don't have a recipe
for reproducing the problem in a real upgrade situation.

My tentative assumption is that the breakage only bites when a version of
liblocale-gettext-perl lacking the perl pre-dependency (that's 1.05-2,
1.05-3 and 1.05-3+b1) is involved, or when the upgrade is done by
installing some packages 'manually' with dpkg. I haven't seen any bug
reports to the contrary yet.

One recipe for breaking update-alternatives and dpkg-divert in such a
'manual' way starting from a minimal Etch install is:

# apt-get install liblocale-gettext-perl
# perl -pi -e 's/etch/lenny/' /etc/apt/sources.list
# apt-get update
# apt-get install libc6
# apt-get -d install perl-base
# dpkg -i /var/cache/apt/archives/perl-base_5.10.0-11.1_amd64.deb 

Note that dpkg doesn't refuse to do this: the dependencies of the old
liblocale-gettext-perl package are violated, but they aren't checked
because liblocale-gettext-perl is not being configured.

> In general a package offers no guaranty to be functionnal until it is
> successfully configured... so the perl module rely on this assumption.
> The problem is that some of the non-core modules are used by part of our
> essential infrastructure. Locale::gettext is the most important one.
> Any script using this module is potentially broken when called in
> some preinst script.

... or a prerm one ("old-prerm upgrade new-version"), and "only" if the
script doesn't set $ENV{PERL_DL_NONLAZY}.

> > * Suppressing lazy symbol resolution may work in this case, but it is
> >   not correct.  ABI changes may result in random crashes due to
> >   different structure sizes and do not necessarily involve missing
> >   symbols - so the problem may go undetected.  If we think that we
> >   want to fix it in etch->lenny by suppressing lazy symbol resolution,
> >   we need to:
> >     (a) check what the actual ABI differences are and that either
> >         there aren't any others besides missing symbols, or that
> >         every module will definitely fail to load
> >     (b) regard this as a workaround and do something sensible next
> >         time.
> 
> I leave that to perl maintainers. :)

FWIW, modifying Perl to always disable lazy symbol resolution in "eval"
blocks was explicitly discouraged upstream because it conceivably could
break existing setups using partly broken binary modules. See

 http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2008-05/msg00426.html

> > * One of the Perl upstream commenters in #479711 suggests that the
> >   answer is to use a `pre-inst dependency' which apparently none of
> >   the submitters have realised is what dpkg already has and calls
> >   Pre-Depends.  However, a Pre-Depends doesn't solve this problem
> >   because there is no correct order to upgrade the packages:
> >   regardless of whether you upgrade Perl first, or the modules first,
> >   something may break.
> 
> This is why I suggested to integrate liblocale-gettext-perl in perl-base
> itself. This would be the simplest/nicest solution IMO. It would always be
> synchronized with the current perl.
> 
> See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=479681

There's also the suggestion in #489132 to make perl-base Pre-Depend on
dpkg (>= 1.14.20), whose scripts set $ENV{PERL_DL_NONLAZY} to avoid the
breakage.  As far as I can see, this should work too for the immediate
problem, and it would be even simpler. But maybe I'm missing something?

Brendan already acked the liblocale-gettext-perl/perl-base integration
option in #479681, so I'll work on that for now. If somebody thinks it's
not enough for lenny (for example because other Perl module packages
beside liblocale-gettext-perl need attention too), please speak up.
-- 
Niko Tyni   ntyni@debian.org




Information forwarded to debian-bugs-dist@lists.debian.org, Brendan O'Dea <bod@debian.org>:
Bug#479681; Package perl. Full text and rfc822 format available.

Acknowledgement sent to Niko Tyni <ntyni@debian.org>:
Extra info received and forwarded to list. Copy sent to Brendan O'Dea <bod@debian.org>. Full text and rfc822 format available.

Message #20 received at 479681@bugs.debian.org (full text, mbox):

From: Niko Tyni <ntyni@debian.org>
To: debian-devel@lists.debian.org
Cc: 489132@bugs.debian.org, 488300@bugs.debian.org, 479681@bugs.debian.org
Subject: Re: Perl symbol problem - release critical (Re: Bug#489132)
Date: Thu, 21 Aug 2008 00:52:44 +0300
On Thu, Jul 03, 2008 at 08:11:05PM +0100, Ian Jackson wrote:
> Raphael Hertzog writes ("Bug#489132: lenny release notes, upgrade dpkg first"):
> > To work-around a problem that can happen in the perl 5.10 upgrade (see
> > #479711), the perl scripts contained in dpkg (update-alternatives,
> > dpkg-divert) have been modified... but for the work-around to be used, the
> > new dpkg must obviously be installed first, before the dist-upgrade.
> 
> I don't think this is the right solution.  To be honest I'm just
> astonished at this situation, which is terrible.  It is the
> consequence of a mistake in the Debian Perl policy - a mistake which
> has caused trouble on every previous upgrade, too.

Revisiting this; #489132 and #488300 are still open and I'm (hopefully)
less confused about the issue now than in my earlier reply to #489132
and others.

Summary: I think making perl-base Pre-Depend on dpkg (>= 1.4.20) is enough
to fix this for lenny. 

> Possible solutions that I see for lenny:

> 2. Find out which modules are used in this way by Essential packages.
>    Arrange somehow for those modules to fail at `require' when loaded
>    with Perl 5.8 from etch.  This might involve rebuilding only
>    those modules.

The only perl scripts provided by Essential packages are

/usr/bin/scriptreplay
/usr/lib/dpkg/mksplit
/usr/sbin/cleanup-info
/usr/sbin/install-info
/usr/sbin/update-alternatives
/usr/sbin/dpkg-statoverride
/usr/sbin/dpkg-divert
/usr/bin/chkdupexe

All but chkdupexe are in the dpkg package. No external modules are used
by scriptreplay, chkdupexe, and mksplit. The only module outside
perl-base that is used by the others is Locale::gettext. 

All but cleanup-info set PERL_DL_NONLAZY in their Lenny versions, which
makes the "eval 'use Locale::gettext'" call fail due to missing symbols
when liblocale-gettext-perl and perl-base are out of sync.

The Lenny version of liblocale-gettext-perl Pre-Depends on 
perl-base (>= 5.10.0-9). This makes it impossible for the Etch version
of /usr/bin/perl to see the Lenny version of Locale::gettext.

The other way around is still possible: unpacking perl-base on an Etch
system (after upgrading libc6 etc.) makes Perl 5.10 see the 5.8 version of
Locale::gettext. This breaks the Etch version of the dpkg utilities. 

The breakage could be prevented by making perl-base Pre-Depend on dpkg
(>= 1.14.20). I think this would be enough to solve the issue for lenny
and fix #488300 (and possibly #489132, but that one includes some concerns
about the need to upgrade apt manually first.)

If /usr/sbin/cleanup-info is considered part of the Essential
functionality of dpkg, it also needs to set PERL_DL_NONLAZY.
Judging by /usr/share/doc/dpkg/README.feature-removal-schedule,
that is probably not the case.

> * Suppressing lazy symbol resolution may work in this case, but it is
>   not correct.  ABI changes may result in random crashes due to
>   different structure sizes and do not necessarily involve missing
>   symbols - so the problem may go undetected.  If we think that we
>   want to fix it in etch->lenny by suppressing lazy symbol resolution,
>   we need to:
>     (a) check what the actual ABI differences are and that either
>         there aren't any others besides missing symbols, or that
>         every module will definitely fail to load

I think it's clear that Locale::gettext fails to load both ways
when PERL_DL_NONLAZY is set:

- when compiled for 5.10.0 it needs Perl_Istack_sp_ptr from perl/libperl,
  which is not present in 5.8.8
- when compiled for 5.8.8  it needs Perl_Tstack_sp_ptr instead,
  which is not present in 5.10.0

>     (b) regard this as a workaround and do something sensible next
>         time.

Post-lenny, I see two options that don't involve changing the module path:

- mandate that ABI changes in the Perl XS module interface
  will always be accompanied with a symbol rename caught by
  PERL_DL_NONLAZY, and artificially do that for Debian if needed in the
  future. This practically means "just carry on and hope we don't have
  to deviate from perl upstream".

- integrate Locale::gettext in perl-base (#479681) and mandate that
  Essential:yes programs may not load non-Essential XS modules even
  opportunistically (inside an eval block) because PERL_DL_NONLAZY
  can't be trusted.  This seems to be the safer option of the two.

-- 
Niko Tyni   ntyni@debian.org




Information forwarded to debian-bugs-dist@lists.debian.org, Niko Tyni <ntyni@debian.org>:
Bug#479681; Package perl. (Sat, 21 May 2011 17:27:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Dominic Hargreaves <dom@earth.li>:
Extra info received and forwarded to list. Copy sent to Niko Tyni <ntyni@debian.org>. (Sat, 21 May 2011 17:27:05 GMT) Full text and rfc822 format available.

Message #25 received at 479681@bugs.debian.org (full text, mbox):

From: Dominic Hargreaves <dom@earth.li>
To: 479681@bugs.debian.org
Subject: Re: Bug#479681: Perl symbol problem - release critical (Re: Bug#489132)
Date: Sat, 21 May 2011 18:23:25 +0100
On Thu, Aug 21, 2008 at 12:52:44AM +0300, Niko Tyni wrote:
> Post-lenny, I see two options that don't involve changing the module path:
> 
> - mandate that ABI changes in the Perl XS module interface
>   will always be accompanied with a symbol rename caught by
>   PERL_DL_NONLAZY, and artificially do that for Debian if needed in the
>   future. This practically means "just carry on and hope we don't have
>   to deviate from perl upstream".
> 
> - integrate Locale::gettext in perl-base (#479681) and mandate that
>   Essential:yes programs may not load non-Essential XS modules even
>   opportunistically (inside an eval block) because PERL_DL_NONLAZY
>   can't be trusted.  This seems to be the safer option of the two.

I think this bug probably represents the same issues as #479711 now.
I wonder whether they should be merged.

I just commented there that we should just see how the squeeze/wheezy
upgrade works out (ie the first option above).

Suggesting upstream that Locale::gettext be integrated into core might
not be a bad idea for the future, but at this point it probably doesn't
make sense to deviate from upstream at this stage.

Dominic.

-- 
Dominic Hargreaves | http://www.larted.org.uk/~dom/
PGP key 5178E2A5 from the.earth.li (keyserver,web,email)




Information forwarded to debian-bugs-dist@lists.debian.org:
Bug#479681; Package perl. (Thu, 02 Jun 2011 10:57:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Niko Tyni <ntyni@debian.org>:
Extra info received and forwarded to list. (Thu, 02 Jun 2011 10:57:04 GMT) Full text and rfc822 format available.

Message #30 received at 479681@bugs.debian.org (full text, mbox):

From: Niko Tyni <ntyni@debian.org>
To: 479681@bugs.debian.org
Cc: "Brendan O'Dea" <bod@debian.org>
Subject: Re: Bug#479681: Perl symbol problem - release critical (Re: Bug#489132)
Date: Thu, 2 Jun 2011 13:56:01 +0300
On Sat, May 21, 2011 at 06:23:25PM +0100, Dominic Hargreaves wrote:
> On Thu, Aug 21, 2008 at 12:52:44AM +0300, Niko Tyni wrote:
> > Post-lenny, I see two options that don't involve changing the module path:
> > 
> > - mandate that ABI changes in the Perl XS module interface
> >   will always be accompanied with a symbol rename caught by
> >   PERL_DL_NONLAZY, and artificially do that for Debian if needed in the
> >   future. This practically means "just carry on and hope we don't have
> >   to deviate from perl upstream".
> > 
> > - integrate Locale::gettext in perl-base (#479681) and mandate that
> >   Essential:yes programs may not load non-Essential XS modules even
> >   opportunistically (inside an eval block) because PERL_DL_NONLAZY
> >   can't be trusted.  This seems to be the safer option of the two.
> 
> I think this bug probably represents the same issues as #479711 now.
> I wonder whether they should be merged.

I'm not sure about the merge either. Probably yes.

Reviewing the discussion, I think that Ian had a valid point and that
we should revisit the vendorarch directory choice. Brendan (cc'd):
I'd love to have your thoughts on this.

First, I see all the dpkg commands in /usr/bin have been rewritten in C
for squeeze so I don't think this issue concerns Essential functionality
at all anymore.

As a consequence, I think Locale::gettext is no longer quite as special as
it used to be. The remaining cases that I'm aware of where XS modules get
loaded opportunistically during upgrades (without package dependencies
fulfilled) are 'preinst upgrade' maintainer scripts and debconf config
scripts.

The debconf case is sourcing /usr/share/debconf/confmodule.sh, which
currently tries to load three XS modules: Locale::Gettext, Text::Iconv
and Text::CharWidth.

Therefore, if any problems remain, they are not going to be solved
just by integrating Locale::gettext into perl-base.

Now, as long as PERL_DL_NONLAZY catches ABI skew problems, I think we're
safe. However, as Ian pointed out [1] there may well be ABI changes
that don't involve missing or renamed symbols. A prime example of those
would be backing out the -Duse64bitint change we made for 5.12. [2]

Ideally, the XS modules would have an ABI signature that would be checked
by DynaLoader first. I doubt this would fly upstream as they already
rejected making the PERL_DL_NONLAZY=1 behaviour the default in eval{}
blocks (see  #479711) because some people might be relying on partially
loading ABI incompatible modules. Also, I believe they encode some
Configure options into the default @INC [3], which helps to avoid
situations like this.

Lacking such a signature, switching to a versioned @INC for
$Config{vendorarch} (currently /usr/lib/perl5) while keeping
$Config{vendorlib} as /usr/share/perl5 seems to me the way to go.

It would be lovely to only have one vendorarch directory per ABI,
but I think upstream's policy [4] of not breaking the ABI in minor
releases doesn't cover extending it. Such extensions would break forward
compatibility (i.e. 5.12.3 would not be able load XS modules built
for 5.12.4.)

That would leave us with something like (say)
 /usr/lib/perl5.12/5.12.3
 /usr/lib/perl5.12/5.12.4
which could be renamed after a hypothetical -Duse64bitint reversion to
(for example)
 /usr/lib/perl5.12/5.12.4-32bitint

Given the upstream new schedule of yearly major releases, I'd hope
there would not be too many subdirectories.

The 5.14 transition is probably a good time to do this change if we decide
to go forward with it. Naturally we need a discussion on the debian-perl /
debian-devel lists before pushing changes to the policy.

I suppose the alternative is to carry on as usual and worry about
these things when they actually become a problem. Hopefully we aren't
in too much of a hurry at that point so that the necessary changes can
be reviewed properly.

One real problem I'm seeing now is that building a Debian 5.14.0 on a
system with 5.12.3 hits test failures if certain XS module packages are
installed. This is because the modules get opportunistically loaded from
/usr/lib/perl5 but fail to work due to ABI incompatibilities. This is
not a big thing but it's rather annoying and may need Build-Conflicts
that would otherwise be unnecessary.

[1] http://lists.debian.org/debian-devel/2008/07/msg00067.html 
[2] observed behaviour in such a situation:
    # perl -MDevel::Refcount=refcount -le 'print refcount($a=[])'
    583208145925439490
[3] for instance 'x86_64-linux-thread-multi' or 'i686-linux-64int';
    I don't think there's enough information to uniquely define the ABI though
[4] see perlpolicy.pod
-- 
Niko Tyni   ntyni@debian.org




Information forwarded to debian-bugs-dist@lists.debian.org, Niko Tyni <ntyni@debian.org>:
Bug#479681; Package perl. (Thu, 02 Jun 2011 13:18:37 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Brendan O'Dea" <bod@debian.org>:
Extra info received and forwarded to list. Copy sent to Niko Tyni <ntyni@debian.org>. (Thu, 02 Jun 2011 13:18:37 GMT) Full text and rfc822 format available.

Message #35 received at 479681@bugs.debian.org (full text, mbox):

From: "Brendan O'Dea" <bod@debian.org>
To: Niko Tyni <ntyni@debian.org>
Cc: 479681@bugs.debian.org, joeyh@debian.org
Subject: Re: Bug#479681: Perl symbol problem - release critical (Re: Bug#489132)
Date: Thu, 2 Jun 2011 23:17:54 +1000
On 2 June 2011 20:56, Niko Tyni <ntyni@debian.org> wrote:
> Reviewing the discussion, I think that Ian had a valid point and that
> we should revisit the vendorarch directory choice. Brendan (cc'd):
> I'd love to have your thoughts on this.

vendorarch was chosen for simplicity, and consistency with vendorlib.
So long as maintainer scripts use only essential packages, it works
rather well.  The situation of attempting to use packages if possible
was not anticipated, and I'll admit that I was somewhat surprised when
the "eval { require ... }" didn't work as expected.

You certainly could make vendorarch ABI dependent, and perhaps should
to avoid some of the more subtle failures (such as the 64bit issue you
mentioned).  I'm not entirely sure however that it fixes this
particular problem correctly.

The conditional inclusion of Locale::gettext is intended to provide
translations for those who want it, but not force a dependancy on the
package for those who don't.  I suspect that it is not intended to
work around upgrades--there are people for whom the configuration
dialogs suddenly reverting to English mid-upgrade could be as much a
bug as the dynamic link failure.

> First, I see all the dpkg commands in /usr/bin have been rewritten in C
> for squeeze [...]
> The debconf case is sourcing /usr/share/debconf/confmodule.sh, which
> currently tries to load three XS modules: Locale::Gettext, Text::Iconv
> and Text::CharWidth.
>
> Therefore, if any problems remain, they are not going to be solved
> just by integrating Locale::gettext into perl-base.

That is unfortunate.  Changing vendorarch would work around that, but
with the caveats discussed above.

Joey may be able to comment on this issue from the debconf side.  Such
as the status of cdebconf, or the possibility of providing pure-Perl
implementations of the debconf dependencies.

--bod




Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Mon Apr 21 12:57:15 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.