Debian Bug report logs - #843073
dpkg-shlibdeps: broken on i386 with merged /usr

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Ross Vandegrift <ross@kallisti.us>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: dpkg-shlibdeps: broken on i386 with merged /usr

Date: Thu, 03 Nov 2016 13:14:35 -0400

Package: dpkg-dev
Version: 1.18.10
Severity: normal

Dear Maintainer,

debootstrap 1.0.85 began deploying with --merged-usr by default in
response to #839046.  On i386, this causes dpkg-shlibdeps to fail on
(some?) shared libraries.

An example error:
dpkg-shlibdeps: error: no dependency information found for /usr/lib/ld-linux.so.2 (used by debian/libevas1/usr/lib/i386-linux-gnu/libevas.so.1.18.2)
Hint: check if the library actually comes from a package.

In this case:
# ls -l /usr/lib/ld-linux.so.2
lrwxrwxrwx 1 root root 25 Oct 18 21:10 /usr/lib/ld-linux.so.2 -> i386-linux-gnu/ld-2.24.so

To workaround with a pbuilder chroot, when creating, supply:
  --debootstrapopts --no-merged-usr.

I'm not sure if this is a debootstrap or dpkg-dev issue.  Apologies in
advance if I'm incorrect.  The same underlying problem was reported
against usrmerge in #810499.  For some reason, debootstrap does not
create this symlink on amd64.

Ross.

-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (500, 'testing'), (40, 'unstable'), (30, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.7.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages dpkg-dev depends on:
ii  base-files    9.6
ii  binutils      2.27-9+b1
ii  bzip2         1.0.6-8
ii  libdpkg-perl  1.18.10
ii  make          4.1-9
ii  patch         2.7.5-1
ii  tar           1.29b-1
ii  xz-utils      5.2.2-1.2

Versions of packages dpkg-dev recommends:
ii  build-essential          12.2
ii  fakeroot                 1.21-2
ii  gcc [c-compiler]         4:6.1.1-1
ii  gcc-5 [c-compiler]       5.4.1-3
ii  gcc-6 [c-compiler]       6.2.0-9
ii  gnupg                    2.1.15-4
ii  gnupg2                   2.1.15-4
ii  gpgv                     2.1.15-4
ii  libalgorithm-merge-perl  0.08-3

Versions of packages dpkg-dev suggests:
ii  debian-keyring  2016.09.04

-- no debconf information

Message #10 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: "843073@bugs.debian.org" <843073@bugs.debian.org>

Subject: Re: dpkg-shlibdeps: broken on i386 with merged /usr

Date: Sat, 5 Nov 2016 20:58:06 +0100

[Message part 1 (text/plain, inline)]

Control: severity -1 serious

On Thu, 03 Nov 2016 13:14:35 -0400 Ross Vandegrift <ross@kallisti.us> wrote:
> Package: dpkg-dev
> Version: 1.18.10
> Severity: normal
> 
> Dear Maintainer,
> 
> debootstrap 1.0.85 began deploying with --merged-usr by default in
> response to #839046.  On i386, this causes dpkg-shlibdeps to fail on
> (some?) shared libraries.
> 
> An example error:
> dpkg-shlibdeps: error: no dependency information found for /usr/lib/ld-linux.so.2 (used by debian/libevas1/usr/lib/i386-linux-gnu/libevas.so.1.18.2)
> Hint: check if the library actually comes from a package.
> 
> In this case:
> # ls -l /usr/lib/ld-linux.so.2
> lrwxrwxrwx 1 root root 25 Oct 18 21:10 /usr/lib/ld-linux.so.2 -> i386-linux-gnu/ld-2.24.so
> 
> To workaround with a pbuilder chroot, when creating, supply:
>   --debootstrapopts --no-merged-usr.
> 
> I'm not sure if this is a debootstrap or dpkg-dev issue.  Apologies in
> advance if I'm incorrect.  The same underlying problem was reported
> against usrmerge in #810499.  For some reason, debootstrap does not
> create this symlink on amd64.
> 

As this breaks packages to build successfully, I'm bumping this to RC.
E.g. I'm unable to successfully build systemd in a freshly created
chroot on i386. I suspect once the buildds update their chroot, it will
affect a lot more packages.

Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #17 received at 843073@bugs.debian.org (full text, mbox, reply):

From: md@Linux.IT (Marco d'Itri)

To: Ross Vandegrift <ross@kallisti.us>, 843073@bugs.debian.org

Subject: Re: Bug#843073: dpkg-shlibdeps: broken on i386 with merged /usr

Date: Sat, 5 Nov 2016 22:40:30 +0100

[Message part 1 (text/plain, inline)]

On Nov 03, Ross Vandegrift <ross@kallisti.us> wrote:

> debootstrap 1.0.85 began deploying with --merged-usr by default in
> response to #839046.  On i386, this causes dpkg-shlibdeps to fail on
> (some?) shared libraries.
This is a more complex issue, since it does not happen on my i386 
system.
It has been discussed in #810499 but I am not sure about the best way to 
fix this.

-- 
ciao,
Marco

[signature.asc (application/pgp-signature, inline)]

Message #22 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: "843073@bugs.debian.org" <843073@bugs.debian.org>, "md@linux.it" <md@linux.it>

Subject: Re: Bug#843073: dpkg-shlibdeps: broken on i386 with merged /usr

Date: Sun, 6 Nov 2016 11:59:31 +0100

[Message part 1 (text/plain, inline)]

On Sat, 5 Nov 2016 22:40:30 +0100 md@Linux.IT (Marco d'Itri) wrote:
> On Nov 03, Ross Vandegrift <ross@kallisti.us> wrote:
> 
> > debootstrap 1.0.85 began deploying with --merged-usr by default in
> > response to #839046.  On i386, this causes dpkg-shlibdeps to fail on
> > (some?) shared libraries.
> This is a more complex issue, since it does not happen on my i386 
> system.

If you build systemd in a newly created (pbuilder) chroot, are you able
to reproduce the issue? I'd be suprised if not.

Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #27 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Felipe Sateler <fsateler@debian.org>

To: 843073@bugs.debian.org, "Marco d'Itri" <md@linux.it>, Michael Biebl <biebl@debian.org>

Subject: More information

Date: Mon, 7 Nov 2016 13:42:36 -0300

Hi,

The problem seems to occur in the Shlibs perl module[1]. In
particular, the function find_library looks in a list of configured
paths. But for each path, it checks if the path is a link to another
known path, and if so, it does not use the original link but the
resolved path. Thus for /lib, it detects that it is a link to
/usr/lib and uses the latter instead. Complicating things:

1. Many libraries live in /usr/lib/$triplet, but /lib/$triplet is not
a link to the latter. Thus this link resolving is not done.
2. On at least my (usrmerged) amd64 machine, /usr/lib64 is not a
symlink to /usr/lib, and ld-linux-86_64 lives there instead of
/usr/lib.

It's unclear what the best solution for this is. The indirection is
there due to bug #453885. Maybe find_library should be changed to
return a list of possible matches, and have dpkg-shlibdeps decide
which one to use by using the first one for which dpkg-query returns a
positive result. That would be an API break on libdpkg-perl though.

[1] https://sources.debian.net/src/dpkg/1.18.13/scripts/Dpkg/Shlibs.pm/#L161-L166

-- 

Saludos,
Felipe Sateler

Message #32 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: Michael Biebl <biebl@debian.org>, 843073@bugs.debian.org

Subject: Re: Bug#843073: dpkg-shlibdeps: broken on i386 with merged /usr

Date: Wed, 9 Nov 2016 17:10:30 +0000

[Message part 1 (text/plain, inline)]

Hi,

On Sat, Nov 05, 2016 at 08:58:06PM +0100, Michael Biebl wrote:
> As this breaks packages to build successfully, I'm bumping this to RC.
> E.g. I'm unable to successfully build systemd in a freshly created
> chroot on i386. I suspect once the buildds update their chroot, it will
> affect a lot more packages.

https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/systemd.html
shows that we could build systemd on sid. The builds are being done in
recently created pbuilder setups though.


-- 
cheers,
	Holger

[signature.asc (application/pgp-signature, inline)]

Message #37 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Holger Levsen <holger@layer-acht.org>, "843073@bugs.debian.org" <843073@bugs.debian.org>

Subject: Re: Bug#843073: dpkg-shlibdeps: broken on i386 with merged /usr

Date: Wed, 9 Nov 2016 19:16:53 +0100

[Message part 1 (text/plain, inline)]

Am 09.11.2016 um 18:10 schrieb Holger Levsen:
> Hi,
> 
> On Sat, Nov 05, 2016 at 08:58:06PM +0100, Michael Biebl wrote:
>> As this breaks packages to build successfully, I'm bumping this to RC.
>> E.g. I'm unable to successfully build systemd in a freshly created
>> chroot on i386. I suspect once the buildds update their chroot, it will
>> affect a lot more packages.
> 
> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/systemd.html
> shows that we could build systemd on sid. The builds are being done in
> recently created pbuilder setups though.

Do you also have builds for i386? (which this issue is about, amd64 is fine)


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #42 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Holger Levsen <holger@layer-acht.org>, "843073@bugs.debian.org" <843073@bugs.debian.org>

Subject: Re: Bug#843073: dpkg-shlibdeps: broken on i386 with merged /usr

Date: Wed, 9 Nov 2016 19:28:39 +0100

[Message part 1 (text/plain, inline)]

Hi Holger,

Am 09.11.2016 um 18:10 schrieb Holger Levsen:
> On Sat, Nov 05, 2016 at 08:58:06PM +0100, Michael Biebl wrote:
>> As this breaks packages to build successfully, I'm bumping this to RC.
>> E.g. I'm unable to successfully build systemd in a freshly created
>> chroot on i386. I suspect once the buildds update their chroot, it will
>> affect a lot more packages.
> 
> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/systemd.html
> shows that we could build systemd on sid. The builds are being done in
> recently created pbuilder setups though.

Are you sure those buildds actually use a merged-usr setup?
I've just checked the build log of 232-2, and the configure log has

checking for quotaon... /sbin/quotaon
checking for quotacheck... /sbin/quotacheck
checking for setcap... /usr/sbin/setcap
checking for kill... /bin/kill
checking for kmod... /bin/kmod
checking for kexec... /sbin/kexec
checking for sulogin... /sbin/sulogin
checking for mount... /bin/mount
checking for umount... /bin/umount

This indicates, that the chroot is not actually using a merged-usr
setup, otherwise it would have detected the binaries in /usr/bin.

Can you check the buildds and post the output of ls -l from the chroot's
root directory?
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #47 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: Michael Biebl <biebl@debian.org>

Cc: "843073@bugs.debian.org" <843073@bugs.debian.org>, reproducible-builds@lists.alioth.debian.org

Subject: Re: Bug#843073: dpkg-shlibdeps: broken on i386 with merged /usr

Date: Wed, 9 Nov 2016 18:41:31 +0000

[Message part 1 (text/plain, inline)]

Hi Michael,

On Wed, Nov 09, 2016 at 07:16:53PM +0100, Michael Biebl wrote:
> > https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/systemd.html
> Do you also have builds for i386? (which this issue is about, amd64 is fine)

yes, they are linked from that url as well, though you can also go
directly to
https://tests.reproducible-builds.org/debian/rb-pkg/unstable/i386/systemd.html

I've just checked
https://jenkins.debian.net/view/reproducible/view/Debian_setup_i386/job/reproducible_setup_pbuilder_unstable_i386_profitbricks2/257/console
and found nothing of usrmerge in there, which makes sense, as we use
debootstrap from stable…

what we could maybe do, is to use debootstrap from jessie-backports and vary this
regularily, so we would setup half of our pbuilders with merged /usr and the other
half without. cc:ing the reproducible builds list for feedback.


-- 
cheers,
	Holger

[signature.asc (application/pgp-signature, inline)]

Message #52 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: debian-boot@lists.debian.org, debian-devel@lists.debian.org

Subject: Re: Debian Installer Stretch Alpha 8 release

Date: Sat, 12 Nov 2016 19:17:28 +0100

Control: severity 843073 important

On Sat, 2016-11-12 at 18:37:52 +0100, Guillem Jover wrote:
> On Sat, 2016-11-12 at 16:24:32 +0100, Cyril Brulebois wrote:
> > Important change in this release of the installer
> > =================================================
> > 
> >  * debootstrap now defaults to merged-/usr, that is with /bin, /sbin,
> >    /lib* being symlinks to their counterpart in /usr (more details on:
> >    https://lists.debian.org/debian-devel/2016/09/msg00269.html).
> 
> This has caused build issues with dpkg-shlibdeps (#843073) on some
> architectures, which was a known problem (#810499) at the time of
> the default switch. :(
> 
> As I'm told the buildds regenerate the chroots every couple of weeks (?),
> which means several of those buildds will most probably start failing,
> I'd appreciate if the proponents of this change could either help
> Felipe Sateler (many thanks to him for working on this!) to track down
> and find an acceptable solution for dpkg-shlibdeps, or revert the change
> in debootstrap.

Ok, Mattia Rizzolo tells me (thanks!) that it's actually twice per
week, and that the debootstrap used is from jessie so this should not
be such a big deal for the Debian buildds. I'm thus lowering the
severity. All other comments still apply though.

> Otherwise I guess I might just reassign the dpkg-shlibdeps bug to both
> usrmerge and debootstrap.

Thanks,
Guillem

Message #59 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Helmut Grohne <helmut@subdivi.de>

To: Marco d'Itri <md@Linux.IT>, 843073@bugs.debian.org

Subject: Re: Bug#843073: dpkg-shlibdeps: broken on i386 with merged /usr

Date: Sat, 12 Nov 2016 22:05:56 +0100

On Sat, Nov 05, 2016 at 10:40:30PM +0100, Marco d'Itri wrote:
> This is a more complex issue, since it does not happen on my i386 
> system.
> It has been discussed in #810499 but I am not sure about the best way to 
> fix this.

It depends on the search order for library directories. dpkg-shlibdeps
parses /etc/ld.so.conf and has its own idea on which directories to
search built into the Dpkg::Shlibdeps module. Thus the failure comes and
goes as the search order changes. By changing the search order, you can
even reproduce in on amd64:

sbuild -d sid --add-depends=usrmerge --chroot-setup-commands="sed -i 's#^/usr##;t;s#^/lib#/usr&#' /etc/ld.so.conf.d/x86_64-linux-gnu.conf" dash

The crucial bit here is that dpkg-shlibdeps assumes that every single
library is provided in exactly one location and that it can use that
location to look up which package it belongs to. The /usr merge breaks
that assumption by providing e.g. libc.so.6 via two locations. Depending
on which location is found first, dpkg-shlibdeps gives up.

I conclude that dpkg-shlibdeps only works on merged /usr by sheer luck.
 
Helmut

Message #64 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Michael Biebl <biebl@debian.org>, debian-devel@lists.debian.org

Cc: 843073@bugs.debian.org

Subject: Re: Debian Installer Stretch Alpha 8 release

Date: Sun, 13 Nov 2016 16:14:26 +0100

Control: clone 843073 -1
Control: reassign -1 debootstrap 1.0.85
Control: retitle -1 debootstrap: Please revert merged-/usr by default as it breaks builds
Control: severity -1 serious
Control: affects -1 dpkg-dev
Control: severity 843073 wishlist
Control: block 810499 by 843073
Control: severity 810499 serious
Control: affects 810499 dpkg-dev

Hi!

On Sun, 2016-11-13 at 08:38:53 +0100, Helmut Grohne wrote:
> On Sat, Nov 12, 2016 at 08:55:43PM +0100, Michael Biebl wrote:
> > Is this really so for all buildds?
> > See #843433, the sparc64 buildds apparently do use a merged-usr chroot.
> 
> The issue depends on the loader path of the architecture. Although I do
> not understand why, it seems that /lib64 is less prone to exposing it
> than /lib is. We have people saying:
>  * not reproducible on amd64 /lib64 (Marco)
>  * not reproducible on sparc64 /lib64 (Michael)
>  * reproducible on i386 /lib (#810499)
>  * reproducible on armhf /lib (Uwe)
> 
> So the expectation I have is that it'll break:
>  * armel
>  * armhf
>  * i386
>  * mips
>  * mipsel
>  * s390x
> 
> Plus a number of non-release architectures.

Thanks for taking a look!

> > Hm, I would still consider it release critical, i.e. something which
> > needs to be fixed before we can release stretch. Otherwise this will
> > bite us later.
> 
> I agree.

Ok fine, this looks pretty worse than it looked before. So I'm rearranging
the bugs to where they belong.

> The /usr merge violates core assumptions in dpkg-shlibdeps. The reason
> that amd64 isn't broken is sheer luck.
> /etc/ld.so.conf.d/x86_64-linux-gnu.conf lists /lib before /usr/lib, so
> dpkg-shlibdeps considers that first. Swapping them or simply removing
> /lib (as seems reasonable on a merged /usr), breaks almost any build on
> amd64 (e.g. dash). The breakage on amd64 is simply hidden.

Right. I'm happy to assist people who want to fix this to try to find
a proper solution in dpkg-dev, but I'm not planning on spending time
on this on my own.

On Sun, 2016-11-13 at 12:04:07 +0100, Marco d'Itri wrote:
> On Nov 13, Helmut Grohne <helmut@subdivi.de> wrote:
> > Thus I think that debootstrap should revert to unmerged /usr until
> > dpkg-shlibdeps has been fixed. Fixing is non-trivial and likely requires
> > an archive rebuild on several architectures.

> Not really: dpkg-shlibdeps just needs to be fixed to search for 
> libraries in $directory and /usr/$directory, and then everything will 
> work again as usual.
> And yes, hacks like this one are a side effect of maintaining support 
> for non-merged systems.

Err, well exactly because usemerge is a major hack, and I'm actually
surprised we are deploying systems by default with that. As an
interesting thing to install and try out on ones system that's
perfectly fine though. But IMO if the merge needs to be done it should
be done by installing things into their final desired destination on
each package. Of course that makes split installations not possible,
but oh well.

Thanks,
Guillem

Message #75 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Michael Biebl <biebl@debian.org>, debian-devel@lists.debian.org, 843073@bugs.debian.org, Guillem Jover <guillem@debian.org>

Subject: Re: Debian Installer Stretch Alpha 8 release

Date: Mon, 14 Nov 2016 12:22:58 +0100

[Message part 1 (text/plain, inline)]

Control: tag -1 + patch
Control: severity -1 serious

Hi Guillem,

On Sun, 13 Nov 2016, Guillem Jover wrote:
> > The /usr merge violates core assumptions in dpkg-shlibdeps. The reason
> > that amd64 isn't broken is sheer luck.
> > /etc/ld.so.conf.d/x86_64-linux-gnu.conf lists /lib before /usr/lib, so
> > dpkg-shlibdeps considers that first. Swapping them or simply removing
> > /lib (as seems reasonable on a merged /usr), breaks almost any build on
> > amd64 (e.g. dash). The breakage on amd64 is simply hidden.
> 
> Right. I'm happy to assist people who want to fix this to try to find
> a proper solution in dpkg-dev, but I'm not planning on spending time
> on this on my own.

Please find two patches attached.

I checked that the command below was failing with the current dpkg-dev
and it did no longer fail with the updated one.

$ sbuild -d sid --add-depends=usrmerge --chroot-setup-commands="sed -i 's#^/usr##;t;s#^/lib#/usr&#' /etc/ld.so.conf.d/x86_64-linux-gnu.conf" dash

I believe my patch to be correct and clean.

> Err, well exactly because usemerge is a major hack, and I'm actually
> surprised we are deploying systems by default with that. As an

It's easier to push work upon others... to be honest the code (that I
wrote) in dpkg-dev that tries to identifiy the canonical version of the
library is also somewhat hackish.

I think the updated code that I submitted works better in all weird
corner cases that we could think of.

I'm happy to assist you shall any regression be found.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

[0001-Dpkg-Shlibs-add-find_library_locations.patch (text/x-diff, attachment)]

[0002-dpkg-shlibdeps-improve-logic-to-identify-package-own.patch (text/x-diff, attachment)]

Message #84 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: "debian-devel@lists.debian.org" <debian-devel@lists.debian.org>, "843073@bugs.debian.org" <843073@bugs.debian.org>, Guillem Jover <guillem@debian.org>, "844221@bugs.debian.org" <844221@bugs.debian.org>, Raphael Hertzog <hertzog@debian.org>

Subject: Re: Debian Installer Stretch Alpha 8 release

Date: Mon, 14 Nov 2016 12:50:05 +0100

[Message part 1 (text/plain, inline)]

Am 14.11.2016 um 12:22 schrieb Raphael Hertzog:

> Please find two patches attached.
> 
> I checked that the command below was failing with the current dpkg-dev
> and it did no longer fail with the updated one.
> 
> $ sbuild -d sid --add-depends=usrmerge --chroot-setup-commands="sed -i 's#^/usr##;t;s#^/lib#/usr&#' /etc/ld.so.conf.d/x86_64-linux-gnu.conf" dash
> 
> I believe my patch to be correct and clean.
> 
>> Err, well exactly because usemerge is a major hack, and I'm actually
>> surprised we are deploying systems by default with that. As an
> 
> It's easier to push work upon others... to be honest the code (that I
> wrote) in dpkg-dev that tries to identifiy the canonical version of the
> library is also somewhat hackish.
> 
> I think the updated code that I submitted works better in all weird
> corner cases that we could think of.
> 
> I'm happy to assist you shall any regression be found.

That's great, thanks a lot Raphael!

I debugged this issue myself this evening and came up with a patch on my
own. Your's is nicer und more generic though. So I'm happy to scrap mine.

Just for the record: I can confirm it fixes the problem in dpkg-shlibdeps.

I've CCed the debootrap bug, so the debian-boot people are aware that a
fix for dpkg is available and a revert in debootstrap is not necessary.

Guillem, it would be great if you can upload a fixed dpkg soon.
Julien Cristau told me, that he didn't want to see this issue unresolved
for too long.

Thanks everyone involved in getting this resolved.

Regards,
Michael



-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #89 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Felipe Sateler <fsateler@debian.org>

To: 843073@bugs.debian.org, Raphael Hertzog <hertzog@debian.org>

Subject: Re: Debian Installer Stretch Alpha 8 release

Date: Mon, 14 Nov 2016 11:19:41 -0300

[Message part 1 (text/plain, inline)]

On Mon, 14 Nov 2016 12:22:58 +0100 Raphael Hertzog <hertzog@debian.org>
wrote:
> Control: tag -1 + patch
> Control: severity -1 serious
>
> Hi Guillem,
>
> On Sun, 13 Nov 2016, Guillem Jover wrote:
> > > The /usr merge violates core assumptions in dpkg-shlibdeps. The reason
> > > that amd64 isn't broken is sheer luck.
> > > /etc/ld.so.conf.d/x86_64-linux-gnu.conf lists /lib before /usr/lib, so
> > > dpkg-shlibdeps considers that first. Swapping them or simply removing
> > > /lib (as seems reasonable on a merged /usr), breaks almost any build
on
> > > amd64 (e.g. dash). The breakage on amd64 is simply hidden.
> >
> > Right. I'm happy to assist people who want to fix this to try to find
> > a proper solution in dpkg-dev, but I'm not planning on spending time
> > on this on my own.
>
> Please find two patches attached.

Thanks for the patch. This looks better than what I had been working on.

Saludos

[Message part 2 (text/html, inline)]

Message #94 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: 843073@bugs.debian.org

Subject: Re: Debian Installer Stretch Alpha 8 release

Date: Mon, 14 Nov 2016 15:37:28 +0100

On Mon, 14 Nov 2016, Raphael Hertzog wrote:
> Please find two patches attached.

Given Felipe's comment about not breaking API for find_library()
I opted to create a new function, but a quick codesearch seems
to indicate that no other package is relying on this function.

https://codesearch.debian.net/search?q=find_library+filetype%3Aperl&perpkg=1

And the versioning of the module is 0.x indicating that the API
had not been declared stable.

So you can either get rid of find_library() entirely or replace
it entirely with the code find_library_locations() (and then drop
find_library_locations) to clean up the codebase.

dpkg-shlibdeps is the only user of that function within dpkg.

If you want me to rework the patches to do one of the suggested
cleanup, let me know.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

Message #101 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Felipe Sateler <fsateler@debian.org>

To: 844430@bugs.debian.org, 843073@bugs.debian.org, Wookey <wookey@wookware.org>

Subject: Re: dpkg: dpkg-shlibdeps failing due to /usr/lib/ld-linux-armhf.so.3 symlink

Date: Tue, 15 Nov 2016 15:53:28 -0300

Control: forcemerge 843073 844430

Hi,

On Tue, 15 Nov 2016 17:22:12 +0000 Wookey <wookey@wookware.org> wrote:
> Package: dpkg
> Version: 1.18.10
> Severity: normal
>
> I built a large package (mythtv) with this recipie:
>  git clone https://github.com/MythTV/packaging -b fixes/0.28
>  cd packaging/deb
>  ./build-debs.sh fixes/0.28
>
> This all works fine until the dh_shlibdeps when packing it up at the
> end.
>
> which complains about missing dependency info for
> /usr/lib/ld-linux-armhf.so.3

This is the same as #843073, so merging. Raphael has proposed a patch
on that bug, but it needs more testing before it is accepted.


Saludos,
Felipe Sateler

Message #110 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Wookey <wookey@wookware.org>

To: Felipe Sateler <fsateler@debian.org>

Cc: 844430@bugs.debian.org, 843073@bugs.debian.org

Subject: Re: dpkg: dpkg-shlibdeps failing due to /usr/lib/ld-linux-armhf.so.3 symlink

Date: Tue, 15 Nov 2016 21:17:33 +0000

[Message part 1 (text/plain, inline)]

On 2016-11-15 15:53 -0300, Felipe Sateler wrote:
> On Tue, 15 Nov 2016 17:22:12 +0000 Wookey <wookey@wookware.org> wrote:
> > Package: dpkg
> > Version: 1.18.10
> > Severity: normal
> >
> > I built a large package (mythtv) with this recipie:
> >  git clone https://github.com/MythTV/packaging -b fixes/0.28
> >  cd packaging/deb
> >  ./build-debs.sh fixes/0.28
> >
> > This all works fine until the dh_shlibdeps when packing it up at the
> > end.
> >
> > which complains about missing dependency info for
> > /usr/lib/ld-linux-armhf.so.3
> 
> This is the same as #843073, so merging. Raphael has proposed a patch
> on that bug, but it needs more testing before it is accepted.

Right. Cheers for that. (I failed to check the dpkg bugs first because
I started filing against glibc and checked that list, then realised
part way through that this was best filed against dpkg).

How is the usrmerge done? bind-mounting? hardlinks?

Is dpkg-shlibdeps the only thing that is going to wrong if files stop
having a canonical location in the system? It's a pretty major change
having every library (and a load of binaries?) appear twice? (Or do I
misunderstand how this is implemented?).

Like Guillem, I'd be a lot happier if files, especially libraries,
only existed in the place they existed, and if we want to move that
then we should move it in the packaging. Isn't there potential for
autoconf or libtool to get confused too?

Anyway, I'll test raphael's patch. 

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/

[signature.asc (application/pgp-signature, inline)]

Message #115 received at 843073@bugs.debian.org (full text, mbox, reply):

From: md@Linux.IT (Marco d'Itri)

To: Wookey <wookey@wookware.org>, 843073@bugs.debian.org

Cc: 844430@bugs.debian.org

Subject: Re: Bug#843073: dpkg: dpkg-shlibdeps failing due to /usr/lib/ld-linux-armhf.so.3 symlink

Date: Wed, 16 Nov 2016 00:16:02 +0100

[Message part 1 (text/plain, inline)]

On Nov 15, Wookey <wookey@wookware.org> wrote:

> How is the usrmerge done? bind-mounting? hardlinks?
rename(2)

> Is dpkg-shlibdeps the only thing that is going to wrong if files stop
> having a canonical location in the system? It's a pretty major change
> having every library (and a load of binaries?) appear twice? (Or do I
> misunderstand how this is implemented?).
This has been the only significant issue so far.
Red Hat released RHEL7 with a merged /usr and I am not aware of any 
troubles.

> Like Guillem, I'd be a lot happier if files, especially libraries,
> only existed in the place they existed, and if we want to move that
> then we should move it in the packaging. Isn't there potential for
> autoconf or libtool to get confused too?
Apparently not.
We will never be able to remove the /{lib*,bin,sbin} symlinks (they are 
more or less all requires by ABIs), but sooner or later we should drop 
support for non-merged systems and just install everything in /usr.

-- 
ciao,
Marco

[signature.asc (application/pgp-signature, inline)]

Message #120 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Wookey <wookey@wookware.org>

To: Felipe Sateler <fsateler@debian.org>

Cc: 844430@bugs.debian.org, 843073@bugs.debian.org

Subject: Re: dpkg: dpkg-shlibdeps failing due to /usr/lib/ld-linux-armhf.so.3 symlink

Date: Wed, 16 Nov 2016 03:44:57 +0000

Wookey wrote:
Anyway, I'll test raphael's patch. 

OK, so building dpkg is a bit tricky becuase it FTBFS with the same
ld-linux-armhf.so.3 dpkg-shlibdeps error, but adding a -l/lib/ in the
rules file fixed it.

With the new dpkg installed, mythtv did indeed run through
dh_shlibdeps OK, although there were a lot of "dpkg-shlibdeps:
warning: tried to merge the same object ($library) twice in a symfile"

So, yes that patch seems to work (if rather too noisily). 

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/

Message #125 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Felipe Sateler <fsateler@debian.org>

To: Wookey <wookey@wookware.org>

Cc: 844430@bugs.debian.org, 843073@bugs.debian.org

Subject: Re: dpkg: dpkg-shlibdeps failing due to /usr/lib/ld-linux-armhf.so.3 symlink

Date: Wed, 16 Nov 2016 08:59:02 -0300

[Message part 1 (text/plain, inline)]

On 15 November 2016 at 18:17, Wookey <wookey@wookware.org> wrote:
> On 2016-11-15 15:53 -0300, Felipe Sateler wrote:
>> On Tue, 15 Nov 2016 17:22:12 +0000 Wookey <wookey@wookware.org> wrote:
>> > Package: dpkg
>> > Version: 1.18.10
>> > Severity: normal
>> >
>> > I built a large package (mythtv) with this recipie:
>> > git clone https://github.com/MythTV/packaging -b fixes/0.28
>> > cd packaging/deb
>> > ./build-debs.sh fixes/0.28
>> >
>> > This all works fine until the dh_shlibdeps when packing it up at the
>> > end.
>> >
>> > which complains about missing dependency info for
>> > /usr/lib/ld-linux-armhf.so.3
>>
>> This is the same as #843073, so merging. Raphael has proposed a patch
>> on that bug, but it needs more testing before it is accepted.
>
> Right. Cheers for that. (I failed to check the dpkg bugs first because
> I started filing against glibc and checked that list, then realised
> part way through that this was best filed against dpkg).
>
> How is the usrmerge done? bind-mounting? hardlinks?

Softlinks. /bin => usr/bin, /sbin => usr/sbin , /lib => usr/lib,
/lib${multilib} => usr/lib/lib${multilib} .

> Is dpkg-shlibdeps the only thing that is going to wrong if files stop
> having a canonical location in the system? It's a pretty major change
> having every library (and a load of binaries?) appear twice? (Or do I
> misunderstand how this is implemented?).

Guillem pointed out on IRC that dpkg-query --search will also be confused.
For most things it shouldn't matter, as things will be where people expect
them. Some other programs are bound to be confused. systemd-delta, for
example, was also confused, but it was fixed already. To my knowledge,
dpkg-query --search and dlocate are the only ones (in fact, the problem in
dpkg-shlibdeps is because it uses dpkg-query --search).

> Like Guillem, I'd be a lot happier if files, especially libraries,
> only existed in the place they existed, and if we want to move that
> then we should move it in the packaging.

That would be great from my POV. However, it has some drawbacks: Stuff in
/bin cannot be moved to /usr/bin until /bin is a link to /usr/bin (ditto
sbin). Otherwise, for example fsck won't find the /sbin/fsck.$fs programs.
In general, way too many things hardcode paths in /bin or /sbin. So the
options are a period where package metadata does not match the filesystem,
or we have a long period of uninstallability until all packages + usrmerge
are installed.

> Isn't there potential for
> autoconf or libtool to get confused too?

Yes, there is some potential for confusion. If you program detects the path
of binaries or libraries and then hardcodes that, it may get paths wrong,
as it will probably choose the / path over the /usr one. This won't be a
problem for usrmerged systems, but it will for non-merged ones.

> Anyway, I'll test raphael's patch.

Thanks!

-- 

Saludos,
Felipe Sateler

[Message part 2 (text/html, inline)]

Message #130 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Michael Biebl <biebl@debian.org>

Cc: "debian-devel@lists.debian.org" <debian-devel@lists.debian.org>, "843073@bugs.debian.org" <843073@bugs.debian.org>, Guillem Jover <guillem@debian.org>

Subject: Re: Debian Installer Stretch Alpha 8 release

Date: Mon, 21 Nov 2016 16:38:07 +0100

Hello Guillem,

On Mon, 14 Nov 2016, Michael Biebl wrote:
> Just for the record: I can confirm it fixes the problem in dpkg-shlibdeps.
[...]
> Guillem, it would be great if you can upload a fixed dpkg soon.

A full week went by already. What's your plan?

I can offer to upload dpkg 1.18.15.1 to sid with just those patches if
it relieves you from handling an intermediary upload until you
release 1.18.16 on your usual schedule.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

Message #135 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Michael Biebl <biebl@debian.org>, "debian-devel@lists.debian.org" <debian-devel@lists.debian.org>, "843073@bugs.debian.org" <843073@bugs.debian.org>

Subject: Re: Debian Installer Stretch Alpha 8 release

Date: Mon, 21 Nov 2016 17:09:50 +0100

Just a brief reply.

On Mon, 2016-11-21 at 16:38:07 +0100, Raphael Hertzog wrote:
> On Mon, 14 Nov 2016, Michael Biebl wrote:
> > Just for the record: I can confirm it fixes the problem in dpkg-shlibdeps.
> [...]
> > Guillem, it would be great if you can upload a fixed dpkg soon.
> 
> A full week went by already. What's your plan?

First I have to go over a list of queued pending items and then I'll
get to this during this week. I have not yet reviewed the patches (in
part because I didn't do much Debian stuff last week due to lack of
motivation after an unpleasant interaction precisely due to this
issue, and TBH it has become one of those energey drainers), there's
a pending run in rebootstrap by Helmut (which is now waiting for a
gixed gcc), and then I need to write a mail summary of the current
situation and implications.

> I can offer to upload dpkg 1.18.15.1 to sid with just those patches if
> it relieves you from handling an intermediary upload until you
> release 1.18.16 on your usual schedule.

No, thanks. That's not the blocker. Also the release intervals will be
shorter for a while, at least until deeper freeze stages.

Regards,
Guillem

Message #140 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Michael Biebl <biebl@debian.org>, "debian-devel@lists.debian.org" <debian-devel@lists.debian.org>, "843073@bugs.debian.org" <843073@bugs.debian.org>

Subject: Re: Debian Installer Stretch Alpha 8 release

Date: Mon, 21 Nov 2016 17:17:57 +0100

On Mon, 2016-11-21 at 17:09:50 +0100, Guillem Jover wrote:
> On Mon, 2016-11-21 at 16:38:07 +0100, Raphael Hertzog wrote:
> > I can offer to upload dpkg 1.18.15.1 to sid with just those patches if
> > it relieves you from handling an intermediary upload until you
> > release 1.18.16 on your usual schedule.
> 
> No, thanks. That's not the blocker. Also the release intervals will be
> shorter for a while, at least until deeper freeze stages.

Oh, and forgot to mention, this issue has been known for over 8
months, and now there's this need to be pushy and rush things, etc.
I certainly do not appreciate that.

Regards,
Guillem

Message #145 received at 843073@bugs.debian.org (full text, mbox, reply):

From: md@Linux.IT (Marco d'Itri)

To: debian-devel@lists.debian.org, 843073@bugs.debian.org

Subject: Re: Bug#843073: Debian Installer Stretch Alpha 8 release

Date: Mon, 21 Nov 2016 17:23:19 +0100

[Message part 1 (text/plain, inline)]

On Nov 21, Guillem Jover <guillem@debian.org> wrote:

> Oh, and forgot to mention, this issue has been known for over 8
> months, and now there's this need to be pushy and rush things, etc.
> I certainly do not appreciate that.
No, not really: it was not clear (e.g. I could never reproduce it) until 
very recently.

-- 
ciao,
Marco

[signature.asc (application/pgp-signature, inline)]

Message #150 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Michael Biebl <biebl@debian.org>, "debian-devel@lists.debian.org" <debian-devel@lists.debian.org>, "843073@bugs.debian.org" <843073@bugs.debian.org>

Subject: Re: Debian Installer Stretch Alpha 8 release

Date: Mon, 21 Nov 2016 17:35:23 +0100

On Mon, 21 Nov 2016, Guillem Jover wrote:
> Oh, and forgot to mention, this issue has been known for over 8
> months, and now there's this need to be pushy and rush things, etc.
> I certainly do not appreciate that.

I have not been involved in this project so I don't know its history
but #843073 is not 8 months old and I only discovered the problem
when I read about in on debian-devel and yet I have followed former
discussions on debian-devel about merged-/usr.

I can understand that this project was of no interest to you and that
while you were aware of the problem for 8 months, you did not want to fix
it by yourself. But then I'm also sympathetic to people who want
to get their project moving forward and I'm always happy to assist them
when it touches code that I wrote in dpkg. Feel free to direct such people
towards me when you are not interested yourself.

In this specific case, the merged-/usr was mostly done and ready before
the freeze but then this problem (which was possibly underrated by Marco
or whoever else was aware of it) came into light. I would not like that we
end up with this feature reverted at freeze time because you did not find
the time to merge the patch. Hence my query.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

Message #155 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: debian-devel@lists.debian.org, 843073@bugs.debian.org

Subject: Re: Bug#843073: Debian Installer Stretch Alpha 8 release

Date: Wed, 30 Nov 2016 04:20:07 +0100

On Mon, 2016-11-21 at 17:23:19 +0100, Marco d'Itri wrote:
> On Nov 21, Guillem Jover <guillem@debian.org> wrote:
> > Oh, and forgot to mention, this issue has been known for over 8
> > months, and now there's this need to be pushy and rush things, etc.
> > I certainly do not appreciate that.

> No, not really: it was not clear (e.g. I could never reproduce it) until 
> very recently.

Well, it seems it was reproducible enough, that dpkg in Tanglu got a
workaround for it:

  <http://deriv.debian.net/patches/d/dpkg/Debian_dpkg_1.18.4_Tanglu_dpkg_1.18.4tanglu2.patch>

Regards,
Guillem

Message #162 received at 843073@bugs.debian.org (full text, mbox, reply):

From: md@Linux.IT (Marco d'Itri)

To: Guillem Jover <guillem@debian.org>, debian-devel@lists.debian.org, 843073@bugs.debian.org

Subject: Re: Bug#843073: dpkg-shlibdeps fix for merged /usr

Date: Mon, 12 Dec 2016 04:26:20 +0100

[Message part 1 (text/plain, inline)]

On Nov 21, Guillem Jover <guillem@debian.org> wrote:

> First I have to go over a list of queued pending items and then I'll
> get to this during this week. I have not yet reviewed the patches (in
> part because I didn't do much Debian stuff last week due to lack of
> motivation after an unpleasant interaction precisely due to this
> issue, and TBH it has become one of those energey drainers), there's
> a pending run in rebootstrap by Helmut (which is now waiting for a
> gixed gcc), and then I need to write a mail summary of the current
> situation and implications.
Do you have any updates about reviewing this patch?

-- 
ciao,
Marco

[signature.asc (application/pgp-signature, inline)]

Message #167 received at 843073@bugs.debian.org (full text, mbox, reply):

From: David Bremner <david@tethera.net>

To: 843073@bugs.debian.org

Subject: reproducible on amd64 for me

Date: Sun, 11 Dec 2016 20:30:07 -0800

I have a new stretch/amd64 install created by the alpha 8 installer.

Here is my recipe to reproduce

% sudo apt-get build-dep notmuch
% debcheckout notmuch
% cd notmuch
% make debian-snapshot

This failed for me 4/4 times.

It failed once on libc, twice on libz, and once on glib.

Message #172 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Helmut Grohne <helmut@subdivi.de>

To: 843073@bugs.debian.org, Raphael Hertzog <hertzog@debian.org>

Subject: Re: Bug#843073: Debian Installer Stretch Alpha 8 release

Date: Mon, 12 Dec 2016 09:02:22 +0100

Control: tags -1 - patch

On Mon, Nov 14, 2016 at 12:22:58PM +0100, Raphael Hertzog wrote:
> Please find two patches attached.
> 
> I checked that the command below was failing with the current dpkg-dev
> and it did no longer fail with the updated one.
> 
> $ sbuild -d sid --add-depends=usrmerge --chroot-setup-commands="sed -i 's#^/usr##;t;s#^/lib#/usr&#' /etc/ld.so.conf.d/x86_64-linux-gnu.conf" dash
> 
> I believe my patch to be correct and clean.

Thank you for taking the time to come up with a solution. Still, the
approach is wrong. dpkg-shlibdeps relies on dpkg -S to look for the
owning package of a library. We know that dpkg -S works badly with
symlinks and that has been why using symlinks for critical paths has
never been supported nor the default. The /usr merge changes that. On a
merged system, you get things like:

$ dpkg -S `which false`
dpkg-query: no path found matching pattern /usr/bin/false
$

This confuses both human and automated users of dpkg -S including
dpkg-shlibdeps. Once dpkg -S produces reasonable output for such paths,
dpkg-shlibdeps will just work. Patching dpkg-shlibdeps papers over the
bug, it doesn't fix it. Your patch is an improvement in clarity and
pinpoints the cause, but that's about it. dpkg -S has users beyond
dpkg-shlibdeps such as needrestart, massxpert, pcp, lxc,
debian-handbook, etc (as a quick codesearch "dpkg .*-S" and manual
inspection reveals).

Papering over the issue in dpkg-shlibdeps and trying to fix all
consumers does not look viable to me. Thus I am removing the patch tag
in the sense that it doesn't solve the problem in a way suitable for
stretch.

> It's easier to push work upon others... to be honest the code (that I
> wrote) in dpkg-dev that tries to identifiy the canonical version of the
> library is also somewhat hackish.

That's what you do, indeed. dpkg-shlibdeps is known to be very fragile.
Last time Guillem touched it in fairly trivial ways, stuff broke. Your
patch hasn't seen extensive testing yet and it changes the search order,
which is a key aspect of its fragility (in particular due to multilib).
Felipe has run some tests, but that's about it. Part of what resulted in
a delay here is that nobody ran the tests and when I had time to run
them, other changes in unstable (gnutls28 Build-Depends: libunbound-dev)
broke my test cases. Testing it now, but I'm quite annoyed by the amount
of breakage pushed on me.

Helmut

Message #179 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Helmut Grohne <helmut@subdivi.de>, 843073@bugs.debian.org

Subject: Re: Bug#843073: Debian Installer Stretch Alpha 8 release

Date: Mon, 12 Dec 2016 12:46:21 +0100

Control: tag -1 + patch
Control: severity 134758 + important

I add the patch tag back because this bug is about "dpkg-shlibdeps" being
broken by usrmerge. Not about dpkg -S being broken. I agree that dpkg -S
ought to be fixed too... but this corresponds to bug #134758. I'm thus
raising the severity of #134758 to important.

On Mon, 12 Dec 2016, Helmut Grohne wrote:
> Thank you for taking the time to come up with a solution. Still, the
> approach is wrong. dpkg-shlibdeps relies on dpkg -S to look for the
> owning package of a library.
[...]
> dpkg-shlibdeps. Once dpkg -S produces reasonable output for such paths,
> dpkg-shlibdeps will just work. Patching dpkg-shlibdeps papers over the
> bug, it doesn't fix it.

Yes, dpkg-shlibdeps relies on dpkg -S and knows how to parse its output.
dpkg -S being a very old interface, I believe that any "fix" in that
interface will break something... we will have to adapt code to understand
the new lines of output or we will have to make those new lines
conditional on a new command line option.

So I don't buy your argument of let's just fix "dpkg -S" and everything
will be fine. We will have to fix "dpkg -S" and any code that parses the
output of "dpkg -S".

> Your patch is an improvement in clarity and pinpoints the cause, but
> that's about it.

The patch replaces a fragile approach trying to avoid non-canonical paths
with a more solid one where we look up all possible paths against the
dpkg database.

It improves dpkg-shlibdeps whether or not we end up using usrmerge for
stretch. It should be merged.

Following your suggestion on IRC, I am fine for it to be committed
with limited review and then being reverted if it breaks anything (I'm
also available to handle any subsequent bugreport).

> dpkg -S has users beyond
> dpkg-shlibdeps such as needrestart, massxpert, pcp, lxc,
> debian-handbook, etc (as a quick codesearch "dpkg .*-S" and manual
> inspection reveals).

debian-handbook (my book) documents it :-)

> Papering over the issue in dpkg-shlibdeps and trying to fix all
> consumers does not look viable to me. Thus I am removing the patch tag

As I said, I believe that all consumers will have to be fixed (or at least
reviewed to ensure they are not confused by the new output we would have).

> > It's easier to push work upon others... to be honest the code (that I
> > wrote) in dpkg-dev that tries to identifiy the canonical version of the
> > library is also somewhat hackish.
> 
> That's what you do, indeed. dpkg-shlibdeps is known to be very fragile.
> Last time Guillem touched it in fairly trivial ways, stuff broke. Your
> patch hasn't seen extensive testing yet and it changes the search order,
> which is a key aspect of its fragility (in particular due to multilib).

What search order does it change? It still looks up libraries in the same
ordered list of directories.

> broke my test cases. Testing it now, but I'm quite annoyed by the amount
> of breakage pushed on me.

Sorry for this, but that's just our life of free software contributors, we
constantly have to adapt to changes made by others. It's hard to avoid it.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

Message #186 received at 843073@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: debian-devel@lists.debian.org, 843073@bugs.debian.org

Subject: Re: Bug#843073: dpkg-shlibdeps fix for merged /usr

Date: Thu, 15 Dec 2016 13:37:40 +0100

On Mon, 2016-12-12 at 04:26:20 +0100, Marco d'Itri wrote:
> On Nov 21, Guillem Jover <guillem@debian.org> wrote:
> > First I have to go over a list of queued pending items and then I'll
> > get to this during this week. I have not yet reviewed the patches (in
> > part because I didn't do much Debian stuff last week due to lack of
> > motivation after an unpleasant interaction precisely due to this
> > issue, and TBH it has become one of those energey drainers), there's
> > a pending run in rebootstrap by Helmut (which is now waiting for a
> > gixed gcc), and then I need to write a mail summary of the current
> > situation and implications.

> Do you have any updates about reviewing this patch?

Now that Helmut has done an initial successful build with rebootstrap,
I'm planning on reviewing and merging this after having sent that summary
mail, either for .16 or .17, which I'd like to upload in quick succession,
starting this week.

Thanks,
Guillem

Message #189 received at 843073-submitter@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: 843073-submitter@bugs.debian.org

Subject: Bug#843073 in package dpkg marked as pending

Date: Mon, 19 Dec 2016 02:45:19 +0000

Control: tag 843073 pending

Hi!

Bug #843073 in package dpkg reported by you has been fixed in
the dpkg/dpkg.git Git repository. You can see the changelog below, and
you can check the diff of the fix at:

    https://anonscm.debian.org/cgit/dpkg/dpkg.git/diff/?id=a927295

---
commit a927295c93fb7a17742441aa863aaffcf4a351b5
Author: Raphaël Hertzog <hertzog@debian.org>
Date:   Mon Nov 14 11:46:09 2016 +0100

    dpkg-shlibdeps: Improve logic to identify packages owning a library
    
    With things like merged-/usr, a system might have libraries that are
    stored for example in /usr/lib but that dpkg knows under /lib. This
    breaks some of the initial assumptions made in dpkg-shlibdeps.
    
    We now scan all possible paths for a given library (instead of trying
    to guess which one is the canonical one) and whenever we find a match
    in the dpkg database, we also associate the package for the associated
    realpath(). That way when a library is not properly identified, we can
    fallback on looking if its realpath is known and be confident that if
    the library was packaged, we did identify it correctly.
    
    [guillem@debian.org:
     - Fold find_library_locations() into find_library().
     - Minor coding style fixes.
     - Squash the two separate commits and cleanup the commit message. ]
    
    Closes: #843073
    Signed-off-by: Guillem Jover <guillem@debian.org>

diff --git a/debian/changelog b/debian/changelog
index 6ee9b0e..e85be87 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -5,6 +5,9 @@ dpkg (1.18.17) UNRELEASED; urgency=medium
     are optional. Regression introduced in dpkg 1.18.16. Closes: #848422
   * Validate versions too in dpkg-maintscript-helper symlink_to_dir and
     dir_to_symlink commands.
+  * Improve logic to identify packages owning a library in dpkg-shlibdeps,
+    by handling the same library appearing more than once on the filesystem
+    Thanks to Raphaël Hertzog <hertzog@debian.org>. Closes: #843073
   * Documentation:
     - Clarify that dpkg-buildpackage does not run «dpkg-source --before-build»
       when using the -T option. Closes: #649531

Message #196 received at 843073-close@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: 843073-close@bugs.debian.org

Subject: Bug#843073: fixed in dpkg 1.18.17

Date: Mon, 19 Dec 2016 03:07:28 +0000

Source: dpkg
Source-Version: 1.18.17

We believe that the bug you reported is fixed in the latest version of
dpkg, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 843073@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Guillem Jover <guillem@debian.org> (supplier of updated dpkg package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Mon, 19 Dec 2016 02:43:10 +0100
Source: dpkg
Binary: dpkg libdpkg-dev dpkg-dev libdpkg-perl dselect
Architecture: source
Version: 1.18.17
Distribution: unstable
Urgency: medium
Maintainer: Dpkg Developers <debian-dpkg@lists.debian.org>
Changed-By: Guillem Jover <guillem@debian.org>
Description:
 dpkg       - Debian package management system
 dpkg-dev   - Debian package development tools
 dselect    - Debian package management front-end
 libdpkg-dev - Debian package management static library
 libdpkg-perl - Dpkg perl modules
Closes: 649531 843073 848422
Changes:
 dpkg (1.18.17) unstable; urgency=medium
 .
   [ Guillem Jover ]
   * Handle empty versions on validation in dpkg-maintscript-helper as they
     are optional. Regression introduced in dpkg 1.18.16. Closes: #848422
   * Validate versions too in dpkg-maintscript-helper symlink_to_dir and
     dir_to_symlink commands.
   * Improve logic to identify packages owning a library in dpkg-shlibdeps,
     by handling the same library appearing more than once on the filesystem
     Thanks to Raphaël Hertzog <hertzog@debian.org>. Closes: #843073
   * Documentation:
     - Clarify that dpkg-buildpackage does not run «dpkg-source --before-build»
       when using the -T option. Closes: #649531
     - Document that dpkg sets DPKG_ADMINDIR on maintainer script environment.
   * Packaging:
     - Do not let dh_perl insert a dependency for libdpkg-perl, because it
       emits perl instead of perl:any. Instead hardcode a perl:any dependency
       to comply with the perl policy and make the Multi-Arch hinter happy.
 .
   [ Updated scripts translations ]
   * German (Helge Kreutzmann).
 .
   [ Updated man pages translations ]
   * German (Helge Kreutzmann).
Checksums-Sha1:
 7dcc4a6c3a418c821817f913228a26e1e352102c 2032 dpkg_1.18.17.dsc
 6075fb692aa80b53283c8f913b2b8a1549443f91 4501292 dpkg_1.18.17.tar.xz
Checksums-Sha256:
 3af8240f15f0dc542afc8f906eddf167dd7f96e5695ca786f1bc11e0204e4dcb 2032 dpkg_1.18.17.dsc
 7cd322f667eb2d3cbc96b98f7b9c0a8840e62ff2711e1fce5da85fc43187e51c 4501292 dpkg_1.18.17.tar.xz
Files:
 fa726149cf5c8084c7ff508cca108b46 2032 admin required dpkg_1.18.17.dsc
 e8a3ab93c2b8e882e64bdf9e378221be 4501292 admin required dpkg_1.18.17.tar.xz

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEETz509DYFDBD1aWV0uXK/PqSuV6MFAlhXR8gACgkQuXK/PqSu
V6Nz8xAAufgHA45PxTGfaE17t0sxFj+9nG+v82deO0rJ4hLADLdCAXSdBdMmZlD0
dYV3qNrZHbwIVI/tDqWFjrphrKw62TNVuMmaLBywAb7e2ftcNpRDj+F2ryYqBVkX
4taNBa/oS2NajcPuFoCe9u8L7g2zyBVUADnS1rxf3xQ+LDKmjrXBZXurE9Ig5jCi
bAbUllJkmE+u6mNHJYtTDqSdd5wcJjI7mO4VoGPgsnHupq2wqlzdM+015cQzRjvI
rGx51tM4reXQZL2QZIwuR8aOQ4WlAsib8BTYnJPVer2GEN/SP5766sNxCuUoV/cj
6ghZhNqZ504Zoqnuf/u5oYpbTH6DXWI7bq+c8HzuIT6VejZpSG4OqmMouNerBV7u
5K03KaNGl06AaRQvzmYOeSVPB+0P6Ec1eOYn5VHEJbOUuV5fEq4tSHlOR0t0l9My
6/hQY9e/3EZa4AYIPv9D/tCcEWB4tamxZFnpmlCqOYRjLDJhQd1035jCHinvg1hY
lFiMgi0wL47fMHZFli2e4moW3kHwzAfOrE7tYGv9YIy0C/k6AOoRswMg1hX75RRN
9rho1tK4kXeVd699muRzjyVa+RyuFYKc8RrMkyBOKvjf4j8wlrRIVkYxkopMale2
HkUx3jOkvjG+gHKbY2tUb1njpqKrq87IvAZL2Wwp4PRHUHr1dJQ=
=bcJM
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.