Debian Bug report logs - #340904
Inclusion of .private's "-I"s

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Loic Minier <lool@dooz.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Inclusion of .private's "-I"s

Date: Sat, 26 Nov 2005 22:23:16 +0100

Package: pkg-config
Version: 0.20-1
Severity: normal

        Hi,

 (we had this discussion on IRC already)

 It might be nice to use "-I" of the .private pseudo-headers in .pc
 files each time some -I flags are requested, eg. not only for static
 builds.

 As I understand it, the introduction of .private pseudo-headers was
 meant to lower the number of ELF DEPENDS as caused by the use of "-l"
 flags when linking objects with the help of pkg-config.
   These flags are only needed when linking statically, and pkg-config
 offers a "--static" flag to choose the type of linking one wants.

 The problems arise for header files of a package distributing a .pc
 file when these are including third party headers;  for example, the
 vte headers include the Atk, Pango, Gtk and other headers.

 While there might be some discussion on whether the ELF DEPENDS should
 reflect these header inclusions, it seems quite safe to assume that all
 -I, from private or non private headers, should be present when
 building objets, for static linking or not.

   Bye,

-- 
Loïc Minier <lool@dooz.org>

Message #10 received at 340904@bugs.debian.org (full text, mbox, reply):

From: Loïc Minier <lool@dooz.org>

To: 340904@bugs.debian.org

Subject: Re: Inclusion of .private's "-I"s

Date: Wed, 14 Dec 2005 22:39:30 +0100

On sam, nov 26, 2005, Loic Minier wrote:
>  (we had this discussion on IRC already)

 More bits, the new information here is that people involved in
 pkg-config think that using headers of another lib in your headers
 means you have a public dep on this other lib, and that will result in
 an ELF dep.  It also clarifies that the improvement is because of the
 removal of the static flags which caused dynamic deps, and won't
 improve gtk-ish/gnome-ish libs.

21:55 < lool> vorlon: if you have two minutes, could you please send a word to 
              <http://bugzilla.gnome.org/show_bug.cgi?id=322240>?
21:57 < Q_> lool: Still waiting for a fix so I can build gtk-sharp2.
21:58 < lool> Q_: well, I'm still waiting for a consensus to be reached, I 
              think the stronger opinion was to add only -Is from private fields
21:58 < lool> but this is all too borderline for me to take the decision, one 
              way or the other
22:00 < Q_> Mithrandir: Around?
22:01 < lool> Q_: BTW, I filed #340904 to track the story
22:02 < vorlon> lool: what word are you looking for? :)
22:03 < lool> vorlon: the kind of word carrying final-person opinion (eg where 
              I don't relay opinions from third parties :)
22:04 < lool> this is really a pkg-config design question for me, and the 
              technical implementation is not the problem, so I'd rather have 
              you or Mithrandir or who ever feels he knows what is correct 
              express himself there, than me
22:05 < vorlon> lool: well, Mithrandir and I seem to disagree here on what the 
                correct design is, and one of is is pkg-config upstream whereas 
                the other is not. :)
22:06 < Q_> lool: And nothing happened to it yet.
22:06 < lool> vorlon: exactly, and since I'm not strongly convinced one way or 
              the other, I can't represent either party in the discussion  O:-)
22:08 < vorlon> lool: well, I don't see much point in trying to argue the 
                pkg-config design details in a bug report against *vte*: that's 
                not fair to the vte maintainers, and it's not a very effective 
                means of getting pkg-config changes made.  I don't feel 
                strongly enough about this corner case today to argue it at 
                *all*, really.
22:10 < lool> vorlon: well, I wanted your take on the situation to vte's 
              upstream which is now victim of the change too, I can try to 
              explain the situation myself to vte's upstream, but I will be 
              short of arguments what ever camp vte's upstream picks
22:10  * lool bravely goes telling vte maintainer that we've pulled them in a 
          mid-term problem
22:12 < Mithrandir> Q_: yes
22:13 < vorlon> lool: I don't think vte upstream is going to pick the "I'll let 
                my software continue to be broken" camp
22:14 < lool> exactly
22:14 < Q_> Mithrandir: So, did you have some time to think about the problem, 
            and the best way to solve it?
22:14 < dilinger> i wish i could admin some AIX machines
22:15 < dilinger> i bet it would cause me to get a lot less frustrated when i'm 
                  faced w/ crap like redhat and suse
22:15 < Mithrandir> Q_: oh, the requires.private stuff?
22:16 < Q_> Mithrandir: Yes, the -I
22:17 < Mithrandir> 02:39 < jamesh> Mithrandir: about your pkg-config question: 
                    no.  If a package is a private dependency, then its 
                    interface shouldn't be exposed at all.
22:17 < Mithrandir> 02:39 < jamesh> Mithrandir: so the -I flags shouldn't be 
                    included
22:18 < Q_> You mean we should try to convince jamesh?
22:18 < Mithrandir> I'm somewhat inclined to agree with jamesh, but neuro had 
                    some comments which went in the other direction.  I think 
                    I'll have to sit down and look at what can go wrong in the 
                    different situations.
22:18 < Mithrandir> yes, if you convince jamesh, I'm ok with it.  He knows a 
                    bloody lot about how pkg-config works, is supposed to work 
                    and why it works that way.
22:19  * lool wishes for a clearer definition of the goal of the 
          requires.private header
22:21 < vorlon> lool: well, this opinion from jamesh is consistent with 
                treating glib as a *public* dependency of gtk just because 
                software can cheat and use glib API calls directly when only 
                including gtk. :P
22:23 < Mithrandir> vorlon: yes, but what if glib was a private dependency of 
                    gtk?  Would it be ok to expose, say, gchar*?
22:24 < Mithrandir> (or better, gchar, since a pointer is always ok)
22:25 < vorlon> Mithrandir: if gchar is used in the gtk headers, yes. :)
22:25 < Q_> Mithrandir: A pointer isn't any better.
22:25 < vorlon> Q_: it's legal to have pointers in C that you don't have enough 
                information to be able to dereference.
22:26 < Mithrandir> Q_: sure it is, as long as you don't expose the typedef.
22:26 < Mithrandir> so you can't dereference it, as vorlon says.
22:27 < Q_> vorlon: But you still need to say that it's a type.
22:28 < Mithrandir> Q_: a struct would be a better example and then you can 
                    just say struct foo; and use struct foo* without any 
                    trouble.
22:28 < Q_> so something like "struct foo *bar;" works, "struct foo bar;" 
            doesn't.
22:29 < Q_> But at that point you don't even need the include in the first 
            place.
22:30 < lool> vorlon: if you look at private/public deps in this way, then my 
              feeling is that this clarifies deps for static linking, but won't 
              help reducing dynamic deps
22:31 < vorlon> lool: ?
22:31 < Mithrandir> lool: and we want to reduce dynamic deps.  Possibly not 
                    perfectly (stuff linked to gtk will still link to glib, 
                    since it's an actual public dependency), but better than 
                    today.
22:31 < vorlon> lool: which way?
22:32 < lool> vorlon: in the way of jamesh proposing to treat such interfaces 
              as public deps
22:32 < lool> anyone minds if I copy paste this log to the bug?
22:33 < Mithrandir> lool: feel free
22:33 < lool> Mithrandir: true, it's just sad this helps so little in the case 
              of gtk-ish libs
22:35 < Q_> Anyway, I think the point was if that gchar is part of gtk headers, 
            and gchar changes in such a way that it breaks gtk, and so also 
            things linked to gtk, that both got an api change, and both should 
            do an soname change.
22:35 < vorlon> lool: it's less than ideal, but there are still *some* lib 
                savings.
22:36 < Mithrandir> I would like to accomodate the gnome/gtk model of handling 
                    headers, though.
22:36 < Q_> And that might not be something the gnome people would like to do?
22:37 < lool> Q_: well they included vorlon's patch quite rapidly
22:37 < lool> Q_: I suppose they will use any pkg-config coolness you explain 
              them to use
22:38 < Q_> lool: I think they don't want to and change all their sonames 
            because something in glib changed.
22:38 < Q_> s/to/to go/

-- 
Loïc Minier <lool@dooz.org>

Message #15 received at 340904@bugs.debian.org (full text, mbox, reply):

From: Loïc Minier <lool@dooz.org>

To: 340904@bugs.debian.org

Subject: Re: Inclusion of .private's "-I"s

Date: Thu, 15 Dec 2005 19:59:37 +0100

 (End of the IRC discussion:)

22:42 < Q_> But it's the right way to do it, just because they can relink 
            everything to the new glib, it's not the right way to do it.
22:43 < vorlon> Q_: actually, IME they'll use any excuse to change sonames 
                between stable branches
22:44 < Q_> They do?
22:44 < Q_> Then I don't see why they would have a problem with it.
22:46 < Q_> Mithrandir: Do you have any idea around what time jamesh is around 
            ussualy?
22:46 < Mithrandir> if you have App A, lib B and lib C.  B links to C.  A uses 
                    B directly, but also dereferences structs which it has 
                    gotten from C, through B.  C then changes soname (new ABI, 
                    same API), but not removed from the system, B is recompiled 
                    on the system without any soname changes.  Is there any way 
                    you can make A continue working?
22:46 < Mithrandir> Q_: he lives in western .au, so he might be around in a 
                    couple of hours.
22:47 < lool> Mithrandir: ah you make me think of something really painful, 
              there are weird macros accessing structs in headers
22:48 < lool> so including headers of lib C in lib B might be enough for app A 
              to get smoe code dependent on the structs in C added to it and 
              hence crashing with a newer C
22:49 < Mithrandir> I'm wondering if my case will work correctly if A is linked 
                    with both B and C and C has versioned symbols.
22:49 < Q_> Mithrandir: I don't think you can keep A working without also 
            relinking it.
22:49 < lool> I'm not sure I have a real life example, but I think the fact 
              it's possible might be enough to try to guard against it
22:56 < vorlon> Mithrandir: in that scenario, why is it *not* a bug for B to be 
                recompiled without an soname change?
22:57 < Q_> Mithrandir: A will get linked to 2 lib C's in that case.
22:57 < Mithrandir> Q_: that's fine if they have versioned symbols, AIUI?
22:57 < vorlon> Mithrandir: if the headers from C are being included into B, it 
                must be for a reason; I can't think of any non-ABI-breaking 
                reasons for this other than including trivial enums or defines, 
                and that just tells me C's headers should be more granular and 
                B shouldn't be linking to C either.
22:58 < vorlon> hmm, ok.  possibility: C makes ABI changes to parts that aren't 
                referenced by B. Meh.
22:59 < Mithrandir> vorlon: I didn't talk about headers, I talked about 
                    linking.  A might very well include C's headers itself, to 
                    get struct definitions.
22:59 < Mithrandir> I was wondering if you could make it work or not.
22:59 < Q_> Mithrandir: If A include's C's headers, it should link to it itself.
22:59 < vorlon> oh.  then I guess you're only talking about the usual 
                "versioned symbols" case.
23:00 < Mithrandir> vorlon: yes, I might very well be, and that works, doesn't 
                    it?
23:00 < Mithrandir> Q_: yes, I meant to write that, but didn't
23:00 < vorlon> Mithrandir: yes, it does.
23:00 < Q_> Mithrandir: And versioned symbols might make it work, but I don't 
            think in all cases.
23:01 < vorlon> Q_: it works unless B's API is pathological.
23:01 < vorlon> (takes as arguments objects which are defined by C)
23:03 < Mithrandir> I'm thinking that jamesh is correct, but I can't really 
                    tell you why. :-/
23:07 < Mithrandir> well, I gotta sleep.  I have some vacation in a little less 
                    than a week, I'll see if I can get to it then.  Or in the 
                    weekend.

-- 
Loïc Minier <lool@dooz.org>

Message #24 received at 340904@bugs.debian.org (full text, mbox, reply):

From: Steve Langasek <vorlon@debian.org>

To: David Nusinow <david_nusinow@verizon.net>

Cc: 349318@bugs.debian.org, 340904@bugs.debian.org

Subject: Re: Bug#349318: fixed in xft 2.1.8.2-1

Date: Sun, 22 Jan 2006 22:59:24 -0800

[Message part 1 (text/plain, inline)]

reopen 349318
block 349318 by 340904
thanks

>    * Add 001_no_export_freetype.diff. This is a modified patch from Steve
>      Langasek that causes xft.pc not to export freetype or xrender as part of
>      the interface, but hide them in Requires.private instead (closes: #349318)

The patch is:

Index: libXft-X11R7.0-2.1.8.2/xft.pc.in
===================================================================
--- libXft-X11R7.0-2.1.8.2.orig/xft.pc.in      2005-12-30 14:50:49.000000000 -0500
+++ libXft-X11R7.0-2.1.8.2/xft.pc.in   2006-01-22 22:12:27.000000000 -0500
@@ -6,7 +6,7 @@
 Name: Xft
 Description: X FreeType library
 Version: @VERSION@
-Requires: xproto, xrender, fontconfig, freetype2
-Requires.private: xrender, fontconfig, freetype2
+Requires: xproto, fontconfig
+Requires.private: xrender, freetype2
 Cflags: -I${includedir}
 Libs: -L${libdir} -lXft

This won't work because of the aforementioned bug #340904:

$ grep -h include include/X11/Xft/*.h
#include <X11/Xfuncproto.h>
#include <stdarg.h>
#include <ft2build.h>
#include FT_FREETYPE_H
#include <fontconfig/fontconfig.h>
#include <X11/extensions/Xrender.h>
#include <X11/Xfuncproto.h>
/* #include <X11/Xosdefs.h>*/
#include <X11/Xft/XftCompat.h>
$

Both the xrender and freetype2 headers are included in Xft/Xft.h, and types
from both of these headers are exported in the Xft API -- so any software
using Xft.h needs to be able to find these other headers as well.
Currently, pkg-config does *not* pass cflags from packages listed in
Requires.private when called as pkg-config --cflags <foo>.

So far, the maintainer has resisted changing pkg-config to export cflags
(which should really be called cppflags...) from Requires.private due to a
fallacious argument regarding the nature of "private" dependencies.  There
are three real use cases for libraries which depend on other libraries:

- the library intentionally exports the API and ABI of its dependencies when
  linked to, and therefore both the ldflags and the cflags of its
  dependencies should be exported by pkg-config in all cases[1]
- the library intentionally includes headers from dependencies in its own
  headers in order to inherit type definitions, but these definitions are
  not intended for direct consumption by users of this library alone;
  therefore pkg-config must export the cflags from dependencies in all
  cases, but the ldflags only when linking statically
- the library's API includes no headers from its dependencies; pkg-config
  needs to export the ldflags of private dependencies when statically
  linking but not when dynamically linking, and should *never* need to
  export the cflags of these headers.

Please note that today, the handling of Requires.private in pkg-config maps
to *none* of these cases -- I can't think of a single situation in which
cflags of dependencies are needed when statically linking, and not needed
when dynamically linking!  Instead, the Requires.private handling is
adequate for the third case; handling of Requires is correct for the first
case; and there is no combination of options that is appropriate for the
second case.

This is unfortunate, because there are a great many packages that are
inheriting dependencies this way on libraries they don't use.  While it is
true that an ABI change in the dependent library will *sometimes* mean an
ABI change in the depending library, this is not always the case.  As a
result, this behavior of pkg-config causes unnecessary churn for packages
depending on libraries in this scenario.  In the case of libxft2, that's
over 400 packages in Debian that are potentially affected.

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
vorlon@debian.org                                   http://www.debian.org/

[1] Personally, I think this is completely specious, but there are GNOME
people who insist that this describes the relationship between gtk+2.0
glib.

[signature.asc (application/pgp-signature, inline)]

Message #31 received at 340904@bugs.debian.org (full text, mbox, reply):

From: James Henstridge <james@jamesh.id.au>

To: Tollef Fog Heen <tfheen@err.no>, Steve Langasek <vorlon@debian.org>

Cc: 349318@bugs.debian.org, 340904@bugs.debian.org, David Nusinow <david_nusinow@verizon.net>

Subject: Re: [Steve Langasek] Bug#340904: Bug#349318: fixed in xft 2.1.8.2-1

Date: Wed, 08 Feb 2006 19:28:39 +0800

Tollef Fog Heen wrote:

>Hi, any chance you could comment on this?  I'm somewhat inclined to
>agree with Steve, but you know this stuff better than I do, so I'd
>value a second opinion.
>
>Also, if there's anything unclear or anything, please do mail Steve
>(and the bug) with me in Cc.
>  
>
[snip]

>  
>
>Both the xrender and freetype2 headers are included in Xft/Xft.h, and types
>from both of these headers are exported in the Xft API -- so any software
>using Xft.h needs to be able to find these other headers as well.
>Currently, pkg-config does *not* pass cflags from packages listed in
>Requires.private when called as pkg-config --cflags <foo>.
>  
>

The fact that the headers are included makes them part of the _public_
ABI of Xft.  There are Xft functions that accept or return types defined
in the Xrender and freetype2 headers.

If Xft is updated to a new version of either of those libraries such
that those types are defined differently (altered struct layout,
different type sizes, etc), then the app also needs to be updated to the
new version.

>So far, the maintainer has resisted changing pkg-config to export cflags
>(which should really be called cppflags...) from Requires.private due to a
>fallacious argument regarding the nature of "private" dependencies.  There
>are three real use cases for libraries which depend on other libraries:
>
>- the library intentionally exports the API and ABI of its dependencies when
>  linked to, and therefore both the ldflags and the cflags of its
>  dependencies should be exported by pkg-config in all cases[1]
>  
>
Yep, this is the use case addressed by "Requires".

>- the library intentionally includes headers from dependencies in its own
>  headers in order to inherit type definitions, but these definitions are
>  not intended for direct consumption by users of this library alone;
>  therefore pkg-config must export the cflags from dependencies in all
>  cases, but the ldflags only when linking statically
>  
>
Changes to type definitions _do_ change the ABI.  If library A uses
library B's types in its ABI, then it's ABI will break if library B
changes those types.  An app using library A should definitely record
the version of library B being used.

>- the library's API includes no headers from its dependencies; pkg-config
>  needs to export the ldflags of private dependencies when statically
>  linking but not when dynamically linking, and should *never* need to
>  export the cflags of these headers.
>  
>
This is the use case for "Requires.private".  If the dependency doesn't
form part of the library's API, then it obviously doesn't require direct
linkage and is private.

>Please note that today, the handling of Requires.private in pkg-config maps
>to *none* of these cases -- I can't think of a single situation in which
>cflags of dependencies are needed when statically linking, and not needed
>when dynamically linking!
>
The cflags of dependencies listed in "Requires.private" are not included
for either dynamic or static modes of pkg-config, so maps to case 3
quite well.

Consider the following two .pc files.  First "foo.pc":
  Name: foo
  Description: foo
  Version: 1.0
  Cflags: foo-cflags
  Libs: foo-libs

And "bar.pc":
  Name: bar
  Description: bar
  Version: 1.0
  Cflags: bar-cflags
  Libs: bar-libs
  Requires.private: foo

For the dynamic linking case:
  $ pkg-config --cflags bar
  bar-cflags
  $ pkg-config --libs bar
  bar-libs

For the static linking case:
  $ pkg-config --static --cflags bar
  bar-cflags
  $ pkg-config --static --libs bar
  bar-libs foo-libs

>  Instead, the Requires.private handling is
>adequate for the third case; handling of Requires is correct for the first
>case; and there is no combination of options that is appropriate for the
>second case.
>  
>
I'd argue that the second case is a strawman.  If a particular library
exposes another in its API/ABI, then it is part of case 1.

>This is unfortunate, because there are a great many packages that are
>inheriting dependencies this way on libraries they don't use.  While it is
>true that an ABI change in the dependent library will *sometimes* mean an
>ABI change in the depending library, this is not always the case.  As a
>result, this behavior of pkg-config causes unnecessary churn for packages
>depending on libraries in this scenario.  In the case of libxft2, that's
>over 400 packages in Debian that are potentially affected.
>  
>
Sure, unnecessary churn is bad and should be avoided.  But you do want
to make sure that the churn happens when required.  Relying on indirect
linkage in a lot of cases just results in more fragile applications.

There are cases where it might make sense to move a library to
"Requires.private" even when it is used in some headers, but I don't
think this is one of them.  Some examples where it would be appropriate
include:

    * Cairo's dependency on glitz, xlib and freetype (this one has been
      done upstream).  Apps only depend on glitz if they include
      <cairo-glitz.h> and use those APIs.  Apps that don't use those
      APIs shouldn't be linked to glitz so that they'll work on systems
      where Cairo hasn't been built with glitz support.  The Cairo
      developers advise app developers to direct link to both cairo and
      glitz if they are using the glitz features.
    * GTK does not expose the X API through its headers except for
      <gdk/gdkx.h>.  You might argue that it should list the xlib
      dependency as private.

James.

Message #36 received at 340904@bugs.debian.org (full text, mbox, reply):

From: Steve Langasek <vorlon@debian.org>

To: James Henstridge <james@jamesh.id.au>

Cc: Tollef Fog Heen <tfheen@err.no>, 349318@bugs.debian.org, 340904@bugs.debian.org, David Nusinow <david_nusinow@verizon.net>

Subject: Re: [Steve Langasek] Bug#340904: Bug#349318: fixed in xft 2.1.8.2-1

Date: Wed, 8 Feb 2006 17:30:27 -0800

[Message part 1 (text/plain, inline)]

Hi James,

Nice to meet you!

On Wed, Feb 08, 2006 at 07:28:39PM +0800, James Henstridge wrote:
> Tollef Fog Heen wrote:

> >Hi, any chance you could comment on this?  I'm somewhat inclined to
> >agree with Steve, but you know this stuff better than I do, so I'd
> >value a second opinion.

> >Also, if there's anything unclear or anything, please do mail Steve
> >(and the bug) with me in Cc.

> [snip]

> >Both the xrender and freetype2 headers are included in Xft/Xft.h, and types
> >from both of these headers are exported in the Xft API -- so any software
> >using Xft.h needs to be able to find these other headers as well.
> >Currently, pkg-config does *not* pass cflags from packages listed in
> >Requires.private when called as pkg-config --cflags <foo>.

> The fact that the headers are included makes them part of the _public_
> ABI of Xft.  There are Xft functions that accept or return types defined
> in the Xrender and freetype2 headers.

> If Xft is updated to a new version of either of those libraries such
> that those types are defined differently (altered struct layout,
> different type sizes, etc), then the app also needs to be updated to the
> new version.

Ok, here's the problem with this argument.

Yes, if one of the freetype types that's used in Xft/Xft.h changes, that is
an ABI change... *in libXft* -- that's why we care about it!  Because it's
an ABI change in libXft, the soname of *libXft* should change.  With the
change of libXft's soname, there's no reason an application that is a
consumer of libXft, but *not* a consumer of freetype, should care about
libfreetype *at all*.[1]

On the other hand, there are plenty of cases in which an ABI change in
libfreetype will *not* cause an ABI change in libXft.  Addition or removal
of functions, addition of typedefs, or removal or changing of any typedefs
not used in libXft's ABI are all changes that should require an soname
change in libfreetype but not an soname change in libXft.  The present ABI
transition in libfreetype is such a case.  But because these applications
are being encouraged to link directly to libfreetype, *even though they
don't use it*, they have to care about ABI changes that should not affect
them.

The net result is that pkg-config's handling of Requires/Requires.private is
directly causing churn in response to ABI changes in any indirect library
dependencies, where this should be completely unnecessary on GNU/Linux
platforms.  It increases the chances of segfaults or other failures from
loading two different versions of a library into memory, and correspondingly
increases the frequency with which binaries need to be rebuilt in response
to ABI changes that don't actually concern them.  And it does this entirely
to support a use case which, as explained above, should not actually exist.

> >- the library intentionally includes headers from dependencies in its own
> >  headers in order to inherit type definitions, but these definitions are
> >  not intended for direct consumption by users of this library alone;
> >  therefore pkg-config must export the cflags from dependencies in all
> >  cases, but the ldflags only when linking statically

> Changes to type definitions _do_ change the ABI.  If library A uses
> library B's types in its ABI, then it's ABI will break if library B
> changes those types.  An app using library A should definitely record
> the version of library B being used.

However:

- If library A's soname is correctly changed in sync with library B's,
  linking application C to library B is redundant.
- If library A's soname is not changed when library B's soname changes,
  pkg-config's behavior does not prevent applications from being broken by
  the ABI change in libA.  At most, it makes it easier to detect such
  breakage due to double-linkage of libB.

> >Please note that today, the handling of Requires.private in pkg-config maps
> >to *none* of these cases -- I can't think of a single situation in which
> >cflags of dependencies are needed when statically linking, and not needed
> >when dynamically linking!

> The cflags of dependencies listed in "Requires.private" are not included
> for either dynamic or static modes of pkg-config, so maps to case 3
> quite well.

Ah, guess I should have looked a bit closer at the behavior in that case.

> >This is unfortunate, because there are a great many packages that are
> >inheriting dependencies this way on libraries they don't use.  While it is
> >true that an ABI change in the dependent library will *sometimes* mean an
> >ABI change in the depending library, this is not always the case.  As a
> >result, this behavior of pkg-config causes unnecessary churn for packages
> >depending on libraries in this scenario.  In the case of libxft2, that's
> >over 400 packages in Debian that are potentially affected.

> Sure, unnecessary churn is bad and should be avoided.  But you do want
> to make sure that the churn happens when required.

Er, the whole reason I'm objecting to the current behavior is that it *is*
unnecessary churn.  If the libfreetype ABI change affected the ABI of
libXft, there would be no sense in trying to avoid the double-linking, and I
wouldn't be bothering! :)

> Relying on indirect linkage in a lot of cases just results in more fragile
> applications.

No, recursive linking of indirect dependencies results in more *brittle*
applications, which break any time anything anywhere in the dependency tree
changes.  I'm not sure why you think that "relying" on libraries to take
care of their own messes without bothering the application is fragile.

Thanks,
-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
vorlon@debian.org                                   http://www.debian.org/

[1] This sets aside the issue that you can't reasonably ensure that the
freetype.h included from Xft/Xft.h matches the ABI used when building
libxft.so except through a package manager; that's simply an argument why
you should properly encapsulate types from other libraries instead of
exposing them in your own API, which doesn't help us in the case when that's
not what people do.  It's also not anything that pkg-config's current
behavior actually helps...

[signature.asc (application/pgp-signature, inline)]

Message #41 received at 340904@bugs.debian.org (full text, mbox, reply):

From: James Henstridge <james@jamesh.id.au>

To: Steve Langasek <vorlon@debian.org>

Cc: Tollef Fog Heen <tfheen@err.no>, 349318@bugs.debian.org, 340904@bugs.debian.org, David Nusinow <david_nusinow@verizon.net>

Subject: Re: [Steve Langasek] Bug#340904: Bug#349318: fixed in xft 2.1.8.2-1

Date: Thu, 09 Feb 2006 18:59:02 +0800

Steve Langasek wrote:

>>If Xft is updated to a new version of either of those libraries such
>>that those types are defined differently (altered struct layout,
>>different type sizes, etc), then the app also needs to be updated to the
>>new version.
>>    
>>
>Ok, here's the problem with this argument.
>
>Yes, if one of the freetype types that's used in Xft/Xft.h changes, that is
>an ABI change... *in libXft* -- that's why we care about it!  Because it's
>an ABI change in libXft, the soname of *libXft* should change.  With the
>change of libXft's soname, there's no reason an application that is a
>consumer of libXft, but *not* a consumer of freetype, should care about
>libfreetype *at all*.[1]
>  
>
The simple matter of fact is that when you ask for the Xft API in your
app (#include <Xft/Xft.h>), you also get the freetype API at the same
time.  While some apps don't use it, we can't say that apps don't use
those APIs in the general case.  Until we have a widely used way to
express dependencies at that granularity, you need to look at the
library level.

>On the other hand, there are plenty of cases in which an ABI change in
>libfreetype will *not* cause an ABI change in libXft.  Addition or removal
>of functions, addition of typedefs, or removal or changing of any typedefs
>not used in libXft's ABI are all changes that should require an soname
>change in libfreetype but not an soname change in libXft.  The present ABI
>transition in libfreetype is such a case.  But because these applications
>are being encouraged to link directly to libfreetype, *even though they
>don't use it*, they have to care about ABI changes that should not affect
>them.
>  
>
They are using it in conjunction with the Xft API.  There is no clear
separation between the Xft APIs that use it and those that don't.  Of
course, an app that only depends on Xft indirectly and doesn't use Xft's
APIs doesn't need direct linkage to freetype.

>The net result is that pkg-config's handling of Requires/Requires.private is
>directly causing churn in response to ABI changes in any indirect library
>dependencies, where this should be completely unnecessary on GNU/Linux
>platforms.  It increases the chances of segfaults or other failures from
>loading two different versions of a library into memory, and correspondingly
>increases the frequency with which binaries need to be rebuilt in response
>to ABI changes that don't actually concern them.  And it does this entirely
>to support a use case which, as explained above, should not actually exist.
>  
>
The "app links to multiple versions of library A" problem is quite easy
to diagnose, and also makes it clear that an app needs to be rebuilt if
the old library version is removed (i.e. the app doesn't start).  The
"app depends on the structure layout of the old version of library A but
doesn't directly link to library A" is more difficult to catch.

It sounds like your argument is that "Xft shouldn't expose the freetype
API".  Given that this isn't the case, the direct linkage makes sense. 
The freetype change in this case might not break many Xft using apps,
but that won't necessarily be the case next time this type of situation
occurs.

>>Changes to type definitions _do_ change the ABI.  If library A uses
>>library B's types in its ABI, then it's ABI will break if library B
>>changes those types.  An app using library A should definitely record
>>the version of library B being used.
>>    
>>
>However:
>
>- If library A's soname is correctly changed in sync with library B's,
>  linking application C to library B is redundant.
>- If library A's soname is not changed when library B's soname changes,
>  pkg-config's behavior does not prevent applications from being broken by
>  the ABI change in libA.  At most, it makes it easier to detect such
>  breakage due to double-linkage of libB.
>  
>
I'd argue that this is a useful feature to have, rather than having apps
break with no indication of the cause.

>>The cflags of dependencies listed in "Requires.private" are not included
>>for either dynamic or static modes of pkg-config, so maps to case 3
>>quite well.
>>    
>>
>Ah, guess I should have looked a bit closer at the behavior in that case.
>  
>
Case 3 was exactly the sort of use case I was trying to model with how
Requires.private functions.

>>>This is unfortunate, because there are a great many packages that are
>>>inheriting dependencies this way on libraries they don't use.  While it is
>>>true that an ABI change in the dependent library will *sometimes* mean an
>>>ABI change in the depending library, this is not always the case.  As a
>>>result, this behavior of pkg-config causes unnecessary churn for packages
>>>depending on libraries in this scenario.  In the case of libxft2, that's
>>>over 400 packages in Debian that are potentially affected.
>>>      
>>>
>>Sure, unnecessary churn is bad and should be avoided.  But you do want
>>to make sure that the churn happens when required.
>>    
>>
>Er, the whole reason I'm objecting to the current behavior is that it *is*
>unnecessary churn.  If the libfreetype ABI change affected the ABI of
>libXft, there would be no sense in trying to avoid the double-linking, and I
>wouldn't be bothering! :)
>  
>
I understand that ABI changes are painful, but this doesn't seem like
the right way to try and combat it.  Other options include:

    * don't break the ABI.
    * don't expose unstable dependencies in your own API.
    * increase soname version numbers in lockstep with dependencies.

These all require upstream cooperation though.

>>Relying on indirect linkage in a lot of cases just results in more fragile
>>applications.
>>    
>>
>No, recursive linking of indirect dependencies results in more *brittle*
>applications, which break any time anything anywhere in the dependency tree
>changes.  I'm not sure why you think that "relying" on libraries to take
>care of their own messes without bothering the application is fragile.
>  
>
If the indirect dependency is solely the library's mess, then relying on
indirect linking is the right solution (and is what should happen with
Requires.private).

When details of the indirect dependency leak through the library's API
then the mess belongs to both the app and library, and should be treated
as such.

James.

Message #46 received at 340904@bugs.debian.org (full text, mbox, reply):

From: Steve Langasek <vorlon@debian.org>

To: James Henstridge <james@jamesh.id.au>

Cc: Tollef Fog Heen <tfheen@err.no>, 349318@bugs.debian.org, 340904@bugs.debian.org, David Nusinow <david_nusinow@verizon.net>

Subject: Re: [Steve Langasek] Bug#340904: Bug#349318: fixed in xft 2.1.8.2-1

Date: Sat, 11 Feb 2006 23:33:54 -0800

[Message part 1 (text/plain, inline)]

On Thu, Feb 09, 2006 at 06:59:02PM +0800, James Henstridge wrote:
> Steve Langasek wrote:

> >>If Xft is updated to a new version of either of those libraries such
> >>that those types are defined differently (altered struct layout,
> >>different type sizes, etc), then the app also needs to be updated to the
> >>new version.

> >Ok, here's the problem with this argument.

> >Yes, if one of the freetype types that's used in Xft/Xft.h changes, that is
> >an ABI change... *in libXft* -- that's why we care about it!  Because it's
> >an ABI change in libXft, the soname of *libXft* should change.  With the
> >change of libXft's soname, there's no reason an application that is a
> >consumer of libXft, but *not* a consumer of freetype, should care about
> >libfreetype *at all*.[1]

> The simple matter of fact is that when you ask for the Xft API in your
> app (#include <Xft/Xft.h>), you also get the freetype API at the same
> time.  While some apps don't use it, we can't say that apps don't use
> those APIs in the general case.

You're right, we can't.  But as library authors/maintainers, we can say
whether we intend to *support* such abuses of the API.  You can get a valid
prototype of fprintf() on Linux by doing #include <sql.h> (from UnixODBC)
instead of #include <stdio.h>, too, and we don't know that there isn't
anybody relying on that -- but if they do, and they complain about it
breaking, they are cordially invited to blow it out their ass. ;)

The current policy of pkg-config prevents library authors from making their
own choices about whether to support such API abuses on the part of
applications.  I don't think pkg-config should be imposing policies which
prevent library authors from deciding what uses of their headers they intend
to support.  The current behavior of pkg-config *ensures* that the API
abuses you describe will work, because use of Requires means they'll get the
ldflags as well as the cflags; that makes pkg-config as it stands pretty
suboptimal for anyone who *doesn't* want to get supporting their lib
dependencies as part of the public API.

> >On the other hand, there are plenty of cases in which an ABI change in
> >libfreetype will *not* cause an ABI change in libXft.  Addition or removal
> >of functions, addition of typedefs, or removal or changing of any typedefs
> >not used in libXft's ABI are all changes that should require an soname
> >change in libfreetype but not an soname change in libXft.  The present ABI
> >transition in libfreetype is such a case.  But because these applications
> >are being encouraged to link directly to libfreetype, *even though they
> >don't use it*, they have to care about ABI changes that should not affect
> >them.

> They are using it in conjunction with the Xft API.  There is no clear
> separation between the Xft APIs that use it and those that don't.  Of
> course, an app that only depends on Xft indirectly and doesn't use Xft's
> APIs doesn't need direct linkage to freetype.

They are using data types defined by freetype in conjunction with the Xft
API.  If they're using function calls from freetype without including
freetype headers, they didn't get that idea from Xft's documentation or by
reading the Xft headers themselves.  Xft's maintainers should not be
constrained by a pkg-config design decision to support people who think
#include <Xft/Xft.h> is an acceptable substitute for #include
<freetype/freetype.h>.

> >The net result is that pkg-config's handling of Requires/Requires.private is
> >directly causing churn in response to ABI changes in any indirect library
> >dependencies, where this should be completely unnecessary on GNU/Linux
> >platforms.  It increases the chances of segfaults or other failures from
> >loading two different versions of a library into memory, and correspondingly
> >increases the frequency with which binaries need to be rebuilt in response
> >to ABI changes that don't actually concern them.  And it does this entirely
> >to support a use case which, as explained above, should not actually exist.

> The "app links to multiple versions of library A" problem is quite easy
> to diagnose, and also makes it clear that an app needs to be rebuilt if
> the old library version is removed (i.e. the app doesn't start).  The
> "app depends on the structure layout of the old version of library A but
> doesn't directly link to library A" is more difficult to catch.

The app doesn't depend on the structure layout of the old version of library
A.  It depends on the structure layout of *library B*, which happens to have
*inherited* that structure from the *headers* of library A.  It's entirely
feasible for a library to do this without actually using library A at
runtime; so then you get library B not linking to library A at all, but
because it uses pkg-config to resolve the header reference, app C *does* get
linked against it for no reason...

> It sounds like your argument is that "Xft shouldn't expose the freetype
> API".  Given that this isn't the case, the direct linkage makes sense. 
> The freetype change in this case might not break many Xft using apps,
> but that won't necessarily be the case next time this type of situation
> occurs.

No, that's not my argument.  It's *true* that libraries should generally
strive not to expose the APIs of other libraries in their own; they also
should use linker scripts to avoid exporting internal symbols, use versioned
symbols to avoid problems with multiple versions of the library,
forward-proof their APIs so structs can be extended without breaking the
ABI, and carefully check their ABIs with every release to make sure they
aren't silently incompatible.  But I still live on Earth, where lots of
libraries are sub-optimally maintained. :)  My argument is, *given* that
such suboptimal libraries exist, pkg-config should accomodate this reality
instead of amplifying the problems.

> >- If library A's soname is correctly changed in sync with library B's,
> >  linking application C to library B is redundant.
> >- If library A's soname is not changed when library B's soname changes,
> >  pkg-config's behavior does not prevent applications from being broken by
> >  the ABI change in libA.  At most, it makes it easier to detect such
> >  breakage due to double-linkage of libB.

> I'd argue that this is a useful feature to have, rather than having apps
> break with no indication of the cause.

Possibly so.  I don't believe it outweighs all the other considerations,
such as the large number of cases in which applications wouldn't break *at
all* except for pkg-config pulling in unnecessary libraries.

It's also not as though such breakage isn't debuggable; you just generally
have to use a debugger, instead of merely looking at the output of ldd...

> >>>This is unfortunate, because there are a great many packages that are
> >>>inheriting dependencies this way on libraries they don't use.  While it is
> >>>true that an ABI change in the dependent library will *sometimes* mean an
> >>>ABI change in the depending library, this is not always the case.  As a
> >>>result, this behavior of pkg-config causes unnecessary churn for packages
> >>>depending on libraries in this scenario.  In the case of libxft2, that's
> >>>over 400 packages in Debian that are potentially affected.

> >>Sure, unnecessary churn is bad and should be avoided.  But you do want
> >>to make sure that the churn happens when required.

> >Er, the whole reason I'm objecting to the current behavior is that it *is*
> >unnecessary churn.  If the libfreetype ABI change affected the ABI of
> >libXft, there would be no sense in trying to avoid the double-linking, and I
> >wouldn't be bothering! :)

> I understand that ABI changes are painful, but this doesn't seem like
> the right way to try and combat it.  Other options include:

>     * don't break the ABI.
>     * don't expose unstable dependencies in your own API.
>     * increase soname version numbers in lockstep with dependencies.

> These all require upstream cooperation though.

Sure.  That makes them substantially less scalable. :)

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
vorlon@debian.org                                   http://www.debian.org/

[signature.asc (application/pgp-signature, inline)]

Message #51 received at 340904-close@bugs.debian.org (full text, mbox, reply):

From: Tollef Fog Heen <tfheen@debian.org>

To: 340904-close@bugs.debian.org

Subject: Bug#340904: fixed in pkg-config 0.21-1

Date: Wed, 16 Aug 2006 14:20:12 -0700

Source: pkg-config
Source-Version: 0.21-1

We believe that the bug you reported is fixed in the latest version of
pkg-config, which is due to be installed in the Debian FTP archive:

pkg-config_0.21-1.diff.gz
  to pool/main/p/pkg-config/pkg-config_0.21-1.diff.gz
pkg-config_0.21-1.dsc
  to pool/main/p/pkg-config/pkg-config_0.21-1.dsc
pkg-config_0.21-1_i386.deb
  to pool/main/p/pkg-config/pkg-config_0.21-1_i386.deb
pkg-config_0.21.orig.tar.gz
  to pool/main/p/pkg-config/pkg-config_0.21.orig.tar.gz



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 340904@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Tollef Fog Heen <tfheen@debian.org> (supplier of updated pkg-config package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.7
Date: Wed, 16 Aug 2006 21:11:21 +0200
Source: pkg-config
Binary: pkg-config
Architecture: source i386
Version: 0.21-1
Distribution: unstable
Urgency: low
Maintainer: Tollef Fog Heen <tfheen@debian.org>
Changed-By: Tollef Fog Heen <tfheen@debian.org>
Description: 
 pkg-config - manage compile and link flags for libraries
Closes: 254289 334896 340904 350176 378570
Changes: 
 pkg-config (0.21-1) unstable; urgency=low
 .
   * New upstream release
     - Adds internal pkg-config package.  (closes: #254289)
     - Supports escaping $ rather than %.  (closes: #378570)
     - Always adds all cflags.  (closes: #340904)
   * Includes full NEWS file.  (closes: #334896)
   * Remove libglib2.0-dev build-deps.  (closes: #350176)
Files: 
 e922bea83dadee66fab37c6a83fd0fd4 572 devel optional pkg-config_0.21-1.dsc
 476f45fab1504aac6697aa7785f0ab91 998420 devel optional pkg-config_0.21.orig.tar.gz
 23c25fd98789a5e18d342a39339346b8 4404 devel optional pkg-config_0.21-1.diff.gz
 59f0e1fdc7ba08a4faa1fc4071e05960 67532 devel optional pkg-config_0.21-1_i386.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQFE43EjQSseMYF6mWoRAmDSAKCluBOZcezXRPykQT3O5PGuhlpCIQCfRJOh
y5+NQC2kJ4aADtbzFvmU9lg=
=OwY/
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.