Debian Bug report logs - #837478
Clarify Policy requirements for relocatable (-fPIE or -fPIC) static libraries

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Balint Reczey <balint@balintreczey.hu>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: debian-policy: Allow (encourage?) PIC static libraries

Date: Mon, 12 Sep 2016 00:10:08 +0200

Package: debian-policy
Severity: important

Dear Maintainer,

Current (3.9.8.0) Policy mandates non-PIC static libraries with a few
exceptions:

---
10.2 Libraries
... (paragraph about shared libs)

As to the static libraries, the common case is not to have relocatable
code, since there is no benefit, unless in specific cases; therefore the
static version must not be compiled with the -fPIC flag. Any exception
to this rule should be discussed on the mailing list
debian-devel@lists.debian.org, and the reasons for compiling with the
-fPIC flag must be recorded in the file README.Debian. [86]

In other words, if both a shared and a static library is being built,
each source unit (*.c, for example, for C files) will need to be
compiled twice, for the normal case.

---

I think with the spreading of PIE binaries the "... since there is no
benefit ..." claim does not stand anymore. Non-PIC static libraries
can't be linked to PIE binaries thus they are less useful for code
sharing among packages.

There is also a plan to use a specially configured GCC on several
architectures which builds PIE binaries by default and that needs PIC
static libraries for not statically linked binaries. [1]

Planned archive-wide enabling of bindnow (-Wl,-z,now) hardening setting
in dpkg [3] also decreases the speed advantage of non-PIC static libraries.

I would like to suggest revising the Policy text and at least allowing
shipping PIC static libraries without broader discussion and
documentation. I would be in favor of even encouraging PIC for static
libraries because that would allow compiling them to PIE binaries.

I have already filed many bugs [4] related to the transition to PIE by
defauld where the problem can be solved easily by providing PIC static
libraries. Note that many packages ship only static libs.

Thanks,
Balint


[1] https://wiki.debian.org/Hardening/PIEByDefaultTransition
[2] https://lists.debian.org/debian-devel/2016/05/msg00309.html
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=835146
[4]
https://udd.debian.org/cgi-bin/bts-usertags.cgi?tag=pie-bindnow-20160906&user=balint%40balintreczey.hu

Message #10 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Adrian Bunk <bunk@stusta.de>

To: Bálint Réczey <balint@balintreczey.hu>, 837478@bugs.debian.org

Cc: Debian Release Team <debian-release@lists.debian.org>, debian-gcc@lists.debian.org

Subject: Re: Static libraries - PIC or PIE?

Date: Sun, 23 Oct 2016 14:26:14 +0300

On Sun, Oct 23, 2016 at 12:29:42PM +0200, Bálint Réczey wrote:
> Hi Ardian,

Hi Bláint, ;-)

> 2016-10-23 10:18 GMT+02:00 Adrian Bunk <bunk@stusta.de>:
> > Hi Bálint,
> >
> > there is some confusion regarding how static libraries should be
> > compiled now.
> >
> > Your bugs (e.g. #837350) say "Please build libfoo.a with -fPIC".
> >
> > Why do these say -fPIC and not -fPIE?
> 
> I suggest using -fPIC, because then the shared libraries would be
> usable in shared (PIC) libraries libraries, too.

you have created a lot of confusion by mixing two separate issues.

One of the worst examples:
#837658 libfl-dev: Please build libfl_pic.a with -fPIC
#841203 nmu: flex_2.6.1-1

A _pic.a library compiled without -fPIC sounds like a clear bug,
and a binNMU that will recompile it with -fPIE won't fix that bug.

> This in many cases also simplify debian/rules.

No, it would actually make building static libraries a real pain.

Think of a normal source package building shared libraries,
static libraries and some programs.

How do you want to tell the build system of the package that it should 
build the static libraries with -fPIC, but not the programs?

> I also suggested changing the policy #837478.

Unless I misunderstand something, current policy is perfectly fine
for the PIE change, and your claims in #837478 that building static 
libraries with -fPIC would be required for PIE binaries are not
correct.

> > My current understanding is that a binNMU would recompile the the static
> > library with PIE (not PIC), and that this is sufficient.
> 
> In most of the cases this would be sufficient, but at the time I filed the
> bugs the default was -no-pie, thus it was not an option.

It is clear that the binNMUs have to happen after the change.

It would have created less confusion to not file bugs in cases where no 
maintainer action is required, ask the release team to schedule binNMUs 
for the static libraries known to need them immediately after the 
compiler change, and announce on debian-devel-announce that some 
transient build failures might be observed immediately after the 
compiler change until the binNMUs are done.

I'll sort out what binNMUs are required later today.

> I'm OK with performing binnmu-s and decreasing the severity of the 'solved'
> bugs to wishlist.

With the exception of special cases like #837658 a binNMU will 
completely solve it, and there is no point in having wishlist
bugs for something not really permitted by policy.

> Cheers,
> Balint

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Message #15 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Bálint Réczey <balint@balintreczey.hu>

To: Adrian Bunk <bunk@stusta.de>

Cc: 837478@bugs.debian.org, Debian Release Team <debian-release@lists.debian.org>, debian-gcc@lists.debian.org

Subject: Re: Static libraries - PIC or PIE?

Date: Sun, 23 Oct 2016 15:17:06 +0200

Hi Adrian,

2016-10-23 13:26 GMT+02:00 Adrian Bunk <bunk@stusta.de>:
> On Sun, Oct 23, 2016 at 12:29:42PM +0200, Bálint Réczey wrote:
>> Hi Ardian,
>
> Hi Bláint, ;-)

I'm sorry. :-)

>
>> 2016-10-23 10:18 GMT+02:00 Adrian Bunk <bunk@stusta.de>:
>> > Hi Bálint,
>> >
>> > there is some confusion regarding how static libraries should be
>> > compiled now.
>> >
>> > Your bugs (e.g. #837350) say "Please build libfoo.a with -fPIC".
>> >
>> > Why do these say -fPIC and not -fPIE?
>>
>> I suggest using -fPIC, because then the shared libraries would be
>> usable in shared (PIC) libraries libraries, too.
>
> you have created a lot of confusion by mixing two separate issues.

I'm sorry for creating confusion, I had absolutely no intent to cause
any. As you can see I attempted discussing every change, while in
many cases I got no response.

We also discussed the tradeoffs of using PIE or PIC in debian-devel.

>
> One of the worst examples:
> #837658 libfl-dev: Please build libfl_pic.a with -fPIC
> #841203 nmu: flex_2.6.1-1

I did not file the second one.

>
> A _pic.a library compiled without -fPIC sounds like a clear bug,
> and a binNMU that will recompile it with -fPIE won't fix that bug.

This was the single case I found with non-PIC _pic-postfixed static
library thus I see this something unique, rather than an example
of many bad cases. Recompiling would fix the FTBFS but not the
misleading prefix, indeed.

I also hoped getting response from the maintainer.

>
>> This in many cases also simplify debian/rules.
>
> No, it would actually make building static libraries a real pain.

Could you please show an example?
I went trough many packages and adding -fPIC was really
straight-forward every time. OTOH packages providing both
shared and static libraries build some parts twice like in antlr's
case:

build-stamp:
    dh_testdir
    uudecode -o debian/antlr.snk debian/antlr.snk.uue
    $(MAKE) -f debian/Makefile.debian compile build_antlr
    $(MAKE) -C lib/cpp CXXFLAGS="+ -fPIC -DPIC"
    mv -f lib/cpp/src/libantlr.a debian/libantlr-pic.a
    $(MAKE) -C lib/cpp clean
    $(MAKE) -C lib/cpp
    touch build-stamp


>
> Think of a normal source package building shared libraries,
> static libraries and some programs.
>
> How do you want to tell the build system of the package that it should
> build the static libraries with -fPIC, but not the programs?

It is usually already done by upstream or at least in packaging
since we require -fPIC for shared libs.

>
>> I also suggested changing the policy #837478.
>
> Unless I misunderstand something, current policy is perfectly fine
> for the PIE change, and your claims in #837478 that building static
> libraries with -fPIC would be required for PIE binaries are not
> correct.

True, I stated more than I intended to. This was my original wording:
...
> ---
> 10.2 Libraries
> ... (paragraph about shared libs)
> 
> As to the static libraries, the common case is not to have relocatable
> code, since there is no benefit, unless in specific cases; therefore the
> static version must not be compiled with the -fPIC flag. Any exception
> to this rule should be discussed on the mailing list
> debian-devel@lists.debian.org, and the reasons for compiling with the
> -fPIC flag must be recorded in the file README.Debian. [86]
> 
> In other words, if both a shared and a static library is being built,
> each source unit (*.c, for example, for C files) will need to be
> compiled twice, for the normal case.
> 
> ---
> 
> I think with the spreading of PIE binaries the "... since there is no
> benefit ..." claim does not stand anymore. Non-PIC static libraries
> can't be linked to PIE binaries thus they are less useful for code
> sharing among packages.
> 
> There is also a plan to use a specially configured GCC on several
> architectures which builds PIE binaries by default and that needs PIC
> static libraries for not statically linked binaries. [1]
...

Let me correct myself then:
...
I think with the spreading of PIE binaries the "... since there is no
benefit ..." claim does not stand anymore. Static binaries built
without either of PIC or PIE flags can't be linked to PIE binaries
nor to (PIC) shared libraries thus they are less useful for code
sharing among packages.

GCC is configured to build PIE binaries by default [1] on most
architectures and on those architectures static libraries are built
with PIE enabled. On the rest of the architectures static libraries are
still built without PIE thus the problem described in the previous
paragraph still stands.

I assume non-PIC static library used in a PIC shared library is the
specific case mentioned in the original text, which still does not
work on any architecture.

...

>
>> > My current understanding is that a binNMU would recompile the the static
>> > library with PIE (not PIC), and that this is sufficient.
>>
>> In most of the cases this would be sufficient, but at the time I filed the
>> bugs the default was -no-pie, thus it was not an option.
>
> It is clear that the binNMUs have to happen after the change.

Yes, it was expected, but I think the changes would still be
useful on architectures not enabling PIE because they allow
enabling pie in reverse dependencies selectively.

>
> It would have created less confusion to not file bugs in cases where no
> maintainer action is required, ask the release team to schedule binNMUs
> for the static libraries known to need them immediately after the
> compiler change, and announce on debian-devel-announce that some
> transient build failures might be observed immediately after the
> compiler change until the binNMUs are done.
>
> I'll sort out what binNMUs are required later today.

Thank you very much for the binNMU-s and also for the work on the bugs.

>
>> I'm OK with performing binnmu-s and decreasing the severity of the 'solved'
>> bugs to wishlist.
>
> With the exception of special cases like #837658 a binNMU will
> completely solve it, and there is no point in having wishlist
> bugs for something not really permitted by policy.

I'm OK with closing the bugs since they are not critical from me, but
I think they will still be valid as wishlist bugs for the reasons
described above (non-PIE arches, linking with PIC shared libs).

Cheers,
Balint

Message #20 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Adrian Bunk <bunk@stusta.de>

To: Bálint Réczey <balint@balintreczey.hu>

Cc: 837478@bugs.debian.org, Debian Release Team <debian-release@lists.debian.org>, debian-gcc@lists.debian.org

Subject: Re: Static libraries - PIC or PIE?

Date: Sun, 23 Oct 2016 17:06:28 +0300

On Sun, Oct 23, 2016 at 03:17:06PM +0200, Bálint Réczey wrote:
> Hi Adrian,
> 
> 2016-10-23 13:26 GMT+02:00 Adrian Bunk <bunk@stusta.de>:
> > On Sun, Oct 23, 2016 at 12:29:42PM +0200, Bálint Réczey wrote:
> >> Hi Ardian,
> >
> > Hi Bláint, ;-)
> 
> I'm sorry. :-)

No problem. :-)

>...
> >> This in many cases also simplify debian/rules.
> >
> > No, it would actually make building static libraries a real pain.
> 
> Could you please show an example?
> I went trough many packages and adding -fPIC was really
> straight-forward every time. OTOH packages providing both
> shared and static libraries build some parts twice like in antlr's
> case:

Usually building everything twice is done by the build system of the 
package, and debian/rules just calls $(MAKE)

> build-stamp:
>     dh_testdir
>     uudecode -o debian/antlr.snk debian/antlr.snk.uue
>     $(MAKE) -f debian/Makefile.debian compile build_antlr
>     $(MAKE) -C lib/cpp CXXFLAGS="+ -fPIC -DPIC"
>     mv -f lib/cpp/src/libantlr.a debian/libantlr-pic.a
>     $(MAKE) -C lib/cpp clean
>     $(MAKE) -C lib/cpp
>     touch build-stamp

That's a good example why this is a real pain.

You really don't want to force maintainers to dissect the build of their 
packages, especially in the normal case where just calling $(MAKE) was
working without your proposed requirement.

Just passing normal CFLAGS from dpkg-buildflags through the package 
build to the compiler is still not working in a huge number of packages 
after years, and this would be worse by several orders of magnitude.

> > Think of a normal source package building shared libraries,
> > static libraries and some programs.
> >
> > How do you want to tell the build system of the package that it should
> > build the static libraries with -fPIC, but not the programs?
> 
> It is usually already done by upstream or at least in packaging
> since we require -fPIC for shared libs.

You are completely wrong on that.

-fPIC is required and used for shared libraries.
Static libraries compiled with -fPIC are *very* rare.

Compiling with -fPIC for the shared library and without -fPIC for the 
static library is the one and (usually) only reason why all objects
are compiled twice when building libraries.

>...
> I assume non-PIC static library used in a PIC shared library is the
> specific case mentioned in the original text, which still does not
> work on any architecture.

Why do you want to forvce maintainers to go through great pain to get 
that working?

It is usually a bug when you end up linking a static library into a 
shared library, and in addition to a performance penalty you would
lose the benefit of getting a build failure for such bugs.

There are some rare cases of packages not building shared libraries. 
There might be other exotic situations where linking a static library 
into a shared library makes sense.
Requiring discussion of these on a case-by-case basis on debian-devel
as policy requires sounds pretty appropriate to me.

> >> > My current understanding is that a binNMU would recompile the the static
> >> > library with PIE (not PIC), and that this is sufficient.
> >>
> >> In most of the cases this would be sufficient, but at the time I filed the
> >> bugs the default was -no-pie, thus it was not an option.
> >
> > It is clear that the binNMUs have to happen after the change.
> 
> Yes, it was expected, but I think the changes would still be
> useful on architectures not enabling PIE because they allow
> enabling pie in reverse dependencies selectively.

Is there actually a good reason why PIE was only enabled on the release
architectures?

In any case, this would not provide any kind of reason for requiring 
to build static libraries with PIC.

>...
> Cheers,
> Balint

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Message #27 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: debian-devel@lists.debian.org, 837478@bugs.debian.org

Subject: Re: "PIE by default" transition is underway -- wiki needs updating

Date: Wed, 26 Oct 2016 00:37:18 +0200

Hi,

On 25.10.2016 13:55, Guillem Jover wrote:
> I don't think the reasoning there is sound (as I've mentioned
> elsewhere), and the policy bug should be closed.
> 
> Switching from no-PIE to PIE by default preserves our current behavior
> WRT static libraries vs shared libraries.

The current policy says:
"As to the static libraries, the common case is not to have relocatable code"

As of gcc-6 version 6.2.0-7 this is factually wrong, because the compiler
enables PIE by default, which means it produces relocatable code.
This should definitely be updated to reflect reality.

> For many static libraries,
> making them embeddable into other shared libraries is really not
> desirable. And those should be using the shared libraries instead.

If that's the reason why it shouldn't be done, policy should mention it.
The current policy does not list this as reason not to use -fPIC, merely:
"since there is no benefit"

> I still think the current policy is fine, and if someone wants to build
> a static library with PIC it should be brought up here.

The current ffmpeg packages builds shared and static libraries, the
latter because they are used in the test suite. Both are built from
the same object files compiled with -fPIC.
Do you really think those static libraries should not be included
in the binary lib*-dev packages just because they are not incompatible
with including in other shared libraries?

Best regards,
Andreas

Message #32 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: debian-devel@lists.debian.org, 837478@bugs.debian.org

Subject: Re: "PIE by default" transition is underway -- wiki needs updating

Date: Wed, 26 Oct 2016 05:26:24 +0200

Hi!

On Wed, 2016-10-26 at 00:37:18 +0200, Andreas Cadhalpun wrote:
> On 25.10.2016 13:55, Guillem Jover wrote:
> > I don't think the reasoning there is sound (as I've mentioned
> > elsewhere), and the policy bug should be closed.
> > 
> > Switching from no-PIE to PIE by default preserves our current behavior
> > WRT static libraries vs shared libraries.
> 
> The current policy says:
> "As to the static libraries, the common case is not to have relocatable code"
> 
> As of gcc-6 version 6.2.0-7 this is factually wrong, because the compiler
> enables PIE by default, which means it produces relocatable code.
> This should definitely be updated to reflect reality.

Right, that should be updated, but the bug was not about that. :)

My point was that, yes we have changed to generating relocatable code
but that is still targetted for executables only, which preserves the
current behavior, sorry if the wording was confusing.

> > For many static libraries,
> > making them embeddable into other shared libraries is really not
> > desirable. And those should be using the shared libraries instead.
> 
> If that's the reason why it shouldn't be done, policy should mention it.
> The current policy does not list this as reason not to use -fPIC, merely:
> "since there is no benefit"

I don't think it's "the reason", but personally I think it's one
important reason.

Embedding static libraries into shared libraries can be very
problematic if the shared libraries do not take precautions, such as
explicit symbol visibility or symbol versioning, etc. Which most
shared libraries do not do. And even then it's still prone to symbol
conflicts, etc, even inside the shared library being linked itself in
case of namespace issues, if the static library is sloppy.

So I think this should be in general discouraged.

> > I still think the current policy is fine, and if someone wants to build
> > a static library with PIC it should be brought up here.
> 
> The current ffmpeg packages builds shared and static libraries, the
> latter because they are used in the test suite. Both are built from
> the same object files compiled with -fPIC.
> Do you really think those static libraries should not be included
> in the binary lib*-dev packages just because they are not incompatible
> with including in other shared libraries?

Well, I guess depends on how "clean" they are, what's the intended
usage, etc. But given that in this case the usage is inside the same
project, that seems pretty safe! I'd personally probably not ship them,
and would instead provide non-PIC ones there. Or at most ship them in
addition as _pic.a libraries, to require explicit invocation.

Thanks,
Guillem

Message #37 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: debian-devel@lists.debian.org, 837478@bugs.debian.org

Subject: Re: Bug#837478: "PIE by default" transition is underway -- wiki needs updating

Date: Wed, 26 Oct 2016 20:57:52 +0200

Hi,

On 26.10.2016 05:26, Guillem Jover wrote:
> On Wed, 2016-10-26 at 00:37:18 +0200, Andreas Cadhalpun wrote:
>> On 25.10.2016 13:55, Guillem Jover wrote:
>>> For many static libraries,
>>> making them embeddable into other shared libraries is really not
>>> desirable. And those should be using the shared libraries instead.
>>
>> If that's the reason why it shouldn't be done, policy should mention it.
>> The current policy does not list this as reason not to use -fPIC, merely:
>> "since there is no benefit"
> 
> I don't think it's "the reason", but personally I think it's one
> important reason.

If there are more, they could be mentioned in policy, too.

> Embedding static libraries into shared libraries can be very
> problematic if the shared libraries do not take precautions, such as
> explicit symbol visibility or symbol versioning, etc. Which most
> shared libraries do not do. And even then it's still prone to symbol
> conflicts, etc, even inside the shared library being linked itself in
> case of namespace issues, if the static library is sloppy.

A (possibly shortened) version of this paragraph would be a good addition
to the policy. :)

> So I think this should be in general discouraged.
> 
>>> I still think the current policy is fine, and if someone wants to build
>>> a static library with PIC it should be brought up here.
>>
>> The current ffmpeg packages builds shared and static libraries, the
>> latter because they are used in the test suite. Both are built from
>> the same object files compiled with -fPIC.
>> Do you really think those static libraries should not be included
>> in the binary lib*-dev packages just because they are not incompatible
>> with including in other shared libraries?
> 
> Well, I guess depends on how "clean" they are, what's the intended
> usage, etc. But given that in this case the usage is inside the same
> project, that seems pretty safe!

There might be some confusion here. The ffmpeg package uses these
static libraries only for build-time testing, so doesn't need to
have them included in the binary packages.
I don't know if anyone is using these shipped static libraries.

> I'd personally probably not ship them, 
> and would instead provide non-PIC ones there. Or at most ship them in
> addition as _pic.a libraries, to require explicit invocation.

I'd rather not built everything twice, so I think I'll just drop the
static libraries in the next upload and only worry about this again,
when/if someone files a bug about missing the static libraries.

Best regards,
Andreas

Message #42 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Bill Allombert <ballombe@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>, 837478@bugs.debian.org

Subject: Re: Bug#837478: "PIE by default" transition is underway -- wiki needs updating

Date: Wed, 26 Oct 2016 22:34:05 +0200

On Wed, Oct 26, 2016 at 08:57:52PM +0200, Andreas Cadhalpun wrote:
> Hi,
> 
> On 26.10.2016 05:26, Guillem Jover wrote:
> > On Wed, 2016-10-26 at 00:37:18 +0200, Andreas Cadhalpun wrote:
> >> On 25.10.2016 13:55, Guillem Jover wrote:
> >>> For many static libraries,
> >>> making them embeddable into other shared libraries is really not
> >>> desirable. And those should be using the shared libraries instead.
> >>
> >> If that's the reason why it shouldn't be done, policy should mention it.
> >> The current policy does not list this as reason not to use -fPIC, merely:
> >> "since there is no benefit"
> > 
> > I don't think it's "the reason", but personally I think it's one
> > important reason.
> 
> If there are more, they could be mentioned in policy, too.
> 
> > Embedding static libraries into shared libraries can be very
> > problematic if the shared libraries do not take precautions, such as
> > explicit symbol visibility or symbol versioning, etc. Which most
> > shared libraries do not do. And even then it's still prone to symbol
> > conflicts, etc, even inside the shared library being linked itself in
> > case of namespace issues, if the static library is sloppy.
> 
> A (possibly shortened) version of this paragraph would be a good addition
> to the policy. :)
> 
> > So I think this should be in general discouraged.
> > 
> >>> I still think the current policy is fine, and if someone wants to build
> >>> a static library with PIC it should be brought up here.
> >>
> >> The current ffmpeg packages builds shared and static libraries, the
> >> latter because they are used in the test suite. Both are built from
> >> the same object files compiled with -fPIC.
> >> Do you really think those static libraries should not be included
> >> in the binary lib*-dev packages just because they are not incompatible
> >> with including in other shared libraries?
> > 
> > Well, I guess depends on how "clean" they are, what's the intended
> > usage, etc. But given that in this case the usage is inside the same
> > project, that seems pretty safe!
> 
> There might be some confusion here. The ffmpeg package uses these
> static libraries only for build-time testing, so doesn't need to
> have them included in the binary packages.
> I don't know if anyone is using these shipped static libraries.
> 
> > I'd personally probably not ship them, 
> > and would instead provide non-PIC ones there. Or at most ship them in
> > addition as _pic.a libraries, to require explicit invocation.
> 
> I'd rather not built everything twice, so I think I'll just drop the
> static libraries in the next upload and only worry about this again,
> when/if someone files a bug about missing the static libraries.

It is customary to build everything twice, one with -fPIC, one without.

Waiting for a bug report to do something is unfriendly to people using
the stable release.

Cheers,
-- 
Bill. <ballombe@debian.org>

Imagine a large red swirl here. 

Message #47 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Bill Allombert <ballombe@debian.org>, 837478@bugs.debian.org

Subject: Re: Bug#837478: "PIE by default" transition is underway -- wiki needs updating

Date: Thu, 27 Oct 2016 23:34:08 +0200

On 26.10.2016 22:34, Bill Allombert wrote:
> On Wed, Oct 26, 2016 at 08:57:52PM +0200, Andreas Cadhalpun wrote:
>> I'd rather not built everything twice, so I think I'll just drop the
>> static libraries in the next upload and only worry about this again,
>> when/if someone files a bug about missing the static libraries.
> 
> It is customary to build everything twice, one with -fPIC, one without.
> 
> Waiting for a bug report to do something is unfriendly to people using
> the stable release.

Oh well, I'll just consider your reply as said bug report. ;)

Best regards,
Andreas

Message #52 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Bálint Réczey <balint@balintreczey.hu>

To: Adrian Bunk <bunk@stusta.de>

Cc: 837478@bugs.debian.org, Debian Release Team <debian-release@lists.debian.org>, debian-gcc@lists.debian.org

Subject: Re: Static libraries - PIC or PIE?

Date: Mon, 31 Oct 2016 22:42:28 +0100

2016-10-23 16:06 GMT+02:00 Adrian Bunk <bunk@stusta.de>:
> On Sun, Oct 23, 2016 at 03:17:06PM +0200, Bálint Réczey wrote:
>> Hi Adrian,
>>
>> 2016-10-23 13:26 GMT+02:00 Adrian Bunk <bunk@stusta.de>:
>> > On Sun, Oct 23, 2016 at 12:29:42PM +0200, Bálint Réczey wrote:
>> >> Hi Ardian,
>> >
>> > Hi Bláint, ;-)
>>
>> I'm sorry. :-)
>
> No problem. :-)
>
>>...
>> >> This in many cases also simplify debian/rules.
>> >
>> > No, it would actually make building static libraries a real pain.
>>
>> Could you please show an example?
>> I went trough many packages and adding -fPIC was really
>> straight-forward every time. OTOH packages providing both
>> shared and static libraries build some parts twice like in antlr's
>> case:
>
> Usually building everything twice is done by the build system of the
> package, and debian/rules just calls $(MAKE)
>
>> build-stamp:
>>     dh_testdir
>>     uudecode -o debian/antlr.snk debian/antlr.snk.uue
>>     $(MAKE) -f debian/Makefile.debian compile build_antlr
>>     $(MAKE) -C lib/cpp CXXFLAGS="+ -fPIC -DPIC"
>>     mv -f lib/cpp/src/libantlr.a debian/libantlr-pic.a
>>     $(MAKE) -C lib/cpp clean
>>     $(MAKE) -C lib/cpp
>>     touch build-stamp
>
> That's a good example why this is a real pain.
>
> You really don't want to force maintainers to dissect the build of their
> packages, especially in the normal case where just calling $(MAKE) was
> working without your proposed requirement.
>
> Just passing normal CFLAGS from dpkg-buildflags through the package
> build to the compiler is still not working in a huge number of packages
> after years, and this would be worse by several orders of magnitude.
>
>> > Think of a normal source package building shared libraries,
>> > static libraries and some programs.
>> >
>> > How do you want to tell the build system of the package that it should
>> > build the static libraries with -fPIC, but not the programs?
>>
>> It is usually already done by upstream or at least in packaging
>> since we require -fPIC for shared libs.
>
> You are completely wrong on that.
>
> -fPIC is required and used for shared libraries.
> Static libraries compiled with -fPIC are *very* rare.
>
> Compiling with -fPIC for the shared library and without -fPIC for the
> static library is the one and (usually) only reason why all objects
> are compiled twice when building libraries.
>
>>...
>> I assume non-PIC static library used in a PIC shared library is the
>> specific case mentioned in the original text, which still does not
>> work on any architecture.
>
> Why do you want to forvce maintainers to go through great pain to get
> that working?

Enabling PIC seemed more straightforward to me than asking for PIC
shared libraries and PIE static libraries. With GCC defaulting to PIE
this is easy, but at the beginning it seemed that only a smaller
subset of the architectures was going to use PIE by default.

>
> It is usually a bug when you end up linking a static library into a
> shared library, and in addition to a performance penalty you would
> lose the benefit of getting a build failure for such bugs.
>
> There are some rare cases of packages not building shared libraries.
> There might be other exotic situations where linking a static library
> into a shared library makes sense.
> Requiring discussion of these on a case-by-case basis on debian-devel
> as policy requires sounds pretty appropriate to me.
>
>> >> > My current understanding is that a binNMU would recompile the the static
>> >> > library with PIE (not PIC), and that this is sufficient.
>> >>
>> >> In most of the cases this would be sufficient, but at the time I filed the
>> >> bugs the default was -no-pie, thus it was not an option.
>> >
>> > It is clear that the binNMUs have to happen after the change.
>>
>> Yes, it was expected, but I think the changes would still be
>> useful on architectures not enabling PIE because they allow
>> enabling pie in reverse dependencies selectively.
>
> Is there actually a good reason why PIE was only enabled on the release
> architectures?

AFAIK the porters needed to opt-in which makes sense to me:
https://lists.debian.org/debian-devel/2016/08/msg00324.html

Cheers,
Balint

>
> In any case, this would not provide any kind of reason for requiring
> to build static libraries with PIC.
>
>>...
>> Cheers,
>> Balint
>
> cu
> Adrian
>
> --
>
>        "Is there not promise of rain?" Ling Tan asked suddenly out
>         of the darkness. There had been need of rain for many days.
>        "Only a promise," Lao Er said.
>                                        Pearl S. Buck - Dragon Seed
>

Message #57 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Phillip Hellewell <sshock@gmail.com>

To: 837478@bugs.debian.org

Subject: Re: Static libraries - PIC or PIE?

Date: Sat, 19 Nov 2016 19:12:19 -0700

[Message part 1 (text/plain, inline)]

Consider this use case for an end user with 64-bit Debian 9:
    - Compiles an executable with gcc, linking a few libraries like ICU,
openssl, bz2, etc.  Works fine.
    - Now tries to link a few of the libraries statically (e.g.,
libicuuc.a).  ld blows up with a bunch of relocation R_X86_64_32S, blah
blah blah, recompile with -fPIC errors.

This was me a few hours ago, when I found about all this the hard way.

Problem is, an end user has absolutely no idea what they're doing wrong or
how to fix it.  They have no idea that GCC 6 in Debian 9 now has PIE on by
default.  They have no idea what PIE is.  They have no idea if they must
recompile their own code with -fPIC or rebuild the 3rdparty libraries with
it.  They do not believe they should have to rebuild 3rdparty libraries
(why isn't the distro's version of them working)?  They also have no idea
that they can work around the problem using -no-pie if they don't care
about PIE.

All they know is that for some reason they can't link most libraries
statically without getting a bunch of weird errors.  It could take hours of
research to figure out a workaround like -no-pie.

Given that GCC 6 on Debian 9 now does PIE executables by default, I think
it becomes very necessary to consider that Debian out to build all static
libraries with -fPIE (or -fPIC).  Otherwise you're going to get thousands
and thousands of users having no clue why they can't link anything
statically.  Plus if you did that you wouldn't have to build everything
twice.  Win-Win !!!  So why not?

Thanks,
Phillip

[Message part 2 (text/html, inline)]

Message #62 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Bill Allombert <ballombe@debian.org>

To: Phillip Hellewell <sshock@gmail.com>, 837478@bugs.debian.org

Subject: Re: Bug#837478: Static libraries - PIC or PIE?

Date: Sun, 20 Nov 2016 12:08:39 +0100

On Sat, Nov 19, 2016 at 07:12:19PM -0700, Phillip Hellewell wrote:
> Consider this use case for an end user with 64-bit Debian 9:
>     - Compiles an executable with gcc, linking a few libraries like ICU,
> openssl, bz2, etc.  Works fine.
>     - Now tries to link a few of the libraries statically (e.g.,
> libicuuc.a).  ld blows up with a bunch of relocation R_X86_64_32S, blah
> blah blah, recompile with -fPIC errors.
> 
> This was me a few hours ago, when I found about all this the hard way.
> 
> Problem is, an end user has absolutely no idea what they're doing wrong or
> how to fix it.  They have no idea that GCC 6 in Debian 9 now has PIE on by
> default.  They have no idea what PIE is.  They have no idea if they must
> recompile their own code with -fPIC or rebuild the 3rdparty libraries with
> it.  They do not believe they should have to rebuild 3rdparty libraries
> (why isn't the distro's version of them working)?  They also have no idea
> that they can work around the problem using -no-pie if they don't care
> about PIE.
> 
> All they know is that for some reason they can't link most libraries
> statically without getting a bunch of weird errors.  It could take hours of
> research to figure out a workaround like -no-pie.
> 
> Given that GCC 6 on Debian 9 now does PIE executables by default, I think
> it becomes very necessary to consider that Debian out to build all static
> libraries with -fPIE (or -fPIC).  Otherwise you're going to get thousands
> and thousands of users having no clue why they can't link anything
> statically.  Plus if you did that you wouldn't have to build everything
> twice.  Win-Win !!!  So why not?

Worth, PIE code is measurably slower than normal code.
The fact that the Debian default compiler generate slow code is quite annoying
for HPC and anything where performance comparaison are important.

At the very least this break the principle of leat surprise. -fPIE
should only be activated by default when building Debian package, not
for users compiling their own code.

Cheers,
-- 
Bill. <ballombe@debian.org>

Imagine a large red swirl here. 

Message #67 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Adrian Bunk <bunk@stusta.de>

To: Phillip Hellewell <sshock@gmail.com>, 837478@bugs.debian.org

Subject: Re: Bug#837478: Static libraries - PIC or PIE?

Date: Mon, 21 Nov 2016 18:21:32 +0200

On Sat, Nov 19, 2016 at 07:12:19PM -0700, Phillip Hellewell wrote:
>...
>     - Now tries to link a few of the libraries statically (e.g.,
> libicuuc.a).  ld blows up with a bunch of relocation R_X86_64_32S, blah
> blah blah, recompile with -fPIC errors.

The error message is misleading, PIE is sufficient.

>...
> All they know is that for some reason they can't link most libraries
> statically without getting a bunch of weird errors.  It could take hours of
> research to figure out a workaround like -no-pie.

Submitting a bug against release-notes for having everything PIE 
releated properly documented would make sense.

> Given that GCC 6 on Debian 9 now does PIE executables by default, I think
> it becomes very necessary to consider that Debian out to build all static
> libraries with -fPIE (or -fPIC).

All static libraries in Debian will be recompiled with -fPIE before the 
release of stretch, for the ones where no uploads are happening for
other reasons binNMUs will be done.

> Otherwise you're going to get thousands
> and thousands of users having no clue why they can't link anything
> statically.  Plus if you did that you wouldn't have to build everything
> twice.  Win-Win !!!  So why not?

Building all static libraries with -fPIC (not just -fPIE) would be an 
insane amount of work, since you don't want to also build the binaries
with -fPIC.

> Thanks,
> Phillip

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Message #72 received at 837478@bugs.debian.org (full text, mbox, reply):

From: Phillip Hellewell <sshock@gmail.com>

To: Adrian Bunk <bunk@stusta.de>

Cc: 837478@bugs.debian.org

Subject: Re: Bug#837478: Static libraries - PIC or PIE?

Date: Mon, 21 Nov 2016 10:03:59 -0700

On Mon, Nov 21, 2016 at 9:21 AM, Adrian Bunk <bunk@stusta.de> wrote:
> On Sat, Nov 19, 2016 at 07:12:19PM -0700, Phillip Hellewell wrote:
>>...
>>     - Now tries to link a few of the libraries statically (e.g.,
>> libicuuc.a).  ld blows up with a bunch of relocation R_X86_64_32S, blah
>> blah blah, recompile with -fPIC errors.
>
> The error message is misleading, PIE is sufficient.

Ah yes, it would be nice if gcc could alter this error message slightly.

>> All they know is that for some reason they can't link most libraries
>> statically without getting a bunch of weird errors.  It could take hours of
>> research to figure out a workaround like -no-pie.
>
> Submitting a bug against release-notes for having everything PIE
> releated properly documented would make sense.

Agreed.  I'll try to submit that if I remember and get time; unless
you beat me to it, which I wouldn't mind at all :)

>> Given that GCC 6 on Debian 9 now does PIE executables by default, I think
>> it becomes very necessary to consider that Debian out to build all static
>> libraries with -fPIE (or -fPIC).
>
> All static libraries in Debian will be recompiled with -fPIE before the
> release of stretch, for the ones where no uploads are happening for
> other reasons binNMUs will be done.

Awesome!  That works for me :)

>> Otherwise you're going to get thousands
>> and thousands of users having no clue why they can't link anything
>> statically.  Plus if you did that you wouldn't have to build everything
>> twice.  Win-Win !!!  So why not?
>
> Building all static libraries with -fPIC (not just -fPIE) would be an
> insane amount of work, since you don't want to also build the binaries
> with -fPIC.

I'm happy with -fPIE for the static libraries.  Thanks!

Phillip

Send a report that this bug log contains spam.