Debian Bug report logs - #1006863
tevent: reproducible-builds: build path embedded in various libraries

[Message part 1 (text/plain, inline)]
Source: tevent
Severity: normal
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: buildpath
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org

The build path and resulting Build ID for various libraries is embedded:

  https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/tevent.html

  /usr/lib/x86_64-linux-gnu/libtevent.so.0.11.0

  /build/1st/tevent-0.11.0/bin/default/../../tevent.c:303
  vs.
  /build/2/tevent-0.11.0/2nd/bin/default/../../tevent.c:303

The attached patch to debian/rules fixes this by passing
-ffile-prefix-map via CFLAGS in the dh_auto_configure override.

Alternately, updating to use the CFLAGS passed via dh/debhelper would
also likely fix this.

With this patch applied tevent should build reproducibly on
tests.reproducible-builds.org!

live well,
  vagrant

[0001-debian-rules-Add-ffile-prefix-map-to-CFLAGS.patch (text/x-diff, inline)]
From b07f0c48683e16712d21147feee2d5f85df67d9b Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian <vagrant@reproducible-builds.org>
Date: Mon, 7 Mar 2022 00:37:54 +0000
Subject: [PATCH] debian/rules: Add -ffile-prefix-map to CFLAGS.

Without this, the full path used during the build environment gets
embedded in various binaries.

https://reproducible-builds.org/docs/build-path/
---
 debian/rules | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/debian/rules b/debian/rules
index 856ec0e..a52d533 100755
--- a/debian/rules
+++ b/debian/rules
@@ -14,6 +14,8 @@ else
 CFLAGS += -O2
 endif
 
+CFLAGS += -ffile-prefix-map=$(CURDIR)=.
+
 LDFLAGS += -Wl,--as-needed
 
 DESTDIR=$(CURDIR)/debian/tmp
-- 
2.35.1


[signature.asc (application/pgp-signature, inline)]
I would rather this be discussed and implemented upstream.

For one, the tevent build system is shared with the rest of Samba, and
if possible this should be implemented by default for all 'make
install' runs, just as we do to strip out the bin/default from -rpath.

Andrew Bartlett

On Sun, 2022-03-06 at 16:43 -0800, Vagrant Cascadian wrote:
> Source: tevent
> Severity: normal
> Tags: patch
> User: reproducible-builds@lists.alioth.debian.org
> Usertags: buildpath
> X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org
> 
> The build path and resulting Build ID for various libraries is
> embedded:
> 
>   
> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/tevent.html
> 
>   /usr/lib/x86_64-linux-gnu/libtevent.so.0.11.0
> 
>   /build/1st/tevent-0.11.0/bin/default/../../tevent.c:303
>   vs.
>   /build/2/tevent-0.11.0/2nd/bin/default/../../tevent.c:303
> 
> The attached patch to debian/rules fixes this by passing
> -ffile-prefix-map via CFLAGS in the dh_auto_configure override.
> 
> Alternately, updating to use the CFLAGS passed via dh/debhelper would
> also likely fix this.
> 
> With this patch applied tevent should build reproducibly on
> tests.reproducible-builds.org!
> 
> live well,
>   vagrant
> _______________________________________________
> Pkg-samba-maint mailing list
> Pkg-samba-maint@alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-samba-maint
-- 
Andrew Bartlett (he/him)       https://samba.org/~abartlet/
Samba Team Member (since 2001) https://samba.org
Samba Team Lead, Catalyst IT   https://catalyst.net.nz/services/samba

Samba Development and Support, Catalyst IT - Expert Open Source
Solutions




[Message part 1 (text/plain, inline)]
On 2022-03-07, Andrew Bartlett wrote:
> I would rather this be discussed and implemented upstream.
>
> For one, the tevent build system is shared with the rest of Samba, and
> if possible this should be implemented by default for all 'make
> install' runs, just as we do to strip out the bin/default from -rpath.

Thanks for the quick response, I'll try and come up with something
upstreamable...

live well,
  vagrant

> On Sun, 2022-03-06 at 16:43 -0800, Vagrant Cascadian wrote:
>> Source: tevent
>> Severity: normal
>> Tags: patch
>> User: reproducible-builds@lists.alioth.debian.org
>> Usertags: buildpath
>> X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org
>> 
>> The build path and resulting Build ID for various libraries is
>> embedded:
>> 
>>   
>> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/tevent.html
>> 
>>   /usr/lib/x86_64-linux-gnu/libtevent.so.0.11.0
>> 
>>   /build/1st/tevent-0.11.0/bin/default/../../tevent.c:303
>>   vs.
>>   /build/2/tevent-0.11.0/2nd/bin/default/../../tevent.c:303
>> 
>> The attached patch to debian/rules fixes this by passing
>> -ffile-prefix-map via CFLAGS in the dh_auto_configure override.
>> 
>> Alternately, updating to use the CFLAGS passed via dh/debhelper would
>> also likely fix this.
>> 
>> With this patch applied tevent should build reproducibly on
>> tests.reproducible-builds.org!
>> 
>> live well,
>>   vagrant
>> _______________________________________________
>> Pkg-samba-maint mailing list
>> Pkg-samba-maint@alioth-lists.debian.net
>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-samba-maint
> -- 
> Andrew Bartlett (he/him)       https://samba.org/~abartlet/
> Samba Team Member (since 2001) https://samba.org
> Samba Team Lead, Catalyst IT   https://catalyst.net.nz/services/samba
>
> Samba Development and Support, Catalyst IT - Expert Open Source
> Solutions

[signature.asc (application/pgp-signature, inline)]
[Message part 1 (text/plain, inline)]
On 2022-03-07, Andrew Bartlett wrote:
> I would rather this be discussed and implemented upstream.
>
> For one, the tevent build system is shared with the rest of Samba, and
> if possible this should be implemented by default for all 'make
> install' runs, just as we do to strip out the bin/default from -rpath.

Originally reported as https://bugs.debian.org/1006863 where I proposed
passing additional arguments via CFLAGS in the debian build system.

Attached is a proof of concept patch that works by adding the argument
to CFLAGS by patching the upstream buildsystem.

The patch is bit ugly in how it derives the top level source directory
and likely error-prone... a cleaner way of going about that would be
much appreciated!

It also requires gcc 8+ or clang 10+ ... making it detect weather the
argument was supported and only adding it conditionally might be
desireable.

I am not too familiar with samba project processes, let me know if
there's a better place to send this!


live well,
  vagrant

> On Sun, 2022-03-06 at 16:43 -0800, Vagrant Cascadian wrote:
>> Source: tevent
>> Severity: normal
>> Tags: patch
>> User: reproducible-builds@lists.alioth.debian.org
>> Usertags: buildpath
>> X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org
>> 
>> The build path and resulting Build ID for various libraries is
>> embedded:
>> 
>>   
>> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/tevent.html
>> 
>>   /usr/lib/x86_64-linux-gnu/libtevent.so.0.11.0
>> 
>>   /build/1st/tevent-0.11.0/bin/default/../../tevent.c:303
>>   vs.
>>   /build/2/tevent-0.11.0/2nd/bin/default/../../tevent.c:303
>> 
>> The attached patch to debian/rules fixes this by passing
>> -ffile-prefix-map via CFLAGS in the dh_auto_configure override.
>> 
>> Alternately, updating to use the CFLAGS passed via dh/debhelper would
>> also likely fix this.
>> 
>> With this patch applied tevent should build reproducibly on
>> tests.reproducible-builds.org!


[0001-buildtools-wafsamba-samba_autoconf.py-Add-ffile-pref.patch (text/x-diff, inline)]
From 8324603d80051d6ad2f3f5fcdcc0a110f159b5aa Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian <vagrant@reproducible-builds.org>
Date: Sat, 12 Mar 2022 21:46:46 +0000
Subject: [PATCH] buildtools/wafsamba/samba_autoconf.py: Add -ffile-prefix-map
 to CFLAGS.

Without this, the build path is embedded in various binaries, which
requires additional knowledge about the build environment in order to
reproduce the build.
---
 buildtools/wafsamba/samba_autoconf.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/buildtools/wafsamba/samba_autoconf.py b/buildtools/wafsamba/samba_autoconf.py
index 4d2aea6..d618182 100644
--- a/buildtools/wafsamba/samba_autoconf.py
+++ b/buildtools/wafsamba/samba_autoconf.py
@@ -725,6 +725,13 @@ def SAMBA_CONFIG_H(conf, path=None):
     if Options.options.debug:
         conf.ADD_CFLAGS('-g', testflags=True)
 
+    # Remove build path using -ffile-prefix-map. Requires gcc 8+ or clang 10+
+    # https://reproducible-builds.org/docs/build-path/
+
+    # Determine the top-level source directory by stripping out the relative path that waf was called as.
+    sourcedir=os.path.abspath(sys.argv[0]).split(sys.argv[0].lstrip('.'))[0]
+    conf.ADD_CFLAGS('-ffile-prefix-map=%s=.' % sourcedir, testflags=True)
+
     if Options.options.pidl_developer:
         conf.env.PIDL_DEVELOPER_MODE = True
 
-- 
2.35.1


[signature.asc (application/pgp-signature, inline)]
On Sat, 2022-03-12 at 13:53 -0800, Vagrant Cascadian wrote:
> On 2022-03-07, Andrew Bartlett wrote:
> > I would rather this be discussed and implemented upstream.
> > 
> > For one, the tevent build system is shared with the rest of Samba, and
> > if possible this should be implemented by default for all 'make
> > install' runs, just as we do to strip out the bin/default from -rpath.
> 
> Originally reported as https://bugs.debian.org/1006863 where I proposed
> passing additional arguments via CFLAGS in the debian build system.
> 
> Attached is a proof of concept patch that works by adding the argument
> to CFLAGS by patching the upstream buildsystem.
> 
> The patch is bit ugly in how it derives the top level source directory
> and likely error-prone... a cleaner way of going about that would be
> much appreciated!
> 
> It also requires gcc 8+ or clang 10+ ... making it detect weather the
> argument was supported and only adding it conditionally might be
> desireable.

testflags=True should be doing that, the CI should help determine if
that works for this option.

> I am not too familiar with samba project processes, let me know if
> there's a better place to send this!

https://wiki.samba.org/index.php/Contribute shows how to open a Merge
Request for samba.  Once you get a gitlab username let me know and you
can skip to using our shared development repo for a full CI.

conf.env.srcdir should get you the srcdir you need.

I guess my main concern is once Samba is packaged etc, can we still get
a full backtrace?  How does this interact with debug packages etc?

Andrew Bartlett

-- 
Andrew Bartlett (he/him)        https://samba.org/~abartlet/
Samba Team Member (since 2001)  https://samba.org
Samba Developer, Catalyst IT    https://catalyst.net.nz/services/samba




[Message part 1 (text/plain, inline)]
On 2022-03-13, Andrew Bartlett wrote:
> On Sat, 2022-03-12 at 13:53 -0800, Vagrant Cascadian wrote:
>> Originally reported as https://bugs.debian.org/1006863 where I proposed
>> passing additional arguments via CFLAGS in the debian build system.
>> 
>> Attached is a proof of concept patch that works by adding the argument
>> to CFLAGS by patching the upstream buildsystem.
>> 
>> The patch is bit ugly in how it derives the top level source directory
>> and likely error-prone... a cleaner way of going about that would be
>> much appreciated!
>> 
>> It also requires gcc 8+ or clang 10+ ... making it detect weather the
>> argument was supported and only adding it conditionally might be
>> desireable.
>
> testflags=True should be doing that, the CI should help determine if
> that works for this option.

Excellent!


>> I am not too familiar with samba project processes, let me know if
>> there's a better place to send this!
>
> https://wiki.samba.org/index.php/Contribute shows how to open a Merge
> Request for samba.  Once you get a gitlab username let me know and you
> can skip to using our shared development repo for a full CI.

Ok, I've got:

  https://gitlab.com/vagrantc

I'll work on som e updated patches.

Should it be a merge request into the samba repository or tevent, or is
there a better place for it?


> conf.env.srcdir should get you the srcdir you need.

Yes, that worked well, thanks!


> I guess my main concern is once Samba is packaged etc, can we still get
> a full backtrace?  How does this interact with debug packages etc?

I think the short answer is yes, although it may require passing
arguments to the debugger:

   https://stackoverflow.com/questions/67191647/using-ffile-prefix-map-breaks-debugging

Both gdb and lldb allow you to set the path to the source code.


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]
Control: tag -1 + moreinfo

On Sun, 06 Mar 2022 16:43:09 -0800 Vagrant Cascadian <vagrant@reproducible-builds.org> wrote:
> Source: tevent
> Severity: normal
> Tags: patch
> User: reproducible-builds@lists.alioth.debian.org
> Usertags: buildpath
> X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org
> 
> The build path and resulting Build ID for various libraries is embedded:
> 
>   https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/tevent.html
> 
>   /usr/lib/x86_64-linux-gnu/libtevent.so.0.11.0
> 
>   /build/1st/tevent-0.11.0/bin/default/../../tevent.c:303
>   vs.
>   /build/2/tevent-0.11.0/2nd/bin/default/../../tevent.c:303
> 
> The attached patch to debian/rules fixes this by passing
> -ffile-prefix-map via CFLAGS in the dh_auto_configure override.

Hm. It looks like this flags is already being in use on unstable these days.

At least I see it in all recent unstable builds, eg:

 https://buildd.debian.org/status/fetch.php?pkg=tevent&arch=amd64&ver=0.12.0-1&stamp=1651527651&raw=0

make[3]: Entering directory '/<<PKGBUILDDIR>>'
CC="x86_64-linux-gnu-gcc" CFLAGS="-g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat ...

I don't really know where it is coming from (my guess is dpkg-buildflags),
but it looks like this issue has already been solved in a more general
way meanwhile.

Can we perhaps close this bugreport now?

Thanks,

/mjt



On Sun, 06 Mar 2022 16:43:09 -0800 Vagrant Cascadian <vagrant@reproducible-builds.org> wrote:
> Source: tevent
> Severity: normal
> Tags: patch
> User: reproducible-builds@lists.alioth.debian.org
> Usertags: buildpath
> X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org
> 
> The build path and resulting Build ID for various libraries is embedded:
> 
>   https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/tevent.html
> 
>   /usr/lib/x86_64-linux-gnu/libtevent.so.0.11.0
> 
>   /build/1st/tevent-0.11.0/bin/default/../../tevent.c:303
>   vs.
>   /build/2/tevent-0.11.0/2nd/bin/default/../../tevent.c:303
> 
> The attached patch to debian/rules fixes this by passing
> -ffile-prefix-map via CFLAGS in the dh_auto_configure override.
> 
> Alternately, updating to use the CFLAGS passed via dh/debhelper would
> also likely fix this.

It looks like current debian build environment already enables -ffile-prefix-map=$(CURDIR)=.
option, so nothing is needed to be done on the tevent/talloc/etc side.
Closing this bugreport now.  Please reopen it if you think this is incorrect.

Thanks,

/mjt



[Message part 1 (text/plain, inline)]
Control: fixed 1006863 0.12.0-1

> On Sun, 06 Mar 2022 16:43:09 -0800 Vagrant Cascadian <vagrant@reproducible-builds.org> wrote:
>> The build path and resulting Build ID for various libraries is embedded:
>> 
>>   https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/tevent.html
>> 
>>   /usr/lib/x86_64-linux-gnu/libtevent.so.0.11.0
>> 
>>   /build/1st/tevent-0.11.0/bin/default/../../tevent.c:303
>>   vs.
>>   /build/2/tevent-0.11.0/2nd/bin/default/../../tevent.c:303
>> 
>> The attached patch to debian/rules fixes this by passing
>> -ffile-prefix-map via CFLAGS in the dh_auto_configure override.
>> 
>> Alternately, updating to use the CFLAGS passed via dh/debhelper would
>> also likely fix this.
>
> It looks like current debian build environment already enables -ffile-prefix-map=$(CURDIR)=.
> option, so nothing is needed to be done on the tevent/talloc/etc side.
> Closing this bugreport now.  Please reopen it if you think this is incorrect.

Yeah, appears to have been fixed:

  https://tests.reproducible-builds.org/debian/history/tevent.html

tevent (0.12.0-1) unstable; urgency=medium
...
    - use buildflags.mk instead of our own CFLAGS

That is probably what did it.

Thanks!


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Debian Bug report logs - #1006863 tevent: reproducible-builds: build path embedded in various libraries

Debian Bug report logs - #1006863
tevent: reproducible-builds: build path embedded in various libraries