Debian Bug report logs - #985187
ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: submit@bugs.debian.org

Subject: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c

Date: Sat, 13 Mar 2021 20:05:47 -0800

[Message part 1 (text/plain, inline)]

Source: ffmpeg
Severity: normal
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: buildpath
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org

The build path is embedded in various files generated with tools/cl2c:

  https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/ffmpeg.html

  avgblur.o (shipped in libavfilter.a)

  #line·1·"/build/1st/ffmpeg-4.3.2/libavfilter/opencl/avgblur.cl"
  vs.
  #line·1·"/build/2/ffmpeg-4.3.2/2nd/libavfilter/opencl/avgblur.cl"


The attached patch fixes this by patching tools/cl2c to use a basename
in the generated file rather than the full path.


It builds fine with this patch applied, but someone more familiar with
ffmpeg should ensure ffmpeg still functions correctly with the patch
applied.


Thanks for maintaining ffmpeg!

live well,
  vagrant

[0001-tools-cl2c-Strip-full-path-from-input-file-in-embedd.patch (text/x-diff, inline)]

From ec49e459797d6e112358566ae083c3bfae801023 Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian <vagrant@reproducible-builds.org>
Date: Sat, 13 Mar 2021 03:52:51 +0000
Subject: [PATCH] tools/cl2c: Strip full path from input file in embedded
 output.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Without this patch, the full build path gets embedded into various
binaries shipped in the package, for example, libavfilter.a contains
a references to:

  #line·1·"/build/1st/ffmpeg-4.3.2/libavfilter/opencl/avgblur.cl"

https://tests.reproducible-builds.org/debian/issues/unstable/captures_build_path_issue.html
---
 tools/cl2c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tools/cl2c b/tools/cl2c
index e3f92bab..48444e61 100755
--- a/tools/cl2c
+++ b/tools/cl2c
@@ -23,11 +23,13 @@ input="$1"
 output="$2"
 
 name=$(basename "$input" | sed 's/.cl$//')
+# Avoid embedded the build path, using only the basename of the input file.
+base_input=$(basename "$input")
 
 cat >$output <<EOF
 // Generated from $input
 const char *ff_opencl_source_$name =
-"#line 1 \"$input\"\n"
+"#line 1 \"$base_input\"\n"
 EOF
 
 # Convert \ to \\ and " to \", then add " to the start and end of the line.
-- 
2.20.1

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 985187@bugs.debian.org (full text, mbox, reply):

From: Sebastian Ramacher <sramacher@debian.org>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>, 985187@bugs.debian.org

Subject: Re: Bug#985187: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c

Date: Sun, 8 Aug 2021 21:46:11 +0200

[Message part 1 (text/plain, inline)]

Control: tags -1 upstream

Hi Vagrant

On 2021-03-13 20:05:47 -0800, Vagrant Cascadian wrote:
> Source: ffmpeg
> Severity: normal
> Tags: patch
> User: reproducible-builds@lists.alioth.debian.org
> Usertags: buildpath
> X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org
> 
> The build path is embedded in various files generated with tools/cl2c:
> 
>   https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/ffmpeg.html
> 
>   avgblur.o (shipped in libavfilter.a)
> 
>   #line·1·"/build/1st/ffmpeg-4.3.2/libavfilter/opencl/avgblur.cl"
>   vs.
>   #line·1·"/build/2/ffmpeg-4.3.2/2nd/libavfilter/opencl/avgblur.cl"
> 
> 
> The attached patch fixes this by patching tools/cl2c to use a basename
> in the generated file rather than the full path.
> 
> 
> It builds fine with this patch applied, but someone more familiar with
> ffmpeg should ensure ffmpeg still functions correctly with the patch
> applied.

As this patch touches upstream's build system, please submit it
upstream:

https://ffmpeg.org/developer.html#Contributing

Thanks

Cheers

> 
> 
> Thanks for maintaining ffmpeg!
> 
> live well,
>   vagrant

> From ec49e459797d6e112358566ae083c3bfae801023 Mon Sep 17 00:00:00 2001
> From: Vagrant Cascadian <vagrant@reproducible-builds.org>
> Date: Sat, 13 Mar 2021 03:52:51 +0000
> Subject: [PATCH] tools/cl2c: Strip full path from input file in embedded
>  output.
> MIME-Version: 1.0
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 8bit
> 
> Without this patch, the full build path gets embedded into various
> binaries shipped in the package, for example, libavfilter.a contains
> a references to:
> 
>   #line·1·"/build/1st/ffmpeg-4.3.2/libavfilter/opencl/avgblur.cl"
> 
> https://tests.reproducible-builds.org/debian/issues/unstable/captures_build_path_issue.html
> ---
>  tools/cl2c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/cl2c b/tools/cl2c
> index e3f92bab..48444e61 100755
> --- a/tools/cl2c
> +++ b/tools/cl2c
> @@ -23,11 +23,13 @@ input="$1"
>  output="$2"
>  
>  name=$(basename "$input" | sed 's/.cl$//')
> +# Avoid embedded the build path, using only the basename of the input file.
> +base_input=$(basename "$input")
>  
>  cat >$output <<EOF
>  // Generated from $input
>  const char *ff_opencl_source_$name =
> -"#line 1 \"$input\"\n"
> +"#line 1 \"$base_input\"\n"
>  EOF
>  
>  # Convert \ to \\ and " to \", then add " to the start and end of the line.
> -- 
> 2.20.1
> 




-- 
Sebastian Ramacher

[signature.asc (application/pgp-signature, inline)]

Message #17 received at 985187@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: ffmpeg-devel@ffmpeg.org

Cc: 985187@bugs.debian.org

Subject: Patch to remove build path from tools/cl2c

Date: Fri, 15 Oct 2021 18:16:07 -0700

[Message part 1 (text/plain, inline)]

The attached patch removes build paths from the resulting binaries,
which makes it easier to verify the reproducibility of a build without
needing to know the path from which it was originally built.

I ran "make fate" with and without the patch applied, and all of the
tests that passed as of commit 3cc3f5de2afda5b8f880c0817e9d67c2dafbfe1e
still passed (fate-sws-slice-yuv422-12bit-rgb48 and
fate-sws-slice-bgr0-nv12 both failed regardless of weather the patch was
applied).

This was originally reported at https://bugs.debian.org/985187

With this change applied, the ffmpeg package in Debian should build
reproducibly:

  https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/ffmpeg.html


Thanks for maintaining and developing ffmpeg, it is a very useful tool!


live well,
  vagrant

[0002-tools-cl2c-Strip-full-path-from-input-file-in-embedd.patch (text/x-diff, inline)]

From b86db13adc6a27337cd93af174356d80ead1a5a9 Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian <vagrant@reproducible-builds.org>
Date: Sat, 13 Mar 2021 03:52:51 +0000
Subject: [PATCH 1/1] tools/cl2c: Strip full path from input file in embedded
 output.

Without this patch, the full build path gets embedded into various
binaries shipped in the package, for example, libavfilter.a contains
a reference to:

  #line 1 "/build/1st/ffmpeg-4.3.2/libavfilter/opencl/avgblur.cl"

By not embedding the build path, it makes it easier to recreate the
build environment and reproduce the build:

  https://reproducible-builds.org/docs/build-path/

Originally submitted to Debian as:

  https://bugs.debian.org/985187

Signed-off-by: Vagrant Cascadian <vagrant@reproducible-builds.org>
---
 tools/cl2c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tools/cl2c b/tools/cl2c
index e3f92bab1c..48444e61a7 100755
--- a/tools/cl2c
+++ b/tools/cl2c
@@ -23,11 +23,13 @@ input="$1"
 output="$2"
 
 name=$(basename "$input" | sed 's/.cl$//')
+# Avoid embedded the build path, using only the basename of the input file.
+base_input=$(basename "$input")
 
 cat >$output <<EOF
 // Generated from $input
 const char *ff_opencl_source_$name =
-"#line 1 \"$input\"\n"
+"#line 1 \"$base_input\"\n"
 EOF
 
 # Convert \ to \\ and " to \", then add " to the start and end of the line.
-- 
2.33.0

[signature.asc (application/pgp-signature, inline)]

Message #24 received at 985187@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: Sebastian Ramacher <sramacher@debian.org>, 985187@bugs.debian.org

Subject: Re: Bug#985187: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c

Date: Thu, 14 Apr 2022 15:45:26 -0700

[Message part 1 (text/plain, inline)]

On 2021-08-08, Sebastian Ramacher wrote:
> On 2021-03-13 20:05:47 -0800, Vagrant Cascadian wrote:
>> Source: ffmpeg
>> Severity: normal
>> Tags: patch
>> User: reproducible-builds@lists.alioth.debian.org
>> Usertags: buildpath
>> X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org
>> 
>> The build path is embedded in various files generated with tools/cl2c:
>> 
>>   https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/ffmpeg.html
>> 
>>   avgblur.o (shipped in libavfilter.a)
>> 
>>   #line·1·"/build/1st/ffmpeg-4.3.2/libavfilter/opencl/avgblur.cl"
>>   vs.
>>   #line·1·"/build/2/ffmpeg-4.3.2/2nd/libavfilter/opencl/avgblur.cl"
>> 
>> 
>> The attached patch fixes this by patching tools/cl2c to use a basename
>> in the generated file rather than the full path.
>> 
>> 
>> It builds fine with this patch applied, but someone more familiar with
>> ffmpeg should ensure ffmpeg still functions correctly with the patch
>> applied.
>
> As this patch touches upstream's build system, please submit it
> upstream:
>
> https://ffmpeg.org/developer.html#Contributing

Did so, haven't really heard anything back:

  https://patchwork.ffmpeg.org/project/ffmpeg/patch/87sfwyotmg.fsf@yucca/

Any suggestions?

live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #29 received at 985187@bugs.debian.org (full text, mbox, reply):

From: Diederik de Haas <didi.debian@cknow.org>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>

Cc: 985187@bugs.debian.org

Subject: Re: Bug#985187: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c

Date: Tue, 03 Jan 2023 10:17:57 +0100

[Message part 1 (text/plain, inline)]

On 14 Apr 2022 15:45:26 -0700 Vagrant Cascadian <vagrant@reproducible-
builds.org> wrote:
> On 2021-08-08, Sebastian Ramacher wrote:
> > On 2021-03-13 20:05:47 -0800, Vagrant Cascadian wrote:
> >> Source: ffmpeg
> >> Severity: normal
> >> Tags: patch
> >> User: reproducible-builds@lists.alioth.debian.org
> >> 
> >> The build path is embedded in various files generated with tools/cl2c:
> >> 
> >>   https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/
> >>   diffoscope-results/ffmpeg.html

FWIW, in https://salsa.debian.org/diederik/ffmpeg/-/pipelines/478274 I added 
your patch, but the 'reprotest' job still failed.
I don't know how to read diffoscope's result (and currently don't have time to 
learn it), but it looks like more is needed to make it reproducible.
Happy to add more patches to my branch and if 'reprotest' then succeeds, 
you're free to add a "Tested-By: " tag with my name+email

> >> The attached patch fixes this by patching tools/cl2c to use a basename
> >> in the generated file rather than the full path.
> >
> > As this patch touches upstream's build system, please submit it
> > upstream:
> 
> Did so, haven't really heard anything back:
> 
>   https://patchwork.ffmpeg.org/project/ffmpeg/patch/87sfwyotmg.fsf@yucca/
> 
> Any suggestions?

I have no particular knowledge wrt ffmpeg-devel, but with the kernel patches 
sometimes fall through the cracks and the best way is just to resend them.

I did notice that the patch itself looks better then the email I saw on the 
'Forwarded' link. I _think_ that with 'git send-email' it would look cleaner.

I would suggest to slightly reword the commit message so that it doesn't 
contain "Without this patch". Maybe it's just me, but if the secondary commit 
message starts with "To make builds reproducible ..." that would heighten my 
interest in the patch. 'But' I really like the reproducible builds effort :-)

And possibly update the reference to ffmpeg-4.3.2 to point to 5.1 or something.
And I'd replace "Originally submitted to Debian as:\n\n https://b.d.o/985187" 
with a "Link:" tag.

HTH,
  Diederik

[signature.asc (application/pgp-signature, inline)]

Message #34 received at 985187@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: Diederik de Haas <didi.debian@cknow.org>

Cc: 985187@bugs.debian.org

Subject: Re: Bug#985187: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c

Date: Tue, 03 Jan 2023 10:06:41 -0800

[Message part 1 (text/plain, inline)]

On 2023-01-03, Diederik de Haas wrote:
> On 14 Apr 2022 15:45:26 -0700 Vagrant Cascadian <vagrant@reproducible-
> builds.org> wrote:
>> On 2021-08-08, Sebastian Ramacher wrote:
>> > On 2021-03-13 20:05:47 -0800, Vagrant Cascadian wrote:
>> >> Source: ffmpeg
>> >> Severity: normal
>> >> Tags: patch
>> >> User: reproducible-builds@lists.alioth.debian.org
>> >> 
>> >> The build path is embedded in various files generated with tools/cl2c:
>> >> 
>> >>   https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/
>> >>   diffoscope-results/ffmpeg.html
>
> FWIW, in https://salsa.debian.org/diederik/ffmpeg/-/pipelines/478274 I added 
> your patch, but the 'reprotest' job still failed.
> I don't know how to read diffoscope's result (and currently don't have time to 
> learn it), but it looks like more is needed to make it reproducible.
> Happy to add more patches to my branch and if 'reprotest' then succeeds, 
> you're free to add a "Tested-By: " tag with my name+email

Looks like some new source of non-determinism was added in 5.0.x, you
can see bookworm (which does not test build paths) was reproducible
until 5.0.x started getting tested in bookworm in June of 2022:

  https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/diffoscope-results/ffmpeg.html

Fixing the build path issue mentioned in this patch may dramatically
reduce the size of the diffoscope output from ~71MB to ~29KB ... so
might be nice to apply still even if it does not fix all the
reproducibility issues. I'll take another stab at it...


>> >> The attached patch fixes this by patching tools/cl2c to use a basename
>> >> in the generated file rather than the full path.
>> >
>> > As this patch touches upstream's build system, please submit it
>> > upstream:
>> 
>> Did so, haven't really heard anything back:
>> 
>>   https://patchwork.ffmpeg.org/project/ffmpeg/patch/87sfwyotmg.fsf@yucca/
>> 
>> Any suggestions?
>
> I have no particular knowledge wrt ffmpeg-devel, but with the kernel patches 
> sometimes fall through the cracks and the best way is just to resend them.
>
> I did notice that the patch itself looks better then the email I saw on the 
> 'Forwarded' link. I _think_ that with 'git send-email' it would look cleaner.
>
> I would suggest to slightly reword the commit message so that it doesn't 
> contain "Without this patch". Maybe it's just me, but if the secondary commit 
> message starts with "To make builds reproducible ..." that would heighten my 
> interest in the patch. 'But' I really like the reproducible builds effort :-)
>
> And possibly update the reference to ffmpeg-4.3.2 to point to 5.1 or something.
> And I'd replace "Originally submitted to Debian as:\n\n https://b.d.o/985187" 
> with a "Link:" tag.

Thanks for the suggestions!

live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #39 received at 985187@bugs.debian.org (full text, mbox, reply):

From: Diederik de Haas <didi.debian@cknow.org>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>

Cc: 985187@bugs.debian.org

Subject: Re: Bug#985187: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c

Date: Tue, 03 Jan 2023 19:27:01 +0100

[Message part 1 (text/plain, inline)]

On Tuesday, 3 January 2023 19:06:41 CET Vagrant Cascadian wrote:
> Looks like some new source of non-determinism was added in 5.0.x, you
> can see bookworm (which does not test build paths) was reproducible
> until 5.0.x started getting tested in bookworm in June of 2022:
>  
> https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/diffosco
> pe-results/ffmpeg.html
> 
> Fixing the build path issue mentioned in this patch may dramatically
> reduce the size of the diffoscope output from ~71MB to ~29KB ... so
> might be nice to apply still even if it does not fix all the
> reproducibility issues. I'll take another stab at it...

I just added "SALSA_CI_REPROTEST_ENABLE_DIFFOSCOPE: 1" to my salsa-ci.yml file 
which I assume means it'll now also produce a diffoscope output.

Feel free to use it if it helps:
https://salsa.debian.org/diederik/ffmpeg/-/pipelines
https://salsa.debian.org/diederik/ffmpeg/-/commits/fix-reprotest-issue

Cheers,
  Diederik

[signature.asc (application/pgp-signature, inline)]

Send a report that this bug log contains spam.