Debian Bug report logs - #894476
rcc: please honour SOURCE_DATE_EPOCH

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Thomas Preud'homme <robotux@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: RCC: provide option to encode EPOCH timestamp

Date: Fri, 30 Mar 2018 23:42:36 +0100

Package: qtbase5-dev-tools
Version: 5.9.2+dfsg-12
Severity: wishlist
Tags: upstream
User: reproducible-builds@lists.alioth.debian.org
Usertags: timestamps toolchain

Hi,

While investigating ultracopier's lack of build reproducibility, I found
out that rcc encodes the timestamp of the files the QRC file being
compiled references (see end of RCCFileInfo::writeDataInfo()). This
becomes a problem for generated files because the output of rcc is then
different in 2 different builds. It would be nice if rcc had an option
to encode a stable timestamp, eg. EPOCH.

Best regards,

Thomas

-- System Information:
Debian Release: buster/sid
  APT prefers unstable
  APT policy: (900, 'unstable'), (500, 'unstable-debug'), (500, 'stable-updates'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.15.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE=fr_FR:fr (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages qtbase5-dev-tools depends on:
ii  libc6                            2.27-2
ii  libgcc1                          1:8-20180321-1
ii  libqt5core5a [qtbase-abi-5-9-2]  5.9.2+dfsg-12
ii  libqt5dbus5                      5.9.2+dfsg-12
ii  libstdc++6                       8-20180321-1
ii  perl                             5.26.1-5
ii  qtchooser                        64-ga1b6736-5
ii  zlib1g                           1:1.2.8.dfsg-5

qtbase5-dev-tools recommends no packages.

qtbase5-dev-tools suggests no packages.

-- no debconf information

Message #10 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 894476@bugs.debian.org

Cc: "Thomas Preud'homme" <robotux@debian.org>

Subject: Re: RCC: provide option to encode EPOCH timestamp

Date: Sat, 31 Mar 2018 11:05:02 +0100

Hi,

> While investigating ultracopier's lack of build reproducibility, I found
> out that rcc encodes the timestamp of the files the QRC file being
> compiled references

Indeed, we've been tracking this for a while in the Reproducible Builds
project [0] but, without a patch, we did not wish to bother you with a
bug. :)

It is affecting at least 4 or 5 other packages. I had previously tracked
it down to [1].

Regarding the solution, I think I would actually prefer to see something
consuming SOURCE_DATE_EPOCH[1] rather than adding an option to set a
specific timestamp.

  [0] https://reproducible-builds.org/
  [1] https://sources.debian.org/src/qtbase-opensource-src/5.9.2+dfsg-12/src/tools/rcc/rcc.cpp/#L207-L212
  [2] https://reproducible-builds.org/specs/source-date-epoch/


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #15 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Lisandro Damián Nicanor Pérez Meyer <perezmeyer@gmail.com>

To: Chris Lamb <lamby@debian.org>, 894476@bugs.debian.org

Cc: "Thomas Preud'homme" <robotux@debian.org>

Subject: Re: Bug#894476: RCC: provide option to encode EPOCH timestamp

Date: Sat, 31 Mar 2018 13:17:35 +0000

[Message part 1 (text/plain, inline)]

Thanks for the bug! And sorry for the to posting :-(

I'll try to take a look into this.

El sáb., 31 de mar. de 2018 07:09, Chris Lamb <lamby@debian.org> escribió:

> Hi,
>
> > While investigating ultracopier's lack of build reproducibility, I found
> > out that rcc encodes the timestamp of the files the QRC file being
> > compiled references
>
> Indeed, we've been tracking this for a while in the Reproducible Builds
> project [0] but, without a patch, we did not wish to bother you with a
> bug. :)
>
> It is affecting at least 4 or 5 other packages. I had previously tracked
> it down to [1].
>
> Regarding the solution, I think I would actually prefer to see something
> consuming SOURCE_DATE_EPOCH[1] rather than adding an option to set a
> specific timestamp.
>
>   [0] https://reproducible-builds.org/
>   [1]
> https://sources.debian.org/src/qtbase-opensource-src/5.9.2+dfsg-12/src/tools/rcc/rcc.cpp/#L207-L212
>   [2] https://reproducible-builds.org/specs/source-date-epoch/
>
>
> Best wishes,
>
> --
>       ,''`.
>      : :'  :     Chris Lamb
>      `. `'`      lamby@debian.org / chris-lamb.co.uk
>        `-
>
>

[Message part 2 (text/html, inline)]

Message #20 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Thomas Preud'homme <robotux@debian.org>

To: Sune Vuorela <sune@debian.org>,Chris Lamb <lamby@debian.org>

Cc: 894476@bugs.debian.org

Subject: Re: RCC: provide option to encode EPOCH timestamp

Date: Tue, 03 Apr 2018 19:46:51 +0100

[Message part 1 (text/plain, inline)]

On April 3, 2018 7:25:03 PM GMT+01:00, Sune Vuorela <sune@debian.org> wrote:
>On Saturday, March 31, 2018 12:05:02 PM CEST Chris Lamb wrote:
>> > While investigating ultracopier's lack of build reproducibility, I
>found
>> > out that rcc encodes the timestamp of the files the QRC file being
>> > compiled references
>
>I don't actually see why this should be a problem. If input changes,
>output 
>changes.
>I do think that using touch(1) on the input should allow different
>output.
>
>It is quite easy to reproduce:
>
>qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version
>2 ../
>qimage.qrc -o 1
>qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version
>2 ../
>qimage.qrc -o 2
>qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version
>2 ../
>qimage.qrc -o 3
>
>Gives same output.
>Even adding in touch ../qimage.qrc keeps the same output.
>
>qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version
>2 ../
>qimage.qrc -o 4
>
>touch ../images/image.bmp
>
>qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version
>2 ../
>qimage.qrc -o 5
>
>is needed to get different output
>
>14ef6dae8e4992ce907948c1c4af272b  1
>14ef6dae8e4992ce907948c1c4af272b  2
>14ef6dae8e4992ce907948c1c4af272b  3
>14ef6dae8e4992ce907948c1c4af272b  4
>54c6f8c09a347955ae2f36e68bbd2539  5
>
>
>So. What touches the files?
>
>/Sune
>-- 
>I didn’t stop pretending when I became an adult, it’s just that when I
>was a 
>kid I was pretending that I fit into the rules and structures of this
>world. 
>And now that I’m an adult, I pretend that those rules and structures
>exist.
>   - zefrank

Hi Sune,

The problematic files are Qt message files (ie .qm files) generated at build time via lrelease from translation files (ie .ts files). Therefore two different builds will generate those .qm files at different times which will end up with different cpp files generated by rcc. Currently I'm working around it by setting the modified time of those .qm files to EPOCH after they are generated. I think it would be nice if there was a way for rcc to avoid doing that manually. I agree with Chris that honouring SOURCE_DATE_EPOCH in rcc would be a nice solution.

Best regards,

Thomas

[Message part 2 (text/html, inline)]

Message #25 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Sune Vuorela <sune@debian.org>

To: Chris Lamb <lamby@debian.org>

Cc: 894476@bugs.debian.org, Thomas Preud'homme <robotux@debian.org>

Subject: Re: RCC: provide option to encode EPOCH timestamp

Date: Tue, 03 Apr 2018 20:25:03 +0200

On Saturday, March 31, 2018 12:05:02 PM CEST Chris Lamb wrote:
> > While investigating ultracopier's lack of build reproducibility, I found
> > out that rcc encodes the timestamp of the files the QRC file being
> > compiled references

I don't actually see why this should be a problem. If input changes, output 
changes.
I do think that using touch(1) on the input should allow different output.

It is quite easy to reproduce:

qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 1
qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 2
qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 3

Gives same output.
Even adding in touch ../qimage.qrc keeps the same output.

qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 4

touch ../images/image.bmp

qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 5

is needed to get different output

14ef6dae8e4992ce907948c1c4af272b  1
14ef6dae8e4992ce907948c1c4af272b  2
14ef6dae8e4992ce907948c1c4af272b  3
14ef6dae8e4992ce907948c1c4af272b  4
54c6f8c09a347955ae2f36e68bbd2539  5


So. What touches the files?

/Sune
-- 
I didn’t stop pretending when I became an adult, it’s just that when I was a 
kid I was pretending that I fit into the rules and structures of this world. 
And now that I’m an adult, I pretend that those rules and structures exist.
   - zefrank

Message #30 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Sune Vuorela <sune@debian.org>

To: Thomas Preud'homme <robotux@debian.org>

Cc: Chris Lamb <lamby@debian.org>, 894476@bugs.debian.org

Subject: Re: RCC: provide option to encode EPOCH timestamp

Date: Tue, 03 Apr 2018 21:09:51 +0200

On Tuesday, April 3, 2018 8:46:51 PM CEST Thomas Preud'homme wrote:
> The problematic files are Qt message files (ie .qm files) generated at build
> time via lrelease from translation files (ie .ts files). Therefore two

Right. I didn't think about autogenerated files during build.

> I think it would be nice if there was a way for
> rcc to avoid doing that manually. I agree with Chris that honouring
> SOURCE_DATE_EPOCH in rcc would be a nice solution.

I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal 
circumstances, given that I do think that having the actual timestamp of the 
non-generated files makes sense.

I have reached out to upstream about it.

/Sune
-- 
I didn’t stop pretending when I became an adult, it’s just that when I was a 
kid I was pretending that I fit into the rules and structures of this world. 
And now that I’m an adult, I pretend that those rules and structures exist.
   - zefrank

Message #35 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Sune Vuorela <sune@debian.org>, "Thomas Preud'homme" <robotux@debian.org>

Cc: 894476@bugs.debian.org

Subject: Re: RCC: provide option to encode EPOCH timestamp

Date: Tue, 03 Apr 2018 20:14:23 +0100

Hi Sune!

> I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal 
> circumstances

Can you elaborate on what you mean by "normal circumstances"? :)

How about e use S_D_E if it is setup/exported, otherwise use the mtime
of the file as before?

(This is the pattern/logic used extensively elsewhere in toolchains for
this very exact purpose; it would not be unique to rcc)


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #40 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Sune Vuorela <sune@debian.org>

To: Chris Lamb <lamby@debian.org>

Cc: Thomas Preud'homme <robotux@debian.org>, 894476@bugs.debian.org

Subject: Re: RCC: provide option to encode EPOCH timestamp

Date: Tue, 03 Apr 2018 21:20:22 +0200

On Tuesday, April 3, 2018 9:14:23 PM CEST Chris Lamb wrote:
> Hi Sune!
> 
> > I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal
> > circumstances
> 
> Can you elaborate on what you mean by "normal circumstances"? :)

"normal circumstances" is rcc on a source file, as opposed to an autogenerated 
file.

> How about e use S_D_E if it is setup/exported, otherwise use the mtime
> of the file as before?

I think that using S_D_E only makes sense if rcc is run on an autogenerated 
file, but I do think that most rcc runs is run on existing source files.


I don't have a good idea to differentiate those.

/Sune
-- 
I didn’t stop pretending when I became an adult, it’s just that when I was a 
kid I was pretending that I fit into the rules and structures of this world. 
And now that I’m an adult, I pretend that those rules and structures exist.
   - zefrank

Message #45 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Sune Vuorela <sune@debian.org>

Cc: "Thomas Preud'homme" <robotux@debian.org>, 894476@bugs.debian.org

Subject: Re: RCC: provide option to encode EPOCH timestamp

Date: Tue, 03 Apr 2018 20:24:58 +0100

Hi Sune,

> "normal circumstances" is rcc on a source file, as opposed
> to an autogenerated file

I'm not /entirely/ sure what the difference is as I'm not in the
Qt/RCC world too often these days (alas !), but why just not use
SOURCE_DATE_EPOCH *iff* it is exported?

Normal systems simply do not have this envvar set, so there is really
no danger of it leaking elsewhere or causing unintended side-effects.

It would also have the benefit of not having to differentiate between
the two... :)


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #50 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Thomas Preud'homme <robotux@debian.org>

To: Sune Vuorela <sune@debian.org>,Chris Lamb <lamby@debian.org>

Cc: 894476@bugs.debian.org

Subject: Re: RCC: provide option to encode EPOCH timestamp

Date: Tue, 03 Apr 2018 20:36:21 +0100

[Message part 1 (text/plain, inline)]

On April 3, 2018 8:20:22 PM GMT+01:00, Sune Vuorela <sune@debian.org> wrote:
>On Tuesday, April 3, 2018 9:14:23 PM CEST Chris Lamb wrote:
>> Hi Sune!
>> 
>> > I don't think honouring SOURCE_DATE_EPOCH is the right idea under
>normal
>> > circumstances
>> 
>> Can you elaborate on what you mean by "normal circumstances"? :)
>
>"normal circumstances" is rcc on a source file, as opposed to an
>autogenerated 
>file.
>
>> How about e use S_D_E if it is setup/exported, otherwise use the
>mtime
>> of the file as before?
>
>I think that using S_D_E only makes sense if rcc is run on an
>autogenerated 
>file, but I do think that most rcc runs is run on existing source
>files.
>
>
>I don't have a good idea to differentiate those.
>
>/Sune
>-- 
>I didn’t stop pretending when I became an adult, it’s just that when I
>was a 
>kid I was pretending that I fit into the rules and structures of this
>world. 
>And now that I’m an adult, I pretend that those rules and structures
>exist.
>   - zefrank

One small clarification: in my case rcc *is* run on a nongenerated resource file. It's some of the files that the resource file list that are generated and whose timestamp end up in the cpp file generated by rcc.

In other hand:

foo.qrc mentions bar.qm which is generated from bar.ts.

rcc foo.qrc generates resource_bar.cpp which contains constant data that encodes bar.qm timestamp and this create different resource_bar.o at every build.

Best regards,

Thomas

[Message part 2 (text/html, inline)]

Message #55 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Sune Vuorela <sune@debian.org>

To: Chris Lamb <lamby@debian.org>

Cc: Thomas Preud'homme <robotux@debian.org>, 894476@bugs.debian.org

Subject: Re: RCC: provide option to encode EPOCH timestamp

Date: Tue, 03 Apr 2018 21:39:33 +0200

On Tuesday, April 3, 2018 9:24:58 PM CEST Chris Lamb wrote:
> I'm not /entirely/ sure what the difference is as I'm not in the
> Qt/RCC world too often these days (alas !), but why just not use
> SOURCE_DATE_EPOCH *iff* it is exported?
> 
> Normal systems simply do not have this envvar set, so there is really
> no danger of it leaking elsewhere or causing unintended side-effects.

We don't know how application users are using this.

The following is some theoretical example, that I'm quite sure could be used 
in some real world applications:

QFileInfo embeddedFile(":/foo/data.xml"); // fallback downloaded in the 
ancient past
QFileInfo systemfile("/usr/share/foo/data.xml");

if (systemfile.lastModified() > embeddedFile.lastModified())
{
   data = readFromFile(systemFile);
}
else
{
   data = readFromFile(embeddedFile);
}

If S_D_E gets used, rather than the data.xml modified date in the source, this 
will end up using the wrong file if S_D_E is newer than the system copy of the 
file.

This is not a unused databit, but a fully available piece of metadata for the 
files.

/Sune
-- 
I didn’t stop pretending when I became an adult, it’s just that when I was a 
kid I was pretending that I fit into the rules and structures of this world. 
And now that I’m an adult, I pretend that those rules and structures exist.
   - zefrank

Message #60 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Lisandro Damián Nicanor Pérez Meyer <perezmeyer@gmail.com>

To: Sune Vuorela <sune@debian.org>, 894476@bugs.debian.org

Cc: Chris Lamb <lamby@debian.org>, "Thomas Preud'homme" <robotux@debian.org>

Subject: Re: Bug#894476: RCC: provide option to encode EPOCH timestamp

Date: Tue, 03 Apr 2018 21:15:42 +0000

[Message part 1 (text/plain, inline)]

El mar., 3 de abr. de 2018 16:42, Sune Vuorela <sune@debian.org> escribió:

> On Tuesday, April 3, 2018 9:24:58 PM CEST Chris Lamb wrote:
> > I'm not /entirely/ sure what the difference is as I'm not in the
> > Qt/RCC world too often these days (alas !), but why just not use
> > SOURCE_DATE_EPOCH *iff* it is exported?
> >
> > Normal systems simply do not have this envvar set, so there is really
> > no danger of it leaking elsewhere or causing unintended side-effects.
>
> We don't know how application users are using this.
>
> The following is some theoretical example, that I'm quite sure could be
> used
> in some real world applications:
>
> QFileInfo embeddedFile(":/foo/data.xml"); // fallback downloaded in the
> ancient past
> QFileInfo systemfile("/usr/share/foo/data.xml");
>
> if (systemfile.lastModified() > embeddedFile.lastModified())
> {
>    data = readFromFile(systemFile);
> }
> else
> {
>    data = readFromFile(embeddedFile);
> }
>
> If S_D_E gets used, rather than the data.xml modified date in the source,
> this
> will end up using the wrong file if S_D_E is newer than the system copy of
> the
> file.
>
> This is not a unused databit, but a fully available piece of metadata for
> the
> files.


I'm afraid Sune is right here, it might be used that way. Questionable?
Sure thing, but still valid code.

But after all the same can be said from embedding translations into the
binary itself.

So yes, we will need help from upstream with this one.

>
>

[Message part 2 (text/html, inline)]

Message #65 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Thomas Preud'homme <robotux@debian.org>

To: Lisandro Damián Nicanor Pérez Meyer <perezmeyer@gmail.com>,Sune Vuorela <sune@debian.org>,894476@bugs.debian.org

Cc: Chris Lamb <lamby@debian.org>

Subject: Re: Bug#894476: RCC: provide option to encode EPOCH timestamp

Date: Tue, 03 Apr 2018 22:54:39 +0100

[Message part 1 (text/plain, inline)]

On April 3, 2018 10:15:42 PM GMT+01:00, "Lisandro Damián Nicanor Pérez Meyer" <perezmeyer@gmail.com> wrote:
>El mar., 3 de abr. de 2018 16:42, Sune Vuorela <sune@debian.org>
>escribió:
>
>> On Tuesday, April 3, 2018 9:24:58 PM CEST Chris Lamb wrote:
>> > I'm not /entirely/ sure what the difference is as I'm not in the
>> > Qt/RCC world too often these days (alas !), but why just not use
>> > SOURCE_DATE_EPOCH *iff* it is exported?
>> >
>> > Normal systems simply do not have this envvar set, so there is
>really
>> > no danger of it leaking elsewhere or causing unintended
>side-effects.
>>
>> We don't know how application users are using this.
>>
>> The following is some theoretical example, that I'm quite sure could
>be
>> used
>> in some real world applications:
>>
>> QFileInfo embeddedFile(":/foo/data.xml"); // fallback downloaded in
>the
>> ancient past
>> QFileInfo systemfile("/usr/share/foo/data.xml");
>>
>> if (systemfile.lastModified() > embeddedFile.lastModified())
>> {
>>    data = readFromFile(systemFile);
>> }
>> else
>> {
>>    data = readFromFile(embeddedFile);
>> }
>>
>> If S_D_E gets used, rather than the data.xml modified date in the
>source,
>> this
>> will end up using the wrong file if S_D_E is newer than the system
>copy of
>> the
>> file.
>>
>> This is not a unused databit, but a fully available piece of metadata
>for
>> the
>> files.
>
>
>I'm afraid Sune is right here, it might be used that way. Questionable?
>Sure thing, but still valid code.
>
>But after all the same can be said from embedding translations into the
>binary itself.
>
>So yes, we will need help from upstream with this one.
>
>>
>>

Maybe the solution is then not in rcc but in whatever generate the files that the qrc that rcc processes refer to. For instance in the case of ultracopier lrelease could have a mean if generating .qm files with the same modified timestamp as the .ts file it processes. This would guarantee stable output for a stable source and this later on rcc would also generate stable output.

Thoughts?

Best regards,

Thomas

[Message part 2 (text/html, inline)]

Message #70 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: "Thomas Preud'homme" <robotux@debian.org>, Lisandro Damián Nicanor Pérez Meyer <perezmeyer@gmail.com>, Sune Vuorela <sune@debian.org>, 894476@bugs.debian.org

Subject: Re: Bug#894476: RCC: provide option to encode EPOCH timestamp

Date: Tue, 03 Apr 2018 22:59:51 +0100

Hi Thomas,

> Maybe the solution is then not in rcc but in whatever generate the files 
> that the qrc that rcc processes refer to. For instance in the case of 
> ultracopier lrelease could have a mean if generating .qm files with the 
> same modified timestamp as the .ts file it processes

That sounds workable..


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #75 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <t.glaser@tarent.de>

To: Sune Vuorela <sune@debian.org>

Cc: Chris Lamb <lamby@debian.org>, Thomas Preud'homme <robotux@debian.org>, 894476@bugs.debian.org, control@bugs.debian.org

Subject: rcc: please honour SOURCE_DATE_EPOCH (was Re: RCC: provide option to encode EPOCH timestamp)

Date: Sun, 8 Jul 2018 00:58:48 +0200 (CEST)

affects 894476 src:musescore
retitle 894476 rcc: please honour SOURCE_DATE_EPOCH
thanks

Hi *,

I just got bitten by this in an FTBR of src:musescore which I tracked
down to this bug.

On Tue, 3 Apr 2018, Sune Vuorela wrote:

> If S_D_E gets used, rather than the data.xml modified date in the
> source, this will end up using the wrong file if S_D_E is newer than
> the system copy of the file.

AIUI, S_D_E is used as a *delimiter*, not as a timestamp *all* files
are to be set to.

So, basically, instead of

	encode(dst, sb.st_mtime);

you do

	encode(dst, MIN(sb.st_mtime, s_d_e_as_time_t_value));

and not

	encode(dst, s_d_e_as_time_t_value);

which would indeed break things the way you described.


Chris/H01ger, please correct me if I’m wrong, but this is
how I understood S_D_E to work.


I don’t know cmake magic enough to be able to do the same
patch to lrelease et al. as Thomas did, so I unfortunately
will have to live with the FTBR in src:musescore until this
issue is fixed.

Thanks,
//mirabilos
-- 
>> Why don't you use JavaScript? I also don't like enabling JavaScript in
> Because I use lynx as browser.
+1
	-- Octavio Alvarez, me and ⡍⠁⠗⠊⠕ (Mario Lang) on debian-devel

Message #84 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Thorsten Glaser <t.glaser@tarent.de>, Sune Vuorela <sune@debian.org>

Cc: "Thomas Preud'homme" <robotux@debian.org>, 894476@bugs.debian.org, control@bugs.debian.org

Subject: Re: rcc: please honour SOURCE_DATE_EPOCH (was Re: RCC: provide option to encode EPOCH timestamp)

Date: Sun, 08 Jul 2018 07:46:53 +0100

Hi Thorsten,

> AIUI, S_D_E is used as a *delimiter*, not as a timestamp *all* files
> are to be set to.

I've found that having a shared vocabulary is very useful so just in
case it helps we tend to use the term "clamping" when referring to
this MIN(a,b) concept rather than "delimiter" or similar.  eg. "foo
clamps the mtimes to SOURCE_DATE_EPOCH".

> Chris/H01ger, please correct me if I’m wrong, but this is
> how I understood S_D_E to work.

It is not required. Does the canonical specification not explain?

  https://reproducible-builds.org/specs/source-date-epoch/
  
I pedantically refer you to that /only/ because if it is not clear
enough then there is a need to revise it. :)


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #89 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <t.glaser@tarent.de>

To: Chris Lamb <lamby@debian.org>

Cc: Sune Vuorela <sune@debian.org>, Thomas Preud'homme <robotux@debian.org>, 894476@bugs.debian.org

Subject: Re: rcc: please honour SOURCE_DATE_EPOCH

Date: Mon, 9 Jul 2018 14:52:40 +0200 (CEST)

Hi Chris,

> It is not required.

I’m parsing this as “you were not incorrect”, right?
(Sorry, non-English-native speaker here.)

> Does the canonical specification not explain?
>
>   https://reproducible-builds.org/specs/source-date-epoch/
>
> I pedantically refer you to that /only/ because if it is not clear
> enough then there is a need to revise it. :)

It is indeed clear enough. I didn’t have it at hand when writing
my previous message and only vaguely recalled having read it
quite some time before. (Thanks for linking it in this thread.)

> > AIUI, S_D_E is used as a *delimiter*, not as a timestamp *all* files
> > are to be set to.
>
> I've found that having a shared vocabulary is very useful so just in
> case it helps we tend to use the term "clamping" when referring to

True… it’s just that I didn’t recall the word, as “clamping”
is not something normally part of my English vocabulary (and,
having looked it up in the dictionary, nothing I’d normally
associate with this), so it’s probably a “lost in translation”
effect. I’ll try to remember next time.

Thanks,
//mirabilos
-- 
15:41⎜<Lo-lan-do:#fusionforge> Somebody write a testsuite for helloworld :-)

Message #94 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Thorsten Glaser <t.glaser@tarent.de>

Cc: Sune Vuorela <sune@debian.org>, "Thomas Preud'homme" <robotux@debian.org>, 894476@bugs.debian.org

Subject: Re: rcc: please honour SOURCE_DATE_EPOCH

Date: Mon, 09 Jul 2018 14:19:49 +0100

Hi Thorsten,

> > It is not required.
> 
> I’m parsing this as “you were not incorrect”, right?

It doesn't really matter whether you were correct or incorrect in the
past and seems a little silly to debate that; far easier and clearer
IMHO to statelessly state that clamping is not required when using
SOURCE_DATE_EPOCH. :)

> > I've found that having a shared vocabulary is very useful so just in
> > case it helps we tend to use the term "clamping" when referring to
> 
> True… it’s just that I didn’t recall the word, as “clamping”
> is not something normally part of my English vocabulary […]

I might agree it's esoteric word/usage, but not only is it already "out
there", we have more problems with "reproducible" to worry about this
one too much alas.

> I’ll try to remember next time.

Again, no question of blame or apologies here… :)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #99 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <t.glaser@tarent.de>

To: Chris Lamb <lamby@debian.org>

Cc: Sune Vuorela <sune@debian.org>, Thomas Preud'homme <robotux@debian.org>, 894476@bugs.debian.org

Subject: Re: rcc: please honour SOURCE_DATE_EPOCH

Date: Mon, 9 Jul 2018 15:26:13 +0200 (CEST)

On Mon, 9 Jul 2018, Chris Lamb wrote:

> far easier and clearer IMHO to statelessly state that clamping is not
> required when using SOURCE_DATE_EPOCH. :)

OK, now I do not understand _that_…

Doesn’t matter as long as this bug gets fixed, though ☻☺

Have a nice day,
//mirabilos
-- 
>> Why don't you use JavaScript? I also don't like enabling JavaScript in
> Because I use lynx as browser.
+1
	-- Octavio Alvarez, me and ⡍⠁⠗⠊⠕ (Mario Lang) on debian-devel

Message #104 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Thorsten Glaser <t.glaser@tarent.de>

Cc: Sune Vuorela <sune@debian.org>, "Thomas Preud'homme" <robotux@debian.org>, 894476@bugs.debian.org

Subject: Re: rcc: please honour SOURCE_DATE_EPOCH

Date: Mon, 09 Jul 2018 14:36:39 +0100

Thorsten,

> > far easier and clearer IMHO to statelessly state that clamping is not
> > required when using SOURCE_DATE_EPOCH. :)
> 
> OK, now I do not understand _that_…

(It's clearer to say "X is the situation" vs "when you said Y before
that was correct/incorrect")


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #109 received at 894476@bugs.debian.org (full text, mbox, reply):

From: "Bernhard M. Wiedemann" <bernhardout@lsmod.de>

To: 894476@bugs.debian.org

Subject: qt upstream

Date: Wed, 11 Jul 2018 04:51:18 +0000

Is this the same as this upstream bug?
https://bugreports.qt.io/browse/QTBUG-62511
that had this patch merged
https://codereview.qt-project.org/#/c/202999/5/src/tools/rcc/rcc.cpp
with a renamed SOURCE_DATE_EPOCH env var

Message #114 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Lisandro Damián Nicanor Pérez Meyer <perezmeyer@gmail.com>

To: control@bugs.debian.org

Cc: 894476@bugs.debian.org

Subject: Add metatada

Date: Wed, 11 Jul 2018 16:14:27 -0300

[Message part 1 (text/plain, inline)]

forwarded 894476 https://bugreports.qt.io/browse/QTBUG-62511
thanks

Indeed, that's right. Thanks a lot Bernhard!

-- 
Existe un problema cultural, el que lleva -por ejemplo- a los padres a
comprar una computadora y ponerla en la habitación de los varones.  La
computadora tiende a ser vista como un objeto "masculino", por lo cual
el número de mujeres interesadas es menor.
  Margarita Manterola, Debian Developer, en un mail sobre las mujeres
  en el software libre, LugFi.
  http://listas.fi.uba.ar/pipermail/lug/2005-September/020266.html

Lisandro Damián Nicanor Pérez Meyer
http://perezmeyer.com.ar/
http://perezmeyer.blogspot.com/

[signature.asc (application/pgp-signature, inline)]

Message #121 received at 894476-done@bugs.debian.org (full text, mbox, reply):

From: Lisandro Damián Nicanor Pérez Meyer <perezmeyer@gmail.com>

To: 894476-done@bugs.debian.org

Cc: reproducible-builds@lists.alioth.debian.org

Subject: Solved from the Qt side.

Date: Wed, 11 Jul 2018 16:21:49 -0300

[Message part 1 (text/plain, inline)]

Version: 5.11.1+dfsg-1

The bug upstream has been closed as invalid (see https://bugreports.qt.io/
browse/QTBUG-62511) Non the less a workaround has been included in Qt 5.11, 
already in experimental. Setting QT_RCC_SOURCE_DATE_OVERRIDE should be enough 
to solve this issue.

Now the point is: where should this variable be set?

I guess dh's cmake and qmake helpers might be a good place to start with, but 
I think you might have a bteer undesrtanding of this.

After all forcing the variable will mean also already reproducible builds will 
get modified.

Cheers!

-- 
18: Como se pueden evitar los problemas de alimentacion electrica
    * No coma cerca de un enchufe
    Damian Nadales
    http://mx.grulic.org.ar/lurker/message/20080307.141449.a70fb2fc.es.html

Lisandro Damián Nicanor Pérez Meyer
http://perezmeyer.com.ar/
http://perezmeyer.blogspot.com/

[signature.asc (application/pgp-signature, inline)]

Message #126 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@debian.org>

To: Lisandro Damián Nicanor Pérez Meyer <perezmeyer@gmail.com>

Cc: reproducible-builds@lists.alioth.debian.org, 894476@bugs.debian.org

Subject: Re: #894476: Solved from the Qt side. (rcc: please honour SOURCE_DATE_EPOCH)

Date: Wed, 11 Jul 2018 13:08:23 -0700

[Message part 1 (text/plain, inline)]

On 2018-07-11, Lisandro Damián Nicanor Pérez Meyer wrote:
> Version: 5.11.1+dfsg-1
>
> The bug upstream has been closed as invalid (see https://bugreports.qt.io/
> browse/QTBUG-62511) Non the less a workaround has been included in Qt 5.11, 
> already in experimental. Setting QT_RCC_SOURCE_DATE_OVERRIDE should be enough 
> to solve this issue.

The mentioned upstream bug report is simply marked as invalid without
explanation.

This is (currently) an empty search:

  https://codesearch.debian.net/search?q=QT_RCC_SOURCE_DATE_OVERRIDE

Do you have a link for QT_RCC_SOURCE_DATE_OVERRIDE present in Debian and
upstream?  This bug doesn't really seem resolved if that variable isn't
actually getting set by anything or consumed by anything...


Ideally QT_RCC_SOURCE_DATE_OVERRIDE would get set based on
SOURCE_DATE_EPOCH, otherwise build tools have an arbitrary growing
number of variables to keep track of for each library, toolchain,
language, etc. that implements their own way of doing this sort of
thing, which kind of defeats the purpose of SOURCE_DATE_EPOCH. *sigh*


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #131 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Lisandro Damián Nicanor Pérez Meyer <perezmeyer@gmail.com>

To: Vagrant Cascadian <vagrant@debian.org>, 894476@bugs.debian.org

Cc: Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Subject: Re: Bug#894476: #894476: Solved from the Qt side. (rcc: please honour SOURCE_DATE_EPOCH)

Date: Wed, 11 Jul 2018 17:34:52 -0300

[Message part 1 (text/plain, inline)]

El mié., 11 de jul. de 2018 17:12, Vagrant Cascadian <vagrant@debian.org>
escribió:

> On 2018-07-11, Lisandro Damián Nicanor Pérez Meyer wrote:
> > Version: 5.11.1+dfsg-1
> >
> > The bug upstream has been closed as invalid (see
> https://bugreports.qt.io/
> > browse/QTBUG-62511) Non the less a workaround has been included in Qt
> 5.11,
> > already in experimental. Setting QT_RCC_SOURCE_DATE_OVERRIDE should be
> enough
> > to solve this issue.
>
> The mentioned upstream bug report is simply marked as invalid without
> explanation.
>

The point is that stuff like CMake can change this value. See comments.


This is (currently) an empty search:
>
>   https://codesearch.debian.net/search?q=QT_RCC_SOURCE_DATE_OVERRIDE


Because it did not scan experimental.


> Do you have a link for QT_RCC_SOURCE_DATE_OVERRIDE present in Debian and
> upstream?  This bug doesn't really seem resolved if that variable isn't
> actually getting set by anything or consumed by anything...
>

Right now, in experimental. We hope to get the transition going soon.

Ideally QT_RCC_SOURCE_DATE_OVERRIDE would get set based on
> SOURCE_DATE_EPOCH, otherwise build tools have an arbitrary growing
> number of variables to keep track of for each library, toolchain,
> language, etc. that implements their own way of doing this sort of
> thing, which kind of defeats the purpose of SOURCE_DATE_EPOCH. *sigh*


That's sadly something we can only "fix" by making packages have the right
value set. As per Qt policy the environment variable needs to be prefixed
with QT, so no chance of directly using SOURCE_DATE_EPOCH.

Feel free to jump in in the upstream bug report if you feel like this is
not enough.

Cheers, Lisandro.

[Message part 2 (text/html, inline)]

Message #136 received at 894476@bugs.debian.org (full text, mbox, reply):

From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>

To: 894476@bugs.debian.org

Subject: Re: Bug#894476: #894476: Solved from the Qt side. (rcc: please honour SOURCE_DATE_EPOCH)

Date: Thu, 12 Jul 2018 19:00:13 +0200

> That's sadly something we can only "fix" by making packages have the right
> value set. As per Qt policy the environment variable needs to be prefixed
> with QT, so no chance of directly using SOURCE_DATE_EPOCH.

I added a downstream patch to a test-version [1] in openSUSE with

+        static const quint64 sourceDate2 = 1000 *
qgetenv("SOURCE_DATE_EPOCH").toULongLong();
+        if (sourceDate2 != 0)
+            lastmod = sourceDate2;

but somehow this did not help (e.g. with bitcoin's
bitcoin-0.16.1/src/qt/qrc_bitcoin_locale.cpp going into the bitcoin-qt
binary).
So maybe the patch is not the correct solution... or there are more
similar issues somewhere else in rcc.



[1]
https://build.opensuse.org/package/view_file/home:bmwiedemann:reproducible:test/libqt5-qtbase/reproducibletime.patch

Message #141 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Sune Vuorela <sune@vuorela.dk>

To: Vagrant Cascadian <vagrant@debian.org>, 894476@bugs.debian.org

Cc: Lisandro Damián Nicanor Pérez Meyer <perezmeyer@gmail.com>, reproducible-builds@lists.alioth.debian.org

Subject: Re: Bug#894476: #894476: Solved from the Qt side. (rcc: please honour SOURCE_DATE_EPOCH)

Date: Thu, 12 Jul 2018 19:28:21 +0200

On Wednesday, July 11, 2018 10:08:23 PM CEST Vagrant Cascadian wrote:

> Ideally QT_RCC_SOURCE_DATE_OVERRIDE would get set based on
> SOURCE_DATE_EPOCH, otherwise build tools have an arbitrary growing

No. As explained, we need to look at each individual package to check if the 
timestamp is actually used for anything. It can be. It probably is. 

I think a better fix might be to specifically mark in the rcc file if the 
dates are important or not. But it requires involvement upstream. (Or maybe if 
the rcc file is autogenerated or not). I don't think it is a problem for non-
autogenerated rcc files.

/Sune
-- 
I didn’t stop pretending when I became an adult, it’s just that when I was a 
kid I was pretending that I fit into the rules and structures of this world. 
And now that I’m an adult, I pretend that those rules and structures exist.
   - zefrank

Message #146 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Thomas Preud'homme <robotux@debian.org>

To: Sune Vuorela <sune@vuorela.dk>

Cc: Vagrant Cascadian <vagrant@debian.org>, 894476@bugs.debian.org, Lisandro Damián Nicanor Pérez Meyer <perezmeyer@gmail.com>, reproducible-builds@lists.alioth.debian.org

Subject: Re: Bug#894476: #894476: Solved from the Qt side. (rcc: please honour SOURCE_DATE_EPOCH)

Date: Fri, 13 Jul 2018 23:30:12 +0100

[Message part 1 (text/plain, inline)]

On jeudi 12 juillet 2018 18:28:21 BST Sune Vuorela wrote:
> On Wednesday, July 11, 2018 10:08:23 PM CEST Vagrant Cascadian wrote:
> > Ideally QT_RCC_SOURCE_DATE_OVERRIDE would get set based on
> > SOURCE_DATE_EPOCH, otherwise build tools have an arbitrary growing
> 
> No. As explained, we need to look at each individual package to check if the
> timestamp is actually used for anything. It can be. It probably is.
> 
> I think a better fix might be to specifically mark in the rcc file if the
> dates are important or not. But it requires involvement upstream. (Or maybe
> if the rcc file is autogenerated or not). I don't think it is a problem for
> non- autogenerated rcc files.

Agreed, it's only a problem for files autogenerated at build time. rcc on a 
file that's part of the source tarball is gonna give a reproducible result.

Best regards,

Thomas

[signature.asc (application/pgp-signature, inline)]

Message #151 received at 894476@bugs.debian.org (full text, mbox, reply):

From: Peter Wu <peter@lekensteyn.nl>

To: 894476@bugs.debian.org

Subject: Workarounds for rcc reproducibility issues due to mtime

Date: Fri, 18 Jan 2019 22:09:35 +0100

Hi all,

To improve reproducibility for an upstream project that does not rely on
a valid QFileInfo::lastModified value, we used the --format-version 1
option to force the use of the older (pre-Qt 5.8) format that does not
include the timestamp.

This upstream project uses CMake with AUTORCC enabled, the patch was
pretty simple: https://code.wireshark.org/review/31593

For the benefit of other distributions (I also use Arch Linux) and
upstream projects, I have documented this here as well:
https://reproducible-builds.org/docs/deterministic-build-systems/#cmake-notes

If you feel that the recommendations are wrong, feel free to edit the
page or discuss it here or on IRC, I aimed at having the peculiarities
documented somewhere.
-- 
Kind regards,
Peter Wu
https://lekensteyn.nl

Send a report that this bug log contains spam.