Debian Bug report logs - #1035704
proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: submit@bugs.debian.org

Subject: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Sun, 07 May 2023 17:07:02 -0700

[Message part 1 (text/plain, inline)]

Source: proj
Severity: normal
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: timestamps timezone
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org

Various .gsb and .gtx files get touched in the build process in a way
that results in a timezone-dependent timestamp for these files in the
shipped package:

  https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/diffoscope-results/proj.html

  -rw-r--r--···0·root·········(0)·root·········(0)····83696·2018-02-22·07:28:23.000000·./usr/share/proj/BETA2007.gsb
  vs.
  -rw-r--r--···0·root·········(0)·root·········(0)····83696·2018-02-21·05:28:23.000000·./usr/share/proj/BETA2007.gsb

The attached patch fixes this by not touching these files during the
build process.

According to my local tests, with this patch applied, proj should build
reproducibly on tests.reproducible-builds.org once it lands in
bookworm/testing!

Unfortunately, there are unresolved issues with build paths, which are
tested in unstable and experimental, so will not show as reproducible
there without further fixes.


Thanks for maintaining proj!


live well,
  vagrant

[0001-debian-datumgrids-.shar-Avoid-updating-timestamps.patch (text/x-diff, inline)]

From 25dbf89dae803e4e5e207c253d9f83c889680ed9 Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian <vagrant@reproducible-builds.org>
Date: Tue, 2 May 2023 16:39:31 -0700
Subject: [PATCH] debian/datumgrids*.shar: Avoid updating timestamps.

This results in timezone-specific differences in various .gsb and .gtx
files.

https://reproducible-builds.org/docs/timestamps/
---
 debian/datumgrids-ch.shar | 21 ++-------------------
 debian/datumgrids.shar    | 21 ++-------------------
 2 files changed, 4 insertions(+), 38 deletions(-)

diff --git a/debian/datumgrids-ch.shar b/debian/datumgrids-ch.shar
index 308ca93..37194b7 100644
--- a/debian/datumgrids-ch.shar
+++ b/debian/datumgrids-ch.shar
@@ -84,25 +84,8 @@ st2=123123592001.59
 st2tr=123123592001.5 # old SysV 14-char limit
 st3=1231235901
 
-if   touch -am -t ${st1} ${f} >/dev/null 2>&1 && \
-     test ! -f ${st1} && test -f ${f}; then
-  shar_touch='touch -am -t $1$2$3$4$5$6.$7 "$8"'
-
-elif touch -am ${st2} ${f} >/dev/null 2>&1 && \
-     test ! -f ${st2} && test ! -f ${st2tr} && test -f ${f}; then
-  shar_touch='touch -am $3$4$5$6$1$2.$7 "$8"'
-
-elif touch -am ${st3} ${f} >/dev/null 2>&1 && \
-     test ! -f ${st3} && test -f ${f}; then
-  shar_touch='touch -am $3$4$5$6$2 "$8"'
-
-else
-  shar_touch=:
-  echo
-  ${echo} 'WARNING: not restoring timestamps.  Consider getting and
-installing GNU '\''touch'\'', distributed in GNU coreutils...'
-  echo
-fi
+# Avoid mangling timestamps
+shar_touch=:
 rm -f ${st1} ${st2} ${st2tr} ${st3} ${f}
 #
 if test ! -d ${lock_dir} ; then :
diff --git a/debian/datumgrids.shar b/debian/datumgrids.shar
index d9a0161..f41ae56 100644
--- a/debian/datumgrids.shar
+++ b/debian/datumgrids.shar
@@ -97,25 +97,8 @@ st2=123123592001.59
 st2tr=123123592001.5 # old SysV 14-char limit
 st3=1231235901
 
-if   touch -am -t ${st1} ${f} >/dev/null 2>&1 && \
-     test ! -f ${st1} && test -f ${f}; then
-  shar_touch='touch -am -t $1$2$3$4$5$6.$7 "$8"'
-
-elif touch -am ${st2} ${f} >/dev/null 2>&1 && \
-     test ! -f ${st2} && test ! -f ${st2tr} && test -f ${f}; then
-  shar_touch='touch -am $3$4$5$6$1$2.$7 "$8"'
-
-elif touch -am ${st3} ${f} >/dev/null 2>&1 && \
-     test ! -f ${st3} && test -f ${f}; then
-  shar_touch='touch -am $3$4$5$6$2 "$8"'
-
-else
-  shar_touch=:
-  echo
-  ${echo} 'WARNING: not restoring timestamps.  Consider getting and
-installing GNU '\''touch'\'', distributed in GNU coreutils...'
-  echo
-fi
+# avoid mangling timestamps on files
+shar_touch=:
 rm -f ${st1} ${st2} ${st2tr} ${st3} ${f}
 #
 if test ! -d ${lock_dir} ; then :
-- 
2.39.2

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 1035704@bugs.debian.org (full text, mbox, reply):

From: Sebastiaan Couwenberg <sebastic@xs4all.nl>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>, 1035704@bugs.debian.org

Subject: Re: Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Mon, 8 May 2023 06:00:34 +0200

Control: tags -1 moreinfo

On 5/8/23 02:07, Vagrant Cascadian wrote:
> The attached patch fixes this by not touching these files during the
> build process.

From shar(1):

"
  -m, --no-timestamp
         do not restore modification times.

         Avoid generating 'touch' commands to restore the file
         modification dates when unpacking files from the archive.

         When file modification times are not preserved, project build
         programs like "make" will see built files older than the files
         they get built from.  This is why, when this option is not
         used, a special effort is made to restore timestamps.
"

That should be used when generating the archives instead of your patch 
to not have the mtimes touched when unpacking the archives.

But the diffoscope-results only show a few of the files from the shar 
archives with different mtimes, and all of them get touched when 
unpacking the archive just before the configure target is executed.

shar actually makes the timestamps reproducible by always using the 
mtime recorded in the archive instead of the time the files were unpacked.

Something else is likely changing the mtime after the files are 
unpacked. Some of these grids are used by the testsuite for example.

Kind Regards,

Bas

-- 
 GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1

Message #17 received at 1035704@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: Sebastiaan Couwenberg <sebastic@xs4all.nl>, 1035704@bugs.debian.org

Subject: Re: Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Mon, 08 May 2023 13:43:31 -0700

[Message part 1 (text/plain, inline)]

On 2023-05-08, Sebastiaan Couwenberg wrote:
> On 5/8/23 02:07, Vagrant Cascadian wrote:
>> The attached patch fixes this by not touching these files during the
>> build process.
>
>  From shar(1):
>
> "
>    -m, --no-timestamp
>           do not restore modification times.
>
>           Avoid generating 'touch' commands to restore the file
>           modification dates when unpacking files from the archive.
>
>           When file modification times are not preserved, project build
>           programs like "make" will see built files older than the files
>           they get built from.  This is why, when this option is not
>           used, a special effort is made to restore timestamps.
> "
>
> That should be used when generating the archives instead of your patch 
> to not have the mtimes touched when unpacking the archives.

Is it actually a problem to allow dpkg to normalize the timestamps on
these files rather than forcefully setting them to ... a value from a
shar archive? It is perhaps a naive question; I really do not know.


> But the diffoscope-results only show a few of the files from the shar 
> archives with different mtimes, and all of them get touched when 
> unpacking the archive just before the configure target is executed.

Well, I too was perplexed why other files were not affected, but it is
consistently those .gsb and .gtx files, and the submitted patch allows
to consistently build reproducibly in the dozens of times I've build
it...


> shar actually makes the timestamps reproducible by always using the 
> mtime recorded in the archive instead of the time the files were unpacked.
>
> Something else is likely changing the mtime after the files are 
> unpacked. Some of these grids are used by the testsuite for example.

I will try to look into it further, but without really being familiar
with the proj codebase (or even what proj itself does)... any additional
very specific suggestions where to look next would definitely be
appreciated!  :)


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #22 received at 1035704@bugs.debian.org (full text, mbox, reply):

From: Sebastiaan Couwenberg <sebastic@xs4all.nl>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>, 1035704@bugs.debian.org

Subject: Re: Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Tue, 9 May 2023 05:58:21 +0200

On 5/9/23 05:47, Sebastiaan Couwenberg wrote:
> On 5/8/23 22:43, Vagrant Cascadian wrote:
>> On 2023-05-08, Sebastiaan Couwenberg wrote:
>>> On 5/8/23 02:07, Vagrant Cascadian wrote:
>>>> The attached patch fixes this by not touching these files during the
>>>> build process.
>>>
>>>   From shar(1):
>>>
>>> "
>>>     -m, --no-timestamp
>>>            do not restore modification times.
>>>
>>>            Avoid generating 'touch' commands to restore the file
>>>            modification dates when unpacking files from the archive.
>>>
>>>            When file modification times are not preserved, project build
>>>            programs like "make" will see built files older than the 
>>> files
>>>            they get built from.  This is why, when this option is not
>>>            used, a special effort is made to restore timestamps.
>>> "
>>>
>>> That should be used when generating the archives instead of your patch
>>> to not have the mtimes touched when unpacking the archives.
>>
>> Is it actually a problem to allow dpkg to normalize the timestamps on
>> these files rather than forcefully setting them to ... a value from a
>> shar archive? It is perhaps a naive question; I really do not know.
> 
> Where does dpkg normalize the timestamps?
> 
> shar sets the timestamps when the archive is unpacked before the package 
> built starts.
> 
> Some of the files in the diffoscope-results are only installed in 
> proj-data and not used otherwise during the build.
> 
>   * BETA2007.gsb is used in test/gie/DHDN_ETRS89.gie
> 
>   * CHENYX06.gsb/CHENYX06_etrs.gsb/CHENYX06a.gsb are only installed
> 
>   * egm96_15.gtx is used in test/gie/deformation.gie,
>     test/gie/more_builtins.gie, test/gie/4D-API_cs2cs-style.gie, and
>     test/cli/testdatumfile
> 
>   * ntf_r93.gsb is used in test/gie/more_builtins.gie,
>     test/gie/4D-API_cs2cs-style.gie, and test/cli/testdatumfile
> 
>   * nzgd2kgrid0005.gsb is used in unit tests
> 
>>> But the diffoscope-results only show a few of the files from the shar
>>> archives with different mtimes, and all of them get touched when
>>> unpacking the archive just before the configure target is executed.
>>
>> Well, I too was perplexed why other files were not affected, but it is
>> consistently those .gsb and .gtx files, and the submitted patch allows
>> to consistently build reproducibly in the dozens of times I've build
>> it...
>>
>>
>>> shar actually makes the timestamps reproducible by always using the
>>> mtime recorded in the archive instead of the time the files were 
>>> unpacked.
>>>
>>> Something else is likely changing the mtime after the files are
>>> unpacked. Some of these grids are used by the testsuite for example.
>>
>> I will try to look into it further, but without really being familiar
>> with the proj codebase (or even what proj itself does)... any additional
>> very specific suggestions where to look next would definitely be
>> appreciated!  :)
> 
> CMake's configure_file() is used to copy the .gsb & .gtx files from 
> CMAKE_CURRENT_SOURCE_DIR to CMAKE_CURRENT_BINARY_DIR, that might be 
> touching the mtimes too. See: data/CMakeLists.txt

Seeing how the mtimes are off by two hours, this could be the difference 
between UTC and CEST. The latter was in effect when the archives were 
created:

 $ grep "Made on" debian/datumgrids*.shar
 debian/datumgrids-ch.shar:# Made on 2018-02-26 22:27 CET by <bas@anubis>.
 debian/datumgrids.shar:# Made on 2018-09-15 20:13 CEST by <bas@anubis>.

But why does it only affect the binary GSB & GTX files, and not also the 
binary ntv1_can.dat file or text files like README.DATUMGRID and the 
init files (the ones without extensions)?

Kind Regards,

Bas

-- 
 GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1

Message #27 received at 1035704@bugs.debian.org (full text, mbox, reply):

From: Sebastiaan Couwenberg <sebastic@xs4all.nl>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>, 1035704@bugs.debian.org

Subject: Re: Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Tue, 9 May 2023 05:47:58 +0200

On 5/8/23 22:43, Vagrant Cascadian wrote:
> On 2023-05-08, Sebastiaan Couwenberg wrote:
>> On 5/8/23 02:07, Vagrant Cascadian wrote:
>>> The attached patch fixes this by not touching these files during the
>>> build process.
>>
>>   From shar(1):
>>
>> "
>>     -m, --no-timestamp
>>            do not restore modification times.
>>
>>            Avoid generating 'touch' commands to restore the file
>>            modification dates when unpacking files from the archive.
>>
>>            When file modification times are not preserved, project build
>>            programs like "make" will see built files older than the files
>>            they get built from.  This is why, when this option is not
>>            used, a special effort is made to restore timestamps.
>> "
>>
>> That should be used when generating the archives instead of your patch
>> to not have the mtimes touched when unpacking the archives.
> 
> Is it actually a problem to allow dpkg to normalize the timestamps on
> these files rather than forcefully setting them to ... a value from a
> shar archive? It is perhaps a naive question; I really do not know.

Where does dpkg normalize the timestamps?

shar sets the timestamps when the archive is unpacked before the package 
built starts.

Some of the files in the diffoscope-results are only installed in 
proj-data and not used otherwise during the build.

 * BETA2007.gsb is used in test/gie/DHDN_ETRS89.gie

 * CHENYX06.gsb/CHENYX06_etrs.gsb/CHENYX06a.gsb are only installed

 * egm96_15.gtx is used in test/gie/deformation.gie,
   test/gie/more_builtins.gie, test/gie/4D-API_cs2cs-style.gie, and
   test/cli/testdatumfile

 * ntf_r93.gsb is used in test/gie/more_builtins.gie,
   test/gie/4D-API_cs2cs-style.gie, and test/cli/testdatumfile

 * nzgd2kgrid0005.gsb is used in unit tests

>> But the diffoscope-results only show a few of the files from the shar
>> archives with different mtimes, and all of them get touched when
>> unpacking the archive just before the configure target is executed.
> 
> Well, I too was perplexed why other files were not affected, but it is
> consistently those .gsb and .gtx files, and the submitted patch allows
> to consistently build reproducibly in the dozens of times I've build
> it...
> 
> 
>> shar actually makes the timestamps reproducible by always using the
>> mtime recorded in the archive instead of the time the files were unpacked.
>>
>> Something else is likely changing the mtime after the files are
>> unpacked. Some of these grids are used by the testsuite for example.
> 
> I will try to look into it further, but without really being familiar
> with the proj codebase (or even what proj itself does)... any additional
> very specific suggestions where to look next would definitely be
> appreciated!  :)

CMake's configure_file() is used to copy the .gsb & .gtx files from 
CMAKE_CURRENT_SOURCE_DIR to CMAKE_CURRENT_BINARY_DIR, that might be 
touching the mtimes too. See: data/CMakeLists.txt

Kind Regards,

Bas

-- 
 GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1

Message #32 received at 1035704@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: Sebastiaan Couwenberg <sebastic@xs4all.nl>, 1035704@bugs.debian.org

Subject: Re: Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Mon, 08 May 2023 23:07:08 -0700

[Message part 1 (text/plain, inline)]

On 2023-05-09, Sebastiaan Couwenberg wrote:
> On 5/9/23 05:47, Sebastiaan Couwenberg wrote:
>> On 5/8/23 22:43, Vagrant Cascadian wrote:
>>> On 2023-05-08, Sebastiaan Couwenberg wrote:
>>>> On 5/8/23 02:07, Vagrant Cascadian wrote:
>>>>> The attached patch fixes this by not touching these files during the
>>>>> build process.
>>>>
>>>>   From shar(1):
>>>>
>>>> "
>>>>     -m, --no-timestamp
>>>>            do not restore modification times.
>>>>
>>>>            Avoid generating 'touch' commands to restore the file
>>>>            modification dates when unpacking files from the archive.
>>>>
>>>>            When file modification times are not preserved, project build
>>>>            programs like "make" will see built files older than the 
>>>> files
>>>>            they get built from.  This is why, when this option is not
>>>>            used, a special effort is made to restore timestamps.
>>>> "
>>>>
>>>> That should be used when generating the archives instead of your patch
>>>> to not have the mtimes touched when unpacking the archives.
>>>
>>> Is it actually a problem to allow dpkg to normalize the timestamps on
>>> these files rather than forcefully setting them to ... a value from a
>>> shar archive? It is perhaps a naive question; I really do not know.
>> 
>> Where does dpkg normalize the timestamps?

I thought it did as part of dpkg-deb, from the dpkg-deb.1 manpage:

       SOURCE_DATE_EPOCH
           If set, it will be used as the timestamp (as seconds since
           the epoch) in the deb(5)'s ar(5) container and used to clamp
           the mtime in the tar(5) file entries.

The other adjacent files appear to use a timestamp consistent with the
last debian/changelog entry:

  -rw-r--r--···0·root·········(0)·root·········(0)·····1097·2022-12-01·08:50:03.000000·./usr/share/proj/CH

Which is what the dpkg-buildpackage.1 manpage says is used to
SOURCE_DATE_EPOCH...

       SOURCE_DATE_EPOCH
           This variable is set to the Unix timestamp since the epoch of
           the latest entry in debian/changelog, if it is not already
           defined.


>> shar sets the timestamps when the archive is unpacked before the package 
>> built starts.
>> 
>> Some of the files in the diffoscope-results are only installed in 
>> proj-data and not used otherwise during the build.
>> 
>>   * BETA2007.gsb is used in test/gie/DHDN_ETRS89.gie
>> 
>>   * CHENYX06.gsb/CHENYX06_etrs.gsb/CHENYX06a.gsb are only installed
>> 
>>   * egm96_15.gtx is used in test/gie/deformation.gie,
>>     test/gie/more_builtins.gie, test/gie/4D-API_cs2cs-style.gie, and
>>     test/cli/testdatumfile
>> 
>>   * ntf_r93.gsb is used in test/gie/more_builtins.gie,
>>     test/gie/4D-API_cs2cs-style.gie, and test/cli/testdatumfile
>> 
>>   * nzgd2kgrid0005.gsb is used in unit tests

This seems like a strong lead, but I would expect the test suite to run
(and thus modify the timestamps) before dpkg-deb sets the timestamps on
the built packages... so I am still a bit perplexed, but probably just
misunderstanding exactly what happens when and where. :)


>>>> But the diffoscope-results only show a few of the files from the shar
>>>> archives with different mtimes, and all of them get touched when
>>>> unpacking the archive just before the configure target is executed.
>>>
>>> Well, I too was perplexed why other files were not affected, but it is
>>> consistently those .gsb and .gtx files, and the submitted patch allows
>>> to consistently build reproducibly in the dozens of times I've build
>>> it...
>>>
>>>
>>>> shar actually makes the timestamps reproducible by always using the
>>>> mtime recorded in the archive instead of the time the files were 
>>>> unpacked.
>>>>
>>>> Something else is likely changing the mtime after the files are
>>>> unpacked. Some of these grids are used by the testsuite for example.
>>>
>>> I will try to look into it further, but without really being familiar
>>> with the proj codebase (or even what proj itself does)... any additional
>>> very specific suggestions where to look next would definitely be
>>> appreciated!  :)
>> 
>> CMake's configure_file() is used to copy the .gsb & .gtx files from 
>> CMAKE_CURRENT_SOURCE_DIR to CMAKE_CURRENT_BINARY_DIR, that might be 
>> touching the mtimes too. See: data/CMakeLists.txt

Thanks, that is definitely worth taking a look at...


> Seeing how the mtimes are off by two hours, this could be the difference 
> between UTC and CEST.

For clarity, it is almost definitely timezone related, but actually
UTC-14 and UTC+12 (e.g. off by 26 hours), which is used for the TZ
variable on tests.reproducible-builds.org:

  ···83696·2018-02-22·07:28:23.000000·./usr/share/proj/BETA2007.gsb
  vs.
  ···83696·2018-02-21·05:28:23.000000·./usr/share/proj/BETA2007.gsb

Note the difference of date...


> The latter was in effect when the archives were 
> created:
>
>   $ grep "Made on" debian/datumgrids*.shar
>   debian/datumgrids-ch.shar:# Made on 2018-02-26 22:27 CET by <bas@anubis>.
>   debian/datumgrids.shar:# Made on 2018-09-15 20:13 CEST by <bas@anubis>.
>
> But why does it only affect the binary GSB & GTX files, and not also the 
> binary ntv1_can.dat file or text files like README.DATUMGRID and the 
> init files (the ones without extensions)?

That is the question, eh? :)

Will try to poke at it more tomorrow...


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #37 received at 1035704@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: Sebastiaan Couwenberg <sebastic@xs4all.nl>, 1035704@bugs.debian.org

Subject: Re: Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Sun, 28 May 2023 14:38:59 -0700

[Message part 1 (text/plain, inline)]

On 2023-05-08, Vagrant Cascadian wrote:
> On 2023-05-09, Sebastiaan Couwenberg wrote:
>> On 5/9/23 05:47, Sebastiaan Couwenberg wrote:
>>> On 5/8/23 22:43, Vagrant Cascadian wrote:
>>>> On 2023-05-08, Sebastiaan Couwenberg wrote:
>>>>> On 5/8/23 02:07, Vagrant Cascadian wrote:
>>>>>> The attached patch fixes this by not touching these files during the
>>>>>> build process.
>>>>>
>>>>>   From shar(1):
>>>>>
>>>>> "
>>>>>     -m, --no-timestamp
>>>>>            do not restore modification times.
...
>>>>> That should be used when generating the archives instead of your patch
>>>>> to not have the mtimes touched when unpacking the archives.
...
>>>>> But the diffoscope-results only show a few of the files from the shar
>>>>> archives with different mtimes, and all of them get touched when
>>>>> unpacking the archive just before the configure target is executed.
>>>>
>>>> Well, I too was perplexed why other files were not affected, but it is
>>>> consistently those .gsb and .gtx files, and the submitted patch allows
>>>> to consistently build reproducibly in the dozens of times I've build
>>>> it...
>>>>
>>>>
>>>>> shar actually makes the timestamps reproducible by always using the
>>>>> mtime recorded in the archive instead of the time the files were 
>>>>> unpacked.
>>>>>
>>>>> Something else is likely changing the mtime after the files are
>>>>> unpacked. Some of these grids are used by the testsuite for example.
>>>>
>>>> I will try to look into it further, but without really being familiar
>>>> with the proj codebase (or even what proj itself does)... any additional
>>>> very specific suggestions where to look next would definitely be
>>>> appreciated!  :)
>>> 
>>> CMake's configure_file() is used to copy the .gsb & .gtx files from 
>>> CMAKE_CURRENT_SOURCE_DIR to CMAKE_CURRENT_BINARY_DIR, that might be 
>>> touching the mtimes too. See: data/CMakeLists.txt
>
> Thanks, that is definitely worth taking a look at...
...
> Will try to poke at it more tomorrow...

I had no luck with poking at that approach... though did not have great
ideas what to even try there.

That said, I think it really is the touch commands in debian/datumgrids*
as touch's timestamp modification is timezone dependent in many cases...

The attached patch fixes this by setting the TZ=UTC as an environment
variable in the debian/datumgrids*.shar files.

I also had success with a patch where the touch calls are done with
--date=${SOURCE_DATE_EPOCH} which also worked for me (as touch assumes
to be TZ=UTC in this case)... if that would be preferable, I can also
provide a patch for that.


live well,
  vagrant

[0001-debian-datumgrids-.shar-Use-UTC-timezone-when-touchi.patch (text/x-diff, inline)]

From 5e019591780bcf0b61511cc242d1636c9ec909ca Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian <vagrant@reproducible-builds.org>
Date: Sun, 28 May 2023 12:54:59 -0700
Subject: [PATCH] debian/datumgrids*.shar: Use UTC timezone when touching
 files.

Without this, the touch calls result in timezone-specific differences
in various .gsb and .gtx files.

https://reproducible-builds.org/docs/timezones/
---
 debian/datumgrids-ch.shar | 4 ++++
 debian/datumgrids.shar    | 4 ++++
 2 files changed, 8 insertions(+)

diff --git a/debian/datumgrids-ch.shar b/debian/datumgrids-ch.shar
index 308ca93..313a3c6 100644
--- a/debian/datumgrids-ch.shar
+++ b/debian/datumgrids-ch.shar
@@ -35,6 +35,10 @@ gettext_dir=
 locale_dir=
 set_echo=false
 
+# reproducible builds: Use UTC timezone to ensure consistent
+# timestamps on touched files.
+export TZ=UTC
+
 for dir in $PATH
 do
   if test -f $dir/gettext \
diff --git a/debian/datumgrids.shar b/debian/datumgrids.shar
index d9a0161..2dec48e 100644
--- a/debian/datumgrids.shar
+++ b/debian/datumgrids.shar
@@ -48,6 +48,10 @@ gettext_dir=
 locale_dir=
 set_echo=false
 
+# reproducible builds: Use UTC timezone to ensure consistent
+# timestamps on touched files.
+export TZ=UTC
+
 for dir in $PATH
 do
   if test -f $dir/gettext \
-- 
2.39.2

[signature.asc (application/pgp-signature, inline)]

Message #42 received at 1035704@bugs.debian.org (full text, mbox, reply):

From: Sebastiaan Couwenberg <sebastic@xs4all.nl>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>, 1035704@bugs.debian.org

Subject: Re: Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Mon, 29 May 2023 05:43:47 +0200

On 5/28/23 23:38, Vagrant Cascadian wrote:
> On 2023-05-08, Vagrant Cascadian wrote:
>> On 2023-05-09, Sebastiaan Couwenberg wrote:
>>> On 5/9/23 05:47, Sebastiaan Couwenberg wrote:
>>>> On 5/8/23 22:43, Vagrant Cascadian wrote:
>>>>> On 2023-05-08, Sebastiaan Couwenberg wrote:
>>>>>> On 5/8/23 02:07, Vagrant Cascadian wrote:
>>>>>>> The attached patch fixes this by not touching these files during the
>>>>>>> build process.
>>>>>>
>>>>>>    From shar(1):
>>>>>>
>>>>>> "
>>>>>>      -m, --no-timestamp
>>>>>>             do not restore modification times.
> ...
>>>>>> That should be used when generating the archives instead of your patch
>>>>>> to not have the mtimes touched when unpacking the archives.
> ...
>>>>>> But the diffoscope-results only show a few of the files from the shar
>>>>>> archives with different mtimes, and all of them get touched when
>>>>>> unpacking the archive just before the configure target is executed.
>>>>>
>>>>> Well, I too was perplexed why other files were not affected, but it is
>>>>> consistently those .gsb and .gtx files, and the submitted patch allows
>>>>> to consistently build reproducibly in the dozens of times I've build
>>>>> it...
>>>>>
>>>>>
>>>>>> shar actually makes the timestamps reproducible by always using the
>>>>>> mtime recorded in the archive instead of the time the files were
>>>>>> unpacked.
>>>>>>
>>>>>> Something else is likely changing the mtime after the files are
>>>>>> unpacked. Some of these grids are used by the testsuite for example.
>>>>>
>>>>> I will try to look into it further, but without really being familiar
>>>>> with the proj codebase (or even what proj itself does)... any additional
>>>>> very specific suggestions where to look next would definitely be
>>>>> appreciated!  :)
>>>>
>>>> CMake's configure_file() is used to copy the .gsb & .gtx files from
>>>> CMAKE_CURRENT_SOURCE_DIR to CMAKE_CURRENT_BINARY_DIR, that might be
>>>> touching the mtimes too. See: data/CMakeLists.txt
>>
>> Thanks, that is definitely worth taking a look at...
> ...
>> Will try to poke at it more tomorrow...
> 
> I had no luck with poking at that approach... though did not have great
> ideas what to even try there.
> 
> That said, I think it really is the touch commands in debian/datumgrids*
> as touch's timestamp modification is timezone dependent in many cases...
> 
> The attached patch fixes this by setting the TZ=UTC as an environment
> variable in the debian/datumgrids*.shar files.
> 
> I also had success with a patch where the touch calls are done with
> --date=${SOURCE_DATE_EPOCH} which also worked for me (as touch assumes
> to be TZ=UTC in this case)... if that would be preferable, I can also
> provide a patch for that.

Patching the shar files is not ideal, when their content is modified 
these changes will be lost.

shar/unshar should be more likely be patched.

Does TZ=UTC also work when set in the environment? If so, that could be 
passed to the unshar commands in d/rules.

Kind Regards,

Bas

-- 
 GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1

Message #47 received at 1035704@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: Sebastiaan Couwenberg <sebastic@xs4all.nl>, 1035704@bugs.debian.org

Subject: Re: Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Sun, 28 May 2023 21:13:16 -0700

[Message part 1 (text/plain, inline)]

On 2023-05-29, Sebastiaan Couwenberg wrote:
> On 5/28/23 23:38, Vagrant Cascadian wrote:
>> That said, I think it really is the touch commands in debian/datumgrids*
>> as touch's timestamp modification is timezone dependent in many cases...
>> 
>> The attached patch fixes this by setting the TZ=UTC as an environment
>> variable in the debian/datumgrids*.shar files.
>> 
>> I also had success with a patch where the touch calls are done with
>> --date=${SOURCE_DATE_EPOCH} which also worked for me (as touch assumes
>> to be TZ=UTC in this case)... if that would be preferable, I can also
>> provide a patch for that.
>
> Patching the shar files is not ideal, when their content is modified 
> these changes will be lost.
>
> shar/unshar should be more likely be patched.
>
> Does TZ=UTC also work when set in the environment? If so, that could be 
> passed to the unshar commands in d/rules.

I would expect that to work as well, which I though of shortly after
sending the updated patch... though did not yet test it!

live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #52 received at 1035704@bugs.debian.org (full text, mbox, reply):

From: Sebastiaan Couwenberg <sebastic@xs4all.nl>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>, 1035704@bugs.debian.org

Subject: Re: Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Mon, 29 May 2023 06:20:48 +0200

On 5/29/23 06:13, Vagrant Cascadian wrote:
> On 2023-05-29, Sebastiaan Couwenberg wrote:
>> On 5/28/23 23:38, Vagrant Cascadian wrote:
>>> That said, I think it really is the touch commands in debian/datumgrids*
>>> as touch's timestamp modification is timezone dependent in many cases...
>>>
>>> The attached patch fixes this by setting the TZ=UTC as an environment
>>> variable in the debian/datumgrids*.shar files.
>>>
>>> I also had success with a patch where the touch calls are done with
>>> --date=${SOURCE_DATE_EPOCH} which also worked for me (as touch assumes
>>> to be TZ=UTC in this case)... if that would be preferable, I can also
>>> provide a patch for that.
>>
>> Patching the shar files is not ideal, when their content is modified
>> these changes will be lost.
>>
>> shar/unshar should be more likely be patched.
>>
>> Does TZ=UTC also work when set in the environment? If so, that could be
>> passed to the unshar commands in d/rules.
> 
> I would expect that to work as well, which I though of shortly after
> sending the updated patch... though did not yet test it!

Can you test that? Otherwise we'll have to upload to experimental.

Kind Regards,

Bas

-- 
 GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1

Message #57 received at 1035704@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: Sebastiaan Couwenberg <sebastic@xs4all.nl>, 1035704@bugs.debian.org

Subject: Re: Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Mon, 29 May 2023 15:35:41 -0700

[Message part 1 (text/plain, inline)]

On 2023-05-29, Sebastiaan Couwenberg wrote:
> On 5/29/23 06:13, Vagrant Cascadian wrote:
>> On 2023-05-29, Sebastiaan Couwenberg wrote:
>>> On 5/28/23 23:38, Vagrant Cascadian wrote:
>>>> That said, I think it really is the touch commands in debian/datumgrids*
>>>> as touch's timestamp modification is timezone dependent in many cases...
>>>>
>>>> The attached patch fixes this by setting the TZ=UTC as an environment
>>>> variable in the debian/datumgrids*.shar files.
...
>>> Patching the shar files is not ideal, when their content is modified
>>> these changes will be lost.
>>>
>>> shar/unshar should be more likely be patched.

I am not familiar with shar/unshar, but sure, adding support for
SOURCE_DATE_EPOCH would be welcome...

>>> Does TZ=UTC also work when set in the environment? If so, that could be
>>> passed to the unshar commands in d/rules.
>> 
>> I would expect that to work as well, which I though of shortly after
>> sending the updated patch... though did not yet test it!
>
> Can you test that?

Tested! Works! Patch attached!

> Otherwise we'll have to upload to experimental.

As much as I would love to see it fixed in time for bookworm, my guess
is that it is a bit late already...


live well,
  vagrant

[0001-debian-rules-Use-UTC-timezone-when-calling-unshar.-C.patch (text/x-diff, inline)]

From 24865b8c2cda67a39774415e540dfc24ad243458 Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian <vagrant@reproducible-builds.org>
Date: Mon, 29 May 2023 15:28:36 -0700
Subject: [PATCH] debian/rules: Use UTC timezone when calling unshar. (Closes:
 #1035704)

Without this, the touch calls in debian/datumgrids*.shar result in
timezone-specific differences in various .gsb and .gtx files.

https://reproducible-builds.org/docs/timezones/
---
 debian/rules | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/debian/rules b/debian/rules
index a778f4d..3a624ff 100755
--- a/debian/rules
+++ b/debian/rules
@@ -20,8 +20,8 @@ UPSTREAM_VERSION = $(shell echo $(DEB_VERSION_UPSTREAM) | sed -e 's/\+.*//')
 
 datumgrids: datumgrids-stamp
 datumgrids-stamp:
-	unshar -c -d $(CURDIR)/data $(CURDIR)/debian/datumgrids.shar
-	unshar -c -d $(CURDIR)/data $(CURDIR)/debian/datumgrids-ch.shar
+	TZ=UTC unshar -c -d $(CURDIR)/data $(CURDIR)/debian/datumgrids.shar
+	TZ=UTC unshar -c -d $(CURDIR)/data $(CURDIR)/debian/datumgrids-ch.shar
 	touch $@
 
 %:
-- 
2.39.2

[signature.asc (application/pgp-signature, inline)]

Message #62 received at 1035704@bugs.debian.org (full text, mbox, reply):

From: Sebastiaan Couwenberg <sebastic@xs4all.nl>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>, 1035704@bugs.debian.org

Subject: Re: Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Tue, 30 May 2023 05:22:29 +0200

Control: tags -1 - moreinfo + pending

On 5/30/23 00:35, Vagrant Cascadian wrote:
> On 2023-05-29, Sebastiaan Couwenberg wrote:
>> On 5/29/23 06:13, Vagrant Cascadian wrote:
>>> On 2023-05-29, Sebastiaan Couwenberg wrote:
>>>> Does TZ=UTC also work when set in the environment? If so, that could be
>>>> passed to the unshar commands in d/rules.
>>>
>>> I would expect that to work as well, which I though of shortly after
>>> sending the updated patch... though did not yet test it!
>>
>> Can you test that?
> 
> Tested! Works! Patch attached!

Thanks for confirming the theory. Applied in git.

Kind Regards,

Bas

-- 
 GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1

Message #71 received at 1035704-quiet@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: 1035704-quiet@bugs.debian.org

Subject: Re: Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

Date: Tue, 30 May 2023 14:48:37 -0700

[Message part 1 (text/plain, inline)]

Control: found 1035704 9.1.1-1

On 2023-05-07, Vagrant Cascadian wrote:
> Various .gsb and .gtx files get touched in the build process in a way
> that results in a timezone-dependent timestamp for these files in the
> shipped package:
>
>   https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/diffoscope-results/proj.html
>
>   -rw-r--r--···0·root·········(0)·root·········(0)····83696·2018-02-22·07:28:23.000000·./usr/share/proj/BETA2007.gsb
>   vs.
>   -rw-r--r--···0·root·········(0)·root·········(0)····83696·2018-02-21·05:28:23.000000·./usr/share/proj/BETA2007.gsb

Marking as affecting version 9.1.1-1.

live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #78 received at 1035704-close@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 1035704-close@bugs.debian.org

Subject: Bug#1035704: fixed in proj 9.2.1-1~exp1

Date: Thu, 01 Jun 2023 08:49:31 +0000

Source: proj
Source-Version: 9.2.1-1~exp1
Done: Bas Couwenberg <sebastic@debian.org>

We believe that the bug you reported is fixed in the latest version of
proj, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1035704@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Bas Couwenberg <sebastic@debian.org> (supplier of updated proj package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Thu, 01 Jun 2023 09:38:42 +0200
Source: proj
Architecture: source
Version: 9.2.1-1~exp1
Distribution: experimental
Urgency: medium
Maintainer: Debian GIS Project <pkg-grass-devel@lists.alioth.debian.org>
Changed-By: Bas Couwenberg <sebastic@debian.org>
Closes: 1035704 1036939
Changes:
 proj (9.2.1-1~exp1) experimental; urgency=medium
 .
   * New upstream release.
   * Set TZ=UTC to make datumgrid timestamps reproducible.
     (closes: #1035704)
   * Enable CMAKE_BUILD_RPATH_USE_ORIGIN for reproducible builds.
     (closes: #1036939)
   * Update symbols for other architectures.
   * Strip pre-releases from symbols version.
Checksums-Sha1:
 3429f507e426e84ff529a364f07b1c52a56af9f6 2256 proj_9.2.1-1~exp1.dsc
 0ee9398015bc2aaf4ec4606c2edece1c30d02024 5536575 proj_9.2.1.orig.tar.gz
 499f5e2addec21d61a59cb266c802644c9beb7db 10212588 proj_9.2.1-1~exp1.debian.tar.xz
 ec7faee126e31f9ffb21c009e2b90cf579ab9a71 10275 proj_9.2.1-1~exp1_amd64.buildinfo
Checksums-Sha256:
 35d10b0d9464d54a46c4f561108ffb7d49e68c813870c6038d6a8d2966db1516 2256 proj_9.2.1-1~exp1.dsc
 15ebf4afa8744b9e6fccb5d571fc9f338dc3adcf99907d9e62d1af815d4971a1 5536575 proj_9.2.1.orig.tar.gz
 36709814c0954558426c467658530d40207fd6f642e4350c7caa582533a22756 10212588 proj_9.2.1-1~exp1.debian.tar.xz
 fa4693e258c5cd9c1cba79079b201f70179d9a45ef308c4e4372fc4e263b091d 10275 proj_9.2.1-1~exp1_amd64.buildinfo
Files:
 2ddd4a65b96d4a79e329ede28f396cad 2256 science optional proj_9.2.1-1~exp1.dsc
 c8e878049ef27330ac94624e1a75b0db 5536575 science optional proj_9.2.1.orig.tar.gz
 135fd2558db92e7693e21816bc8d632b 10212588 science optional proj_9.2.1-1~exp1.debian.tar.xz
 8f518f08166aacd64801c150c1b37b83 10275 science optional proj_9.2.1-1~exp1_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEgYLeQXBWQI1hRlDRZ1DxCuiNSvEFAmR4VJ4ACgkQZ1DxCuiN
SvE/zBAAwjQqohtoJ0VlmhkA4thwOjksUvGF7mNJafxttJmzCIs2xb7tmM5qK8PK
ZFCGjOIEw9ZJTwaVkEukLmgI/ckakIqxtGlOOfLspe+44yv322a3KVINRljIcQ7E
er70kJaWKN3OL047xN4UBUd0ofnGSkv80iTcVDppkWAib3BJc/CJ3SBUikhbFqH9
WxxTdoZKWW2mWblsDgbnNwAY7gFa/hj6xWO7ebDNmCZdSvlEI3OML3Vq8s7FPHDE
8RCzfrWWvmcldvIEelU0xkSIyJttzVcm/ly63DGS78ClLx/7lpMDYUvvpMO98bSN
Nn4z4okX6ZuCmnOuxQgTeGBLyi43hDQ5wIWjJVKbGEYJq/mFUEdiQNTxianlmmml
XJubWVP5Y+ZmweF26qTn9Z7NXmQpUCyTLbqDlI4ZZSrKO2A3KoZMkdzPCcurs5nN
/tqXuOT0M7GPRn/gyF+AXZYEkLK4481pTYfbWbo1wVDfGeLm+df+9K5l5zC1vW7N
BCYh1AWaWrfaR9bMB/A+eGAFcdrWFlFnIennUGstKdH1Ev5xDV1NXu9KEPj8au3i
59FaCxcNgDSGPftxsiaxGhp4efqwjjyY4jbkWNj2pVLsnl8gypfnu0/PII6eWHAs
L26AVAJSW2N+oqDSmf+fGvNeqZQkDQTaorm6Dcp1oeTkJ/vmwPY=
=T7ay
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.