Debian Bug report logs - #774031
r-base-dev: please do not write username and current time when building packages

[Message part 1 (text/plain, inline)]
Package: r-base-dev
Version: 3.1.2-2
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: toolchain username timestamps

Hi!

While working on the “reproducible builds” effort [1], we have noticed
that R packages could not be built reproducibly.

The username and build time gets written to the `Packaged` field in
`package.rds`. The build time also gets written to the `Built` field of
the `DESCRIPTION` file.

This data does not look very useful in the context of Debian packages.
The attached patch simply stops writing the `Packaged` field entirely
and remove the build time from the `Built` field. R packages can then be
built reproducibly according to some preliminary tests.

If this approach is seen as too broad, using the timestamp in the latest
debian/changelog entry through an environment variable would allow to
keep a timestamp. My current R skills are not up to such task, though.
The username should probably be removed or made constant in any cases.

 [1]: https://wiki.debian.org/ReproducibleBuilds

-- 
Lunar                                .''`. 
lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
                                    `. `'` 
                                      `-   

[r-base_3.1.2-2_reproducible0.diff (text/x-diff, attachment)]
[signature.asc (application/pgp-signature, inline)]
(CCing upstream)

Salut Jérémy,

On 27 December 2014 at 19:26, Jérémy Bobbio wrote:
| Package: r-base-dev
| Version: 3.1.2-2
| Severity: wishlist
| Tags: patch
| User: reproducible-builds@lists.alioth.debian.org
| Usertags: toolchain username timestamps
| 
| Hi!
| 
| While working on the “reproducible builds” effort [1], we have noticed
| that R packages could not be built reproducibly.
| 
| The username and build time gets written to the `Packaged` field in
| `package.rds`. The build time also gets written to the `Built` field of
| the `DESCRIPTION` file.

That's standard R behaviour which I'd rather not deviate from (as I don't
believe in maintaining local patches for "a long time" -- and I have looked
after this for a decade or more).
 
| This data does not look very useful in the context of Debian packages.
| The attached patch simply stops writing the `Packaged` field entirely
| and remove the build time from the `Built` field. R packages can then be
| built reproducibly according to some preliminary tests.
| 
| If this approach is seen as too broad, using the timestamp in the latest
| debian/changelog entry through an environment variable would allow to
| keep a timestamp. My current R skills are not up to such task, though.
| The username should probably be removed or made constant in any cases.

I think the idea may have some merit. 

If someone from R Core has some sympathy for the request, I can probably work
up a similarly small patch which suppresses this output if an option or flag
has been set.

But without cooperation from R Core, I don't think the Debian package should
deviate. 

Cheers,  Dirk

| 
|  [1]: https://wiki.debian.org/ReproducibleBuilds
| 
| -- 
| Lunar                                .''`. 
| lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
|                                     `. `'` 
|                                       `-   
| x[DELETED ATTACHMENT r-base_3.1.2-2_reproducible0.diff, text/x-diff]
| x[DELETED ATTACHMENT signature.asc, application/pgp-signature]

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org



Hi Jeremy,

Thanks again for the bug report. I can't recall if I replied to you
immediately (which I may not have, my bad).  I did however contact upstream
immediately, and following a brief and cordial discussion it was decided I
should send a patch upstream to R.

And I have been trying to make time ever since ...

Two quick comments below:

On 27 December 2014 at 19:26, Jérémy Bobbio wrote:
| Package: r-base-dev
| Version: 3.1.2-2
| Severity: wishlist
| Tags: patch
| User: reproducible-builds@lists.alioth.debian.org
| Usertags: toolchain username timestamps
| 
| Hi!
| 
| While working on the “reproducible builds” effort [1], we have noticed
| that R packages could not be built reproducibly.
| 
| The username and build time gets written to the `Packaged` field in
| `package.rds`. The build time also gets written to the `Built` field of
| the `DESCRIPTION` file.

There are two issues here. The first one is 'Packaged: ...' (done by R CMD
build and done once and only once by the _code_ upstream maintainer at the
time of package upload to CRAN). The second one is 'Built: ..'  which is done
by R CMD INSTALL ... at time of a Debian upload, and then again at rebuilds.

So thinking this through, it seemd to me that do not need the second part of
your patch (which removes the time stamp from Packaged: ...). That is
upstream to all of us, so we don't care.

The first issue is fine, and upstream (via Martin, CC'ed) agreed that a new
command-line flag could be suitable.  As I see it, this would only need for R
CMD INSTALL --- which is called by r-cran.mk, which it iself is called by
debian/rules for any r-cran-* build in the Archive.

And just so that we are on the same page: our r-base package would then also
have to modify the call to R CMD INSTALL in r-cran.mk so that we never write
a timestamp _from the built_ into DESCRIPTION -- so that replication can be
bit-by-bit identical.  

Do I have the thinking correct then?

In that case I'll work up a patch against 

    src/script/INSTALL

to accept such an option, and then following your patch to

    src/library/tools/R/admin.R

in case of such an option, change from the default of

    Built: R 3.1.2; x86_64-pc-linux-gnu; 2015-01-11 03:34:57 UTC; unix

to, say, a record free of time-stamps such as

    Built: R 3.1.2; x86_64-pc-linux-gnu; ; unix

Do I have the scope correctly?   To recap, and from the same (local)
DESCRIPTION file I just I see


Packaged: 2014-09-27 18:09:31.513913 UTC; edd
NeedsCompilation: yes
Repository: CRAN
Date/Publication: 2014-09-29 09:05:29
Built: R 3.1.2; x86_64-pc-linux-gnu; 2015-01-11 03:34:57 UTC; unix

where
    Packaged: ....  
was written upstream (me) create the (upstream source) tarball, and
    NeedsCompilation:, Repository:, Date/Publication:
were added by CRAN when they accepted my source upload, and
    Built: $version, $arch, $time, $buildos
were added at R CMD INSTALL time.  

So we only need to suppress the time.  That is eminently doable.

Comments?
 
| This data does not look very useful in the context of Debian packages.
| The attached patch simply stops writing the `Packaged` field entirely
| and remove the build time from the `Built` field. R packages can then be
| built reproducibly according to some preliminary tests.

As stated above, we can leave 'Packaged:' alone, and should make removing of
the timestampt from 'Built:' optional.
 
| If this approach is seen as too broad, using the timestamp in the latest
| debian/changelog entry through an environment variable would allow to
| keep a timestamp. My current R skills are not up to such task, though.
| The username should probably be removed or made constant in any cases.

Yes, we could be more complicated and pass a timestamp through.  The one from
ChangeLog is a clever idea as it would still signal a 'when' this was
prepared which "we" (Debian builds) have and could reuse.  

That makes it two-steps.  Patch R (upstream) to provide such a new option,
and patch r-base in Debian to have all builds use it.

Dirk


|  [1]: https://wiki.debian.org/ReproducibleBuilds
| 
| -- 
| Lunar                                .''`. 
| lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
|                                     `. `'` 
|                                       `-   
| x[DELETED ATTACHMENT r-base_3.1.2-2_reproducible0.diff, text/x-diff]
| x[DELETED ATTACHMENT signature.asc, application/pgp-signature]

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org



I have the first part done, which was pretty straightforward. Patch below --
it accepts a new option to R CMD INSTALL and pass it on; if set its value is
used.  Tested when set and not set and the right thing happens.

But in doing this also learned that the worker function is called from the
mkdesc target in share/make/basepkg.mk to deal with the base packages when R
itself is built.  So now I may have to resort to configure to pick up a
compile time override passed around by make to R :-/

Any good alternative ideas, Martin?

Dirk

Index: src/library/tools/R/admin.R
===================================================================

--- src/library/tools/R/admin.R	(revision 67410)
+++ src/library/tools/R/admin.R	(working copy)
@@ -21,7 +21,7 @@
 
 ## called from basepkg.mk and .install_packages
 .install_package_description <-
-function(dir, outDir)
+function(dir, outDir, builtStamp=character())
 {
     ## Function for taking the DESCRIPTION package meta-information,
     ## checking/validating it, and installing it with the 'Built:'
@@ -66,8 +66,9 @@
 	       "; ",
 	       if(dir.exists(file.path(dir, "src"))) OStype else "",
 	       "; ",
+               ## Some build systems want to supply a package-build timestamp for reproducibility
 	       ## Prefer date in ISO 8601 format, UTC.
-	       format(Sys.time(), tz = "UTC", usetz = TRUE),
+	       if (length(builtStamp)==0) format(Sys.time(), tz = "UTC", usetz = TRUE) else builtStamp,
 	       ## Sys.time(),
 	       "; ",
 	       .OStype())
Index: src/library/tools/R/install.R
===================================================================
--- src/library/tools/R/install.R	(revision 67410)
+++ src/library/tools/R/install.R	(working copy)
@@ -180,6 +180,8 @@
             "      --configure-vars=VARS",
             "			set variables for the configure scripts (if any)",
             "      --dsym            (OS X only) generate dSYM directory",
+            "      --built-timestamp=STAMP",
+            "                   set timestamp for Built: entry in DESCRIPTION",
             "\nand on Windows only",
             "      --force-biarch	attempt to build both architectures",
             "			even if there is a non-empty configure.win",
@@ -652,7 +654,7 @@
 		    Sys.chmod(file.path(instdir, f), fmode)
                 }
 
-            res <- try(.install_package_description('.', instdir))
+            res <- try(.install_package_description('.', instdir, built_stamp))
             if (inherits(res, "try-error"))
                 pkgerrmsg("installing package DESCRIPTION failed", pkg_name)
             if (!file.exists(namespace <- file.path(instdir, "NAMESPACE")) ) {
@@ -1274,6 +1276,7 @@
     resave_data <- FALSE
     compact_docs <- FALSE
     keep.source <- getOption("keep.source.pkgs")
+    built_stamp <- character()
 
     install_libs <- TRUE
     install_R <- TRUE
@@ -1401,6 +1404,8 @@
             byte_compile <- FALSE
         } else if (a == "--dsym") {
             dsym <- TRUE
+        } else if (substr(a, 1, 18) == "--built-timestamp=") {
+            built_stamp <- substr(a, 19, 1000)
         } else if (substr(a, 1, 1) == "-") {
             message("Warning: unknown option ", sQuote(a), domain = NA)
         } else pkgs <- c(pkgs, a)


-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org



On 11 January 2015 at 14:18, Dirk Eddelbuettel wrote:
| 
| I have the first part done, which was pretty straightforward. Patch below --
| it accepts a new option to R CMD INSTALL and pass it on; if set its value is
| used.  Tested when set and not set and the right thing happens.
| 
| But in doing this also learned that the worker function is called from the
| mkdesc target in share/make/basepkg.mk to deal with the base packages when R
| itself is built.  So now I may have to resort to configure to pick up a
| compile time override passed around by make to R :-/
| 
| Any good alternative ideas, Martin?

Got it, I think -- simply via an env var.  

Will post in a fresh reply to the old thread with R Core.

Dirk
 
| Dirk
| 
| Index: src/library/tools/R/admin.R
| ===================================================================
| --- src/library/tools/R/admin.R	(revision 67410)
| +++ src/library/tools/R/admin.R	(working copy)
| @@ -21,7 +21,7 @@
|  
|  ## called from basepkg.mk and .install_packages
|  .install_package_description <-
| -function(dir, outDir)
| +function(dir, outDir, builtStamp=character())
|  {
|      ## Function for taking the DESCRIPTION package meta-information,
|      ## checking/validating it, and installing it with the 'Built:'
| @@ -66,8 +66,9 @@
|  	       "; ",
|  	       if(dir.exists(file.path(dir, "src"))) OStype else "",
|  	       "; ",
| +               ## Some build systems want to supply a package-build timestamp for reproducibility
|  	       ## Prefer date in ISO 8601 format, UTC.
| -	       format(Sys.time(), tz = "UTC", usetz = TRUE),
| +	       if (length(builtStamp)==0) format(Sys.time(), tz = "UTC", usetz = TRUE) else builtStamp,
|  	       ## Sys.time(),
|  	       "; ",
|  	       .OStype())
| Index: src/library/tools/R/install.R
| ===================================================================
| --- src/library/tools/R/install.R	(revision 67410)
| +++ src/library/tools/R/install.R	(working copy)
| @@ -180,6 +180,8 @@
|              "      --configure-vars=VARS",
|              "			set variables for the configure scripts (if any)",
|              "      --dsym            (OS X only) generate dSYM directory",
| +            "      --built-timestamp=STAMP",
| +            "                   set timestamp for Built: entry in DESCRIPTION",
|              "\nand on Windows only",
|              "      --force-biarch	attempt to build both architectures",
|              "			even if there is a non-empty configure.win",
| @@ -652,7 +654,7 @@
|  		    Sys.chmod(file.path(instdir, f), fmode)
|                  }
|  
| -            res <- try(.install_package_description('.', instdir))
| +            res <- try(.install_package_description('.', instdir, built_stamp))
|              if (inherits(res, "try-error"))
|                  pkgerrmsg("installing package DESCRIPTION failed", pkg_name)
|              if (!file.exists(namespace <- file.path(instdir, "NAMESPACE")) ) {
| @@ -1274,6 +1276,7 @@
|      resave_data <- FALSE
|      compact_docs <- FALSE
|      keep.source <- getOption("keep.source.pkgs")
| +    built_stamp <- character()
|  
|      install_libs <- TRUE
|      install_R <- TRUE
| @@ -1401,6 +1404,8 @@
|              byte_compile <- FALSE
|          } else if (a == "--dsym") {
|              dsym <- TRUE
| +        } else if (substr(a, 1, 18) == "--built-timestamp=") {
| +            built_stamp <- substr(a, 19, 1000)
|          } else if (substr(a, 1, 1) == "-") {
|              message("Warning: unknown option ", sQuote(a), domain = NA)
|          } else pkgs <- c(pkgs, a)
| 
| 
| -- 
| http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org



[Message part 1 (text/plain, inline)]
On 27 December 2014 at 15:29, Dirk Eddelbuettel wrote:
| 
| Hi Martin,
| 
| On 27 December 2014 at 22:12, Martin Maechler wrote:
| | Dear Dirk,
| | 
| | is this about R itself, or rather (I think)  about building "external"
| | (e.g. CRAN)  R packages?
| | Martin
| 
| As I understand (and the full text is still below) Jeremy is concerned about
| R packages (ie r-cran-*) created by R showing "who built it when".
| 
| Which would make simple bit by bit comparison / replication impossible.

I got some time to work on this today.

Earlier discussion about this is at     http://bugs.debian.org/774031

The issue (for Debian) is that bit-by-bit comparison for reproducibility is
made impossible by the timestamps added at build-time by R.

Jeremy, the bug report submitter, confused Packaged: (created when an R
package authors finalizes his tarball before submitting to CRAN) and Built:
(added at INSTALL time), but submitted a first simple patch removing these
timestamps.  I don't like the R in Debian to be different from the general
one, so I hoping you can see merit in the two simple patches below.

They do the following:

- accept a new option to INSTALL in src/library/tools/install.r and passing
  the value of that option to .install_package_description() in the sibbling
  file admin.R

- also accept a new env var to MAKE at R-base compile time to pass that to
  the mkdesc target in share/make/basepkg.mk for the base packages.

Please let me know if you want me to expand on this, or add more tests.  

The two patches are attached, and were made against R-devel's SVN revision
r67410 from yesterday.


[R-set-buildstamp-via-INSTALL.diff (application/octet-stream, attachment)]
[R-set-buildstamp-via-MAKE_env_var.diff (application/octet-stream, attachment)]
[Message part 4 (text/plain, inline)]

Dirk





| 
| Dirk
| 
| | 
| | On Sat, Dec 27, 2014 at 8:30 PM, Dirk Eddelbuettel <edd@debian.org> wrote:
| | >
| | > (CCing upstream)
| | >
| | > Salut Jérémy,
| | >
| | > On 27 December 2014 at 19:26, Jérémy Bobbio wrote:
| | > | Package: r-base-dev
| | > | Version: 3.1.2-2
| | > | Severity: wishlist
| | > | Tags: patch
| | > | User: reproducible-builds@lists.alioth.debian.org
| | > | Usertags: toolchain username timestamps
| | > |
| | > | Hi!
| | > |
| | > | While working on the “reproducible builds” effort [1], we have noticed
| | > | that R packages could not be built reproducibly.
| | > |
| | > | The username and build time gets written to the `Packaged` field in
| | > | `package.rds`. The build time also gets written to the `Built` field of
| | > | the `DESCRIPTION` file.
| | >
| | > That's standard R behaviour which I'd rather not deviate from (as I don't
| | > believe in maintaining local patches for "a long time" -- and I have looked
| | > after this for a decade or more).
| | >
| | > | This data does not look very useful in the context of Debian packages.
| | > | The attached patch simply stops writing the `Packaged` field entirely
| | > | and remove the build time from the `Built` field. R packages can then be
| | > | built reproducibly according to some preliminary tests.
| | > |
| | > | If this approach is seen as too broad, using the timestamp in the latest
| | > | debian/changelog entry through an environment variable would allow to
| | > | keep a timestamp. My current R skills are not up to such task, though.
| | > | The username should probably be removed or made constant in any cases.
| | >
| | > I think the idea may have some merit.
| | >
| | > If someone from R Core has some sympathy for the request, I can probably work
| | > up a similarly small patch which suppresses this output if an option or flag
| | > has been set.
| | >
| | > But without cooperation from R Core, I don't think the Debian package should
| | > deviate.
| | >
| | > Cheers,  Dirk
| | >
| | > |
| | > |  [1]: https://wiki.debian.org/ReproducibleBuilds
| | > |
| | > | --
| | > | Lunar                                .''`.
| | > | lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
| | > |                                     `. `'`
| | > |                                       `-
| | > | x[DELETED ATTACHMENT r-base_3.1.2-2_reproducible0.diff, text/x-diff]
| | > | x[DELETED ATTACHMENT signature.asc, application/pgp-signature]
| | >
| | > --
| | > http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org
| | >
| | > _______________________________________________
| | > R-core list: https://stat.ethz.ch/mailman/listinfo/r-core
| 
| -- 
| http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

[Message part 1 (text/plain, inline)]
On 11 January 2015 at 17:49, Dirk Eddelbuettel wrote:
| 
| On 27 December 2014 at 15:29, Dirk Eddelbuettel wrote:
| | 
| | Hi Martin,
| | 
| | On 27 December 2014 at 22:12, Martin Maechler wrote:
| | | Dear Dirk,
| | | 
| | | is this about R itself, or rather (I think)  about building "external"
| | | (e.g. CRAN)  R packages?
| | | Martin
| | 
| | As I understand (and the full text is still below) Jeremy is concerned about
| | R packages (ie r-cran-*) created by R showing "who built it when".
| | 
| | Which would make simple bit by bit comparison / replication impossible.
| 
| I got some time to work on this today.
| 
| Earlier discussion about this is at     http://bugs.debian.org/774031
| 
| The issue (for Debian) is that bit-by-bit comparison for reproducibility is
| made impossible by the timestamps added at build-time by R.
| 
| Jeremy, the bug report submitter, confused Packaged: (created when an R
| package authors finalizes his tarball before submitting to CRAN) and Built:
| (added at INSTALL time), but submitted a first simple patch removing these
| timestamps.  I don't like the R in Debian to be different from the general
| one, so I hoping you can see merit in the two simple patches below.
| 
| They do the following:
| 
| - accept a new option to INSTALL in src/library/tools/install.r and passing
|   the value of that option to .install_package_description() in the sibbling
|   file admin.R
| 
| - also accept a new env var to MAKE at R-base compile time to pass that to
|   the mkdesc target in share/make/basepkg.mk for the base packages.
| 
| Please let me know if you want me to expand on this, or add more tests.  
| 
| The two patches are attached, and were made against R-devel's SVN revision
| r67410 from yesterday.

Resending.  The earlier post has an unforunate M-x untabify before the second
diff was created.


[R-set-buildstamp-via-INSTALL.diff (application/octet-stream, attachment)]
[R-set-buildstamp-via-MAKE-env-var.diff (application/octet-stream, attachment)]
[Message part 4 (text/plain, inline)]
Dirk


| 
| x[DELETED ATTACHMENT R-set-buildstamp-via-INSTALL.diff, Untyped binary data]
| x[DELETED ATTACHMENT R-set-buildstamp-via-MAKE_env_var.diff, Untyped binary data]
| 
| 
| Dirk
| 
| 
| 
| 
| 
| | 
| | Dirk
| | 
| | | 
| | | On Sat, Dec 27, 2014 at 8:30 PM, Dirk Eddelbuettel <edd@debian.org> wrote:
| | | >
| | | > (CCing upstream)
| | | >
| | | > Salut Jérémy,
| | | >
| | | > On 27 December 2014 at 19:26, Jérémy Bobbio wrote:
| | | > | Package: r-base-dev
| | | > | Version: 3.1.2-2
| | | > | Severity: wishlist
| | | > | Tags: patch
| | | > | User: reproducible-builds@lists.alioth.debian.org
| | | > | Usertags: toolchain username timestamps
| | | > |
| | | > | Hi!
| | | > |
| | | > | While working on the “reproducible builds” effort [1], we have noticed
| | | > | that R packages could not be built reproducibly.
| | | > |
| | | > | The username and build time gets written to the `Packaged` field in
| | | > | `package.rds`. The build time also gets written to the `Built` field of
| | | > | the `DESCRIPTION` file.
| | | >
| | | > That's standard R behaviour which I'd rather not deviate from (as I don't
| | | > believe in maintaining local patches for "a long time" -- and I have looked
| | | > after this for a decade or more).
| | | >
| | | > | This data does not look very useful in the context of Debian packages.
| | | > | The attached patch simply stops writing the `Packaged` field entirely
| | | > | and remove the build time from the `Built` field. R packages can then be
| | | > | built reproducibly according to some preliminary tests.
| | | > |
| | | > | If this approach is seen as too broad, using the timestamp in the latest
| | | > | debian/changelog entry through an environment variable would allow to
| | | > | keep a timestamp. My current R skills are not up to such task, though.
| | | > | The username should probably be removed or made constant in any cases.
| | | >
| | | > I think the idea may have some merit.
| | | >
| | | > If someone from R Core has some sympathy for the request, I can probably work
| | | > up a similarly small patch which suppresses this output if an option or flag
| | | > has been set.
| | | >
| | | > But without cooperation from R Core, I don't think the Debian package should
| | | > deviate.
| | | >
| | | > Cheers,  Dirk
| | | >
| | | > |
| | | > |  [1]: https://wiki.debian.org/ReproducibleBuilds
| | | > |
| | | > | --
| | | > | Lunar                                .''`.
| | | > | lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
| | | > |                                     `. `'`
| | | > |                                       `-
| | | > | x[DELETED ATTACHMENT r-base_3.1.2-2_reproducible0.diff, text/x-diff]
| | | > | x[DELETED ATTACHMENT signature.asc, application/pgp-signature]
| | | >
| | | > --
| | | > http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org
| | | >
| | | > _______________________________________________
| | | > R-core list: https://stat.ethz.ch/mailman/listinfo/r-core
| | 
| | -- 
| | http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org
| 
| -- 
| http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

>>>>> Dirk Eddelbuettel <edd@debian.org>
>>>>>     on Sun, 11 Jan 2015 18:34:10 -0600 writes:

    > On 11 January 2015 at 17:49, Dirk Eddelbuettel wrote:
    > | 
    > | On 27 December 2014 at 15:29, Dirk Eddelbuettel wrote:
    > | | 
    > | | Hi Martin,
    > | | 
    > | | On 27 December 2014 at 22:12, Martin Maechler wrote:
    > | | | Dear Dirk,
    > | | | 
    > | | | is this about R itself, or rather (I think)  about building "external"
    > | | | (e.g. CRAN)  R packages?
    > | | | Martin
    > | | 
    > | | As I understand (and the full text is still below) Jeremy is concerned about
    > | | R packages (ie r-cran-*) created by R showing "who built it when".
    > | | 
    > | | Which would make simple bit by bit comparison / replication impossible.
    > | 
    > | I got some time to work on this today.
    > | 
    > | Earlier discussion about this is at     http://bugs.debian.org/774031
    > | 
    > | The issue (for Debian) is that bit-by-bit comparison for reproducibility is
    > | made impossible by the timestamps added at build-time by R.
    > | 
    > | Jeremy, the bug report submitter, confused Packaged: (created when an R
    > | package authors finalizes his tarball before submitting to CRAN) and Built:
    > | (added at INSTALL time), but submitted a first simple patch removing these
    > | timestamps.  I don't like the R in Debian to be different from the general
    > | one, so I hoping you can see merit in the two simple patches below.
    > | 
    > | They do the following:
    > | 
    > | - accept a new option to INSTALL in src/library/tools/install.r and passing
    > |   the value of that option to .install_package_description() in the sibbling
    > |   file admin.R
    > | 
    > | - also accept a new env var to MAKE at R-base compile time to pass that to
    > |   the mkdesc target in share/make/basepkg.mk for the base packages.
    > | 
    > | Please let me know if you want me to expand on this, or add more tests.  
    > | 
    > | The two patches are attached, and were made against R-devel's SVN revision
    > | r67410 from yesterday.

    > Resending.  The earlier post has an unforunate M-x untabify before the second
    > diff was created.

Thank you, Dirk!

I've committed these now to R-devel, svn rev r67422 and r67423.
(R devel should become R 3.2.0 in April)

Martin



[Message part 1 (text/plain, inline)]
Martin Maechler:
> I've committed these now to R-devel, svn rev r67422 and r67423.
> (R devel should become R 3.2.0 in April)

That's great news. :) Dirk, thanks for your work. Tell me if you want to
deploy your patches to our experimental framework. I could either do the
work or give you access to the repositories.

-- 
Lunar                                .''`. 
lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
                                    `. `'` 
                                      `-   

[signature.asc (application/pgp-signature, inline)]
On 12 January 2015 at 11:06, Jérémy Bobbio wrote:
| Martin Maechler:
| > I've committed these now to R-devel, svn rev r67422 and r67423.
| > (R devel should become R 3.2.0 in April)

Thanks so much Martin!
 
| That's great news. :) Dirk, thanks for your work. Tell me if you want to
| deploy your patches to our experimental framework. I could either do the
| work or give you access to the repositories.

Let's talk offline (or maybe extend this bug report thread?).

I have never put r-devel into Debian (as there is work in making them
coexist). I could just do a one-off for experimental -- or can you deploy
Docker by chance?  We have daily builds of r-devel in the Rocker repository
(Rocker-org on GitHub; it is 'rocker' on Docker's hub.docker.com site).

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org



[Message part 1 (text/plain, inline)]
Dirk Eddelbuettel:
> On 12 January 2015 at 11:06, Jérémy Bobbio wrote:
> | That's great news. :) Dirk, thanks for your work. Tell me if you want to
> | deploy your patches to our experimental framework. I could either do the
> | work or give you access to the repositories.
> 
> I have never put r-devel into Debian (as there is work in making them
> coexist). I could just do a one-off for experimental -- or can you deploy
> Docker by chance?  We have daily builds of r-devel in the Rocker repository
> (Rocker-org on GitHub; it is 'rocker' on Docker's hub.docker.com site).

We have a continuous integration platform [1] which test reproducibility
using our custom toolchain [2]. The custom toolchain currently contains
a modified r-base with my original patch. What I'm offering is to
replace this package with one with your patches. We could then test all R
packages again to make sure things are well.

 [1]: https://reproducible.debian.net/
 [2]: https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

-- 
Lunar                                .''`. 
lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
                                    `. `'` 
                                      `-   

[signature.asc (application/pgp-signature, inline)]
On 12 January 2015 at 14:50, Jérémy Bobbio wrote:
| Dirk Eddelbuettel:
| > On 12 January 2015 at 11:06, Jérémy Bobbio wrote:
| > | That's great news. :) Dirk, thanks for your work. Tell me if you want to
| > | deploy your patches to our experimental framework. I could either do the
| > | work or give you access to the repositories.
| > 
| > I have never put r-devel into Debian (as there is work in making them
| > coexist). I could just do a one-off for experimental -- or can you deploy
| > Docker by chance?  We have daily builds of r-devel in the Rocker repository
| > (Rocker-org on GitHub; it is 'rocker' on Docker's hub.docker.com site).
| 
| We have a continuous integration platform [1] which test reproducibility
| using our custom toolchain [2]. The custom toolchain currently contains

Nice work.

| a modified r-base with my original patch. What I'm offering is to
| replace this package with one with your patches. We could then test all R
| packages again to make sure things are well.

You need both my patches, and while I only tried them against r-devel they
should work as is on r-release.  

There is one thing I need to stick into debian/rules to set the env var for
make (patch 2), and we need to modify the r-cran.mk snippet to get timestamp
from debian/changelog and pass it on. 

I can prepare both of these this evening. Ok?

Dirk

| 
|  [1]: https://reproducible.debian.net/
|  [2]: https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain
| 
| -- 
| Lunar                                .''`. 
| lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
|                                     `. `'` 
|                                       `-   
| [DELETED ATTACHMENT signature.asc, application/pgp-signature]

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org



[Message part 1 (text/plain, inline)]
Dirk Eddelbuettel:
> You need both my patches, and while I only tried them against r-devel they
> should work as is on r-release.  
> 
> There is one thing I need to stick into debian/rules to set the env var for
> make (patch 2), and we need to modify the r-cran.mk snippet to get timestamp
> from debian/changelog and pass it on. 
> 
> I can prepare both of these this evening. Ok?

Sure, no hurry! :)

-- 
Lunar                                .''`. 
lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
                                    `. `'` 
                                      `-   

[signature.asc (application/pgp-signature, inline)]
On 12 January 2015 at 15:23, Jérémy Bobbio wrote:
| Dirk Eddelbuettel:
| > You need both my patches, and while I only tried them against r-devel they
| > should work as is on r-release.  
| > 
| > There is one thing I need to stick into debian/rules to set the env var for
| > make (patch 2), and we need to modify the r-cran.mk snippet to get timestamp
| > from debian/changelog and pass it on. 
| > 
| > I can prepare both of these this evening. Ok?
| 
| Sure, no hurry! :)

r-base_3.1.2-3 was just uploaded to experimental.  It has the two patches of mine
which are already in r-devel thanks to Martin.  It also has (commented-out)
hooks to to use them

First, in debian/rules:

  ## Support for #774031 uncomment the two assignment to pass a non-empty string for this to hold
  #builttime       := $(shell dpkg-parsechangelog -l$(CURDIR)/debian/changelog | awk -F': ' '/Date/ {print $$2}')
  #builttimestamp  := "$(builttime)"
  buildtimestamp  := ""

Comment out the last one and active the two before, and you should get
buildtimestamp which is use below as

			PKG_BUILT_STAMP="$(buildtimestamp)"	\

in the call to $(MAKE).  That is part one.

Second, in debian/r-cran.mk, very similarly

  ## set built-time in DESCRIPTION time of created binary package based on stamp in changelog
  ## cf discussion in http://bugs.debian.org/774031 --- and uncomment two assignments here
  ##
  ## extract built-timestamp from entry changelog and use as argument 
  #builttime       := $(shell dpkg-parsechangelog -l$(CURDIR)/debian/changelog | awk -F': ' '/Date/ {print $$2}')
  ##
  #builttimeStamp  := "--built-timestamp=\"$(builttime)\""
  ##
  ## else
  builttimeStamp  := ""

which is used via 

					$(builttimeStamp)   	        \

So you would have to make a minimal change, build and see how it is
deployed.  

I can quite make up my mind if I want these 'always on'.  But we should in
any event see where you get with this for reproducibility.

Let me know how it goes.  

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org



Source: r-base
Source-Version: 3.1.2-3

We believe that the bug you reported is fixed in the latest version of
r-base, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 774031@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Dirk Eddelbuettel <edd@debian.org> (supplier of updated r-base package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Mon, 12 Jan 2015 21:03:42 -0600
Source: r-base
Binary: r-base r-base-core r-base-dev r-mathlib r-base-html r-doc-pdf r-doc-html r-doc-info r-recommended r-base-core-dbg
Architecture: source i386 all
Version: 3.1.2-3
Distribution: experimental
Urgency: low
Maintainer: Dirk Eddelbuettel <edd@debian.org>
Changed-By: Dirk Eddelbuettel <edd@debian.org>
Description:
 r-base     - GNU R statistical computation and graphics system
 r-base-core - GNU R core of statistical computation and graphics system
 r-base-core-dbg - GNU R debug symbols for statistical comp. language and environmen
 r-base-dev - GNU R installation of auxiliary GNU R packages
 r-base-html - GNU R html docs for statistical computing system functions
 r-doc-html - GNU R html manuals for statistical computing system
 r-doc-info - GNU R info manuals statistical computing system
 r-doc-pdf  - GNU R pdf manuals for statistical computing system
 r-mathlib  - GNU R standalone mathematics library
 r-recommended - GNU R collection of recommended packages [metapackage]
Closes: 774031
Changes:
 r-base (3.1.2-3) experimental; urgency=low
 .
   * debian/reproducibility-patch/*.patch: Added two patches submitted to R
     upstream and already integrated into upcoming R 3.2.0 releases; these
     help setting a timestamp for the DESCRIPTION file of a) base R package
     via a new env var PKG_BUILT_TIME passed down from debian/rules, and
     also b) r-cran-* package built via r-cran.mk through a new option to
     R CMD INSTALL                                        (Closes: #774031)
 .
   * debian/rules: Added support for PKG_BUILT_TIME, but left commented-out
 .
   * debian/r-cran.mk: Added support for --built-timestampe, also inactive
Checksums-Sha1:
 58319ce8955c1c03897fd35de0d558e7d349ac56 2785 r-base_3.1.2-3.dsc
 93809368e5735a630611633ac1fa99010020c5d6 28655941 r-base_3.1.2.orig.tar.gz
 2ec1f63055cfa15c6a5fe9b78ac836f6fb94a843 86594 r-base_3.1.2-3.diff.gz
 8f390573e939d2713db679dfb8f33cfb65188225 19923714 r-base-core_3.1.2-3_i386.deb
 5da61ea857c3fae9854625d8c7c4a141320599cc 1245696 r-mathlib_3.1.2-3_i386.deb
 fa2e8426ab506aa0d6764d3a701e4db79e56fb02 2145134 r-base-core-dbg_3.1.2-3_i386.deb
 74db3ca8bf219cdb0ab4900eebe46f5b11f7611c 37866 r-base_3.1.2-3_all.deb
 46c698b7e2217cb4c9e61d5089fe21d7d7cf8a68 4240 r-base-dev_3.1.2-3_all.deb
 260aeedb74217ce3a91f8582e11af409bbcbc9a8 81732 r-base-html_3.1.2-3_all.deb
 d8e4cf705a083b601aeb9e3d51ab6485f8e26d90 14260554 r-doc-pdf_3.1.2-3_all.deb
 2ad4fa5d7396b67ef5b1e295c3c6cbd75b1ffcc2 532852 r-doc-html_3.1.2-3_all.deb
 0b23609bf4102b050efdffa07a5efc79372fc778 589784 r-doc-info_3.1.2-3_all.deb
 a35acc79382d25baaad765b5f496205046280cb3 2848 r-recommended_3.1.2-3_all.deb
Checksums-Sha256:
 bbd5f3dfd97f3032290ac68c8d5ed4307f81b65f1b22c5c73fb8854249fe66a4 2785 r-base_3.1.2-3.dsc
 bcd150afcae0e02f6efb5f35a6ab72432be82e849ec52ce0bb89d8c342a8fa7a 28655941 r-base_3.1.2.orig.tar.gz
 29c5b9a3ba3889e7426f2c850dd49ccf3d1e4f4acd3c86787cb27ff80c0c2e2e 86594 r-base_3.1.2-3.diff.gz
 4a3b2637df79221f1149786170775767451250cf15346ef9734330cc6dd52c48 19923714 r-base-core_3.1.2-3_i386.deb
 17847f8be5ff5936ad4ba6ac24c02a778b2b7c78099b4dd21feb2ab7aaf90fb7 1245696 r-mathlib_3.1.2-3_i386.deb
 280cc8f88b9be125652a2c77f03020990046f134adef44e07287ec7e465e6180 2145134 r-base-core-dbg_3.1.2-3_i386.deb
 6aee5deca158faf31a2ec786e3c5331a70780fb063416eeaa7757d09a61587f4 37866 r-base_3.1.2-3_all.deb
 968e28daa163dc30b4c183fa6afec2d1bad621a6015b57215f0de3498a92e03d 4240 r-base-dev_3.1.2-3_all.deb
 e4cbfd2a563853aa52a15947e9f465fb14f0bd99a796edfebc4f9789a9a423b9 81732 r-base-html_3.1.2-3_all.deb
 bcd1b7ce7082b29786233b5ac66f0c424ae3cb73a7ea6675533efaea76f41113 14260554 r-doc-pdf_3.1.2-3_all.deb
 6c04409c3bc93a74e092ba7d256f06fd9e0d253390430c0fa4435bab05d61fc9 532852 r-doc-html_3.1.2-3_all.deb
 6106ac1aff905dd39e67f56cd028938c749eabd2c2b77c282600bc35be6bbdfa 589784 r-doc-info_3.1.2-3_all.deb
 4b681cc0dce07c529bdafc7123f58b405cb1f9047f9f28ca67403c0a9be8e1fc 2848 r-recommended_3.1.2-3_all.deb
Files:
 796d9e87c6b7bda81923c084e1e66bce 2785 gnu-r optional r-base_3.1.2-3.dsc
 3af29ec06704cbd08d4ba8d69250ae74 28655941 gnu-r optional r-base_3.1.2.orig.tar.gz
 3ca8218e827b8ae17ab28b29cd5b73a0 86594 gnu-r optional r-base_3.1.2-3.diff.gz
 07f899080989022b64abe3ff55be41b1 19923714 gnu-r optional r-base-core_3.1.2-3_i386.deb
 675c99822fdf945f144c4be5acc8094e 1245696 gnu-r optional r-mathlib_3.1.2-3_i386.deb
 dda60c5a743af628d55490abfa421384 2145134 debug extra r-base-core-dbg_3.1.2-3_i386.deb
 ffdf00bb9e6ed908de859665555d3f25 37866 gnu-r optional r-base_3.1.2-3_all.deb
 37ca327bc983b0b447edc9d7f17b0740 4240 gnu-r optional r-base-dev_3.1.2-3_all.deb
 fcaf5b58759c8abdc606bfdc35ebe02e 81732 doc extra r-base-html_3.1.2-3_all.deb
 0f3557aa0839398fa14535805c99980f 14260554 doc optional r-doc-pdf_3.1.2-3_all.deb
 7b002b6c7568019884e1d0021f4ade0d 532852 doc optional r-doc-html_3.1.2-3_all.deb
 7c2a09e6a9e4c0abc7ed4a2cba5bee14 589784 doc optional r-doc-info_3.1.2-3_all.deb
 318cb0aa966f1b2679947dbd89f9812d 2848 gnu-r optional r-recommended_3.1.2-3_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIVAwUBVLSScKFIn+KrmaIaAQLN5w/8DpVH0hgPhvAXD3vdAOKrLEZFj8btrJbA
jsA1McSYMxTjH3QHPIsI0llbbXWGHg8p7cwA5Pa/8tTZRF+0xSLPGZRLZeTclGuq
+bI8TRxN/LtXW0wlGZvq1LvLXSUn238tvqHiJ0L2OOZNcnGnFbY5SZ7ToVZ89igk
IUeR8KHMy4yWwieHI5nUkOUr8VyQWYHegbgYDXQQud2Oiltwd/PESxcD8Ma1LFz3
xqMISLI0uiEEBr+LuuGkpd+8puaYxTDEsAVw+vyAjHAz0joEY9OGzhLmt2s9WFDh
YYSLR1FpegxSosUbcWYiERUt/0lNvrj1NIsCT+Wpm6SS+aVPg9FzTMuguRmWsRN7
GF4t+73Nqdy++oXNKmd/VtEsmfekTaSDm/IZDJOuRfY4cIK0YsSOWJsBJf3YHsK5
KFibMVQaIfJdFOo9eEz/r6HWrP6UDaRBlX4uLQ5rLsQ6R7s82wgwuKNIApDXmcoA
ipKIECMeuSwKDX3qdcgnfsMAzUO2YJQs1ZJW9JdKyY/HB7ipki29YEy/PySQxq2W
8UrPooSWftSqJqx0KRknlsB6bVd05WCENeHHXO2V3qO7EIYJieafvnUS8kUpHVp/
at7OuxOaoCT2iHWuRwmfZSqCLjeuzKsZhDLNzoTN2RBYxaPMYbUBO5BpzCFr6eUR
lge5f0b3h/U=
=HueK
-----END PGP SIGNATURE-----




[Message part 1 (text/plain, inline)]
Dirk Eddelbuettel:
> So you would have to make a minimal change, build and see how it is
> deployed.

Done in our experimental repository.

> Let me know how it goes.  

Sure!

-- 
Lunar                                .''`. 
lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
                                    `. `'` 
                                      `-   

[signature.asc (application/pgp-signature, inline)]

Debian Bug report logs - #774031 r-base-dev: please do not write username and current time when building packages

Debian Bug report logs - #774031
r-base-dev: please do not write username and current time when building packages