Debian Bug report logs - #941309
node-browserify-lite: unreproducible dependency map order

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: "Rebecca N. Palmer" <rebecca_palmer@zoho.com>

To: submit@bugs.debian.org

Subject: node-browserify-lite: unreproducible dependency map order

Date: Sat, 28 Sep 2019 14:08:03 +0100

Package: node-browserify-lite
Version: 0.5.0-8
Control: tags -1 upstream patch
Control: affects -1 theano jssip libjs-webrtc-adapter
User: reproducible-builds@lists.alioth.debian.org
Usertags: toolchain fileordering
X-Debbugs-Cc: reproducible-builds@alioth-lists.debian.net

My #918631 patch didn't completely fix 
randomness_in_browserify_lite_output / 
nondeterminstic_output_from_uglifyjs (sorry): while it did fix the order 
in which the input files are numbered and output, it doesn't fix the 
order of the dependency map produced for each input file.

The below patch (which *replaces* the existing reproducibility patch)
attempts to fix this.

I think this is a file system ordering issue (rather than internal
(pseudo)randomness) as it previously didn't show up in testing multiple
builds from the same source tree, but this patch deliberately doesn't
assume this.

Testing done:
- theano (dagre-d3/graphviz-dot) - builds, appears functional in 
Firefox, code looks like it has sorted dependency maps
- node-debug, node-timers-browserify - builds, passes autopkgtests 
(these are already reproducible as they have <=1 entry in each map)

A proper reproducibility test was *not* done, as reprotest/disorderfs 
fails on my system (I apparently don't have a fuse group even though the 
fuse package is installed; I don't know why).

Description: Ease reproducible build

Sort module list and dependency lists in order to be reproducible

Author: Rebecca N. Palmer <rebecca_palmer@zoho.com>
Forwarded: https://github.com/andrewrk/browserify-lite/issues/12

Index: node-browserify-lite/index.js
===================================================================
--- node-browserify-lite.orig/index.js
+++ node-browserify-lite/index.js
@@ -79,6 +79,8 @@ function renderBundle(options, cb) {
 
   function render(entrySourcePath, cb) {
     var modules = Object.keys(sources);
+    // for reproducibility
+    modules.sort();
     var aliases = {};
     modules.forEach(function(canonicalSourcePath, index) {
       aliases[canonicalSourcePath] = index;
@@ -116,7 +118,17 @@ function renderBundle(options, cb) {
         out += "module.exports = ";
       }
       out += sources[canonicalSourcePath];
-      out += "\n}, " + JSON.stringify(depMap[canonicalSourcePath]) + "],";
+      out += "\n}, {";
+      // for reproducibility, as JSON.stringify(depMap[canonicalSourcePath]) does not guarantee order - https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify
+      var depMapKeys = Object.keys(depMap[canonicalSourcePath]);
+      depMapKeys.sort();
+      depMapKeys.forEach(function(depPath, index) {
+          if (index != 0) {
+              out += ",\n"; // separate to not have a trailing comma
+          }
+          out += "\"" + depPath + "\": " + depMap[canonicalSourcePath][depPath];
+      });
+      out += "\n}],";
     });
 
     out += "}, {}, " + aliases[entrySourcePath] + ");\n";

Message #14 received at 941309@bugs.debian.org (full text, mbox, reply):

From: "Rebecca N. Palmer" <rebecca_palmer@zoho.com>

To: 941309@bugs.debian.org

Subject: Re: node-browserify-lite: unreproducible dependency map order

Date: Sun, 29 Sep 2019 11:51:14 +0100

Control: forwarded -1 https://github.com/andrewrk/browserify-lite/pull/13

Upstream pointed out that this doesn't escape special characters in file 
names; corrected version follows.

Subject: Ease reproducible build
author: Rebecca N. Palmer" <rebecca_palmer@zoho.com>

Sort module list and dependency lists in order to be reproducible

Forwarded: https://github.com/andrewrk/browserify-lite/issues/12

Index: node-browserify-lite/index.js
===================================================================
--- node-browserify-lite.orig/index.js
+++ node-browserify-lite/index.js
@@ -79,6 +79,8 @@ function renderBundle(options, cb) {
 
   function render(entrySourcePath, cb) {
     var modules = Object.keys(sources);
+    // for reproducibility
+    modules.sort();
     var aliases = {};
     modules.forEach(function(canonicalSourcePath, index) {
       aliases[canonicalSourcePath] = index;
@@ -116,7 +118,17 @@ function renderBundle(options, cb) {
         out += "module.exports = ";
       }
       out += sources[canonicalSourcePath];
-      out += "\n}, " + JSON.stringify(depMap[canonicalSourcePath]) + "],";
+      out += "\n}, {";
+      // for reproducibility, as JSON.stringify(depMap[canonicalSourcePath]) does not guarantee order - https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify
+      var depMapKeys = Object.keys(depMap[canonicalSourcePath]);
+      depMapKeys.sort();
+      depMapKeys.forEach(function(depPath, index) {
+          if (index != 0) {
+              out += ","; // separate to not have a trailing comma
+          }
+          out += JSON.stringify(depPath) + ":" + depMap[canonicalSourcePath][depPath];
+      });
+      out += "}],";
     });
 
     out += "}, {}, " + aliases[entrySourcePath] + ");\n";

Message #23 received at 941309-done@bugs.debian.org (full text, mbox, reply):

From: "Rebecca N. Palmer" <rebecca_palmer@zoho.com>

To: 941309-done@bugs.debian.org

Subject: appears fixed

Date: Sat, 4 Apr 2020 15:01:17 +0100

Version: 0.5.1-1

theano is now reproducible.  (#954409 was the same bug as this.)

Send a report that this bug log contains spam.