Debian Bug report logs - #950444
intltool: race condition with cache files

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: intltool: race condition when generating FIXME

Date: Sat, 01 Feb 2020 11:44:28 -0800

[Message part 1 (text/plain, inline)]

Package: intltool
Version: 0.51.0-5
Severity: normal
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: toolchain randomness
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org
Control: affects -1 exo
Control: affects -1 garcon

A fix for a race condition was submitted upstream. The effects result in
partial or completely missing translations. The Reproducible Builds
folks have identified over twenty packages which may be affected:

  https://tests.reproducible-builds.org/debian/issues/unstable/translations_missing_in_desktop_files_issue.html


The fix was submitted and merged upstream some years ago, but a new
release was never made:

  https://bugs.launchpad.net/intltool/+bug/1687644
  https://bazaar.launchpad.net/~intltool/intltool/trunk/revision/748


I've done several tests building "exo" and "garcon" packages on multiple
machines with varying environments, and the attached patch appears to
work.


Thanks for maintaining intltool!


live well,
  vagrant

[0001-debian-patches-Add-patch-from-upstream-to-fix-race-i.patch (text/x-diff, inline)]

From ea5b70f1aae8115595226a9f2e23d3bb2c98bdd4 Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian <vagrant@reproducible-builds.org>
Date: Sat, 1 Feb 2020 10:13:30 -0800
Subject: [PATCH] debian/patches: Add patch from upstream to fix race intltool
 race condition end up using a partial cache, resulting in missing
 translations in some cases.

---
 ...re-some-processes-try-to-use-a-parti.patch | 57 +++++++++++++++++++
 debian/patches/series                         |  1 +
 2 files changed, 58 insertions(+)
 create mode 100644 debian/patches/0001-Avoid-a-race-where-some-processes-try-to-use-a-parti.patch

diff --git a/debian/patches/0001-Avoid-a-race-where-some-processes-try-to-use-a-parti.patch b/debian/patches/0001-Avoid-a-race-where-some-processes-try-to-use-a-parti.patch
new file mode 100644
index 0000000..d5998c9
--- /dev/null
+++ b/debian/patches/0001-Avoid-a-race-where-some-processes-try-to-use-a-parti.patch
@@ -0,0 +1,57 @@
+From 97c9854c9ffe34d5fb4c8e928530c9a41b8e1a35 Mon Sep 17 00:00:00 2001
+From: "Bernhard M. Wiedemann" <bmwiedemann+bzrcommit@suse.de>
+Date: Thu, 18 May 2017 21:09:18 +0200
+Origin: https://bazaar.launchpad.net/~intltool/intltool/trunk/revision/748
+ https://bugs.launchpad.net/intltool/+bug/1687644
+Subject: [PATCH] Avoid a race where some processes try to use a partial cache
+ file that is still being written to. Note that we release the lock before
+ load_cache, because if we got the lock, the cache is already completely
+ written and it is OK to have multiple parallel readers
+
+Without this patch, translation files would randomly miss translations
+for some or all languages.
+
+fixes bug #1687644
+maybe also bug #986897
+---
+ intltool-merge.in | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/intltool-merge.in b/intltool-merge.in
+index 05db7cf..89923a7 100644
+--- a/intltool-merge.in
++++ b/intltool-merge.in
+@@ -43,6 +43,7 @@ use Getopt::Long;
+ use Text::Wrap;
+ use File::Basename;
+ use Encode;
++use Fcntl qw(:flock);
+ 
+ my $must_end_tag      = -1;
+ my $last_depth        = -1;
+@@ -392,11 +393,14 @@ sub load_cache
+ 
+ sub get_cached_translation_database
+ {
++    open(my $lockfh, ">", "$cache_file.lock") or die $!;
++    flock($lockfh, LOCK_EX) or die "Could not lock '$cache_file.lock' - $!";
+     my $cache_file_age = -M $cache_file;
+     if (defined $cache_file_age) 
+     {
+         if ($cache_file_age <= &get_newest_po_age) 
+         {
++            close($lockfh);
+             &load_cache;
+             return;
+         }
+@@ -404,6 +408,7 @@ sub get_cached_translation_database
+     }
+ 
+     &create_cache;
++    close($lockfh);
+ }
+ 
+ sub add_translation
+-- 
+2.20.1
+
diff --git a/debian/patches/series b/debian/patches/series
index 4ca6c6d..7ccf586 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -1 +1,2 @@
 perl5.26-regex-fixes.patch
+0001-Avoid-a-race-where-some-processes-try-to-use-a-parti.patch
-- 
2.20.1

[signature.asc (application/pgp-signature, inline)]

Message #16 received at 950444@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: 950444@bugs.debian.org

Subject: #950444: intltool: race condition with cache files

Date: Sat, 01 Feb 2020 11:53:01 -0800

[Message part 1 (text/plain, inline)]

Control: retitle 950444 intltool: race condition with cache files

Oops, fix the title to something a little less FIXME.


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #23 received at 950444@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: 950444@bugs.debian.org

Subject: #950444: intltool: race condition with cache files

Date: Mon, 28 Sep 2020 10:53:15 -0700

[Message part 1 (text/plain, inline)]

On 2020-02-01, Vagrant Cascadian wrote:
> A fix for a race condition was submitted upstream. The effects result in
> partial or completely missing translations. The Reproducible Builds
> folks have identified over twenty packages which may be affected:
>
>   https://tests.reproducible-builds.org/debian/issues/unstable/translations_missing_in_desktop_files_issue.html

Updated url for affected packages:

  https://tests.reproducible-builds.org/debian/issues/unstable/translations_missing_due_to_intltool_merge_issue.html


> The fix was submitted and merged upstream some years ago, but a new
> release was never made:
>
>   https://bugs.launchpad.net/intltool/+bug/1687644
>   https://bazaar.launchpad.net/~intltool/intltool/trunk/revision/748

I've submitted a merge request to the gnome-team for the original patch,
with some minor typos fixed in the commit message:

  https://salsa.debian.org/gnome-team/intltool/-/merge_requests/2


While there has been no upstream release of intltool in quite some
years, this patch has been in upstream revision control since 2017 (as
of now, it is the *only* patch that isn't already in the version in
Debian).

Please consider merging this patch and uploading, as it should fix
reproducibility issues for several other packages. Or would you consider
it acceptible to NMU?


Thanks!


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #28 received at 950444@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: 950444@bugs.debian.org

Cc: reproducible-builds@lists.alioth.debian.org

Subject: Re: #950444: intltool: race condition with cache files

Date: Sun, 15 Nov 2020 17:14:24 -0800

[Message part 1 (text/plain, inline)]

On 2020-09-28, Vagrant Cascadian wrote:
> On 2020-02-01, Vagrant Cascadian wrote:
>> A fix for a race condition was submitted upstream. The effects result in
>> partial or completely missing translations. The Reproducible Builds
>> folks have identified over twenty packages which may be affected:
...
>   https://tests.reproducible-builds.org/debian/issues/unstable/translations_missing_due_to_intltool_merge_issue.html
>
>
>> The fix was submitted and merged upstream some years ago, but a new
>> release was never made:
>>
>>   https://bugs.launchpad.net/intltool/+bug/1687644
>>   https://bazaar.launchpad.net/~intltool/intltool/trunk/revision/748
>
> I've submitted a merge request to the gnome-team for the original patch,
> with some minor typos fixed in the commit message:
>
>   https://salsa.debian.org/gnome-team/intltool/-/merge_requests/2
>
>
> While there has been no upstream release of intltool in quite some
> years, this patch has been in upstream revision control since 2017 (as
> of now, it is the *only* patch that isn't already in the version in
> Debian).
>
> Please consider merging this patch and uploading, as it should fix
> reproducibility issues for several other packages. Or would you consider
> it acceptible to NMU?

I would like to proceed with an NMU in a few days, as this is a
non-deterministic race condition that affects reproducibility in several
other packages in Debian, and would be very nice to have resolved in
time for bullseye. Please let me know if I should hold off.


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #33 received at 950444@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: 950444@bugs.debian.org

Cc: reproducible-builds@lists.alioth.debian.org

Subject: Re: #950444: intltool: race condition with cache files

Date: Fri, 20 Nov 2020 13:08:58 -0800

[Message part 1 (text/plain, inline)]

On 2020-11-15, Vagrant Cascadian wrote:
> On 2020-09-28, Vagrant Cascadian wrote:
>> On 2020-02-01, Vagrant Cascadian wrote:
>>> A fix for a race condition was submitted upstream. The effects result in
>>> partial or completely missing translations. The Reproducible Builds
>>> folks have identified over twenty packages which may be affected:
> ...
>>   https://tests.reproducible-builds.org/debian/issues/unstable/translations_missing_due_to_intltool_merge_issue.html
>>
>>
>>> The fix was submitted and merged upstream some years ago, but a new
>>> release was never made:
>>>
>>>   https://bugs.launchpad.net/intltool/+bug/1687644
>>>   https://bazaar.launchpad.net/~intltool/intltool/trunk/revision/748
>>
>> I've submitted a merge request to the gnome-team for the original patch,
>> with some minor typos fixed in the commit message:
>>
>>   https://salsa.debian.org/gnome-team/intltool/-/merge_requests/2
>>
>>
>> While there has been no upstream release of intltool in quite some
>> years, this patch has been in upstream revision control since 2017 (as
>> of now, it is the *only* patch that isn't already in the version in
>> Debian).
>>
>> Please consider merging this patch and uploading, as it should fix
>> reproducibility issues for several other packages. Or would you consider
>> it acceptible to NMU?
>
> I would like to proceed with an NMU in a few days, as this is a
> non-deterministic race condition that affects reproducibility in several
> other packages in Debian, and would be very nice to have resolved in
> time for bullseye. Please let me know if I should hold off.

Uploaded to DELAYED 10.

Merge request:

  https://salsa.debian.org/gnome-team/intltool/-/merge_requests/3

Debdiff:

diff -Nru intltool-0.51.0/debian/changelog intltool-0.51.0/debian/changelog
--- intltool-0.51.0/debian/changelog	2017-12-21 19:23:13.000000000 -0800
+++ intltool-0.51.0/debian/changelog	2020-11-20 12:53:23.000000000 -0800
@@ -1,3 +1,17 @@
+intltool (0.51.0-5.1) unstable; urgency=medium
+
+  * Non-maintainer upload.
+
+  [ Jeremy Bicha ]
+  * Update Vcs fields for migration to https://salsa.debian.org/
+
+  [ Vagrant Cascadian ]
+  * debian/patches: Add patch from upstream to fix race intltool race
+    condition end up using a partial cache, resulting in missing
+    translations in some cases (Closes: #950444).
+
+ -- Vagrant Cascadian <vagrant@reproducible-builds.org>  Fri, 20 Nov 2020 12:53:23 -0800
+
 intltool (0.51.0-5) unstable; urgency=medium
 
   * Update Vcs fields for conversion to git
diff -Nru intltool-0.51.0/debian/control intltool-0.51.0/debian/control
--- intltool-0.51.0/debian/control	2017-12-21 19:23:13.000000000 -0800
+++ intltool-0.51.0/debian/control	2020-11-20 12:53:23.000000000 -0800
@@ -1,5 +1,5 @@
 # This file is autogenerated. DO NOT EDIT!
-# 
+#
 # Modifications should be made to debian/control.in instead.
 # This file is regenerated automatically in the clean target.
 Source: intltool
@@ -12,8 +12,8 @@
                perl (>= 5.8.1)
 Build-Depends-Indep: libxml-parser-perl
 Standards-Version: 4.1.2
-Vcs-Git: https://anonscm.debian.org/git/pkg-gnome/intltool.git
-Vcs-Browser: https://anonscm.debian.org/git/pkg-gnome/intltool.git
+Vcs-Browser: https://salsa.debian.org/gnome-team/intltool
+Vcs-Git: https://salsa.debian.org/gnome-team/intltool.git
 Homepage: https://launchpad.net/intltool
 
 Package: intltool
diff -Nru intltool-0.51.0/debian/control.in intltool-0.51.0/debian/control.in
--- intltool-0.51.0/debian/control.in	2017-12-21 19:23:13.000000000 -0800
+++ intltool-0.51.0/debian/control.in	2020-11-20 12:51:46.000000000 -0800
@@ -8,8 +8,8 @@
                perl (>= 5.8.1)
 Build-Depends-Indep: libxml-parser-perl
 Standards-Version: 4.1.2
-Vcs-Git: https://anonscm.debian.org/git/pkg-gnome/intltool.git
-Vcs-Browser: https://anonscm.debian.org/git/pkg-gnome/intltool.git
+Vcs-Browser: https://salsa.debian.org/gnome-team/intltool
+Vcs-Git: https://salsa.debian.org/gnome-team/intltool.git
 Homepage: https://launchpad.net/intltool
 
 Package: intltool
diff -Nru intltool-0.51.0/debian/patches/0001-Avoid-a-race-where-some-processes-try-to-use-a-parti.patch intltool-0.51.0/debian/patches/0001-Avoid-a-race-where-some-processes-try-to-use-a-parti.patch
--- intltool-0.51.0/debian/patches/0001-Avoid-a-race-where-some-processes-try-to-use-a-parti.patch	1969-12-31 16:00:00.000000000 -0800
+++ intltool-0.51.0/debian/patches/0001-Avoid-a-race-where-some-processes-try-to-use-a-parti.patch	2020-11-20 12:51:46.000000000 -0800
@@ -0,0 +1,57 @@
+From 97c9854c9ffe34d5fb4c8e928530c9a41b8e1a35 Mon Sep 17 00:00:00 2001
+From: "Bernhard M. Wiedemann" <bmwiedemann+bzrcommit@suse.de>
+Date: Thu, 18 May 2017 21:09:18 +0200
+Origin: https://bazaar.launchpad.net/~intltool/intltool/trunk/revision/748
+ https://bugs.launchpad.net/intltool/+bug/1687644
+Subject: [PATCH] Avoid a race where some processes try to use a partial cache
+ file that is still being written to. Note that we release the lock before
+ load_cache, because if we got the lock, the cache is already completely
+ written and it is OK to have multiple parallel readers
+
+Without this patch, translation files would randomly miss translations
+for some or all languages.
+
+fixes bug #1687644
+maybe also bug #986897
+---
+ intltool-merge.in | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/intltool-merge.in b/intltool-merge.in
+index 05db7cf..89923a7 100644
+--- a/intltool-merge.in
++++ b/intltool-merge.in
+@@ -43,6 +43,7 @@ use Getopt::Long;
+ use Text::Wrap;
+ use File::Basename;
+ use Encode;
++use Fcntl qw(:flock);
+ 
+ my $must_end_tag      = -1;
+ my $last_depth        = -1;
+@@ -392,11 +393,14 @@ sub load_cache
+ 
+ sub get_cached_translation_database
+ {
++    open(my $lockfh, ">", "$cache_file.lock") or die $!;
++    flock($lockfh, LOCK_EX) or die "Could not lock '$cache_file.lock' - $!";
+     my $cache_file_age = -M $cache_file;
+     if (defined $cache_file_age) 
+     {
+         if ($cache_file_age <= &get_newest_po_age) 
+         {
++            close($lockfh);
+             &load_cache;
+             return;
+         }
+@@ -404,6 +408,7 @@ sub get_cached_translation_database
+     }
+ 
+     &create_cache;
++    close($lockfh);
+ }
+ 
+ sub add_translation
+-- 
+2.20.1
+
diff -Nru intltool-0.51.0/debian/patches/series intltool-0.51.0/debian/patches/series
--- intltool-0.51.0/debian/patches/series	2017-12-21 19:23:13.000000000 -0800
+++ intltool-0.51.0/debian/patches/series	2020-11-20 12:51:46.000000000 -0800
@@ -1 +1,2 @@
 perl5.26-regex-fixes.patch
+0001-Avoid-a-race-where-some-processes-try-to-use-a-parti.patch
diff -Nru intltool-0.51.0/debian/rules intltool-0.51.0/debian/rules
--- intltool-0.51.0/debian/rules	2017-12-21 19:23:13.000000000 -0800
+++ intltool-0.51.0/debian/rules	2020-11-20 12:51:46.000000000 -0800
@@ -1,5 +1,4 @@
 #!/usr/bin/make -f
-
 -include /usr/share/gnome-pkg-tools/1/rules/lp-get-source.mk
 
 %:


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #38 received at 950444-close@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 950444-close@bugs.debian.org

Subject: Bug#950444: fixed in intltool 0.51.0-5.1

Date: Mon, 30 Nov 2020 21:34:13 +0000

Source: intltool
Source-Version: 0.51.0-5.1
Done: Vagrant Cascadian <vagrant@reproducible-builds.org>

We believe that the bug you reported is fixed in the latest version of
intltool, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 950444@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Vagrant Cascadian <vagrant@reproducible-builds.org> (supplier of updated intltool package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Fri, 20 Nov 2020 12:53:23 -0800
Source: intltool
Architecture: source
Version: 0.51.0-5.1
Distribution: unstable
Urgency: medium
Maintainer: Debian GNOME Maintainers <pkg-gnome-maintainers@lists.alioth.debian.org>
Changed-By: Vagrant Cascadian <vagrant@reproducible-builds.org>
Closes: 950444
Changes:
 intltool (0.51.0-5.1) unstable; urgency=medium
 .
   * Non-maintainer upload.
 .
   [ Jeremy Bicha ]
   * Update Vcs fields for migration to https://salsa.debian.org/
 .
   [ Vagrant Cascadian ]
   * debian/patches: Add patch from upstream to fix race intltool race
     condition end up using a partial cache, resulting in missing
     translations in some cases (Closes: #950444).
Checksums-Sha1:
 8d5af0d2b269e3c1a01f1c358ec41371d164545f 1613 intltool_0.51.0-5.1.dsc
 3dd4c19e3c28a697b69d4e2edd2e4ad100bc22b8 8288 intltool_0.51.0-5.1.debian.tar.xz
 f5f300169cf147939bcc07c60096bb2bc02d644c 6100 intltool_0.51.0-5.1_amd64.buildinfo
Checksums-Sha256:
 2a9989d5ac9ec3649337792d0356763f284418b7fdf2f91dcb9a3e3e3b473c48 1613 intltool_0.51.0-5.1.dsc
 ddc238dd39dce63df64110303c894ed5612fa1abcfbdc4d9936fd3d3e52e3b3a 8288 intltool_0.51.0-5.1.debian.tar.xz
 52a6fb4625eca746961b0a3a52486506747c7f9bfaa00f834375b5a976d0ee42 6100 intltool_0.51.0-5.1_amd64.buildinfo
Files:
 d1d3293e986e6887ecadd528c12b0d97 1613 devel optional intltool_0.51.0-5.1.dsc
 075d210976aa75291355d8367bb41a97 8288 devel optional intltool_0.51.0-5.1.debian.tar.xz
 96608788989f6de170aede0c5745941b 6100 devel optional intltool_0.51.0-5.1_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iJYEARYKAD4WIQRlgHNhO/zFx+LkXUXcUY/If5cWqgUCX7guOyAcdmFncmFudEBy
ZXByb2R1Y2libGUtYnVpbGRzLm9yZwAKCRDcUY/If5cWqpdbAP9erDsm2PpEQe92
gmH3dO57qWcsDYoVwmKPS5PQTmmsiwEAwUOR1ujWFf8/MaYMn3QUszXudu9D91LQ
wCuGdu6hbQ8=
=kY1n
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.