Debian Bug report logs - #953105
gtk-update-icon-cache does not produce reproducible results on 32-bit architectures

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>

To: submit@bugs.debian.org

Subject: gtk-update-icon-cache does not produce reproducible results on 32-bit architectures

Date: Wed, 04 Mar 2020 10:51:59 -0500

[Message part 1 (text/plain, inline)]

Package: gtk-update-icon-cache
Version: 3.24.14-1
Control: affects -1 balsa
User: reproducible-builds@lists.alioth.debian.org
Usertags: toolchain
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org

The balsa package builds reproducibly on 64-bit architectures arm64 and
amd64.  But it fails to do so on the 32-bit architectures i386 and
armhf:

https://tests.reproducible-builds.org/debian/rb-pkg/unstable/i386/diffoscope-results/balsa.html
https://tests.reproducible-builds.org/debian/rb-pkg/unstable/armhf/diffoscope-results/balsa.html

The only differences are in the /usr/share/balsa/icon-theme.cache file,
which i believe is generated during build by gtk-update-icon-cache.

It seems possible that we could fix this problem in the balsa package by
not shipping an icon cache at all, of course (and maybe running
update-icon-caches in the postinst?  though i'd like to avoid
introducing a postinst script if we don't need to).

But it seems odd that the 32-bit architectures would produce
unreproducible caches when the 64-bit version is reproducible.

               -dkg

[signature.asc (application/pgp-signature, inline)]

Message #12 received at 953105@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: "Daniel Kahn Gillmor" <dkg@fifthhorseman.net>, 953105@bugs.debian.org, rb-general <rb-general@lists.reproducible-builds.org>

Subject: Re: Bug#953105: gtk-update-icon-cache does not produce reproducible results on 32-bit architectures

Date: Wed, 04 Mar 2020 10:22:31 -0800

Hi dkg,

> But it seems odd that the 32-bit architectures would produce
> unreproducible caches when the 64-bit version is reproducible.

I haven't checked the build history but assuming this is not in-of
itself a nondetermistic distinction (in which I would blame the hash
table usage) then this is likely due to filesystem ordering. Indeed if
you look at gtk/updateiconcache.c src:gtk+3.0 then, after a quick
skim, both of these things are apparent in this file.

Anyway, I've added this issue and bug to the Reproducible Builds
"notes" git repository:

  https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/0b871fa18b9accc95c960dacada22820a9a79969

… and tagged the src:balsa package with this issue:

  https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/2ea542dffb997d664a62b0485d6e02209da587ee

(Very rough and ready of course, but even this approximate tagging
is very helpful to us.)


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Message #17 received at 953105@bugs.debian.org (full text, mbox, reply):

From: "Orians, Jeremiah (DTMB)" <OriansJ@michigan.gov>

To: General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>, Daniel Kahn Gillmor <dkg@fifthhorseman.net>, "953105@bugs.debian.org" <953105@bugs.debian.org>

Subject: RE: Bug#953105: gtk-update-icon-cache does not produce reproducible results on 32-bit architectures

Date: Thu, 5 Mar 2020 12:24:49 +0000

> But it seems odd that the 32-bit architectures would produce 
> unreproducible caches when the 64-bit version is reproducible.
Why?
Even such things as alignment in malloc results in different behavior between x86 and amd64
For example this function is reproducible on AMD64 but not on x86
https://github.com/oriansj/mes-m2/blob/master/mes_posix.c#L31

Changing the 12 to 13 works around that subtle difference between the 2 architectures.
- Jeremiah

Message #22 received at 953105@bugs.debian.org (full text, mbox, reply):

From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>

To: "Orians\, Jeremiah \(DTMB\)" <OriansJ@michigan.gov>, General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>, "953105\@bugs.debian.org" <953105@bugs.debian.org>

Subject: RE: Bug#953105: gtk-update-icon-cache does not produce reproducible results on 32-bit architectures

Date: Thu, 05 Mar 2020 09:36:25 -0500

[Message part 1 (text/plain, inline)]

On Thu 2020-03-05 12:24:49 +0000, Orians, Jeremiah (DTMB) wrote:
>> But it seems odd that the 32-bit architectures would produce 
>> unreproducible caches when the 64-bit version is reproducible.
> 
> Why?
> Even such things as alignment in malloc results in different behavior between x86 and amd64

If i'm understanding you correctly, this suggests that the results
created by gtk-update-icon-cache do not belong in any arch-independent
package.

Is this correct?

If so, someone probably ought to review the archive for instances like this:
https://codesearch.debian.net/search?q=gtk-update-icon-cache&literal=1&perpkg=1&page=1

Is there some magic that a tool like lintian could use to identify an
icon-cache file shipped in an arch: all package?  libmagic just
identifies them as "data"

           --dkg

[signature.asc (application/pgp-signature, inline)]

Message #27 received at 953105@bugs.debian.org (full text, mbox, reply):

From: "Orians, Jeremiah (DTMB)" <OriansJ@michigan.gov>

To: Daniel Kahn Gillmor <dkg@fifthhorseman.net>, General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>, "953105@bugs.debian.org" <953105@bugs.debian.org>

Subject: RE: Bug#953105: gtk-update-icon-cache does not produce reproducible results on 32-bit architectures

Date: Thu, 5 Mar 2020 16:43:53 +0000

>> Why?
>> Even such things as alignment in malloc results in different behavior 
>> between x86 and amd64
> If i'm understanding you correctly, this suggests that the results created by gtk-update-icon-cache do not belong in any arch-independent package.
> Is this correct?
Partially, as the requirements for reproducibly cross-platform are a lot more strict than just being reproducible or just being cross-platform.
I learned that the hard way when I ported https://github.com/oriansj/M2-Planet to a big-endian architecture and had to get it to have identical little-endian behavior too.
So you can either admit that the code behaves differently on different architectures and be different or fix the code to behave exactly the identically across different architectures.

> Is there some magic that a tool like lintian could use to identify an icon-cache file shipped in an arch: all package?  libmagic just identifies them as "data"
There is no magic in IT; only sweat, blood and tears.

-Jeremiah

Message #32 received at 953105@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Daniel Kahn Gillmor <dkg@fifthhorseman.net>, 953105@bugs.debian.org, Chris Lamb <lamby@debian.org>

Cc: rb-general <rb-general@lists.reproducible-builds.org>, "Orians, Jeremiah (DTMB)" <OriansJ@michigan.gov>

Subject: Re: Bug#953105: gtk-update-icon-cache does not produce reproducible results on 32-bit architectures

Date: Thu, 5 Mar 2020 20:02:11 +0000

On Wed, 04 Mar 2020 at 10:51:59 -0500, Daniel Kahn Gillmor wrote:
> It seems possible that we could fix this problem in the balsa package by
> not shipping an icon cache at all, of course (and maybe running
> update-icon-caches in the postinst?  though i'd like to avoid
> introducing a postinst script if we don't need to).

It's certainly unusual to want to have your own icon cache. Most packages
that have icons install them into the system-wide hicolor icon theme
directory (for historical reasons "hicolor" is hard-coded to be the
default/fallback icon theme - it was the KDE 3 default). This causes a
dpkg trigger shipped by hicolor-icon-theme to update the icon cache.

According to `apt-file search`, balsa is the only package that
looks like it might possibly be correct to ship this file - but if every
other application manages to get by without having its own private icon
cache, I don't know what's so special about balsa?

medit ships /usr/share/icons/hicolor/icon-theme.cache, which is definitely
a bug: hicolor-icon-theme's trigger will make gtk-update-icon-cache
overwrite it.

numix-icon-theme-circle ships
/usr/share/icons/Numix-Circle/icon-theme.cache, which is definitely a bug:
its own postinst(!) asks update-icon-caches to overwrite that file.

> But it seems odd that the 32-bit architectures would produce
> unreproducible caches when the 64-bit version is reproducible.

I have no idea why this is the case. Having looked at the code that
generates them, I would expect these cache files to be unreproducible
on all architectures. If they're reproducible on 64-bit it's by
luck and coincidence.

I don't know how much sense they make to ship in packages (as I said,
of three packages that do, balsa is possibly wrong and the others are
definitely wrong), but if the cache was reproducible it would reduce
churn when people distribute complete filesystem trees in Flatpak,
ostree, casync, Docker, tarballs and so on.

On Wed, 04 Mar 2020 at 10:22:31 -0800, Chris Lamb wrote:
> I haven't checked the build history but assuming this is not in-of
> itself a nondetermistic distinction (in which I would blame the hash
> table usage) then this is likely due to filesystem ordering. Indeed if
> you look at gtk/updateiconcache.c src:gtk+3.0 then, after a quick
> skim, both of these things are apparent in this file.

Yes, it's both. The order of directory entries is in readdir() order,
the order of insertion of per-file entries into a hash table is also
influenced by readdir() order, and the contents of the hash table
are written into a hash-table-on-disk data structure in arbitrary hash
table iteration order.

Following some late-night hacking, I now have patches that I believe
should fix this, which I need to test (not even compiled yet), and a
Python script to dump the contents of a cache in a readable format,
which I intend to use to check that the result with the patched GTK
is equivalent.

(The file format is more complicated than it probably ought to be, for
compatibility with historical GTK versions.)

On Thu, 05 Mar 2020 at 09:36:25 -0500, Daniel Kahn Gillmor wrote:
> If i'm understanding you correctly, this suggests that the results
> created by gtk-update-icon-cache do not belong in any arch-independent
> package.
> 
> Is this correct?

No, the results of gtk-update-icon-cache are not intentionally
architecture-dependent. It's all done with fixed-size integers and network
endianness, so in principle it ought to be portable and reproducible -
but the implementation doesn't make any attempt to avoid being influenced
by hash and readdir() order.

> Is there some magic that a tool like lintian could use to identify an
> icon-cache file shipped in an arch: all package?

No. The only fixed string is that it starts with 0x00 0x01 0x00 0x00
(major version 1, minor version 0) which probably collides with dozens
of other ad-hoc binary file formats. You'd be better off recognising
the filename.

    smcv

Message #37 received at 953105@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Daniel Kahn Gillmor <dkg@fifthhorseman.net>, 953105@bugs.debian.org, Chris Lamb <lamby@debian.org>

Cc: rb-general <rb-general@lists.reproducible-builds.org>, "Orians, Jeremiah (DTMB)" <OriansJ@michigan.gov>

Subject: Re: Bug#953105: gtk-update-icon-cache does not produce reproducible results on 32-bit architectures

Date: Thu, 5 Mar 2020 20:10:51 +0000

On Thu, 05 Mar 2020 at 20:02:11 +0000, Simon McVittie wrote:
> medit ships /usr/share/icons/hicolor/icon-theme.cache, which is definitely
> a bug: hicolor-icon-theme's trigger will make gtk-update-icon-cache
> overwrite it.
> 
> numix-icon-theme-circle ships
> /usr/share/icons/Numix-Circle/icon-theme.cache, which is definitely a bug:
> its own postinst(!) asks update-icon-caches to overwrite that file.

Sorry, I was fooled by apt-file showing packages from multiple suites.
Both of these packages *used to* ship icon-theme.cache, but medit
is no longer in unstable, and numix-icon-theme-circle no longer ships
icon-theme.cache. So I think it's only balsa that does this, which makes
me continue to question whether balsa should do so.

    smcv

Message #42 received at 953105@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: "Simon McVittie" <smcv@debian.org>, "Daniel Kahn Gillmor" <dkg@fifthhorseman.net>, 953105@bugs.debian.org

Cc: rb-general <rb-general@lists.reproducible-builds.org>, "Orians, Jeremiah (DTMB)" <OriansJ@michigan.gov>

Subject: Re: Bug#953105: gtk-update-icon-cache does not produce reproducible results on 32-bit architectures

Date: Thu, 05 Mar 2020 12:12:06 -0800

Hi Simon,

> Chris Lamb wrote:
>
> > I haven't checked the build history but assuming this is not in-of
> > itself a nondetermistic distinction (in which I would blame the hash
> > table usage) then this is likely due to filesystem ordering. Indeed if
> > you look at gtk/updateiconcache.c src:gtk+3.0 then, after a quick
> > skim, both of these things are apparent in this file.
> 
> Yes, it's both. The order of directory entries is in readdir() order,
> the order of insertion of per-file entries into a hash table is also
> influenced by readdir() order, and the contents of the hash table
> are written into a hash-table-on-disk data structure in arbitrary hash
> table iteration order.

Your email arrived at a good time as I had just written a reply to the
previous thread to say that we were overly hasty in coming to an
architecture-dependent conclusion given the evidence we had in front
of us at the time.

(As this email has a wider audience, as an obiter dictum I will note
that there are a few other variations between our "i386" and "amd64"
builders unrelated to the architecture itself which can lead to
misleading deductions.)

Furthermore, I fixed some nondeterminism in a separate GTK cache (?)
in 2018 or so which lent some circumstantial weight to my
equivocation. Indeed, Simon: shall I try and dig this out for you?
Your fix to GTK is certainly superior and thus there may be parallel
changes to be made to my past work across the codebase.


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Message #47 received at 953105@bugs.debian.org (full text, mbox, reply):

From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>

To: "Orians\, Jeremiah \(DTMB\)" <OriansJ@michigan.gov>, General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>, "953105\@bugs.debian.org" <953105@bugs.debian.org>

Subject: RE: Bug#953105: gtk-update-icon-cache does not produce reproducible results on 32-bit architectures

Date: Thu, 05 Mar 2020 18:35:45 -0500

[Message part 1 (text/plain, inline)]

On Thu 2020-03-05 16:43:53 +0000, Orians, Jeremiah (DTMB) wrote:
> [dkg wrote:]
>> If i'm understanding you correctly, this suggests that the results created by gtk-update-icon-cache do not belong in any arch-independent package.
>> Is this correct?
>
> Partially, as the requirements for reproducibly cross-platform are a lot more strict than just being reproducible or just being cross-platform.
> I learned that the hard way when I ported https://github.com/oriansj/M2-Planet to a big-endian architecture and had to get it to have identical little-endian behavior too.
> So you can either admit that the code behaves differently on different architectures and be different or fix the code to behave exactly the identically across different architectures.

I'm not asking about reproducibility here; i'm asking about bugginess.

I know nothing about the gtk icon cache data format, but it appears to
be "something mmappable" from the docs i've read.

But if the different arches assume different memory data structures once
the file is mmapped, then shipping such a file in an arch:all package
might be dangerously wrong.

For example, consider building the arch:all package on amd64, and part
of that is building an updated gtk icon cache that we will ship.  When
it is subsequently installed and mmapped on s390x (change of endianness)
or armhf (change of pointer size), will the data still look the same to
the application?  (not to mention alignment, int size, etc.)

>> Is there some magic that a tool like lintian could use to identify an icon-cache file shipped in an arch: all package?  libmagic just identifies them as "data"
> There is no magic in IT; only sweat, blood and tears.

ha ha, but there is indeed a libmagic, the result of years of blood,
sweat and tears.  I was asking a serious question about what the
contents of a gtk icon-cache is supposed to be, and how we might
recognize it.

I'm used to generating arch:all cross-platform "binary" optimization
files (e.g. psl-make-dafsa).  Those are structured in a well-documented
way, and libmagic can even detect them.  Can we do the same for a gtk
icon cache?

      --dkg

[signature.asc (application/pgp-signature, inline)]

Message #52 received at 953105@bugs.debian.org (full text, mbox, reply):

From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>

To: Simon McVittie <smcv@debian.org>, 953105@bugs.debian.org, Chris Lamb <lamby@debian.org>

Cc: rb-general <rb-general@lists.reproducible-builds.org>, "Orians\, Jeremiah \(DTMB\)" <OriansJ@michigan.gov>

Subject: Re: Bug#953105: gtk-update-icon-cache does not produce reproducible results on 32-bit architectures

Date: Thu, 05 Mar 2020 19:12:33 -0500

[Message part 1 (text/plain, inline)]

Control: clone 953105 -2
Control: retitle -2 balsa should not ship an icon cache
Control: reassign -2 balsa

On Thu 2020-03-05 20:10:51 +0000, Simon McVittie wrote:
> On Thu, 05 Mar 2020 at 20:02:11 +0000, Simon McVittie wrote:
>> medit ships /usr/share/icons/hicolor/icon-theme.cache, which is definitely
>> a bug: hicolor-icon-theme's trigger will make gtk-update-icon-cache
>> overwrite it.
>> 
>> numix-icon-theme-circle ships
>> /usr/share/icons/Numix-Circle/icon-theme.cache, which is definitely a bug:
>> its own postinst(!) asks update-icon-caches to overwrite that file.
>
> Sorry, I was fooled by apt-file showing packages from multiple suites.
> Both of these packages *used to* ship icon-theme.cache, but medit
> is no longer in unstable, and numix-icon-theme-circle no longer ships
> icon-theme.cache. So I think it's only balsa that does this, which makes
> me continue to question whether balsa should do so.

Wow, thanks for dropping some serious knowledge on this thread, Simon,
and thanks also for the suggestions for balsa.  I'm cloning this bug
report to remind myself (or anyone else working on balsa) to drop the
icon cache from the shipped files.

balsa is due for an arch:all package split given the size of the
arch-independent data that it ships anyway (we can save something like
1.5MiB/architecture), so maybe we can also drop the icon cache as part
of that cleanup.

     --dkg

[signature.asc (application/pgp-signature, inline)]

Message #59 received at 953105@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Daniel Kahn Gillmor <dkg@fifthhorseman.net>, 953105@bugs.debian.org

Cc: "Orians, Jeremiah (DTMB)" <OriansJ@michigan.gov>, General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>

Subject: Re: Bug#953105: gtk-update-icon-cache does not produce reproducible results on 32-bit architectures

Date: Fri, 6 Mar 2020 08:11:57 +0000

On Thu, 05 Mar 2020 at 18:35:45 -0500, Daniel Kahn Gillmor wrote:
> But if the different arches assume different memory data structures once
> the file is mmapped, then shipping such a file in an arch:all package
> might be dangerously wrong.

From the code, it looks like the different architectures use the same
in-memory data structures. The lack of reproducibility is from not caring
about reproducibility, not from not caring about portability.

It's all done with fixed-size integers, fixed endianness (network byte
order), "pointers" that are offsets relative to the start of the mmapped
buffer, and a hard-coded hash function (which I'm guessing was the string
hash function used in GLib/GTK 1, but it's hard-coded in the icon cache
implementation so that it will continue to work even if GLib changes
its hash function).

> Those are structured in a well-documented
> way, and libmagic can even detect them.  Can we do the same for a gtk
> icon cache?

It's somewhat well-structured, but not documented except in the code,
and doesn't have a "magic number" that libmagic could use. Recognising
them by filename is going to be the least-bad way.

    smcv

Message #64 received at 953105-close@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 953105-close@bugs.debian.org

Subject: Bug#953105: fixed in gtk+3.0 3.24.16-1

Date: Thu, 02 Apr 2020 13:49:16 +0000

Source: gtk+3.0
Source-Version: 3.24.16-1
Done: Simon McVittie <smcv@debian.org>

We believe that the bug you reported is fixed in the latest version of
gtk+3.0, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 953105@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Simon McVittie <smcv@debian.org> (supplier of updated gtk+3.0 package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Thu, 02 Apr 2020 13:15:25 +0100
Source: gtk+3.0
Architecture: source
Version: 3.24.16-1
Distribution: unstable
Urgency: medium
Maintainer: Debian GNOME Maintainers <pkg-gnome-maintainers@lists.alioth.debian.org>
Changed-By: Simon McVittie <smcv@debian.org>
Closes: 953105 954584
Changes:
 gtk+3.0 (3.24.16-1) unstable; urgency=medium
 .
   * Team upload
   * New upstream release
   * Drop patches that came from upstream
   * d/p/supp-Use-a-single-suppressions-file-for-lib-lib64-and-mul.patch:
     Add patch to make valgrind suppressions match Debian libraries
   * Update forwarding status for some patches that were sent upstream
   * d/p/Don-t-test-default-constructed-GdkPixbuf-properties.patch:
     Drop patch. Build-depend on gdk-pixbuf 2.40 instead of working around
     a bug in older versions.
   * d/rules: Mark some reftests as allowed to fail.
     These reftests are sensitive to the build environment (exact versions
     of Pango, fonts etc.) and are disabled in upstream's CI.
     (Closes: #954584)
   * d/p/updateiconcache-Sort-list-of-entries.patch:
     Backport patch from GTK 4 to make the icon cache reproducible
     (Closes: #953105)
   * d/p/build-Generate-gdk.gresource.xml-in-sorted-order.patch:
     Sort GDK resources to improve reproducibility.
     The combination of this and fixing #953105 should hopefully make the
     package reliably reproducible.
   * d/README.source: Write down how to inspect reftest failures
Checksums-Sha1:
 681feca029967a2d0e8581372dc6b643e45d9a0e 4040 gtk+3.0_3.24.16-1.dsc
 407496ffcf762dffb06bd71f40145fcaf941e130 20417592 gtk+3.0_3.24.16.orig.tar.xz
 19492382da45b33b5bda9de642160c984a963eff 157004 gtk+3.0_3.24.16-1.debian.tar.xz
 7d567c9583ac359e2c49095fc935481326a7cc8b 15037 gtk+3.0_3.24.16-1_source.buildinfo
Checksums-Sha256:
 7bdf9e745b7054ba9f1ed3a4c3e0a318e40afaf952df9fc0d060fc1b7a9b2ec2 4040 gtk+3.0_3.24.16-1.dsc
 0d5e1e1494101b8c0c63c0526180780559eee469f021ca0d714018b20fa3d8e8 20417592 gtk+3.0_3.24.16.orig.tar.xz
 387e2cb38a4c685ea13324b512b77c96e3a9007126450e098b474bccdc396eed 157004 gtk+3.0_3.24.16-1.debian.tar.xz
 c79ac28e5d53f0f1bcbf74249f09c200c4ee72755eb2d7da7ce86aa80560b0e4 15037 gtk+3.0_3.24.16-1_source.buildinfo
Files:
 a7f5a798c7a846a6911f29f030c4de10 4040 libs optional gtk+3.0_3.24.16-1.dsc
 cef1dd5f7b78d3d51b983e388ba49af5 20417592 libs optional gtk+3.0_3.24.16.orig.tar.xz
 fc0db6763e47f7fc86d1d08d72a32ea5 157004 libs optional gtk+3.0_3.24.16-1.debian.tar.xz
 746c7adc46589484fd7a0d70548c1736 15037 libs optional gtk+3.0_3.24.16-1_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQJEBAEBCAAuFiEENuxaZEik9e95vv6Y4FrhR4+BTE8FAl6F6f4QHHNtY3ZAZGVi
aWFuLm9yZwAKCRDgWuFHj4FMTwBWD/9m/hz4KwEjIHr39EY9rndAvt+2H3dV6w6X
TGvv4ndUN0uV9ZOSz9PEAbP7xBL0SxWb/uZYDnRpd4OBG1UQvBhugmATzzk0lf64
lfVM0f58zW1V1HfMikQ6nRU1E919+2BR7Cwqbx7PkorqVuPpyTSjL/6NGV66aB9h
++4CDdkJCN3tQBUqh35JYRP3f/MX8cMnjLKwFgu1hn9IIre2DRhea560Oo1BSwyi
vqNkcargfu6YYsnTBRyHjyf8DyZtiFQ6H/M9/jGIE4owfhjkjbSwhQach6FECUTG
kFaCZhzUbaX0n9S5Jg/sKle3CirgCEo1YFozy+b3cbqIOV2hO7+WnsDI2ZGZWUHt
q4053uakshb//oLOGaQpGZpD1OTk9dTwcGkQi20DGKbb7X0nN0/GO90o/rAb30lG
aYwfQNF30S3D9oYncjmWAtbLqvrF4Dx6jgqnQ5T8O4KWVJdqtp0Ail7YbgaI7aId
1KvPzUENGW+/4gIVYXQa8ugZa/M8MTG5PxZSW5lr8xUeNbZajjWTmJzSOXYYQ5yQ
mi9gbRsH/K9fIRjl1IhPu0eAm/uFT+9GqqYIeHPK68I2e+nhLDl9WPJhW4QRTv4v
MOYIupuDfRexjte8gSEz/IOgWTxzPyJVs5tBqKCmQpb+nku5mnfjYh9MrxpvZkOJ
ItMIXFKDgA==
=tXqQ
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.