Debian Bug report logs - #822197
sphinx: please remove memory addresses from generated python documentation

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Alexis Bienvenüe <pado@passoire.fr>

To: submit@bugs.debian.org

Subject: sphinx: please remove memory addresses from generated python documentation

Date: Fri, 22 Apr 2016 00:53:52 +0200

[Message part 1 (text/plain, inline)]

Source: sphinx
Version: 1.3.6-2
Severity: wishlist
Tags: patch upstream
User: reproducible-builds@lists.alioth.debian.org
Usertags: toolchain
X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org
Control: forwarded -1 https://github.com/sphinx-doc/sphinx/pull/2480

Dear Maintainer,

While working on the “reproducible builds” effort [1], we have noticed
that the documentation generated by sphinx can include the memory
address of some python objects that are used at build time. See for
example, in file
/usr/share/doc/python-kombu-doc/html/reference/kombu.transport.qpid.html
from package python-kombu-doc_3.0.35-1.1 :

  Transport.Connection.Channel.codecs = {'base64':
<kombu.transport.virtual.Base64 object at 0x7f10fff1a590>}

This memory address is irrelevant in the documentation, and prevents the
package building to be reproducible.

The attached patch strips these memory addresses, leading to better
reproducibility.

Regards,
Alexis Bienvenüe.

[1] https://wiki.debian.org/ReproducibleBuilds

[sphinx_1.3.6-2_rmmem.diff (text/x-patch, attachment)]

Message #12 received at 822197@bugs.debian.org (full text, mbox, reply):

From: Alexis Bienvenüe <pado@passoire.fr>

To: 822197@bugs.debian.org

Subject: Re: sphinx: please remove memory addresses from generated python documentation

Date: Mon, 9 May 2016 15:39:06 +0200

Control: forwarded -1 https://github.com/sphinx-doc/sphinx/pull/2534

A far simpler patch has been submitted upstream!

Alexis Bienvenüe.

Message #19 received at 822197@bugs.debian.org (full text, mbox, reply):

From: Alexis Bienvenüe <pado@passoire.fr>

To: 822197@bugs.debian.org

Subject: Re: sphinx: please remove memory addresses from generated python documentation

Date: Fri, 10 Jun 2016 07:38:21 +0200

Control: tags -1 + fixed-upstream

Now fixed upstream:
https://github.com/sphinx-doc/sphinx/commit/09c25443ff8bbfb745b717658523f2c218bbf843

Regards,
Alexis Bienvenüe.

Message #26 received at 822197-close@bugs.debian.org (full text, mbox, reply):

From: Dmitry Shachnev <mitya57@debian.org>

To: 822197-close@bugs.debian.org

Subject: Bug#822197: fixed in sphinx 1.4.4-1

Date: Fri, 17 Jun 2016 16:46:23 +0000

Source: sphinx
Source-Version: 1.4.4-1

We believe that the bug you reported is fixed in the latest version of
sphinx, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 822197@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Dmitry Shachnev <mitya57@debian.org> (supplier of updated sphinx package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Fri, 17 Jun 2016 18:42:56 +0300
Source: sphinx
Binary: python-sphinx python3-sphinx sphinx-common sphinx-doc libjs-sphinxdoc
Architecture: source
Version: 1.4.4-1
Distribution: unstable
Urgency: medium
Maintainer: Debian Python Modules Team <python-modules-team@lists.alioth.debian.org>
Changed-By: Dmitry Shachnev <mitya57@debian.org>
Description:
 libjs-sphinxdoc - JavaScript support for Sphinx documentation
 python-sphinx - documentation generator for Python projects (implemented in Pytho
 python3-sphinx - documentation generator for Python projects (implemented in Pytho
 sphinx-common - documentation generator for Python projects - common data
 sphinx-doc - documentation generator for Python projects - documentation
Closes: 822197
Changes:
 sphinx (1.4.4-1) unstable; urgency=medium
 .
   * New upstream bugfix release.
     - Autodoc now removes all memory addresses (closes: #822197).
   * Drop python3-xapian build-dependency for unstable upload: it is not
     yet available in unstable.
   * Build-depend on graphviz so that the corresponding tests are not
     skipped.
   * Do not install the bundled TeX files from texinput directory:
     - fncychap.sty (available in texlive-latex-extra)
     - iftex.sty (available in texlive-generic-extra)
     - needspace.sty (available in texlive-latex-extra)
     - newfloat.sty (available in texlive-latex-recommended)
   * Suggest texlive-generic-extra instead (the other packages are already in
     the Suggests list).
   * Upload to unstable.
Checksums-Sha1:
 716b9769b06eba1a2c28685771f31d29164e4cd8 3037 sphinx_1.4.4-1.dsc
 87ef31c2ce8c556a1644c53d21526eac7ad45f38 4335455 sphinx_1.4.4.orig.tar.gz
 cfca3c8f6c4d18bd0448f3fefa49e8040db41673 33048 sphinx_1.4.4-1.debian.tar.xz
Checksums-Sha256:
 955f04f072829f4da54c8314e9442cd7e20d94018cb3fb9bdfb5e1ccbf532262 3037 sphinx_1.4.4-1.dsc
 3effd6373734bd59f7457fed2f0bd4ba7ec3c70b4598d7c2e5193a42209dbfa0 4335455 sphinx_1.4.4.orig.tar.gz
 9c0ccf8eeb372cec52797067e6025d3443fae87f0dce1806bc50d28ae8687204 33048 sphinx_1.4.4-1.debian.tar.xz
Files:
 dc2de5d81f1c01105e2f4ce9c940f5b8 3037 python optional sphinx_1.4.4-1.dsc
 64ce2ec08d37ed56313a98232cbe2aee 4335455 python optional sphinx_1.4.4.orig.tar.gz
 a638fcda9fe9600c260853ee8b27e8e5 33048 python optional sphinx_1.4.4-1.debian.tar.xz

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJXZBsbAAoJENb+cQNj+F3TipYP+wS2BaJu/p6r318jUDIfpgub
QY1xkyHraa9BeC80G08fgVxcpEmQ86yKYYRaMLOXE4IqcSuv68GuDk5dWD1A5+Fu
EZuqBcjZVbUe2ds3zB4oPHQFG+UhHa2jPIj37TkFATL6qSu9wwZVSn9kkfWXIsBu
tAr0lIOTF0zEBqjsH8wwxTer+xjfBzlvixO72D35C9+PpiygJZpnKwiCBFQr2X2t
Y5hNyMQ7A/TKe142etJQqNcbOenPVObSrC924fv2tx2vgh08JSrOrw7wD3pIIVjD
lhH5QEUy1NZu6vokH5ZJzQIQSuCZhWKOcS4rIMvr0VJTNAsyrNKQv0JXX6aqG5x2
GChb9z+Ob1e4cXaDFz7JTaqNmQPYDdHzNcMYjkwNMTCosf/eYeB3XjMxtmrPOr8B
Ll6JVQiqYmgo9GJE+ZULFsonI4lUPZ1R0cJomaxsxMp2DmigqDxf7pT3D+OOPPp8
sheA45Ra+Yoydhid902rWaY1WUVeCXqUv3AI+J2SQu/R1aNXNLxr96vkNr//G++T
68yR3vSjLP3vV3AKJ+vOlAyquCvlOcu3F03gZxXFDYT0DsZBopMXZPmHdycShZrE
2GUAQg7DhhGsRMk2XGDUrSubH/eZIynfeoJ8e4LjiRoMXOgslhB1yjL2mQoBzudt
rLCcEF5yBOMHM1k7PdWC
=Si0O
-----END PGP SIGNATURE-----

Message #35 received at 822197@bugs.debian.org (full text, mbox, reply):

From: Justin Cappos <jcappos@nyu.edu>

To: 822197@bugs.debian.org

Subject: Fwd: Memory addresses still seem to appear in sphinx docs...

Date: Fri, 5 May 2017 06:11:50 -0400

[Message part 1 (text/plain, inline)]

[Resending to the list.  The bug was archived so the initial email was
rejected]

---------- Forwarded message ----------
From: Justin Cappos R<jcappos@nyu.edu>
Date: Fri, May 5, 2017 at 6:01 AM
Subject: Memory addresses still seem to appear in sphinx docs...
To: 822197@bugs.debian.org
Cc: Dmitry Shachnev <mitya57@debian.org>, Alexis Bienvenüe <pado@passoire.fr
>


I am taking a look at some of the reproducible-builds packages that are
still failing and still see some issues that look like they may be from
memory addresses being output by sphinx.

For example, django-celery outputs the following memory address (see
https://tests.reproducible-builds.org/debian/rb-pkg/
unstable/amd64/diffoscope-results/django-celery.html ):

./usr/share/doc/python-django-celery-doc/html/reference/djcelery.app.html:

<code·​class="descclassname"​>djcelery.​app.​</​code><code·​
class="descname">app<​/​code><em·​class="property">·​=·​&lt;
​Celery·​default:​*0x7f468b76b310*&gt;​</​em><a·​class="
headerlink"·​href="#djcelery.​app.​app"·​title="Permalink·​
to·​this·​definition">¶</​a></​dt>

./usr/share/doc/python-django-celery-doc/html/searchindex.js
​·​·​·​·terms:​·​{
·​·​·​·​·​·​·​·​"*0x7f468b76b310*":​·​8,​
·​·​·​·​·​·​·​·​"2to3":​·​0,​

I checked the github issue tracker and this doesn't seem to be listed
there, but it is possible that this issue is related:
https://github.com/sphinx-doc/sphinx/issues/1721.

Thanks,
Justin

[Message part 2 (text/html, inline)]

Message #40 received at 822197@bugs.debian.org (full text, mbox, reply):

From: Dmitry Shachnev <mitya57@debian.org>

To: Justin Cappos <jcappos@nyu.edu>

Cc: 822197@bugs.debian.org, Alexis Bienvenüe <pado@passoire.fr>

Subject: Re: Memory addresses still seem to appear in sphinx docs...

Date: Sun, 7 May 2017 13:38:14 +0300

Hi Justin,

On Fri, May 05, 2017 at 06:01:22AM -0400, Justin Cappos wrote:
> I am taking a look at some of the reproducible-builds packages that are
> still failing and still see some issues that look like they may be from
> memory addresses being output by sphinx.
>
> For example, django-celery outputs the following memory address
> [...]
>
> I checked the github issue tracker and this doesn't seem to be listed
> there, but it is possible that this issue is related:
> https://github.com/sphinx-doc/sphinx/issues/1721.

Can you please file a new bug for upstream Sphinx, or ping the existing
issue you mentioned?

I am afraid I don’t have time to look at this myself, so help from upstream
would be nice.

--
Dmitry Shachnev

Message #45 received at 822197@bugs.debian.org (full text, mbox, reply):

From: Justin Cappos <jcappos@nyu.edu>

To: Dmitry Shachnev <mitya57@debian.org>

Cc: 822197@bugs.debian.org, Alexis Bienvenüe <pado@passoire.fr>

Subject: Re: Memory addresses still seem to appear in sphinx docs...

Date: Sun, 7 May 2017 11:39:50 -0400

[Message part 1 (text/plain, inline)]

Okay, I have opened a new bug about this in sphinx:
https://github.com/sphinx-doc/sphinx/issues/3722

Thanks,
Justin

On Sun, May 7, 2017 at 6:38 AM, Dmitry Shachnev <mitya57@debian.org> wrote:

> Hi Justin,
>
> On Fri, May 05, 2017 at 06:01:22AM -0400, Justin Cappos wrote:
> > I am taking a look at some of the reproducible-builds packages that are
> > still failing and still see some issues that look like they may be from
> > memory addresses being output by sphinx.
> >
> > For example, django-celery outputs the following memory address
> > [...]
> >
> > I checked the github issue tracker and this doesn't seem to be listed
> > there, but it is possible that this issue is related:
> > https://github.com/sphinx-doc/sphinx/issues/1721.
>
> Can you please file a new bug for upstream Sphinx, or ping the existing
> issue you mentioned?
>
> I am afraid I don’t have time to look at this myself, so help from upstream
> would be nice.
>
> --
> Dmitry Shachnev
>

[Message part 2 (text/html, inline)]

Message #50 received at 822197@bugs.debian.org (full text, mbox, reply):

From: Dmitry Shachnev <mitya57@debian.org>

To: Justin Cappos <jcappos@nyu.edu>, 822197@bugs.debian.org

Subject: Re: Bug#822197: Memory addresses still seem to appear in sphinx docs...

Date: Tue, 23 May 2017 16:05:37 +0300

[Message part 1 (text/plain, inline)]

Hi Justin,

On Sun, May 07, 2017 at 11:39:50AM -0400, Justin Cappos wrote:
> Okay, I have opened a new bug about this in sphinx:
> https://github.com/sphinx-doc/sphinx/issues/3722

According to upstream response, this is because Celery is using a non-standard
format for memory addresses, and the regex in Sphinx does not catch it.

We will not add hacks for every specific project in Sphinx, so I don’t think
I can do anything about this.

--
Dmitry Shachnev

[signature.asc (application/pgp-signature, inline)]

Message #55 received at 822197@bugs.debian.org (full text, mbox, reply):

From: Justin Cappos <jcappos@nyu.edu>

To: Dmitry Shachnev <mitya57@debian.org>

Cc: 822197@bugs.debian.org

Subject: Re: Bug#822197: Memory addresses still seem to appear in sphinx docs...

Date: Tue, 23 May 2017 11:54:07 -0400

[Message part 1 (text/plain, inline)]

Okay, I will try to get this fixed further upstream in celery.

Justin

On Tue, May 23, 2017 at 9:05 AM, Dmitry Shachnev <mitya57@debian.org> wrote:

> Hi Justin,
>
> On Sun, May 07, 2017 at 11:39:50AM -0400, Justin Cappos wrote:
> > Okay, I have opened a new bug about this in sphinx:
> > https://github.com/sphinx-doc/sphinx/issues/3722
>
> According to upstream response, this is because Celery is using a
> non-standard
> format for memory addresses, and the regex in Sphinx does not catch it.
>
> We will not add hacks for every specific project in Sphinx, so I don’t
> think
> I can do anything about this.
>
> --
> Dmitry Shachnev
>

[Message part 2 (text/html, inline)]

Message #60 received at 822197@bugs.debian.org (full text, mbox, reply):

From: Justin Cappos <jcappos@nyu.edu>

To: Dmitry Shachnev <mitya57@debian.org>

Cc: 822197@bugs.debian.org

Subject: Re: Bug#822197: Memory addresses still seem to appear in sphinx docs...

Date: Tue, 23 May 2017 23:33:26 -0400

[Message part 1 (text/plain, inline)]

So I had a look and think there may be an easier way to handle this.  What
if the ' at ' was removed from the memory regex in
https://github.com/sphinx-doc/sphinx/blob/1.5.5/sphinx/util/inspect.py#L23

On Tue, May 23, 2017 at 11:54 AM, Justin Cappos <jcappos@nyu.edu> wrote:

> Okay, I will try to get this fixed further upstream in celery.
>
> Justin
>
> On Tue, May 23, 2017 at 9:05 AM, Dmitry Shachnev <mitya57@debian.org>
> wrote:
>
>> Hi Justin,
>>
>> On Sun, May 07, 2017 at 11:39:50AM -0400, Justin Cappos wrote:
>> > Okay, I have opened a new bug about this in sphinx:
>> > https://github.com/sphinx-doc/sphinx/issues/3722
>>
>> According to upstream response, this is because Celery is using a
>> non-standard
>> format for memory addresses, and the regex in Sphinx does not catch it.
>>
>> We will not add hacks for every specific project in Sphinx, so I don’t
>> think
>> I can do anything about this.
>>
>> --
>> Dmitry Shachnev
>>
>
>

[Message part 2 (text/html, inline)]

Message #65 received at 822197@bugs.debian.org (full text, mbox, reply):

From: Dmitry Shachnev <mitya57@debian.org>

To: Justin Cappos <jcappos@nyu.edu>

Cc: 822197@bugs.debian.org

Subject: Re: Bug#822197: Memory addresses still seem to appear in sphinx docs...

Date: Wed, 24 May 2017 11:45:22 +0300

[Message part 1 (text/plain, inline)]

On Tue, May 23, 2017 at 11:33:26PM -0400, Justin Cappos wrote:
> So I had a look and think there may be an easier way to handle this.  What
> if the ' at ' was removed from the memory regex in
> https://github.com/sphinx-doc/sphinx/blob/1.5.5/sphinx/util/inspect.py#L23

That would introduce many false positives.

What if one has a legit hex constant in the code, i.e. 0x12345678?

Also, please move this discussion upstream — I will not add any patches
unless they are accepted upstream.

--
Dmitry Shachnev

[signature.asc (application/pgp-signature, inline)]

Send a report that this bug log contains spam.