Debian Bug report logs - #802792
Prune Debian artifact directories from SOURCES.txt

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: dh-python: dh_python3 sometimes leaves empty python3.Y/dist-packages/*.egg-info directory

Date: Tue, 13 Oct 2015 13:49:56 -0400

Package: dh-python
Version: 2.20150826
Severity: normal

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Dear Maintainer,

dh_python3 sometimes leaves an empty
usr/lib/python3.Y/dist-packages/*.egg-info directory in python3-
binary packages.  I haven't yet tracked the problem down but I have
two test cases.  This bogus egg-info directory can cause problems
because it is on the 3.Y sys.path and can be found before the
usr/lib/python3/dist-packages/*.egg-info one.

In flufl.lock, the bogus egg-info file is present.  But in wheel, it
is not.  If you clone both those packages' git repos and build from
them you should be able to see the effects.


- -- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.2.0-1-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages dh-python depends on:
ii  libdpkg-perl  1.18.3
pn  python3:any   <none>

dh-python recommends no packages.

dh-python suggests no packages.

- -- no debconf information

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJWHURBAAoJEBJutWOnSwa/I0IP/2Y9ken2Fk9X3mOEvEkLyZ48
Idik73uOejEZJl3epyae5InSCw6BjkXhyR9aKm4MbKUBe265AP76soG6L5D2mq3s
ImCFJj/Zqhz4eIciphJKMDAY9lnZgPxjayUbPIE4JfxgCFocUW/nMN6h7zWiRGHC
oUyB6JIvcSh7Ear6AKaRBdK6H3fPd8WCfLHMj4cU9mnT3oBO6nEisGCaBHBeYXm5
DvuksvWUKCmDoBCDSadqfMLEnIvC7Akp9ta5z2YVkEGg8iVePVAL6aZVCZ4tK8hQ
V90AevIwQNrkIfpO4guFxvIBzDWiRLMvGO1+UnsBtgxJcR3ApCpBLBkLSKHz44/X
Wx/bqeVl2KEHeV+dIpZGFGGnQQmFIphhaceRE9fvFKTPoOkiOSy7B1izljPmVOGp
5wWnuEDv974Ql6WxvcBQ6QIhgkSLvf/GSQ8BhK1e6wdhVUWDeEHVffXM9tNp1J8m
0F+4jjm6mrYrVeKWb3Rq7jZF2wBgDFIZzQMRvYA9Bp3PM1mBQ0YynBeewWlEBUem
5VRE5dYInehO5SnSgBmbZT/XkaMMuXlkeE48hmXyJmc6lXAZK2wEMNuXxJNOF16o
ncGkeWv+GGUpCYfv5t5vzW9Pk/LKTL6RgcXO2+lsz1dUApU9QbYad/Y/hM0avAzS
ZwwOKFiPQLV9r4pwLELi
=Z3zD
-----END PGP SIGNATURE-----

Message #10 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: 801710@bugs.debian.org

Date: Tue, 20 Oct 2015 17:57:36 -0400

[Message part 1 (text/plain, inline)]

Stepping through share_files() in dhpython/fs.py yields a clue.  What appears
to happen is that share_files() is first called on the python3.4/dist-packages
tree and that gets copied into python3/dist-packages.

Then it gets called on the python3.5/dist-packages tree, and we find a
difference in d-p/flufl.lock-2.4.egg-info/SOURCES.txt (f.e.).  Indeed the
files are different:

# diff -u debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/SOURCES.txt debian/python3-flufl.lock/usr/lib/python3/dist-packages/flufl.lock-2.4.egg-info/SOURCES.txt 
--- debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/SOURCES.txt	2015-10-20 17:26:51.368888058 -0400
+++ debian/python3-flufl.lock/usr/lib/python3/dist-packages/flufl.lock-2.4.egg-info/SOURCES.txt	2015-10-20 17:26:51.628886127 -0400
@@ -20,6 +20,13 @@
 debian/python-flufl.lock/usr/lib/python2.7/dist-packages/flufl/lock/NEWS.rst
 debian/python-flufl.lock/usr/lib/python2.7/dist-packages/flufl/lock/README.rst
 debian/python-flufl.lock/usr/lib/python2.7/dist-packages/flufl/lock/docs/using.rst
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/SOURCES.txt
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/dependency_links.txt
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/namespace_packages.txt
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/top_level.txt
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl/lock/NEWS.rst
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl/lock/README.rst
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl/lock/docs/using.rst
 flufl/__init__.py
 flufl.lock.egg-info/PKG-INFO
 flufl.lock.egg-info/SOURCES.txt

It looks like the python3.5 version has 3.5 paths where as the 3.4 version
does not.  So share_files() returns False from the cmpfile() call at line 100,
and the fpath1, which is

debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/SOURCES.txt

doesn't get os.remove()'d.  That seems to keep the whole 3.5/.../egg-info
directory alive, although it does end up empty in the installed binary
package.  I don't know why it's empty, and I don't yet know why the are those
extra paths in the 3.5 version of SOURCES.txt.  My guess right now is that
because 3.4 is built *before* 3.5, the 3.4 version doesn't contain the 3.5
version.

I don't yet know what a good fix for that is.

[Message part 2 (application/pgp-signature, inline)]

Message #15 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: 801710@bugs.debian.org

Date: Tue, 20 Oct 2015 18:16:03 -0400

[Message part 1 (text/plain, inline)]

Actually, it looks like the extra python3.5 files are in the 3.4 dist-packages
and not the 3.5 dist-packages.  ?!

[Message part 2 (application/pgp-signature, inline)]

Message #20 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Piotr Ożarowski <piotr@debian.org>

To: Barry Warsaw <barry@debian.org>

Cc: 801710@bugs.debian.org

Subject: Re: Bug#801710: (no subject)

Date: Wed, 21 Oct 2015 15:13:55 +0200

> I don't yet know what a good fix for that is.

if you use pybuild, then I suggest to:

export PYBUILD_AFTER_INSTALL=find {destdir}{install_dir} -name SOURCES.txt -delete

and send yet another angry email to setuptools authors ;)

Message #25 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: Piotr Ożarowski <piotr@debian.org>

Cc: 801710@bugs.debian.org

Subject: Re: Bug#801710: (no subject)

Date: Wed, 21 Oct 2015 14:04:39 -0400

[Message part 1 (text/plain, inline)]

On Oct 21, 2015, at 03:13 PM, Piotr Ożarowski wrote:

>if you use pybuild, then I suggest to:
>
>export PYBUILD_AFTER_INSTALL=find {destdir}{install_dir} -name SOURCES.txt -delete
>
>and send yet another angry email to setuptools authors ;)

I don't think it's as simple as that, otherwise it should fail on all packages
with fairly simple setup.py's right?  Why do some fail (e.g. flufl.lock) but
others don't (e.g. webob)?

I don't want to send AAE until I understand the root cause.

[Message part 2 (application/pgp-signature, inline)]

Message #30 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: 801710@bugs.debian.org

Date: Thu, 22 Oct 2015 18:13:39 -0400

[Message part 1 (text/plain, inline)]

I know exactly what's going on now.

When setuptools wants to create the egg-info/SOURCES.txt file, it calls into
distutils to get a full manifest of all the files that are going to be
installed.  For better or worse, what distutils does is:

* walk the filesystem from cwd, getting a list of all the files matching the
  default pattern (e.g. .py).
* apply some default filters (e.g. pruning build/ and $vcs directories)
* filter the remaining list based on MANIFEST.in

The filtered list is what gets written to SOURCES.txt.

You can see how the first step will find all the Debian build artifacts in the
.pybuild/ and debian/ directories, and as each Python version-specific build
proceeds, all the previous build artifacts will show up in subsequent
SOURCES.txt files.

The reason this only affects some packages and not others has everything to do
with whether and how their upstreams have written their MANIFEST.in files.
The semantics of these files is defined here:

https://docs.python.org/2/distutils/sourcedist.html#manifest-template

If they've done something like

    include *.py

then you'll end up with every file that matches that pattern, including ones
in .pybuild/ and debian/.

If they've done something like

    recursive-include tests *

then you probably won't (modulo of course other commands in that file).  There
are some standard files and patterns that always get included or pruned.

What are the options for handling this?

1) Require upstreams to explicitly prune .pybuild/ and debian/

I don't like this because we really don't want to impose restrictions on
upstreams.  They might have a good reason for writing the MANIFEST.in as they
have, and they certainly didn't expect additional build artifacts from
downstreams to pollute their filesystem.  This is a Debian issue so Debian
should handle it.

2) Require maintainers to patch their package's MANIFEST.in.

I don't like this because it's difficult to notice, diagnose, and understand
how to fix.  As you'll see, I am proposing adding some diagnostics to
dh_python3, but even with that, I think it will be easy to miss.  Even if
caught, maintainers might not easily know how to fix the problem.  We could
document that, but it still requires additional quilt patches.  The Debian
tools are at the root this issue so I think they should handle it.

3) Modify setuptools/distutils to always prune .pybuild/ and debian/

We could do that this, but it means a delta from upstream that we would have
to carry forever.  Yuck.

4) Build the packages elsewhere.

I don't think this is feasible.  Maybe .pybuild/ could be put in the build/
directory, which is a distutils/setuptools artifact that's automatically
ignored, but we still have the debian/ directory to deal with and I think
that's not easily moved.

4) dh_python3 post-process the SOURCES.txt file.

I'm not a fan of this because of the extra complexity.  Also, see below.

6) Ignore differences in SOURCES.txt.

This seems to make the most sense to me.  We *already* delete the SOURCES.txt
from the final binary packages' egg-info directories, which makes sense,
because apt carries its own manifest of installed files.  If we're ignoring
SOURCES.txt in the final egg-info, why not ignore it for comparison purposes?

The only downside could be if the rewheel/dirtbike idea moves forward and
would require the egg-info/SOURCES.txt file to work, but since those (still
mythical tools) are distro-specific (maybe even Debian-specific), they can use
Debian tools to get a list of installed files to operate on.

So I advocate #6 and have a branch that implements exactly this.  I've
verified this fixes the problem for at least two affected packages.

You'll notice too that the branch adds some additional diagnostic output in
the case where the filecmp fails.  It prints a helpful message and a unified
diff to stderr so you can at least try to figure out why the bogus
python3.X/.../egg-info directory didn't get collapsed.

A proposed fix is in the bug801710 branch at
git+ssh://git.debian.org/git/dh-python/dh-python.git

[Message part 2 (application/pgp-signature, inline)]

Message #35 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Piotr Ożarowski <piotr@debian.org>

To: Barry Warsaw <barry@debian.org>

Cc: 801710@bugs.debian.org

Subject: Re: Bug#801710: egg-info garbage

Date: Fri, 23 Oct 2015 11:17:38 +0200

[Barry Warsaw, 2015-10-23]
> What are the options for handling this?
> 
> 1) Require upstreams to explicitly prune .pybuild/ and debian/

instead of .pybuild, I'd suggest upstream to ignore everything that
starts with a dot - we can create a patch to do that in our setuptools
package and another patch that extends it with "debian" dir (but keep
that one only for ourselves)

Message #40 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Piotr Ożarowski <piotr@debian.org>

To: 801710@bugs.debian.org

Cc: Barry Warsaw <barry@debian.org>

Subject: Re: Bug#801710: egg-info garbage

Date: Fri, 23 Oct 2015 11:25:12 +0200

> instead of .pybuild, I'd suggest upstream to ignore everything that
> starts with a dot - we can create a patch to do that in our setuptools
> package and another patch that extends it with "debian" dir (but keep
> that one only for ourselves)

FTR: I meant "upstream" as in setuptools/distutils authors here, not
every single module author out there 

i.e I think distutils should ignore dot files and we should extend it
to ignore debian dir

Message #53 received at 802792@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: 802792@bugs.debian.org

Date: Fri, 23 Oct 2015 12:41:51 -0400

[Message part 1 (text/plain, inline)]

Attached is a patch that fixes the setuptools egg_info command to ignore
Debian-specific artifacts.  These have to be ignored when creating the
SOURCES.txt manifest, otherwise dh_python3 will not collapse otherwise
equivalent package egg-info directories.  See also the original bug for
details.

[802792.diff (text/x-patch, attachment)]

[Message part 3 (application/pgp-signature, inline)]

Message #58 received at 802792@bugs.debian.org (full text, mbox, reply):

From: Matthias Klose <doko@ubuntu.com>

To: 802792@bugs.debian.org, Debian Bug Tracking System <control@bugs.debian.org>, Barry Warsaw <barry@debian.org>

Subject: Re: Bug#801710: egg-info garbage

Date: Sat, 24 Oct 2015 16:43:39 +0200

tags 802792 - patch
clone 802792 -1
reassign -1 dh-python
clone 802792 -2
reassign -2 python-defaults
thanks

I won't apply this patch to setuptools, however I think we should stop shipping 
SOURCES.txt everywhere.  There is no reason to have this file about sources in a 
binary package.

dh-python is already doing that (removing SOURCES.txt), so there is bug in 
dh-python and/or python-defaults that it first tries to compare that file, and 
only then removes it.  This should be fixed, not worked around like suggested in 
message #20.

I'm adding a patch to setuptools to don't install SOURCES.txt, when setup.py is 
called with --install-layout=deb.  This won't have any effect for pybuild, 
because pybuild nastily ignores --install-layout=deb.

Even for upstream, the patch looks wrong. You should update install_egg_info.py 
(skimmer) with a list of VCS names to ignore.

Message #67 received at 802792-close@bugs.debian.org (full text, mbox, reply):

From: Matthias Klose <doko@debian.org>

To: 802792-close@bugs.debian.org

Subject: Bug#802792: fixed in python-setuptools 18.4-2

Date: Sat, 24 Oct 2015 15:39:12 +0000

Source: python-setuptools
Source-Version: 18.4-2

We believe that the bug you reported is fixed in the latest version of
python-setuptools, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 802792@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Matthias Klose <doko@debian.org> (supplier of updated python-setuptools package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Sat, 24 Oct 2015 16:07:48 +0200
Source: python-setuptools
Binary: python-pkg-resources python-setuptools python-setuptools-doc python3-pkg-resources python3-setuptools pypy-pkg-resources pypy-setuptools python-setuptools-whl
Architecture: source all
Version: 18.4-2
Distribution: unstable
Urgency: medium
Maintainer: Matthias Klose <doko@debian.org>
Changed-By: Matthias Klose <doko@debian.org>
Description:
 pypy-pkg-resources - Package Discovery and Resource Access using pkg_resources
 pypy-setuptools - PyPy Distutils Enhancements
 python-pkg-resources - Package Discovery and Resource Access using pkg_resources
 python-setuptools - Python Distutils Enhancements
 python-setuptools-doc - Python Distutils Enhancements (documentation)
 python-setuptools-whl - Python Distutils Enhancements (wheel package)
 python3-pkg-resources - Package Discovery and Resource Access using pkg_resources
 python3-setuptools - Python3 Distutils Enhancements
Closes: 802792
Changes:
 python-setuptools (18.4-2) unstable; urgency=medium
 .
   * Don't install the SOURCES.txt egg-info file when setup.py is called
     with --install-layout=deb. Closes: #802792.
Checksums-Sha1:
 dd7c6a40a864e28b8142042b776af913344c782e 2399 python-setuptools_18.4-2.dsc
 7812a2cc00a52fbb86ae9365ab27b6efe46ca4f2 13500 python-setuptools_18.4-2.debian.tar.xz
 617bc9519343dff85c75f53c7202bb8a452845ec 70098 pypy-pkg-resources_18.4-2_all.deb
 bbd4427fefb05b0c2e12be4609d518ea6bd3258c 119182 pypy-setuptools_18.4-2_all.deb
 ed39246fb94b12fa4fafecf2ecb5804de0426677 99440 python-pkg-resources_18.4-2_all.deb
 b68e87f5b731a6ffafc3b1da2bfef4aa597d69d3 221466 python-setuptools-doc_18.4-2_all.deb
 e0a44d9dde1aadf7a9906d28cd6821b0f85ce71d 150982 python-setuptools-whl_18.4-2_all.deb
 1ff6812b1b2df489d0e49d1664d2387665061b5c 201030 python-setuptools_18.4-2_all.deb
 1b6162b7f93099735089a079d9acb483cd3f3601 70102 python3-pkg-resources_18.4-2_all.deb
 d8732c2d5f9e12cf4f5a77904f10c121bf0543bf 119146 python3-setuptools_18.4-2_all.deb
Checksums-Sha256:
 b9251dc57b70a5f4cdbf978f0eacf6c1fef89fa09a1f2d824b4dafb81064fc75 2399 python-setuptools_18.4-2.dsc
 244b85f6f2eeac6e128162ed9cd24e107885db90fcfa948ba9e585fe897a09b9 13500 python-setuptools_18.4-2.debian.tar.xz
 b1dd5f00c5aadba5561ce297008587572b312fb6429217db17bd3c9dd53c0d8f 70098 pypy-pkg-resources_18.4-2_all.deb
 b06950a8ff8c62f7f2e046c31e0d4ae8a14b2f57b730ab9e55b803e5e592542e 119182 pypy-setuptools_18.4-2_all.deb
 319714535b3fb3899d9f6429fb8029a24a30675cec87bc96a020df4eb0b9a561 99440 python-pkg-resources_18.4-2_all.deb
 73e38039f9dad0ee8fad418dce80496257cd134834b9b9572170d4b8f5548e5d 221466 python-setuptools-doc_18.4-2_all.deb
 5e8d840a82ddb5ddd7d0c9cbcde620f5a43a98c245cbdfbb3c1f38f6e0be0b35 150982 python-setuptools-whl_18.4-2_all.deb
 447c93929b6c2e6b33bdd0681b8dae1697f985ea4b8bfc1fd3deffea7ba7e704 201030 python-setuptools_18.4-2_all.deb
 979eda8c81cdde02ef87dfdb5f40efe19c176c1805d5bfae7a65d02956eddfff 70102 python3-pkg-resources_18.4-2_all.deb
 55eccb948f09821f8733cad3a82c4ee3125fbe4f95ad1beca05db73759b4afe2 119146 python3-setuptools_18.4-2_all.deb
Files:
 f85d1763ba2d82a124721cfb3219ea4a 2399 python optional python-setuptools_18.4-2.dsc
 284578b1d871c3e08359d6300358e7a4 13500 python optional python-setuptools_18.4-2.debian.tar.xz
 fa40e29f1cacbb19ab166acee791ad21 70098 python optional pypy-pkg-resources_18.4-2_all.deb
 93e219d040c645b3ce169afe54cc9d14 119182 python optional pypy-setuptools_18.4-2_all.deb
 763a0ec1f5a1a1b04058d0a2ba7bd6a4 99440 python optional python-pkg-resources_18.4-2_all.deb
 ac1370e96e473359c689430ebec83a57 221466 doc optional python-setuptools-doc_18.4-2_all.deb
 c0f672fb32a2c3452028998ddeef1285 150982 python optional python-setuptools-whl_18.4-2_all.deb
 798b0ed1ddd5d2d8ba5d8b5c8980088c 201030 python optional python-setuptools_18.4-2_all.deb
 d1c234084cf13c82451d28036cce70b3 70102 python optional python3-pkg-resources_18.4-2_all.deb
 15e0ac0f2b8d6fd86357b3404439c38d 119146 python optional python3-setuptools_18.4-2_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJWK53YAAoJEL1+qmB3j6b1GukQAKmSmjGloXz1Sa9/yBbQXoZn
Vzm+rxkQJE54hVGiWYuPK/MaYQMrEPrcDvQS1oSs5Ppyj5qIZQU7e+G4EDK5bkeP
+XZwrl31Taf2o/fjQAP60SgLj10Ai3qyuR3YtjClPofPpOh/ctEZEAgsM9s5wTBa
MHXojPCDMpto14RezNQOprvQZlnWDI70aaPlp73qFZ0kpkR6u6kVWC+jKsGSaxRd
wK5tZoStEak3K1AqKsAsFcutj5/0eCOJPVpdoM7b+PeX/uc/Ozeo1DHSUsJVI4JN
e+QLInl503bRbjRm7UEpKiHgo7Ro+FZN0vlEgRGhe2cvghKjQHExr8AYHzWPUQhj
6lTrMSpldBWx0eskwxRuMCm1SFAgH6W6jq6QObEOwF9s13o4UJ7av+6zdWYs9Hak
sBQs8BE+vaBNQAIoJuTiCozquvxQ3NK9nKe3+4d9BTQr5zcamx6Xw72u/M9eqbNd
Ot/jwwtQjWCwhJ1YqgGXxgJIoiTKL+R1lD9ktfhDVY8dCqhGU3Y7liBQgdQwFFXW
q6mw9kmloZ+qhZqIyzTsIG9tAmZ3y5A40e2nSrMrJvSkQhGmURUsDWrhkoIkEaNi
uZdH8aJtX/v84Cpt9yRdXvZzsbQeaCR0pBOwnA2tTAXbwSyM/yM73joR000aBTL0
I8RCQs3ibJf0A7HdhS8x
=QUda
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.