Debian Bug report logs - #801710
Provide better diagnostics when python3.x differences are found

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: dh-python: dh_python3 sometimes leaves empty python3.Y/dist-packages/*.egg-info directory

Date: Tue, 13 Oct 2015 13:49:56 -0400

Package: dh-python
Version: 2.20150826
Severity: normal

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Dear Maintainer,

dh_python3 sometimes leaves an empty
usr/lib/python3.Y/dist-packages/*.egg-info directory in python3-
binary packages.  I haven't yet tracked the problem down but I have
two test cases.  This bogus egg-info directory can cause problems
because it is on the 3.Y sys.path and can be found before the
usr/lib/python3/dist-packages/*.egg-info one.

In flufl.lock, the bogus egg-info file is present.  But in wheel, it
is not.  If you clone both those packages' git repos and build from
them you should be able to see the effects.


- -- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.2.0-1-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages dh-python depends on:
ii  libdpkg-perl  1.18.3
pn  python3:any   <none>

dh-python recommends no packages.

dh-python suggests no packages.

- -- no debconf information

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJWHURBAAoJEBJutWOnSwa/I0IP/2Y9ken2Fk9X3mOEvEkLyZ48
Idik73uOejEZJl3epyae5InSCw6BjkXhyR9aKm4MbKUBe265AP76soG6L5D2mq3s
ImCFJj/Zqhz4eIciphJKMDAY9lnZgPxjayUbPIE4JfxgCFocUW/nMN6h7zWiRGHC
oUyB6JIvcSh7Ear6AKaRBdK6H3fPd8WCfLHMj4cU9mnT3oBO6nEisGCaBHBeYXm5
DvuksvWUKCmDoBCDSadqfMLEnIvC7Akp9ta5z2YVkEGg8iVePVAL6aZVCZ4tK8hQ
V90AevIwQNrkIfpO4guFxvIBzDWiRLMvGO1+UnsBtgxJcR3ApCpBLBkLSKHz44/X
Wx/bqeVl2KEHeV+dIpZGFGGnQQmFIphhaceRE9fvFKTPoOkiOSy7B1izljPmVOGp
5wWnuEDv974Ql6WxvcBQ6QIhgkSLvf/GSQ8BhK1e6wdhVUWDeEHVffXM9tNp1J8m
0F+4jjm6mrYrVeKWb3Rq7jZF2wBgDFIZzQMRvYA9Bp3PM1mBQ0YynBeewWlEBUem
5VRE5dYInehO5SnSgBmbZT/XkaMMuXlkeE48hmXyJmc6lXAZK2wEMNuXxJNOF16o
ncGkeWv+GGUpCYfv5t5vzW9Pk/LKTL6RgcXO2+lsz1dUApU9QbYad/Y/hM0avAzS
ZwwOKFiPQLV9r4pwLELi
=Z3zD
-----END PGP SIGNATURE-----

Message #10 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: 801710@bugs.debian.org

Date: Tue, 20 Oct 2015 17:57:36 -0400

[Message part 1 (text/plain, inline)]

Stepping through share_files() in dhpython/fs.py yields a clue.  What appears
to happen is that share_files() is first called on the python3.4/dist-packages
tree and that gets copied into python3/dist-packages.

Then it gets called on the python3.5/dist-packages tree, and we find a
difference in d-p/flufl.lock-2.4.egg-info/SOURCES.txt (f.e.).  Indeed the
files are different:

# diff -u debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/SOURCES.txt debian/python3-flufl.lock/usr/lib/python3/dist-packages/flufl.lock-2.4.egg-info/SOURCES.txt 
--- debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/SOURCES.txt	2015-10-20 17:26:51.368888058 -0400
+++ debian/python3-flufl.lock/usr/lib/python3/dist-packages/flufl.lock-2.4.egg-info/SOURCES.txt	2015-10-20 17:26:51.628886127 -0400
@@ -20,6 +20,13 @@
 debian/python-flufl.lock/usr/lib/python2.7/dist-packages/flufl/lock/NEWS.rst
 debian/python-flufl.lock/usr/lib/python2.7/dist-packages/flufl/lock/README.rst
 debian/python-flufl.lock/usr/lib/python2.7/dist-packages/flufl/lock/docs/using.rst
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/SOURCES.txt
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/dependency_links.txt
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/namespace_packages.txt
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/top_level.txt
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl/lock/NEWS.rst
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl/lock/README.rst
+debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl/lock/docs/using.rst
 flufl/__init__.py
 flufl.lock.egg-info/PKG-INFO
 flufl.lock.egg-info/SOURCES.txt

It looks like the python3.5 version has 3.5 paths where as the 3.4 version
does not.  So share_files() returns False from the cmpfile() call at line 100,
and the fpath1, which is

debian/python3-flufl.lock/usr/lib/python3.5/dist-packages/flufl.lock-2.4.egg-info/SOURCES.txt

doesn't get os.remove()'d.  That seems to keep the whole 3.5/.../egg-info
directory alive, although it does end up empty in the installed binary
package.  I don't know why it's empty, and I don't yet know why the are those
extra paths in the 3.5 version of SOURCES.txt.  My guess right now is that
because 3.4 is built *before* 3.5, the 3.4 version doesn't contain the 3.5
version.

I don't yet know what a good fix for that is.

[Message part 2 (application/pgp-signature, inline)]

Message #15 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: 801710@bugs.debian.org

Date: Tue, 20 Oct 2015 18:16:03 -0400

[Message part 1 (text/plain, inline)]

Actually, it looks like the extra python3.5 files are in the 3.4 dist-packages
and not the 3.5 dist-packages.  ?!

[Message part 2 (application/pgp-signature, inline)]

Message #20 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Piotr Ożarowski <piotr@debian.org>

To: Barry Warsaw <barry@debian.org>

Cc: 801710@bugs.debian.org

Subject: Re: Bug#801710: (no subject)

Date: Wed, 21 Oct 2015 15:13:55 +0200

> I don't yet know what a good fix for that is.

if you use pybuild, then I suggest to:

export PYBUILD_AFTER_INSTALL=find {destdir}{install_dir} -name SOURCES.txt -delete

and send yet another angry email to setuptools authors ;)

Message #25 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: Piotr Ożarowski <piotr@debian.org>

Cc: 801710@bugs.debian.org

Subject: Re: Bug#801710: (no subject)

Date: Wed, 21 Oct 2015 14:04:39 -0400

[Message part 1 (text/plain, inline)]

On Oct 21, 2015, at 03:13 PM, Piotr Ożarowski wrote:

>if you use pybuild, then I suggest to:
>
>export PYBUILD_AFTER_INSTALL=find {destdir}{install_dir} -name SOURCES.txt -delete
>
>and send yet another angry email to setuptools authors ;)

I don't think it's as simple as that, otherwise it should fail on all packages
with fairly simple setup.py's right?  Why do some fail (e.g. flufl.lock) but
others don't (e.g. webob)?

I don't want to send AAE until I understand the root cause.

[Message part 2 (application/pgp-signature, inline)]

Message #30 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: 801710@bugs.debian.org

Date: Thu, 22 Oct 2015 18:13:39 -0400

[Message part 1 (text/plain, inline)]

I know exactly what's going on now.

When setuptools wants to create the egg-info/SOURCES.txt file, it calls into
distutils to get a full manifest of all the files that are going to be
installed.  For better or worse, what distutils does is:

* walk the filesystem from cwd, getting a list of all the files matching the
  default pattern (e.g. .py).
* apply some default filters (e.g. pruning build/ and $vcs directories)
* filter the remaining list based on MANIFEST.in

The filtered list is what gets written to SOURCES.txt.

You can see how the first step will find all the Debian build artifacts in the
.pybuild/ and debian/ directories, and as each Python version-specific build
proceeds, all the previous build artifacts will show up in subsequent
SOURCES.txt files.

The reason this only affects some packages and not others has everything to do
with whether and how their upstreams have written their MANIFEST.in files.
The semantics of these files is defined here:

https://docs.python.org/2/distutils/sourcedist.html#manifest-template

If they've done something like

    include *.py

then you'll end up with every file that matches that pattern, including ones
in .pybuild/ and debian/.

If they've done something like

    recursive-include tests *

then you probably won't (modulo of course other commands in that file).  There
are some standard files and patterns that always get included or pruned.

What are the options for handling this?

1) Require upstreams to explicitly prune .pybuild/ and debian/

I don't like this because we really don't want to impose restrictions on
upstreams.  They might have a good reason for writing the MANIFEST.in as they
have, and they certainly didn't expect additional build artifacts from
downstreams to pollute their filesystem.  This is a Debian issue so Debian
should handle it.

2) Require maintainers to patch their package's MANIFEST.in.

I don't like this because it's difficult to notice, diagnose, and understand
how to fix.  As you'll see, I am proposing adding some diagnostics to
dh_python3, but even with that, I think it will be easy to miss.  Even if
caught, maintainers might not easily know how to fix the problem.  We could
document that, but it still requires additional quilt patches.  The Debian
tools are at the root this issue so I think they should handle it.

3) Modify setuptools/distutils to always prune .pybuild/ and debian/

We could do that this, but it means a delta from upstream that we would have
to carry forever.  Yuck.

4) Build the packages elsewhere.

I don't think this is feasible.  Maybe .pybuild/ could be put in the build/
directory, which is a distutils/setuptools artifact that's automatically
ignored, but we still have the debian/ directory to deal with and I think
that's not easily moved.

4) dh_python3 post-process the SOURCES.txt file.

I'm not a fan of this because of the extra complexity.  Also, see below.

6) Ignore differences in SOURCES.txt.

This seems to make the most sense to me.  We *already* delete the SOURCES.txt
from the final binary packages' egg-info directories, which makes sense,
because apt carries its own manifest of installed files.  If we're ignoring
SOURCES.txt in the final egg-info, why not ignore it for comparison purposes?

The only downside could be if the rewheel/dirtbike idea moves forward and
would require the egg-info/SOURCES.txt file to work, but since those (still
mythical tools) are distro-specific (maybe even Debian-specific), they can use
Debian tools to get a list of installed files to operate on.

So I advocate #6 and have a branch that implements exactly this.  I've
verified this fixes the problem for at least two affected packages.

You'll notice too that the branch adds some additional diagnostic output in
the case where the filecmp fails.  It prints a helpful message and a unified
diff to stderr so you can at least try to figure out why the bogus
python3.X/.../egg-info directory didn't get collapsed.

A proposed fix is in the bug801710 branch at
git+ssh://git.debian.org/git/dh-python/dh-python.git

[Message part 2 (application/pgp-signature, inline)]

Message #35 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Piotr Ożarowski <piotr@debian.org>

To: Barry Warsaw <barry@debian.org>

Cc: 801710@bugs.debian.org

Subject: Re: Bug#801710: egg-info garbage

Date: Fri, 23 Oct 2015 11:17:38 +0200

[Barry Warsaw, 2015-10-23]
> What are the options for handling this?
> 
> 1) Require upstreams to explicitly prune .pybuild/ and debian/

instead of .pybuild, I'd suggest upstream to ignore everything that
starts with a dot - we can create a patch to do that in our setuptools
package and another patch that extends it with "debian" dir (but keep
that one only for ourselves)

Message #40 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Piotr Ożarowski <piotr@debian.org>

To: 801710@bugs.debian.org

Cc: Barry Warsaw <barry@debian.org>

Subject: Re: Bug#801710: egg-info garbage

Date: Fri, 23 Oct 2015 11:25:12 +0200

> instead of .pybuild, I'd suggest upstream to ignore everything that
> starts with a dot - we can create a patch to do that in our setuptools
> package and another patch that extends it with "debian" dir (but keep
> that one only for ourselves)

FTR: I meant "upstream" as in setuptools/distutils authors here, not
every single module author out there 

i.e I think distutils should ignore dot files and we should extend it
to ignore debian dir

Message #49 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: 801710@bugs.debian.org

Date: Fri, 23 Oct 2015 12:34:36 -0400

[Message part 1 (text/plain, inline)]

After talking with Donald Stufft, we agreed that ignoring all dot-directories
isn't a good approach, and not something upstream would be interested in.
I've cloned this bug onto python-setuptools and am preparing a patch that will
prune debian/ and .pybuild/ for that package.  It's a two-line change and I've
verified that it fixes the problem for zope.testing and flufl.enum, and
doesn't otherwise break webob (which was not affected).

However, I still think it's useful to add the additional diagnostics to
dh_python3 and I've retitled this bug accordingly.  This provides valuable
feedback when the unexpected happens.  I've pushed an update to the git branch
previously mentioned, but I'll attach a diff here as well (sans the changelog
change).

[801710.diff (text/x-patch, attachment)]

[Message part 3 (application/pgp-signature, inline)]

Message #54 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Piotr Ożarowski <piotr@debian.org>

To: Barry Warsaw <barry@debian.org>

Cc: 801710@bugs.debian.org

Subject: Re: Bug#801710: (no subject)

Date: Fri, 23 Oct 2015 19:03:21 +0200

[Barry Warsaw, 2015-10-23]
> However, I still think it's useful to add the additional diagnostics to
> dh_python3 and I've retitled this bug accordingly.  This provides valuable

most if not all files that differ outside egg-info dir are valid cases
(version specific changes, so files, ...) so I don't think standard
error is the right place (and I definitely don't want to see changes in
.so files in build log)

Message #59 received at 801710@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

Cc: 801710@bugs.debian.org

Subject: Re: Bug#801710: (no subject)

Date: Fri, 23 Oct 2015 13:13:00 -0400

[Message part 1 (text/plain, inline)]

On Oct 23, 2015, at 07:03 PM, Piotr Ożarowski wrote:

>most if not all files that differ outside egg-info dir are valid cases
>(version specific changes, so files, ...) so I don't think standard
>error is the right place (and I definitely don't want to see changes in
>.so files in build log)

I guess it would be okay to only output this when PYBUILD_VERBOSE=1.  I
suppose that means the diff lines would have to be joined and written to the
log, but at what level?

[Message part 2 (application/pgp-signature, inline)]

Message #64 received at 801710@bugs.debian.org (full text, mbox, reply):

From: "USPS SameDay" <jason.knight@klaudija-kosanovic.de>

To: 801710@bugs.debian.org

Subject: Parcel ID002917650 delivery problems, please review

Date: Wed, 25 Jan 2017 03:10:36 +0100

[Message part 1 (text/plain, inline)]

Dear Customer,

USPS courier was unable to contact you for your parcel delivery.

You can download the shipment label attached!

Thank you,
Jason Knight,
USPS Chief Office Manager.

[Ground-Label-002917650.zip (application/zip, attachment)]

Message #69 received at 801710@bugs.debian.org (full text, mbox, reply):

From: webserver <web@maguji.han-solo.net>

To: 801710@bugs.debian.org

Subject: New status of your UPS delivery (code: 01918601)

Date: Fri, 24 Feb 2017 12:08:21 +0100

[Message part 1 (text/plain, inline)]

Dear Customer,

This is to confirm that your item has been shipped at February 20.

Please check the attachment for details!

With many thanks,
Jerome Holloway,
UPS Chief Support Manager.

[UPS-Delivery-01918601.zip (application/zip, attachment)]

Send a report that this bug log contains spam.