Debian Bug report logs - #685167
py3clean removes *.pyc of foreign packages

Toggle useless messages

Message #3 received at submit@bugs.debian.org (full text, mbox, reply):

From: Jakub Wilk <jwilk@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: py3clean removes *.pyc of foreign packages

Date: Fri, 17 Aug 2012 18:02:02 +0200

Package: python3
Version: 3.2.3-5
Severity: important

py3clean removes *.pyc files that belong to other packages than 
requested:

# apt-get install -qq python3-pkg-resources python3-gnupg
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package python3-gnupg.
(Reading database ... 11713 files and directories currently installed.)
Unpacking python3-gnupg (from .../python3-gnupg_0.3.0-1_all.deb) ...
Selecting previously unselected package python3-pkg-resources.
Unpacking python3-pkg-resources (from .../python3-pkg-resources_0.6.24-1_all.deb) ...
Setting up python3-gnupg (0.3.0-1) ...
Setting up python3-pkg-resources (0.6.24-1) ...

# ls -l /usr/lib/python3/dist-packages/__pycache__/
total 156
-rw-r--r-- 1 root root  41019 Aug 17 15:57 gnupg.cpython-32.pyc
-rw-r--r-- 1 root root 112438 Aug 17 15:57 pkg_resources.cpython-32.pyc

# dpkg -P python3-pkg-resources
(Reading database ... 11722 files and directories currently installed.)
Removing python3-pkg-resources ...

# ls -l /usr/lib/python3/dist-packages/__pycache__/
ls: cannot access /usr/lib/python3/dist-packages/__pycache__/: No such file or directory


-- System Information:
Debian Release: wheezy/sid
  APT prefers unstable
  APT policy: (990, 'unstable'), (500, 'experimental')
Architecture: i386 (x86_64)
Foreign Architectures: amd64

Kernel: Linux 3.5-trunk-amd64 (SMP w/2 CPU cores)
Locale: LANG=C, LC_CTYPE=pl_PL.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages python3 depends on:
ii  python3-minimal  3.2.3-5
ii  python3.2        3.2.3-3

-- 
Jakub Wilk

Message #8 received at 685167@bugs.debian.org (full text, mbox, reply):

From: Dmitry Shachnev <mitya57@gmail.com>

To: 685167@bugs.debian.org

Cc: 685167-submitter@bugs.debian.org, piotr@debian.org

Subject: Re: py3clean removes *.pyc of foreign packages

Date: Thu, 13 Sep 2012 14:04:52 +0400

[Message part 1 (text/plain, inline)]

Control: tags -1 +patch

The attached patch adds a special case for files in
*/python3/dist-packages directories to py3clean.

--
Dmitry Shachnev

[fix-685167.patch (text/x-patch, attachment)]

Message #18 received at 685167@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@python.org>

To: 685167@bugs.debian.org

Date: Tue, 18 Sep 2012 11:05:40 -0400

[Message part 1 (text/plain, inline)]

So I agree that this is a legitimate bug, and the patch does appear to fix the
problem, at least in the use case given.  I'm just not sure it does it in the
right way.

I'm confused by a few things in the original code.  Looking at the docstring
for destroyer() we see:

    """Remove every .py[co] file associated to received .py file.

    :param magic_tag: if None, removes __pycache__ directories,
        if False, removes python3.1's .pyc files only,
        otherwise removes given magic tag from __pycache__ directory
    :type magic_tag: None or False or str"""

Dmitry's patch changes this documented behavior because when magic_tag=None,
it will *not* remove the __pycache__ directory if that happens to be
/usr/lib/python3/dist-packages.  So if the patch is accepted, the docstring at
least has to be updated.

(Aside: if you really wanted to remove the entire __pycache__ directory and
everything in it, I wonder why one call to shutil.rmtree() wasn't used
instead?)

(Aside2: imp.cache_from_source() would give you the tagged .pyc file to
remove, but i guess of course the problem is that the version of Python 3
running py3clean won't necessarily match the version of pyc files you want to
clean up.  Still, I might use imp.cache_from_source() first, then split the
basename and substitute for the tags, or glob for the tags part of the
resulting file name.)

destroyer()'s documented behavior only makes sense when you're dealing with a
private package.  Otherwise, for public packages, it should never be correct
to remove the __pycache__ directory and all its contents.  An empty
/usr/lib/python3/dist-packages/__pycache__ shouldn't ever hurt.

Unfortunately, py3clean doesn't know whether it's dealing with a private or
public package.  I don't think `-p package` gives it enough information,
though perhaps there's a way to find out from the package name that I'm
missing.

The other thing I'd do is instead of hard-coding python3/dist-packages, I'd
probably compare the resulting directory against the list given by
site.getsitepackages():

>>> import site
>>> site.getsitepackages()
['/usr/local/lib/python3.2/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.2/dist-packages', '/usr/lib/dist-python']

If the directory being cleaned were any of these, then don't empty them out.

So I guess if no one else has any other suggestions, then the following
changes should be made to the patch, after which it could be applied:

* compare against site.getsitepackages() for directories that should not be
  removed.
* update the docstring to indicate that site package directories are not
  removed even if magic_tag is None

[signature.asc (application/pgp-signature, attachment)]

Message #23 received at 685167@bugs.debian.org (full text, mbox, reply):

From: Dmitry Shachnev <mitya57@gmail.com>

To: 685167@bugs.debian.org

Cc: Barry Warsaw <barry@python.org>

Subject: Re: Bug#685167: py3clean removes *.pyc of foreign packages

Date: Tue, 18 Sep 2012 19:37:37 +0400

[Message part 1 (text/plain, inline)]

On Tue, 18/09/2012 11:05 -0400, Barry Warsaw wrote:
> So I guess if no one else has any other suggestions, then the following
> changes should be made to the patch, after which it could be applied:
> 
> * compare against site.getsitepackages() for directories that should not be
>   removed.
> * update the docstring to indicate that site package directories are not
>   removed even if magic_tag is None

Done, the updated patch attached. Thanks for the suggestions!

Also, I've updated my branch at lp:~mitya57/+junk/python3-defaults so
that it can be easily merged.

--
Dmitry Shachnev

[fix-685167-v2.patch (text/x-patch, attachment)]

Message #28 received at 685167@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@python.org>

To: Dmitry Shachnev <mitya57@gmail.com>

Cc: 685167@bugs.debian.org

Subject: Re: Bug#685167: py3clean removes *.pyc of foreign packages

Date: Tue, 18 Sep 2012 13:59:16 -0400

[Message part 1 (text/plain, inline)]

On Sep 18, 2012, at 07:37 PM, Dmitry Shachnev wrote:

>On Tue, 18/09/2012 11:05 -0400, Barry Warsaw wrote:
>> So I guess if no one else has any other suggestions, then the following
>> changes should be made to the patch, after which it could be applied:
>> 
>> * compare against site.getsitepackages() for directories that should not be
>>   removed.
>> * update the docstring to indicate that site package directories are not
>>   removed even if magic_tag is None
>
>Done, the updated patch attached. Thanks for the suggestions!
>
>Also, I've updated my branch at lp:~mitya57/+junk/python3-defaults so
>that it can be easily merged.

Thanks for making the changes.  Note that the above branch has some other
changes which aren't related to this bug.  I'm disinclined to include those
fixes here for that reason, and also because i think there are better ways to
do it, e.g. by using a context manager to handle the automatic closing of the
files, instead of relying on potentially buggy explicit .closes()

[signature.asc (application/pgp-signature, attachment)]

Message #33 received at 685167@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@python.org>

To: Dmitry Shachnev <mitya57@gmail.com>

Cc: 685167@bugs.debian.org

Subject: Re: Bug#685167: py3clean removes *.pyc of foreign packages

Date: Tue, 18 Sep 2012 14:37:03 -0400

[Message part 1 (text/plain, inline)]

On Sep 18, 2012, at 07:37 PM, Dmitry Shachnev wrote:

>On Tue, 18/09/2012 11:05 -0400, Barry Warsaw wrote:
>> So I guess if no one else has any other suggestions, then the following
>> changes should be made to the patch, after which it could be applied:
>> 
>> * compare against site.getsitepackages() for directories that should not be
>>   removed.
>> * update the docstring to indicate that site package directories are not
>>   removed even if magic_tag is None
>
>Done, the updated patch attached. Thanks for the suggestions!
>
>Also, I've updated my branch at lp:~mitya57/+junk/python3-defaults so
>that it can be easily merged.

I'll commit this to alioth bzr after I do some additional local testing, but
it generally looks pretty good.  Thanks!  I made a few minor changes, such as
using os.path.splitext() instead of assuming the extension is 3 characters
long (it probably always will be, but splitext() is safer).  I also added a
bunch of comments so that it's clearer what's going on.

I'll have to request sponsorship since I can't upload this myself.

[signature.asc (application/pgp-signature, attachment)]

Message #38 received at 685167@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@python.org>

To: Dmitry Shachnev <mitya57@gmail.com>

Cc: 685167@bugs.debian.org

Subject: Re: Bug#685167: py3clean removes *.pyc of foreign packages

Date: Tue, 18 Sep 2012 15:48:22 -0400

[Message part 1 (text/plain, inline)]

Okay, I don't have permission either to commit changes to pkg-python.  My
branch, ready for merging and sponsorship, is at

bzr branch bzr+ssh://bzr.debian.org/home/users/warsaw-guest/public_bzr/pkg-python/bug-685167

dget http://barry.warsaw.us/debian/python3-defaults_3.2.3-6.dsc

[signature.asc (application/pgp-signature, attachment)]

Message #43 received at 685167@bugs.debian.org (full text, mbox, reply):

From: Dmitry Shachnev <mitya57@gmail.com>

To: 685167@bugs.debian.org

Cc: Barry Warsaw <barry@python.org>, Scott Kitterman <debian@kitterman.com>

Subject: Re: Bug#685167: py3clean removes *.pyc of foreign packages

Date: Wed, 19 Sep 2012 07:20:35 +0400

@Scott: please merge Barry's branch :)

I also think we should use a different version number (something like
3.2.3.6) to fix bug #684431.

--
Dmitry Shachnev

2012/9/18, Barry Warsaw <barry@python.org>:
> Okay, I don't have permission either to commit changes to pkg-python.  My
> branch, ready for merging and sponsorship, is at
>
> bzr branch
> bzr+ssh://bzr.debian.org/home/users/warsaw-guest/public_bzr/pkg-python/bug-685167
>
> dget http://barry.warsaw.us/debian/python3-defaults_3.2.3-6.dsc
>
>

Message #48 received at 685167@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@python.org>

To: Dmitry Shachnev <mitya57@gmail.com>

Cc: 685167@bugs.debian.org, Scott Kitterman <debian@kitterman.com>, Piotr Ożarowski <piotr@debian.org>

Subject: Re: Bug#685167: py3clean removes *.pyc of foreign packages

Date: Wed, 19 Sep 2012 09:44:43 -0400

[Message part 1 (text/plain, inline)]

On Sep 19, 2012, at 07:20 AM, Dmitry Shachnev wrote:

>@Scott: please merge Barry's branch :)
>
>I also think we should use a different version number (something like
>3.2.3.6) to fix bug #684431.

Piotr mentioned he was going to try to get to it today.

[signature.asc (application/pgp-signature, attachment)]

Message #55 received at 685167-close@bugs.debian.org (full text, mbox, reply):

From: Piotr Ożarowski <piotr@debian.org>

To: 685167-close@bugs.debian.org

Subject: Bug#685167: fixed in python3-defaults 3.2.3-6

Date: Sun, 21 Oct 2012 22:03:21 +0000

Source: python3-defaults
Source-Version: 3.2.3-6

We believe that the bug you reported is fixed in the latest version of
python3-defaults, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 685167@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Piotr Ożarowski <piotr@debian.org> (supplier of updated python3-defaults package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Sun, 21 Oct 2012 21:29:45 +0200
Source: python3-defaults
Binary: python3 python3-minimal python3-examples python3-dev idle3 python3-doc python3-dbg python3-all python3-all-dev python3-all-dbg
Architecture: source all
Version: 3.2.3-6
Distribution: unstable
Urgency: low
Maintainer: Matthias Klose <doko@debian.org>
Changed-By: Piotr Ożarowski <piotr@debian.org>
Description: 
 idle3      - IDE for Python using Tkinter (default version)
 python3    - interactive high-level object-oriented language (default python3
 python3-all - package depending on all supported Python 3 runtime versions
 python3-all-dbg - package depending on all supported Python 3 debugging packages
 python3-all-dev - package depending on all supported Python 3 development packages
 python3-dbg - debug build of the Python 3 Interpreter (version 3.2)
 python3-dev - header files and a static library for Python (default)
 python3-doc - documentation for the high-level object-oriented language Python
 python3-examples - examples for the Python language (default version)
 python3-minimal - minimal subset of the Python language (default python3 version)
Closes: 672178 685067 685167 686587 687060 690259
Changes: 
 python3-defaults (3.2.3-6) unstable; urgency=low
 .
   [ Piotr Ożarowski ]
   * dh_python3
     - ignore empty files while trying to normalize shebangs
     - mention the right file (debian/py3dist-overrides) while warning about
       guessed dependency - thanks to Sebastian Ramacher for the patch
       (Closes: #685067)
     - fix generating dependencies when maximum version is specified
       (Closes: #687060)
   * py3versions.py: fix parsing DEBPYTHON3_SUPPORTED env. variable
     (versions should be separated using comma, as in debian_defaults
     config file)
   * py3versions, debpython: close previously opened files - thanks to Dmitry
     Shachnev for the patch (Closes: #686587)
 .
   [ Dmitry Shachnev ]
   [ Barry Warsaw ]
   * py3clean: Don't remove everything in the __pycache__ directory for
     system site package directories. Prevents bogus cleaning of unrelated
     *.pyc files. Original patch by Dmitry, with review, clean up, small
     modifications, and additional comments by Barry. (Closes: #685167)
 .
   [ Barry Warsaw ]
   * dh_python3: Rework calculation of extension tags to add support for
     Python 3.3's different suffixes, and to allow for unadorned .so files
     to assume they are built with the default Python 3 version.
     Closes: 672178
   * README.derivatives: It is no longer necessary to edit
     debian/py3versions.py since the values are taken from
     debian_defaults. Also added some text on how to separate the
     specification when multiple versions are supported.
   * Makefile: Fix the nosetests3 command. Closes: 690259
Checksums-Sha1: 
 f159abbfdeaf73cb305179161c530ff6e0f7a532 2250 python3-defaults_3.2.3-6.dsc
 a04e3aeb1cdc0dfdd2cfbbe7c8ddca0fbf40a381 142250 python3-defaults_3.2.3-6.tar.gz
 bcdc00cbe0918c52991ac2c1297e4956b8619c17 43366 python3_3.2.3-6_all.deb
 1962245c42a83a6ae9239bfbf038aab321cdc6bd 20948 python3-minimal_3.2.3-6_all.deb
 dead844ec2ce82f3486aea045da6e839cfef19d4 856 python3-examples_3.2.3-6_all.deb
 1eb84f3ff0890145d8bd748972d7ef36eb8916cd 1062 python3-dev_3.2.3-6_all.deb
 25299c18e57a7c551993452fe12af8560325163a 3152 idle3_3.2.3-6_all.deb
 d0297693bc3f1ce492262934095591e986e506bd 20580 python3-doc_3.2.3-6_all.deb
 254ea0b8ed0f5727c74d9bfdc0682c898cb98fb1 1130 python3-dbg_3.2.3-6_all.deb
 8e4a671289c84011c19b1395b13c3579cf0b8dfc 866 python3-all_3.2.3-6_all.deb
 3ee1175a2b485e466e2e983b255ad4b38c362090 882 python3-all-dev_3.2.3-6_all.deb
 c1ee758eee0372864534f8831f6e54af5c8152c6 880 python3-all-dbg_3.2.3-6_all.deb
Checksums-Sha256: 
 bc0d268b4d47a45d02178e0fe873c5a3f63a5e483ef13309fa23c81bfd82e8e1 2250 python3-defaults_3.2.3-6.dsc
 a1545f5fd45994817f9cb7fc4fd7fe17dd1108377d9a6fc3e97ef41520565b4a 142250 python3-defaults_3.2.3-6.tar.gz
 00e1191bc68d1d6608cd9cba37ecab3065fd3b899b2467d2ad6e2c7a7e735b87 43366 python3_3.2.3-6_all.deb
 56acc5d4a86138ecac049c8edac41ab37e2f2ba18a6dc4f6b29be395e7b9e581 20948 python3-minimal_3.2.3-6_all.deb
 b6b649ac819e61ad2929230ec3a9717d42a09c4ff75fd746dd5d108c17a4e823 856 python3-examples_3.2.3-6_all.deb
 c4b5cd3897826a10a9881b863574f16e25438bdef16fa825f8bbfb5be4d17364 1062 python3-dev_3.2.3-6_all.deb
 056cd90178b8fc07c6f100cf1f13467e258ed8671bd809500531e88279f614c6 3152 idle3_3.2.3-6_all.deb
 512cc66b7ba522d3641172dcc904c04b03f0ec3388d123573fd2cdeb5ffd5793 20580 python3-doc_3.2.3-6_all.deb
 619b974df2b61a3473fa807f4168de3b7b7824c303dfe153d42865a00c2b3683 1130 python3-dbg_3.2.3-6_all.deb
 a4eaed140d01801ec827a932791785c7db1c2c18245c2682d05c465d09c5e3e8 866 python3-all_3.2.3-6_all.deb
 0ecae0fbb2276620d03ad6c573f18f1eba38d3b0060bb17e2ac82409c4a7cd3e 882 python3-all-dev_3.2.3-6_all.deb
 acd5dd7bec7b31bfd2ec1a37e35d46f5e3a8284882992ba76887ac2b31a01b4c 880 python3-all-dbg_3.2.3-6_all.deb
Files: 
 93ccda45b7f0c2e6bfe3ca52b5884f5b 2250 python optional python3-defaults_3.2.3-6.dsc
 f9fca037e3defc4340bcd90aa38d3afb 142250 python optional python3-defaults_3.2.3-6.tar.gz
 973374176f0bb2866fb597d9ece712b2 43366 python optional python3_3.2.3-6_all.deb
 5149d291609b63e114250a13737ad2f1 20948 python optional python3-minimal_3.2.3-6_all.deb
 c00da7e14a521509df7a1ed395178402 856 python optional python3-examples_3.2.3-6_all.deb
 c44d171fa1bbcd0ef7237b7834e380e9 1062 python optional python3-dev_3.2.3-6_all.deb
 4ed213ae07af26798928b4aadc177fb7 3152 python optional idle3_3.2.3-6_all.deb
 7d06b3075d9d9a9e1b36db20a1cf2eb9 20580 doc optional python3-doc_3.2.3-6_all.deb
 1c54c7c1d78f83c5134a241e26abcce1 1130 debug extra python3-dbg_3.2.3-6_all.deb
 c71ebd5303008b2f062f2bd913f84ae4 866 python optional python3-all_3.2.3-6_all.deb
 a56e15f6fa51626f9a99ca8e86c9f8a1 882 python optional python3-all-dev_3.2.3-6_all.deb
 b94553e282b33595434d1aa32e8f3398 880 debug extra python3-all-dbg_3.2.3-6_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBCgAGBQJQhG6UAAoJEK728aKnRXZFSIQP/jLSB3zJe7gDTGDY1tlMJ/ZS
cQ5GX0T47cMrA+jgsW4dKN4i5eY2bzgwxzyxVpcRFNyqQZUa73Cuu2FdupGuiHTt
tMjDWn1v2OCtjTFJNbuodXJs6hyIGLkumXWKOG7xhOtgSmT5rJ72XsTgwmTnSylW
f17QsCeOaa4fdWeCKlACOAXpEfn2fE8SMYKO7hzSvw1PamF3nKbhX8mZ3LLASdfB
Q0H8zUzdwCUeEb/RJjBdZVuTiTUjMO9KUm8WELS8a4PBqFxjYyJdzRIs/2Tp4Isa
UFVDluR3+qAC3jE4+lfjfKk2azJodlSpaiP7Hemd/t+nxN91AY8i/nrlOM+EpvHG
3wFcCC2ebietQmSf4PEpO0HGH9yVrnciFLmjfaHv9geA/7iYI5Ray95ubN7T6OiN
IMDihDCKsAZPyCPAu6suH4wKCFx1GMstbUqWb3aechPgWHxUFUK3q36QAdgJnI52
qbb7+QEiKX2wNFughf0gmp2mFTjZk/SIpSEGG4KpyfQEkyJHxfXj58TcrROvjCin
bDEvtM+A1RJaohZfdwwN69ATBpC7RSrqr9NE8yR/OhffdMfDnmX6uX3pvzCT6AQN
5bug8oNYTDfRjqDzYtmSm80KGbSVvsLsmno6Bq3+32JYGibvZ7t67zTZOG+CDA0X
y8U+DrpBsiEooqEbwd7Z
=hHhM
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.