Debian Bug report logs - #828833
python3-magic: magic file type double encoded

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Reiner Herrmann <reiner@reiner-h.de>

To: submit@bugs.debian.org

Subject: python3-magic: magic file type double encoded

Date: Tue, 28 Jun 2016 10:59:13 +0200

[Message part 1 (text/plain, inline)]

Package: python3-magic
Version: 1:5.28-1

Hi,

since python3-magic 1:5.28-1 the file type detection is broken.
Instead of returning a string whih contains the magic, it returns
double encoded bytes:

>>> import magic
>>> path = "/usr/share/doc/gdb/refcard.ps.gz"
>>> x = magic.open(magic.NONE)
>>> x.load()
0
>>> x.file(path)
b"b'gzip compressed data, max compression, from Unix'"

(Instead of just 'gzip compressed data, max compression, from Unix' like
before.)

The problem seems to be those lines:

> r = _file(self._magic_t, bi)
> ...
>     return str(r).encode('utf-8')

r is already of type bytes, converting it to string results in the
string "b'gzip compressed data, max compression, from Unix'", which is
then encoded to utf-8:
b"b'gzip compressed data, max compression, from Unix'"

Converting it with str(r, 'utf-8') seems to be the correct way and has
the same result as before.

Kind regards,
  Reiner

[file.patch (text/x-diff, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 828833@bugs.debian.org (full text, mbox, reply):

From: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

To: Reiner Herrmann <reiner@reiner-h.de>, 828833@bugs.debian.org

Subject: Re: Bug#828833: python3-magic: magic file type double encoded

Date: Tue, 28 Jun 2016 11:42:35 +0200

[Message part 1 (text/plain, inline)]

tags 828833 confirmed
thanks

Reiner Herrmann wrote...

> since python3-magic 1:5.28-1 the file type detection is broken.
> Instead of returning a string whih contains the magic, it returns
> double encoded bytes:
(...)
> b"b'gzip compressed data, max compression, from Unix'"

Thanks for catching, and also for the patch. However, this reverts a
change upstream introduced in
https://github.com/file/file/commit/5560418ebc6f94e80953c8646383fdb70758fc7a
to "Restore python-2 compatibility" as reported in
http://bugs.gw.com/view.php?id=511

Still, even after applying your patch a test script works when
using python2.7, and now I'm confused. Since my python knowledge is
rather limited, could you please check for both versions, too - and
perhaps shed some light on it?

    Christoph

[signature.asc (application/pgp-signature, inline)]

Message #19 received at 828833@bugs.debian.org (full text, mbox, reply):

From: Reiner Herrmann <reiner@reiner-h.de>

To: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

Cc: 828833@bugs.debian.org, orestis@oioannou.com

Subject: Re: Bug#828833: python3-magic: magic file type double encoded

Date: Tue, 28 Jun 2016 12:09:03 +0200

[Message part 1 (text/plain, inline)]

[ adding Orestis to CC, who submitted the patch in file's #511 ]

On Tue, Jun 28, 2016 at 11:42:35AM +0200, Christoph Biedl wrote:
> > since python3-magic 1:5.28-1 the file type detection is broken.
> > Instead of returning a string whih contains the magic, it returns
> > double encoded bytes:
> (...)
> > b"b'gzip compressed data, max compression, from Unix'"
> 
> Thanks for catching, and also for the patch. However, this reverts a
> change upstream introduced in
> https://github.com/file/file/commit/5560418ebc6f94e80953c8646383fdb70758fc7a
> to "Restore python-2 compatibility" as reported in
> http://bugs.gw.com/view.php?id=511
> 
> Still, even after applying your patch a test script works when
> using python2.7, and now I'm confused. Since my python knowledge is
> rather limited, could you please check for both versions, too - and
> perhaps shed some light on it?

Hm, for me it also still works in Python 2.
But only because it doesn't reach the str(r, 'utf-8')
(it chooses the other branch).
Python 2 does indeed not have a 2-argument str.
I'm currently not sure in which cases this path is reached
with Python 2.  Perhaps Orestis remembers why the patch was
needed, and perhaps has an idea for restoring Python 3 support.

[signature.asc (application/pgp-signature, inline)]

Message #24 received at 828833@bugs.debian.org (full text, mbox, reply):

From: Reiner Herrmann <reiner@reiner-h.de>

To: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

Cc: 828833@bugs.debian.org

Subject: Re: Bug#828833: python3-magic: magic file type double encoded

Date: Tue, 28 Jun 2016 19:48:08 +0200

[Message part 1 (text/plain, inline)]

On Tue, Jun 28, 2016 at 11:42:35AM +0200, Christoph Biedl wrote:
> Still, even after applying your patch a test script works when
> using python2.7, and now I'm confused. Since my python knowledge is
> rather limited, could you please check for both versions, too - and
> perhaps shed some light on it?

It looks like upstream just commited the same fix:
https://github.com/file/file/commit/73e043d2a986234b187a00ed0c8d1f7bf83df372

[signature.asc (application/pgp-signature, inline)]

Message #29 received at 828833@bugs.debian.org (full text, mbox, reply):

From: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

To: Reiner Herrmann <reiner@reiner-h.de>, 828833@bugs.debian.org

Subject: Re: Bug#828833: python3-magic: magic file type double encoded

Date: Tue, 28 Jun 2016 23:28:30 +0200

[Message part 1 (text/plain, inline)]

tags 828833 pending
thanks

Reiner Herrmann wrote...

> It looks like upstream just commited the same fix:
> https://github.com/file/file/commit/73e043d2a986234b187a00ed0c8d1f7bf83df372

Great. I'll collect some more stuff that needs attention, shouldn't
take longer than a few days.

    Christoph

[signature.asc (application/pgp-signature, inline)]

Message #36 received at 828833@bugs.debian.org (full text, mbox, reply):

From: Iago <iago@espiv.net>

To: 828833@bugs.debian.org

Subject: Re: Bug#828833: python3-magic: magic file type double encoded

Date: Tue, 28 Jun 2016 23:47:51 +0200

Hello,

Thanks for including me in the loop

On Tue, 28 Jun 2016 19:48:08 +0200 Reiner Herrmann <reiner@reiner-h.de>
wrote:
> On Tue, Jun 28, 2016 at 11:42:35AM +0200, Christoph Biedl wrote:
> > Still, even after applying your patch a test script works when
> > using python2.7, and now I'm confused. Since my python knowledge is
> > rather limited, could you please check for both versions, too - and
> > perhaps shed some light on it?
> 
> It looks like upstream just commited the same fix:
> https://github.com/file/file/commit/73e043d2a986234b187a00ed0c8d1f7bf83df372

Well this is going to break the python2 support again :/

I hadn't notice the fix i proposed was breaking python3 support. Wasn't
my intention.

The reason i proposed the patch was that the testsuit of Debsources
(www.sources.debian.net) was reaching the str(r, 'utf-8'). Here's a log
of the incident https://travis-ci.org/Debian/debsources/builds/102972221

It's a pity that they merged this so quickly. unfortunately i don't have
the time before next week to work on a patch that will actually support
both Python version correctly.

Cheers,

Orestis

Message #41 received at 828833@bugs.debian.org (full text, mbox, reply):

From: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

To: Iago <iago@espiv.net>, 828833@bugs.debian.org

Subject: Re: Bug#828833: python3-magic: magic file type double encoded

Date: Wed, 29 Jun 2016 01:28:29 +0200

Iago wrote...

> The reason i proposed the patch was that the testsuit of Debsources
> (www.sources.debian.net) was reaching the str(r, 'utf-8'). Here's a log
> of the incident https://travis-ci.org/Debian/debsources/builds/102972221

Can you reduce that to a reproducer? Also, travis-ci is down and/or
they are incapable of presenting something as a log file simply as
plain text. In other words: I'd like to understand what went wrong but
cannot even see it. 

Sorry for the inconvenience,

    Christoph

Message #44 received at 828833@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>, 828833@bugs.debian.org

Cc: Reiner Herrmann <reiner@reiner-h.de>

Subject: Re: Bug#828833: python3-magic: magic file type double encoded

Date: Wed, 29 Jun 2016 05:54:43 +0000

[Message part 1 (text/plain, inline)]

Hi Christoph,

On Tue, Jun 28, 2016 at 11:28:30PM +0200, Christoph Biedl wrote:
> > It looks like upstream just commited the same fix:
> > https://github.com/file/file/commit/73e043d2a986234b187a00ed0c8d1f7bf83df372
> 
> Great. I'll collect some more stuff that needs attention, shouldn't
> take longer than a few days.

This bug is causing quite some headaches within the reproducible builds
project, as diffoscope relies on python3-magic quite a lot.
Both yesterday and today I downgraded the packages to the previous
versions, but given that the chroot where this runs is recreated at
early morning we have to experience hundreds of failures until I wake up
and paper this over...

Can I ask you to quickly upload the fix as soon as there is a working
py2/py3 patch?

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Message #49 received at 828833@bugs.debian.org (full text, mbox, reply):

From: Orestis Ioannou <orestis@oioannou.com>

To: 828833@bugs.debian.org

Subject: Re: Bug#828833: python3-magic: magic file type double encoded

Date: Wed, 29 Jun 2016 08:12:42 +0200

On Wed, 29 Jun 2016 01:28:29 +0200 Christoph Biedl
<debian.axhn@manchmal.in-ulm.de> wrote:
> 
> > The reason i proposed the patch was that the testsuit of Debsources
> > (www.sources.debian.net) was reaching the str(r, 'utf-8'). Here's a log
> > of the incident https://travis-ci.org/Debian/debsources/builds/102972221
> 
> Can you reduce that to a reproducer? Also, travis-ci is down and/or
> they are incapable of presenting something as a log file simply as
> plain text. In other words: I'd like to understand what went wrong but
> cannot even see it. 
> 
> Sorry for the inconvenience,
> 
>     Christoph
> 
> 

So many tests failed with this stacktrace http://paste.debian.net/770211/

If this is causing a mess in other projects then don't wait for me :).
At debsources the issue was happening only in travis since we pull the
master branch of python magic. In other words you won't break
debsources, at least right now, if you upload.

Sorry for the trouble i caused :/

Orestis

Message #54 received at 828833@bugs.debian.org (full text, mbox, reply):

From: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

To: Mattia Rizzolo <mattia@debian.org>, 828833@bugs.debian.org, Reiner Herrmann <reiner@reiner-h.de>

Subject: Re: Bug#828833: python3-magic: magic file type double encoded

Date: Wed, 29 Jun 2016 10:35:51 +0200

[Message part 1 (text/plain, inline)]

Mattia Rizzolo wrote...

> Can I ask you to quickly upload the fix as soon as there is a working
> py2/py3 patch?

Will do as soon as I'm confident there's a fix that does not create
new hazzle, read: I still wasn't able to reproduce the original
regression when using python2 (PR/511).

As there's some python knowledge around here this should not take
longer than a few hours.

    Christoph

[signature.asc (application/pgp-signature, inline)]

Message #59 received at 828833@bugs.debian.org (full text, mbox, reply):

From: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

To: Orestis Ioannou <orestis@oioannou.com>, 828833@bugs.debian.org

Subject: Re: Bug#828833: python3-magic: magic file type double encoded

Date: Wed, 29 Jun 2016 14:09:05 +0200

[Message part 1 (text/plain, inline)]

Orestis Ioannou wrote...

> So many tests failed with this stacktrace http://paste.debian.net/770211/

What I'd really like to see is the actual code and the circumstances
that triggered the stacktrace, probably a certain file. Still I failed
to create that, and this is important to fix the issue.

> If this is causing a mess in other projects then don't wait for me :).
> At debsources the issue was happening only in travis since we pull the
> master branch of python magic. In other words you won't break
> debsources, at least right now, if you upload.

No, the library should support python2 to the full extent as long as
python2 is part of Debian. Which will be for stretch and perhaps even
for buster.

> Sorry for the trouble i caused :/

Don't worry, you're just the bringer of the news.

    Christoph

[signature.asc (application/pgp-signature, inline)]

Message #68 received at 828833@bugs.debian.org (full text, mbox, reply):

From: Orestis Ioannou <orestis@oioannou.com>

To: 828833@bugs.debian.org

Subject: Re: Bug#828833: python3-magic: magic file type double encoded

Date: Thu, 30 Jun 2016 00:45:13 +0200

Hey

So i took some time to investigate this. Here are my findings:

On Wed, 29 Jun 2016 14:09:05 +0200 Christoph Biedl
<debian.axhn@manchmal.in-ulm.de> wrote:
> Orestis Ioannou wrote...
> 
> > So many tests failed with this stacktrace http://paste.debian.net/770211/
> 
> What I'd really like to see is the actual code and the circumstances
> that triggered the stacktrace, probably a certain file. Still I failed
> to create that, and this is important to fix the issue.
> 

On debsources the code is
http://anonscm.debian.org/cgit/qa/debsources.git/tree/debsources/navigation.py#n176


Steps to reproduce bug i fixed:

clone file repository and checkout to commit e3f9082 (just before my commit)

wget http://sources.debian.net/data/main/g/gnubg/1.04.000-1/debian/copyright

You then inside file/python start python and:

import magic
mime = magic.open(magic.MIME_TYPE)
mime.load()
mime.file(u'path/to/copyright/file')


Please note the u before the path.
I tried it with many files and it always fails. With the fix i did it
wasn't failing anymore.


However on current master i can't reproduce the debsources bug since
Reiner only touched the string conversion and not the bytes one.

I am not sure if it is possible to reach the str(r, 'utf-8'). I can't
seem to be able to do this but i haven't spend much time on it.

Probably now python2/python3 is restored for good.

Cheers,

Orestis

Message #73 received at 828833@bugs.debian.org (full text, mbox, reply):

From: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

To: Orestis Ioannou <orestis@oioannou.com>, 828833@bugs.debian.org

Subject: Re: Bug#828833: python3-magic: magic file type double encoded

Date: Thu, 30 Jun 2016 01:45:17 +0200

[Message part 1 (text/plain, inline)]

Orestis Ioannou wrote...

> However on current master i can't reproduce the debsources bug since
> Reiner only touched the string conversion and not the bytes one.
> 
> I am not sure if it is possible to reach the str(r, 'utf-8'). I can't
> seem to be able to do this but i haven't spend much time on it.

Thanks for your work on this. I finally could reproduce the stack
trace on ancient file commits and can confirm it did not return after
the recent ones.

> Probably now python2/python3 is restored for good.

Seems like it, and cherry-picking that commit will certainly result in
a better state than the current one.

Upload for file 1:5.28-2 will follow in a few minutes.

    Christoph

[signature.asc (application/pgp-signature, inline)]

Message #78 received at 828833-close@bugs.debian.org (full text, mbox, reply):

From: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

To: 828833-close@bugs.debian.org

Subject: Bug#828833: fixed in file 1:5.28-2

Date: Thu, 30 Jun 2016 00:48:47 +0000

Source: file
Source-Version: 1:5.28-2

We believe that the bug you reported is fixed in the latest version of
file, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 828833@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Christoph Biedl <debian.axhn@manchmal.in-ulm.de> (supplier of updated file package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Thu, 30 Jun 2016 01:39:07 +0200
Source: file
Binary: file file-dbg libmagic1 libmagic-dev python-magic python3-magic
Architecture: source armhf all
Version: 1:5.28-2
Distribution: unstable
Urgency: high
Maintainer: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>
Changed-By: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>
Description:
 file       - Determines file type using "magic" numbers
 file-dbg   - Determines file type using "magic" numbers (debug)
 libmagic-dev - File type determination library using "magic" numbers (developmen
 libmagic1  - File type determination library using "magic" numbers
 python-magic - File type determination library using "magic" numbers (Python bin
 python3-magic - File type determination library using "magic" numbers (Python 3 b
Closes: 828833
Changes:
 file (1:5.28-2) unstable; urgency=high
 .
   * Enable full hardening build
   * Cherry-pick commits since 5.28 release:
     - Avoid "can't break line" warnings from lintian
     - Don't copy NULL
     - Avoid double encoding with python3. Closes: #828833
Checksums-Sha1:
 4adf3888d6d84f6dd3c63da3e3cc6ebcbe8bb765 2107 file_5.28-2.dsc
 24ac26bdc633c221bfa93abd33513122c5b48f93 29424 file_5.28-2.debian.tar.xz
 2dbf81d026f601808e8a552bd9198a0be18cc6de 198080 file-dbg_5.28-2_armhf.deb
 4bb1b222667b4c15061c8927570292b93f536a61 62358 file_5.28-2_armhf.deb
 78310b902137d72af7a1850e2b64063a1df5c5e1 113404 libmagic-dev_5.28-2_armhf.deb
 cc40863475d76189fe27532a714177d0bb82d6fe 269210 libmagic1_5.28-2_armhf.deb
 f9e451c9ebd825da06ed2088bec52890bbf9fd71 46954 python-magic_5.28-2_all.deb
 cbe54a08ab6fb72a2f9f481da1e83184ac9f63d2 47014 python3-magic_5.28-2_all.deb
Checksums-Sha256:
 418d3ad7bef876c6a264124035a99bb4e8e78ee2601b2555c598fe8f98a7768a 2107 file_5.28-2.dsc
 e2a112839c3bcb0df981fae41022495776fda01e7a00d96711c7a4eaf1aae302 29424 file_5.28-2.debian.tar.xz
 41e83ff361b8e399c7c9cfa01ae96a3b8cc1797c401cf6439259090dc4b72e35 198080 file-dbg_5.28-2_armhf.deb
 e1b9b93b4c74f3bf5e70572cabc67e284f236aa09b9169ffe409652d9709372d 62358 file_5.28-2_armhf.deb
 cdc94727e2aa72cc62a258ec2471aed03743e6acf616638183c2d0c30ca98c92 113404 libmagic-dev_5.28-2_armhf.deb
 655bfd846ef28c2bf941771ecc37426bead487bb028f8e1b1ba96c4ce6bcd08c 269210 libmagic1_5.28-2_armhf.deb
 0e716968cb254cc61fdaa8b07d5ccfe4c553048e192a37cb746435010680e966 46954 python-magic_5.28-2_all.deb
 0bec96413261d9937d9bb662342d62fd9f51656bd9338e330da48a5b7d01be07 47014 python3-magic_5.28-2_all.deb
Files:
 4683d9f45cfb5fcbeb96d2c87e4973c1 2107 utils standard file_5.28-2.dsc
 63a8b0dd16c8d4af18e47bbb21ffce54 29424 utils standard file_5.28-2.debian.tar.xz
 ff76656d626ebc2d119e5d9f7c18a349 198080 debug extra file-dbg_5.28-2_armhf.deb
 9340dd50239181798faf8bbf3415b73c 62358 utils standard file_5.28-2_armhf.deb
 e274eb4205e1798e8707a2a86c62ed0d 113404 libdevel optional libmagic-dev_5.28-2_armhf.deb
 a7b0fe40318cc87212770fd468273034 269210 libs standard libmagic1_5.28-2_armhf.deb
 5fccb8858e413783585570767b0e534c 46954 python optional python-magic_5.28-2_all.deb
 dac16e0e3ca043d5b212b3722e6d5435 47014 python optional python3-magic_5.28-2_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJXdGFpAAoJEMQsWOtZFJL9d94P/0W35DlUuM9B6Ei5Vsr+zH9r
kFA2/ilSVoGZbsLrUAdSue2Wy8HmnUJf36mLwsG3TdLnWR1I8KkNVYgVQ5v2v5Zp
IDYmGSNdH1+/ERBjQpMV5nihmTtW7Kfc20vauE2a+pTDriE3H1avePY0C776gwRf
PUVEYhnOzbkTx5Tkc+SNb/0YWsU/eaNWEKPqyqlF75ov2+WWDgs6OzAY4kjesyri
bKylEJ+8sHqYBiylKBv+oFW++VZycBYRgiLzLVk8sZWu1B8jBFSX5PIjiOTWL5Xk
fKJTWPAkBjG7luDa48QIVymCu+FhEOhfiSb0iSiTkylUA23r6UDB6k4BSYQlvT7n
wgrvI3QcbaxkuOqO+S/QX5R8310ssccrJ/aCO4rSC8wWYG0Dr5egzwXyTY38FdM2
6sv47wya0JK2zPhxQ1z0zSsCYC5vtJ1Nqxi+e+FQoYhHT1dgbam0rmyuSOh6BLwW
7x3tFLE82pSRmMIPlrP0AMEWaDsvEWfq9KSqIX2EgZBH0hisLQF5hKJqvXOjDRrP
NtXE4zFdT9nZgTLsTlS5Nvq9l8xeuuoFxkmGRJ413dCcET9gZchuZ71QA/wbM081
x3Q1RCJJ2KR1MVFlpePVBix22Esv78YXFe3HOmxqdr9vailKALNXu/JG8+PZT2Zi
co35R/vuP9mEBO26DTPG
=dqJm
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.