Debian Bug report logs - #488562
non-deterministic results if the magic file is a directory

version graph

Package: file; Maintainer for file is Christoph Biedl <debian.axhn@manchmal.in-ulm.de>; Source for file is src:file.

Reported by: Simon McVittie <smcv@debian.org>

Date: Sun, 29 Jun 2008 19:24:01 UTC

Severity: minor

Tags: upstream

Found in versions file/4.24-3, file/4.26-1

Fixed in version 5.04-2

Done: Daniel Baumann <daniel@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Daniel Baumann <daniel@debian.org>:
Bug#488562; Package file. Full text and rfc822 format available.

Acknowledgement sent to Simon McVittie <smcv@debian.org>:
New Bug report received and forwarded. Copy sent to Daniel Baumann <daniel@debian.org>. Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Simon McVittie <smcv@debian.org>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: non-deterministic results if the magic file is a directory
Date: Sun, 29 Jun 2008 20:12:36 +0100
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Package: file
Version: 4.24-3
Severity: normal

The root cause of #481247 is that when magic.mgc is compiled during the
package build, the files in Magdir/ come out in an unpredictable order,
depending on the underlying filesystem (readdir() doesn't sort its
results). By contrast, the order of /usr/share/file/magic is determined
by "echo *" in debian/rules, which (at least on bash and dash) gives
sorted results.

file(1) probably ought to readdir() the whole directory into an array of
strings, sort them lexicographically, and iterate over the result.

As a workaround for #481247 I'll just remove magic.mgc and re-create it
from (the file that will become) /usr/share/file/magic.

- -- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable'), (101, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.25-2-686 (SMP w/2 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages file depends on:
ii  libc6                         2.7-12     GNU C Library: Shared libraries
ii  libmagic1                     4.24-3     File type determination library us

file recommends no packages.

- -- no debconf information
-----BEGIN PGP SIGNATURE-----

iD8DBQFIZ96bWSc8zVUw7HYRAuovAJ4nS/NZ//Lubd+5d9ELvAxbFPJZPgCfW5Sb
jMKsdvAkkRFk7JvKXY9LOPk=
=eULf
-----END PGP SIGNATURE-----




Information forwarded to debian-bugs-dist@lists.debian.org, Daniel Baumann <daniel@debian.org>:
Bug#488562; Package file. Full text and rfc822 format available.

Acknowledgement sent to daniel@debian.org:
Extra info received and forwarded to list. Copy sent to Daniel Baumann <daniel@debian.org>. Full text and rfc822 format available.

Message #10 received at 488562@bugs.debian.org (full text, mbox):

From: Daniel Baumann <daniel@debian.org>
To: Simon McVittie <smcv@debian.org>, 488562@bugs.debian.org
Subject: Re: Bug#488562: non-deterministic results if the magic file is a directory
Date: Sun, 29 Jun 2008 21:46:34 +0200
Simon McVittie wrote:
> The root cause of #481247 is that when magic.mgc is compiled during the
> package build, the files in Magdir/ come out in an unpredictable order,
> depending on the underlying filesystem (readdir() doesn't sort its
> results). By contrast, the order of /usr/share/file/magic is determined
> by "echo *" in debian/rules, which (at least on bash and dash) gives
> sorted results.

as said in #481247, I'll revert the makefile changes in 4.24 in order to
build plain files again.

-- 
Address:        Daniel Baumann, Burgunderstrasse 3, CH-4562 Biberist
Email:          daniel.baumann@panthera-systems.net
Internet:       http://people.panthera-systems.net/~daniel-baumann/




Reply sent to daniel@debian.org:
You have taken responsibility. Full text and rfc822 format available.

Notification sent to Simon McVittie <smcv@debian.org>:
Bug acknowledged by developer. Full text and rfc822 format available.

Message #15 received at 488562-done@bugs.debian.org (full text, mbox):

From: Daniel Baumann <daniel@debian.org>
To: 488562-done@bugs.debian.org, 488562-submitter@bugs.debian.org
Subject: Re: non-deterministic results if the magic file is a directory
Date: Tue, 01 Jul 2008 18:52:30 +0200
Version: 4.24-4

As already implied before, this is a non-issue when building the plain
magic file with the upstream build system rather than with debian
hackery, which as said before I did now, thus closing.

Regards,
Daniel

-- 
Address:        Daniel Baumann, Burgunderstrasse 3, CH-4562 Biberist
Email:          daniel.baumann@panthera-systems.net
Internet:       http://people.panthera-systems.net/~daniel-baumann/




Message sent on to Simon McVittie <smcv@debian.org>:
Bug#488562. Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Daniel Baumann <daniel@debian.org>:
Bug#488562; Package file. Full text and rfc822 format available.

Acknowledgement sent to Simon McVittie <smcv@debian.org>:
Extra info received and forwarded to list. Copy sent to Daniel Baumann <daniel@debian.org>. Full text and rfc822 format available.

Message #23 received at 488562@bugs.debian.org (full text, mbox):

From: Simon McVittie <smcv@debian.org>
To: 488562@bugs.debian.org
Subject: Re: Bug#488562: non-deterministic results if the magic file is a directory
Date: Tue, 1 Jul 2008 19:28:29 +0100
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

reopen 488562
severity 488562 minor
tags 488562 + upstream
thanks

Thanks for uploading a better fix for the RC bug #481247.

However, the non-RC bug I reported still exists: if "file -m" is used to
select an alternative source of magic patterns, and that source is a
directory, then the order in which they are applied is non-deterministic,
since it depends on the order in which readdir() lists files.

With hindsight, I should have filed #488562 as minor, since using "file -m"
like this is rare (at least if you use either your solution to #481247,
or mine, to avoid needing it during the package build). Correcting that now.

I haven't been able to reproduce this bug in a minimal way (I don't have
a filesystem that re-orders readdir() on demand :-) but here's an
illustration that it still exists in 4.24-4.

libRuby.so.5.18 is the same file used to reproduce #481247, ~/tmp/file-4.24
is the unpacked source code of the package, and ~/tmp is on an ext3 filesystem
with hashed directories.

smcv@carbon% dpkg -s file | grep Version
Version: 4.24-4
smcv@carbon% grep -v --no-filename '^:!' ~/tmp/file-4.24/magic/Magdir/* > ~/tmp/concatenated-magic
[1] smcv@carbon% file -m ~/tmp/concatenated-magic ~/tmp/libRuby.so.5.18
/home/smcv/tmp/libRuby.so.5.18: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
[2] smcv@carbon% file -m ~/tmp/file-4.24/magic/Magdir ~/tmp/libRuby.so.5.18
/home/smcv/tmp/libRuby.so.5.18: Linux/i386 core file of '/', dynamically linked, not stripped
[3] smcv@carbon% file ~/tmp/libRuby.so.5.18
/home/smcv/tmp/libRuby.so.5.18: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped

Commentary:

[1] this result is correct

[2] but this result, from input that ought to be equivalent, is wrong -
    that's what this minor bug is about

[3] on the other hand, #481247 has indeed been fixed, so the package is no
    longer RC-buggy

    Simon
-----BEGIN PGP SIGNATURE-----

iD8DBQFIandEWSc8zVUw7HYRAvAEAJ9Rh4wSX2wpJ9pZGC1u3Tl/EHrEDgCgsvkL
eawPG2Gqkc7Q2UCD8BuFkbY=
=/Ttv
-----END PGP SIGNATURE-----




Bug reopened, originator not changed. Request was from Simon McVittie <smcv@debian.org> to control@bugs.debian.org. (Tue, 01 Jul 2008 18:30:08 GMT) Full text and rfc822 format available.

Severity set to `minor' from `normal' Request was from Simon McVittie <smcv@debian.org> to control@bugs.debian.org. (Tue, 01 Jul 2008 18:30:10 GMT) Full text and rfc822 format available.

Tags added: upstream Request was from Simon McVittie <smcv@debian.org> to control@bugs.debian.org. (Tue, 01 Jul 2008 18:30:10 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Daniel Baumann <daniel@debian.org>:
Bug#488562; Package file. (Fri, 11 Sep 2009 22:45:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to Reuben Thomas <rrt@sc3d.org>:
Extra info received and forwarded to list. Copy sent to Daniel Baumann <daniel@debian.org>. (Fri, 11 Sep 2009 22:45:07 GMT) Full text and rfc822 format available.

Message #34 received at 488562@bugs.debian.org (full text, mbox):

From: Reuben Thomas <rrt@sc3d.org>
To: Debian Bug Tracking System <488562@bugs.debian.org>
Subject: This bug will be fixed in the next release of file
Date: Fri, 11 Sep 2009 23:37:22 +0100
Package: file
Version: 4.26-1
Followup-For: Bug #488562


I've just committed a fix for this bug. Contrary to what it says in
the bug thread, I believe that this bug *can* affect building the
system magic file: certainly on my system, readdir does not return the
magic files in sorted order, and hence magic.mgc doesn't get built in
the "right" order. With the fix I've committed, the magic files are
read in lexicographic order.

-- System Information:
Debian Release: 5.0.3
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 2.6.26-2-686 (SMP w/1 CPU core)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages file depends on:
ii  libc6                  2.7-18            GNU C Library: Shared libraries
ii  libmagic1              4.26-1            File type determination library us
ii  zlib1g                 1:1.2.3.3.dfsg-12 compression library - runtime

file recommends no packages.

file suggests no packages.

-- no debconf information




Reply sent to daniel@debian.org:
You have taken responsibility. (Tue, 13 Jul 2010 15:21:05 GMT) Full text and rfc822 format available.

Notification sent to Simon McVittie <smcv@debian.org>:
Bug acknowledged by developer. (Tue, 13 Jul 2010 15:21:05 GMT) Full text and rfc822 format available.

Message #39 received at 488562-done@bugs.debian.org (full text, mbox):

From: Daniel Baumann <daniel@debian.org>
To: 488562-done@bugs.debian.org
Subject: Re: non-deterministic results if the magic file is a directory
Date: Tue, 13 Jul 2010 17:20:07 +0200
Version: 5.04-2

there are two issues here, one is the build-time order which is fixed
upstream, and the second is that the legacy copies of the plain magics
did not match the compiled ones.

the latter is not the case anymore since 5.04-2, the plain magics are
dropped in favour of the compiled once, see changelog entry of 5.04-2
and README.Debian.

therefore, closing the bug.

Regards,
Daniel

-- 
Address:        Daniel Baumann, Burgunderstrasse 3, CH-4562 Biberist
Email:          daniel.baumann@panthera-systems.net
Internet:       http://people.panthera-systems.net/~daniel-baumann/




Reply sent to daniel@debian.org:
You have taken responsibility. (Mon, 19 Jul 2010 09:27:15 GMT) Full text and rfc822 format available.

Notification sent to Simon McVittie <smcv@debian.org>:
Bug acknowledged by developer. (Mon, 19 Jul 2010 09:27:15 GMT) Full text and rfc822 format available.

Message #44 received at 488562-done@bugs.debian.org (full text, mbox):

From: Daniel Baumann <daniel@debian.org>
To: 488562-done@bugs.debian.org
Subject: Re: non-deterministic results if the magic file is a directory
Date: Sun, 11 Jul 2010 01:41:41 +0200
Version: 5.04-2

there are two issues here, one is the build-time order which is fixed
upstream, and the second is that the legacy copies of the plain magics
did not match the compiled ones.

the latter is not the case anymore since 5.04-2, the plain magics are
dropped in favour of the compiled once, see changelog entry of 5.04-2
and README.Debian.

therefore, closing the bug.

Regards,
Daniel

-- 
Address:        Daniel Baumann, Burgunderstrasse 3, CH-4562 Biberist
Email:          daniel.baumann@panthera-systems.net
Internet:       http://people.panthera-systems.net/~daniel-baumann/




Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Fri, 09 Sep 2011 07:33:07 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Thu Apr 17 00:06:52 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.