Debian Bug report logs - #527388
can't handle zip archives with more than 65535 files

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Vincent Lefevre <vincent@vinc17.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: can't handle zip archives with more than 65535 files

Date: Thu, 7 May 2009 15:31:06 +0200

Package: zip
Version: 3.0-1
Severity: important

zip 3.0 cannot handle zip files with more than 65535 files, e.g.

  zip -9ru file.zip directory

gives errors like:

        zip warning: expected 10468 entries but found 76004

  zip error: Zip file structure invalid (file.zip)

I reported the bug upstream in January, but it seems that nothing has
been done (or Debian is not up-to-date). FYI, I gave the following
testcase:

  http://www.vinc17.org/download/archive.zip.bz2

This archive.zip file itself is compressed with bzip2 (1.4 MB),
otherwise this file takes 10 MB.

-- System Information:
Debian Release: squeeze/sid
  APT prefers oldstable
  APT policy: (500, 'oldstable'), (500, 'unstable'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.26.5-20080922 (SMP w/2 CPU cores; PREEMPT)
Locale: LANG=POSIX, LC_CTYPE=en_US.ISO8859-1 (charmap=ISO-8859-1)
Shell: /bin/sh linked to /bin/bash

Versions of packages zip depends on:
ii  libc6                         2.9-10     GNU C Library: Shared libraries

Versions of packages zip recommends:
ii  unzip                         5.52-12    De-archiver for .zip files

zip suggests no packages.

-- no debconf information

Message #10 received at 527388@bugs.debian.org (full text, mbox, reply):

From: Santiago Vila <sanvila@unex.es>

To: Vincent Lefevre <vincent@vinc17.org>, 527388@bugs.debian.org

Subject: Re: Bug#527388: can't handle zip archives with more than 65535 files

Date: Thu, 7 May 2009 17:27:57 +0200 (CEST)

Hello.

You say that you reported this upstream in January.
Did you do that by using the new forum here:

http://www.info-zip.org/board/board.pl?b-zipbugs/

or you did it by any other means? (email, for example)

[ I ask because, if already reported, I would like to send them a
  reminder instead of a duplicate. ]

[ BTW: I see they have finally released unzip 6.0. Hurrah! ]

Thanks.

Message #15 received at 527388@bugs.debian.org (full text, mbox, reply):

From: Vincent Lefevre <vincent@vinc17.org>

To: Santiago Vila <sanvila@unex.es>

Cc: 527388@bugs.debian.org

Subject: Re: Bug#527388: can't handle zip archives with more than 65535 files

Date: Thu, 7 May 2009 17:56:35 +0200

Hi,

On 2009-05-07 17:27:57 +0200, Santiago Vila wrote:
> You say that you reported this upstream in January.
> Did you do that by using the new forum here:

No, I used the web form:

  http://www.info-zip.org/zip-bug.html

and in the discussion that followed (by mail in Info-ZIP-Dev with
a Cc to myself), the subject had:

  Subject: Re: Info-ZIP Bug report [Zip 3.0 not reading Zip 2.32 archive]

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Message #20 received at 527388-done@bugs.debian.org (full text, mbox, reply):

From: Santiago Vila <sanvila@unex.es>

To: Vincent Lefevre <vincent@vinc17.org>, 527388-done@bugs.debian.org

Subject: Re: Bug#527388: can't handle zip archives with more than 65535 files

Date: Fri, 8 May 2009 14:32:24 +0200 (CEST)

On Thu, 7 May 2009, Vincent Lefevre wrote:

> Package: zip
> Version: 3.0-1
> Severity: important
> 
> zip 3.0 cannot handle zip files with more than 65535 files, e.g.
> 
>   zip -9ru file.zip directory
> 
> gives errors like:
> 
>         zip warning: expected 10468 entries but found 76004
> 
>   zip error: Zip file structure invalid (file.zip)
> 
> I reported the bug upstream in January, but it seems that nothing has
> been done (or Debian is not up-to-date). FYI, I gave the following
> testcase:
> 
>   http://www.vinc17.org/download/archive.zip.bz2

Ok. I think this is not a bug in zip 3.0 but in zip 2.3, which you
used to create archive.zip.

The traditional zip file format (if you don't use the Zip64 extensions)
is limited to 64K entries. If you managed to create a zipfile with
more that 64K entries using zip 2.3 (which does not support Zip64)
then that's a bug in zip 2.3.

zip 3.0 uses the Zip64 extensions when they are needed, but if the
zipfile is already corrupt, I would not expect it to fix it.

See what "zipinfo -v" (from the upcoming unzip 6.0) says about archive.zip:


Archive:  archive.zip
There is no zipfile comment.

End-of-central-directory record:
-------------------------------

  Zip archive file size:                  10016830 (000000000098D83Eh)
  Actual end-cent-dir record offset:      10016808 (000000000098D828h)
  Expected end-cent-dir record offset:    10016808 (000000000098D828h)
  (based on the length of the central directory and its expected offset)

  This zipfile constitutes the sole disk of a single-part archive; its
  central directory contains 1465 entries.

[...]

and see what it says if we create archive.zip correctly from the beginning:

unzip archive.zip
zip -r good-archive.zip archive
zipinfo -v good-archive.zip

Output:

Archive:  good-archive.zip
There is no zipfile comment.

End-of-central-directory record:
-------------------------------

  Zip archive file size:                  11222924 (0000000000AB3F8Ch)
  Actual end-cent-dir record offset:      11222826 (0000000000AB3F2Ah)
  Expected end-cent-dir record offset:    11222826 (0000000000AB3F2Ah)
  (based on the length of the central directory and its expected offset)

  This zipfile constitutes the sole disk of a single-part archive; its
  central directory contains 67001 entries.

As opposed to zip 2.3, zip 3.0 has no problems adding more entries to
the archive created that way:

$ zip -94u good-archive.zip directory
  adding: directory/ (stored 0%)


So the bug was in zip 2.3, which you are not using anymore, not in zip 3.0
which is probably right in refusing to update a corrupted zipfile.

I plan to upload unzip 6.0 in short, but unzip 5.52 is apparently still
able to unzip good-archive.zip.

Thanks.

Message #27 received at 527388@bugs.debian.org (full text, mbox, reply):

From: Vincent Lefevre <vincent@vinc17.org>

To: Santiago Vila <sanvila@unex.es>

Cc: 527388@bugs.debian.org

Subject: Re: Bug#527388: can't handle zip archives with more than 65535 files

Date: Sat, 9 May 2009 02:44:34 +0200

reopen 527388
thanks

On 2009-05-08 14:32:24 +0200, Santiago Vila wrote:
> Ok. I think this is not a bug in zip 3.0 but in zip 2.3, which you
> used to create archive.zip.
> 
> The traditional zip file format (if you don't use the Zip64 extensions)
> is limited to 64K entries.

No, it was not limited to 64K entries, but the number of entries was
stored modulo 2^16 (possibly as an extension to the official format
specifications). It has worked like that for years, and this was *not*
a problem to retrieve the files from the archive.

Here's what Ed Gordon said in the January discussion:

  However, since Zip 2.32 archives like this are good, except for this
  count being chopped, Zip 3.0 should accept these archives without
  complaint, or maybe with just a warning.

> If you managed to create a zipfile with more that 64K entries using
> zip 2.3 (which does not support Zip64) then that's a bug in zip 2.3.

Note that this version is distributed in Debian/lenny and if the
archives are not guaranteed to be re-read with future versions,
this is a clear case of critical bug (data loss).

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Message #36 received at 527388@bugs.debian.org (full text, mbox, reply):

From: Santiago Vila <sanvila@unex.es>

To: Vincent Lefevre <vincent@vinc17.org>, 527388@bugs.debian.org

Subject: Re: Bug#527388: can't handle zip archives with more than 65535 files

Date: Sat, 9 May 2009 12:00:34 +0200 (CEST)

Ok, thanks for the additional info, specially the reply given by Ed Gordon.

This is a backwards compatibility issue more than a data loss thing,
as unzip is still able to unpack those odd zipfiles.

I've forwarded this upstream:

http://www.info-zip.org/board/board.pl?m-1241862258/

Message #41 received at 527388@bugs.debian.org (full text, mbox, reply):

From: Santiago Vila <sanvila@unex.es>

To: Vincent Lefevre <vincent@vinc17.org>, 527388@bugs.debian.org

Subject: Re: Bug#527388: can't handle zip archives with more than 65535 files

Date: Sat, 9 May 2009 22:41:37 +0200 (CEST)

Hi.

The authors suggest that you run zip -FF on the defective archive, like this:

zip -FF file.zip --out fixed.zip

Message #46 received at 527388@bugs.debian.org (full text, mbox, reply):

From: Vincent Lefevre <vincent@vinc17.org>

To: Santiago Vila <sanvila@unex.es>

Cc: 527388@bugs.debian.org

Subject: Re: Bug#527388: can't handle zip archives with more than 65535 files

Date: Sun, 10 May 2009 12:02:16 +0200

On 2009-05-09 12:00:34 +0200, Santiago Vila wrote:
> This is a backwards compatibility issue more than a data loss thing,
> as unzip is still able to unpack those odd zipfiles.

OK, I now remember that upstream handled the issue in unzip.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Message #51 received at 527388@bugs.debian.org (full text, mbox, reply):

From: Vincent Lefevre <vincent@vinc17.org>

To: Santiago Vila <sanvila@unex.es>

Cc: 527388@bugs.debian.org

Subject: Re: Bug#527388: can't handle zip archives with more than 65535 files

Date: Sun, 10 May 2009 12:12:11 +0200

On 2009-05-09 22:41:37 +0200, Santiago Vila wrote:
> The authors suggest that you run zip -FF on the defective archive, like this:
> 
> zip -FF file.zip --out fixed.zip

When Ed Gordon asked me to do this test in January, it didn't work.
But perhaps this was fixed in the new unzip version (since, IIRC,
unzip is called by zip for this operation). I'll have to test...
BTW, shouldn't the zip package recommend the *new* unzip version?

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Message #56 received at 527388@bugs.debian.org (full text, mbox, reply):

From: Santiago Vila <sanvila@unex.es>

To: Vincent Lefevre <vincent@vinc17.org>, 527388@bugs.debian.org

Subject: Re: Bug#527388: can't handle zip archives with more than 65535 files

Date: Mon, 11 May 2009 11:08:24 +0200 (CEST)

On Sun, 10 May 2009, Vincent Lefevre wrote:

> On 2009-05-09 22:41:37 +0200, Santiago Vila wrote:
> > The authors suggest that you run zip -FF on the defective archive, like this:
> > 
> > zip -FF file.zip --out fixed.zip
> 
> When Ed Gordon asked me to do this test in January, it didn't work.
> But perhaps this was fixed in the new unzip version (since, IIRC,
> unzip is called by zip for this operation). I'll have to test...
> BTW, shouldn't the zip package recommend the *new* unzip version?

Versioned recommends are almost useless: If the user is using testing
and the recommended package is also in testing, a versioned recommends
is the same as a normal recommends. If the recommended package is not
in testing yet, the user will be annoyed, as he will not understand
why a package recommends another one which simply "does not exist".

In this particular case, only a minority of users manage big zipfiles.
Everybody else is able to unzip packages using any version of unzip.
As unzip 6.0 will be in testing soon, I don't think it's worth to
worry about that.

Message #61 received at 527388@bugs.debian.org (full text, mbox, reply):

From: Vincent Lefevre <vincent@vinc17.org>

To: Santiago Vila <sanvila@unex.es>

Cc: 527388@bugs.debian.org

Subject: Re: Bug#527388: can't handle zip archives with more than 65535 files

Date: Mon, 11 May 2009 12:41:55 +0200

On 2009-05-11 11:08:24 +0200, Santiago Vila wrote:
> Versioned recommends are almost useless: If the user is using testing
> and the recommended package is also in testing, a versioned recommends
> is the same as a normal recommends. If the recommended package is not
> in testing yet, the user will be annoyed, as he will not understand
> why a package recommends another one which simply "does not exist".

Such kind of problems can already happen with normal recommends
or depends. Users should be educated.

> In this particular case, only a minority of users manage big zipfiles.
> Everybody else is able to unzip packages using any version of unzip.
> As unzip 6.0 will be in testing soon, I don't think it's worth to
> worry about that.

There's still a problem: if I "fix" the archive with -FF, the new
archive is not readable by the old unzip that is in Debian/stable.
This is nasty for interoperability.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Send a report that this bug log contains spam.