Debian Bug report logs - #556407
data.tar.xz

Package: ftp.debian.org; Maintainer for ftp.debian.org is Debian FTP Master <ftpmaster@ftp-master.debian.org>;

Reported by: Clint Adams <clint@debian.org>

Date: Sun, 15 Nov 2009 21:12:02 UTC

Severity: wishlist

Done: Ansgar Burchardt <ansgar@43-1.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian FTP Master <ftpmaster@ftp-master.debian.org>:
Bug#556407; Package ftp.debian.org. (Sun, 15 Nov 2009 21:12:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Clint Adams <schizo@debian.org>:
New Bug report received and forwarded. Copy sent to Debian FTP Master <ftpmaster@ftp-master.debian.org>. (Sun, 15 Nov 2009 21:12:05 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Clint Adams <schizo@debian.org>
To: submit@bugs.debian.org
Subject: data.tar.xz
Date: Sun, 15 Nov 2009 21:02:02 +0000
Package: ftp.debian.org

On Sun, Nov 15, 2009 at 08:06:32PM +0000, Mark Hymers wrote:
> Just for the record, at the moment dak only allows data.tar.gz and
> data.tar.bz2 (although I had to check that myself in the source code,
> I've got documenting what is and isn't supported someone more sane on my
> TODO list).
> 
> As I said on -devel recently, the data.tar.bz2 support in dak seems to
> date from 2005, so it's been around for a while.  lzma/xz on the other
> hand hasn't even been asked for yet as far as I know.

I'll ask for xz support.

This won't work until http://bugs.python.org/issue6715 is resolved.

diff --git a/daklib/binary.py b/daklib/binary.py
index c6ee96f..09ae36c 100644
--- a/daklib/binary.py
+++ b/daklib/binary.py
@@ -141,7 +141,7 @@ class Binary(object):
         Check deb contents making sure the .deb contains:
           1. debian-binary
           2. control.tar.gz
-          3. data.tar.gz or data.tar.bz2
+          3. data.tar.gz or data.tar.bz2 or data.tar.xz
         in that order, and nothing else.
         """
         self.__scan_ar()
@@ -160,9 +160,9 @@ class Binary(object):
         if not rejected and self.chunks[1] != "control.tar.gz":
             rejected = True
             self.reject("%s: second chunk is '%s', expected 'control.tar.gz'." % (self.filename, self.chunks[1]))
-        if not rejected and self.chunks[2] not in [ "data.tar.bz2", "data.tar.gz" ]:
+        if not rejected and self.chunks[2] not in [ "data.tar.xz", "data.tar.bz2", "data.tar.gz" ]:
             rejected = True
-            self.reject("%s: third chunk is '%s', expected 'data.tar.gz' or 'data.tar.bz2'." % (self.filename, self.chunks[2]))
+            self.reject("%s: third chunk is '%s', expected 'data.tar.gz' or 'data.tar.bz2' or 'data.tar.xz'." % (self.filename, self.chunks[2]))
 
         return not rejected
 
@@ -198,6 +198,8 @@ class Binary(object):
                         data = tarfile.open(os.path.join(self.tmpdir, "data.tar.gz"), "r:gz")
                     elif self.chunks[2] == "data.tar.bz2":
                         data = tarfile.open(os.path.join(self.tmpdir, "data.tar.bz2" ), "r:bz2")
+                    elif self.chunks[2] == "data.tar.xz":
+                        data = tarfile.open(os.path.join(self.tmpdir, "data.tar.xz" ), "r:xz")
 
                     if bootstrap_id:
                         result = insert_content_paths(bootstrap_id, [tarinfo.name for tarinfo in data if not tarinfo.isdir()], session)
@@ -241,6 +243,8 @@ class Binary(object):
                     data = tarfile.open(os.path.join(self.tmpdir, "data.tar.gz"), "r:gz")
                 elif self.chunks[2] == "data.tar.bz2":
                     data = tarfile.open(os.path.join(self.tmpdir, "data.tar.bz2" ), "r:bz2")
+                elif self.chunks[2] == "data.tar.xz":
+                    data = tarfile.open(os.path.join(self.tmpdir, "data.tar.xz" ), "r:xz")
 
                 for tarinfo in data:
                     try:
diff --git a/daklib/queue.py b/daklib/queue.py
index d97b99f..66d7d5a 100644
--- a/daklib/queue.py
+++ b/daklib/queue.py
@@ -1358,10 +1358,17 @@ class Upload(object):
                         apt_inst.debExtract(deb_file, tar.callback, "data.tar.gz")
                     except SystemError, e:
                         # If we can't find a data.tar.gz, look for data.tar.bz2 instead.
-                        if not re.search(r"Cannot f[ui]nd chunk data.tar.gz$", str(e)):
-                            raise
-                        deb_file.seek(0)
-                        apt_inst.debExtract(deb_file,tar.callback,"data.tar.bz2")
+                        try:
+                            if not re.search(r"Cannot f[ui]nd chunk data.tar.gz$", str(e)):
+                                raise
+                            deb_file.seek(0)
+                            apt_inst.debExtract(deb_file,tar.callback,"data.tar.bz2")
+                        except SystemError, e:
+                            # If we can't find a data.tar.bz2, look for data.tar.xz instead.
+                            if not re.search(r"Cannot f[ui]nd chunk data.tar.bz2$", str(e)):
+                                raise
+                            deb_file.seek(0)
+                            apt_inst.debExtract(deb_file,tar.callback,"data.tar.xz")
 
                     deb_file.close()
 
diff --git a/daklib/utils.py b/daklib/utils.py
index 3cc4053..80ac74d 100644
--- a/daklib/utils.py
+++ b/daklib/utils.py
@@ -387,10 +387,10 @@ def check_dsc_files(dsc_filename, dsc=None, dsc_files=None):
         (r'orig.tar.gz',               ('orig_tar_gz', 'orig_tar')),
         (r'diff.gz',                   ('debian_diff',)),
         (r'tar.gz',                    ('native_tar_gz', 'native_tar')),
-        (r'debian\.tar\.(gz|bz2)',     ('debian_tar',)),
-        (r'orig\.tar\.(gz|bz2)',       ('orig_tar',)),
+        (r'debian\.tar\.(gz|bz2|xz)',     ('debian_tar',)),
+        (r'orig\.tar\.(gz|bz2|xz)',       ('orig_tar',)),
         (r'tar\.(gz|bz2)',             ('native_tar',)),
-        (r'orig-.+\.tar\.(gz|bz2)',    ('more_orig_tar',)),
+        (r'orig-.+\.tar\.(gz|bz2|xz)',    ('more_orig_tar',)),
     )
 
     for f in dsc_files.keys():




Severity set to 'wishlist' from 'normal' Request was from Luca Falavigna <dktrkranz@debian.org> to control@bugs.debian.org. (Thu, 26 Aug 2010 20:30:05 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian FTP Master <ftpmaster@ftp-master.debian.org>:
Bug#556407; Package ftp.debian.org. (Sat, 27 Nov 2010 09:36:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Raphael Hertzog <hertzog@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian FTP Master <ftpmaster@ftp-master.debian.org>. (Sat, 27 Nov 2010 09:36:03 GMT) Full text and rfc822 format available.

Message #12 received at 556407@bugs.debian.org (full text, mbox):

From: Raphael Hertzog <hertzog@debian.org>
To: Clint Adams <schizo@debian.org>, 556407@bugs.debian.org
Subject: Re: Bug#556407: data.tar.xz
Date: Sat, 27 Nov 2010 10:32:49 +0100
On Sun, 15 Nov 2009, Clint Adams wrote:
> I'll ask for xz support.
> 
> This won't work until http://bugs.python.org/issue6715 is resolved.

This is going to be in python3 only apparently. So we should find a
work-around in the mean time I think.

> @@ -198,6 +198,8 @@ class Binary(object):
>                          data = tarfile.open(os.path.join(self.tmpdir, "data.tar.gz"), "r:gz")
>                      elif self.chunks[2] == "data.tar.bz2":
>                          data = tarfile.open(os.path.join(self.tmpdir, "data.tar.bz2" ), "r:bz2")
> +                    elif self.chunks[2] == "data.tar.xz":
> +                        data = tarfile.open(os.path.join(self.tmpdir, "data.tar.xz" ), "r:xz")


That's the only place where it's needed. We could try to extract it on the
fly (xzcat or xz --decompress --stdout) and pass the uncompressed stream with:
tarfile.open(None, "r|", stream)

Quoting http://docs.python.org/library/tarfile.html there's a limitation though:
| For special purposes, there is a second format for mode:
| 'filemode|[compression]'. tarfile.open() will return a TarFile object that
| processes its data as a stream of blocks. No random seeking will be done on the
| file. If given, fileobj may be any object that has a read() or write() method
| (depending on the mode). bufsize specifies the blocksize and defaults to 20 *
| 512 bytes. Use this variant in combination with e.g. sys.stdin, a socket file
| object or a tape device. However, such a TarFile object is limited in that it
| does not allow to be accessed randomly, see Examples. The currently possible
| modes:

I haven't checked whether it requires more changes later in the code.

It would be nice to allow data.tar.xz as soon as wheezy opens.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)




Information forwarded to debian-bugs-dist@lists.debian.org, Debian FTP Master <ftpmaster@ftp-master.debian.org>:
Bug#556407; Package ftp.debian.org. (Wed, 02 Feb 2011 11:33:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Charles Plessy <plessy@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian FTP Master <ftpmaster@ftp-master.debian.org>. (Wed, 02 Feb 2011 11:33:03 GMT) Full text and rfc822 format available.

Message #17 received at 556407@bugs.debian.org (full text, mbox):

From: Charles Plessy <plessy@debian.org>
To: 556407@bugs.debian.org
Cc: Clint Adams <schizo@debian.org>, Raphael Hertzog <hertzog@debian.org>
Subject: Re: Bug#556407: data.tar.xz
Date: Wed, 2 Feb 2011 20:30:09 +0900
Le Sat, Nov 27, 2010 at 10:32:49AM +0100, Raphael Hertzog a écrit :
> 
> > @@ -198,6 +198,8 @@ class Binary(object):
> >                          data = tarfile.open(os.path.join(self.tmpdir, "data.tar.gz"), "r:gz")
> >                      elif self.chunks[2] == "data.tar.bz2":
> >                          data = tarfile.open(os.path.join(self.tmpdir, "data.tar.bz2" ), "r:bz2")
> > +                    elif self.chunks[2] == "data.tar.xz":
> > +                        data = tarfile.open(os.path.join(self.tmpdir, "data.tar.xz" ), "r:xz")
> 
> 
> That's the only place where it's needed. We could try to extract it on the
> fly (xzcat or xz --decompress --stdout) and pass the uncompressed stream with:
> tarfile.open(None, "r|", stream)

Hello everybody,

given that ‘malformed-deb-archive’ is already listed as fatal in
config/debian/lintian.tags, another solution would be to enhance Lintian to
perform all the checks instead of dak itself. If there is an interst
following that way, I volunteer to work on it.

Cheers,

-- 
Charles Plessy
Tsurumi, Kanagawa, Japan




Changed Bug submitter to 'Clint Adams <clint@debian.org>' from 'Clint Adams <schizo@debian.org>' Request was from Clint Adams <clint@debian.org> to control@bugs.debian.org. (Sat, 12 Feb 2011 01:09:46 GMT) Full text and rfc822 format available.

Reply sent to Ansgar Burchardt <ansgar@43-1.org>:
You have taken responsibility. (Thu, 28 Jul 2011 12:49:12 GMT) Full text and rfc822 format available.

Notification sent to Clint Adams <clint@debian.org>:
Bug acknowledged by developer. (Thu, 28 Jul 2011 12:49:16 GMT) Full text and rfc822 format available.

Message #24 received at 556407-done@bugs.debian.org (full text, mbox):

From: Ansgar Burchardt <ansgar@43-1.org>
To: 556407-done@bugs.debian.org
Subject: Re: Bug#556407: data.tar.xz
Date: Thu, 28 Jul 2011 14:39:54 +0200
dak can noe handle XZ in both binary and source packages, however before
packages using it will be accepted by dak we need a new version of
lintian (#632556).

Regards,
Ansgar




Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Fri, 26 Aug 2011 07:31:53 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Thu Apr 17 12:31:39 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.