Debian Bug report logs - #531321
[uscan] support for downloading multiple upstream tarballs

version graph

Package: devscripts; Maintainer for devscripts is Devscripts Devel Team <devscripts-devel@lists.alioth.debian.org>; Source for devscripts is src:devscripts.

Reported by: Stefano Zacchiroli <zack@debian.org>

Date: Sun, 31 May 2009 15:12:02 UTC

Severity: wishlist

Found in version devscripts/2.10.50

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Devscripts Devel Team <pkg-devscripts@teams.debian.net>:
Bug#531321; Package devscripts. (Sun, 31 May 2009 15:12:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Stefano Zacchiroli <zack@debian.org>:
New Bug report received and forwarded. Copy sent to Devscripts Devel Team <pkg-devscripts@teams.debian.net>. (Sun, 31 May 2009 15:12:04 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Stefano Zacchiroli <zack@debian.org>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: [uscan] support for downloading multiple upstream tarballs
Date: Sun, 31 May 2009 17:07:40 +0200
Package: devscripts
Version: 2.10.50
Severity: wishlist
File: /usr/bin/uscan

The context is the "packaging multiple-egg" part of my blog post
available at [1]. The scenario is quite common: in the occasions where
you do have source packages depending on multiple upstream tarballs,
you have to keep track of them.

Now that we have get-orig-source in policy, I believe it makes sense
to have it rely on uscan and then do the extra needed work of tarball
merging.

The part on which uscan can improve its support is automation of the
download of multiple tarballs. My current hack^Wsolution is to rely on
several debian/*.watch files and invoke it from a Makefile snippet
[2]. What about supporting this out of the box in uscan?

I see two possible ways of doing that:

- make uscan understand debian/*.watch files, in the style of devscripts
- extend the debian/watch format to support multiple declarations

Given that debian/watch is already based on single logical lines
(split using continuations), the latter solution seems cleaner to me.

What problems, if any, inhibit adding such support?

I'm sure that not all features currently provided by uscan can be
applied on all downloaded tarballs, but I'm confident most can be.

Cheers.

[1] http://upsilon.cc/~zack/blog/posts/2009/05/kick-starting_turbogears_2_packaging/
[2] http://svn.debian.org/wsvn/python-modules/packages/python-repoze.who-plugins/trunk/debian/get-orig-source.mk

-- Package-specific info:

--- /etc/devscripts.conf ---

--- ~/.devscripts ---
DEBUILD_ROOTCMD=fakeroot
DEBCHANGE_RELEASE_HEURISTIC=changelog

-- System Information:
Debian Release: squeeze/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.29-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages devscripts depends on:
ii  dpkg-dev                      1.15.2     Debian package development tools
ii  libc6                         2.9-13     GNU C Library: Shared libraries
ii  perl                          5.10.0-22  Larry Wall's Practical Extraction 

Versions of packages devscripts recommends:
ii  at                 3.1.10.2              Delayed job execution and batch pr
ii  bsd-mailx [mailx]  8.1.2-0.20081101cvs-2 A simple mail user agent
ii  bzr                1.15~rc1-1            easy to use distributed version co
ii  curl               7.19.5-1              Get a file from an HTTP, HTTPS or 
ii  cvs                1:1.12.13-12          Concurrent Versions System
ii  dctrl-tools        2.14                  Command-line tools to process Debi
ii  debian-keyring     2009.05.28            GnuPG (and obsolete PGP) keys of D
ii  debian-maintainers 1.59                  GPG keys of Debian maintainers
ii  dput               0.9.4                 Debian package upload tool
ii  epiphany-webkit [w 2.27.2-1              Intuitive GNOME web browser - WebK
ii  equivs             2.0.7-0.1             Circumvent Debian package dependen
ii  fakeroot           1.12.2                Gives a fake root environment
ii  galeon [www-browse 2.0.7-1               GNOME web browser for advanced use
ii  git-core           1:1.6.3.1-1           fast, scalable, distributed revisi
ii  gnupg              1.4.9-4               GNU privacy guard - a free PGP rep
ii  iceweasel [www-bro 3.0.9-1               lightweight web browser based on M
ii  libauthen-sasl-per 2.12-1                Authen::SASL - SASL Authentication
ii  libcrypt-ssleay-pe 0.57-1+b1             Support for https protocol in LWP
ii  libparse-debcontro 2.005-2               Easy OO parsing of Debian control-
ii  libsoap-lite-perl  0.710.08-2            Client and server side SOAP implem
ii  libterm-size-perl  0.2-4+b1              Perl extension for retrieving term
ii  libtimedate-perl   1.1600-9              Time and date functions for Perl
ii  liburi-perl        1.37+dfsg-1           Manipulates and accesses URI strin
ii  libwww-perl        5.826-1               WWW client/server library for Perl
ii  libyaml-syck-perl  1.07-1                fast, lightweight YAML loader and 
ii  lintian            2.2.10                Debian package checker
ii  lsb-release        3.2-22                Linux Standard Base version report
ii  lynx-cur [www-brow 2.8.7pre4-1           Text-mode WWW Browser with NLS sup
ii  man-db             2.5.5-2               on-line manual pager
ii  openssh-client [ss 1:5.1p1-5+b1          secure shell client, an rlogin/rsh
ii  patch              2.5.9-5               Apply a diff file to an original
ii  patchutils         0.3.1-2               Utilities to work with patches
ii  strace             4.5.18-1              A system call tracer
ii  subversion         1.5.6dfsg-1           Advanced version control system
ii  unzip              6.0-1                 De-archiver for .zip files
ii  w3m [www-browser]  0.5.2-2+b1            WWW browsable pager with excellent
ii  wdiff              0.5-18                Compares two files word by word
ii  wget               1.11.4-2              retrieves files from the web

Versions of packages devscripts suggests:
ii  build-essential               11.4       Informational list of build-essent
pn  cvs-buildpackage              <none>     (no description available)
pn  devscripts-el                 <none>     (no description available)
ii  gnuplot                       4.2.5-2    A command-line driven interactive 
ii  libfile-desktopentry-perl     0.04-2     Perl module to handle freedesktop 
pn  libnet-smtp-ssl-perl          <none>     (no description available)
ii  mutt                          1.5.19-3   text-based mailreader supporting M
ii  svn-buildpackage              0.6.25     helper programs to maintain Debian

-- no debconf information




Information forwarded to debian-bugs-dist@lists.debian.org, Devscripts Devel Team <pkg-devscripts@teams.debian.net>:
Bug#531321; Package devscripts. (Sun, 31 May 2009 18:42:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Adam D. Barratt" <adam@adam-barratt.org.uk>:
Extra info received and forwarded to list. Copy sent to Devscripts Devel Team <pkg-devscripts@teams.debian.net>. (Sun, 31 May 2009 18:42:02 GMT) Full text and rfc822 format available.

Message #10 received at 531321@bugs.debian.org (full text, mbox):

From: "Adam D. Barratt" <adam@adam-barratt.org.uk>
To: Stefano Zacchiroli <zack@debian.org>, 531321@bugs.debian.org
Subject: Re: Bug#531321: [uscan] support for downloading multiple upstream tarballs
Date: Sun, 31 May 2009 19:39:51 +0100
On Sun, 2009-05-31 at 17:07 +0200, Stefano Zacchiroli wrote:
> The part on which uscan can improve its support is automation of the
> download of multiple tarballs. My current hack^Wsolution is to rely on
> several debian/*.watch files and invoke it from a Makefile snippet
> [2]. What about supporting this out of the box in uscan?
> 
> I see two possible ways of doing that:
> 
> - make uscan understand debian/*.watch files, in the style of devscripts
> - extend the debian/watch format to support multiple declarations

This second option should /mostly/ work already.  Certainly, multiple
entries in a single watch file are supported; the only problem is that
there is an (not altogether unreasonable given the tool's design)
assumption that they are all for the same package.

By way of an example, I concatenated two watchfiles I had to hand.  As
you can see, both upstream tarballs are successfully downloaded.  The
--no-symlink option is required as otherwise one ends up with
foo_1.8.orig.tar.gz as a symlink to the scheme48 tarball and
foo_3.4.3.orig.tar.gz as a symlink to that for v-sim.

$ cat watchmulti 
version=3
opts=downloadurlmangle=s{([^/]+)/download.html$}{$1/scheme48-$1.tgz},filenamemangle=s{([^/]+)/download.html}{scheme48-$1.tgz} \
  http://s48.org/index.html  ([\d\.]+)/download.html

opts=filenamemangle=s/.*v_sim-(.*).tar.bz2/v-sim_$1.orig.tar.bz2/,downloadurlmangle=s{-confirm.en.html\?}{/} \
  http://inac.cea.fr/L_Sim/V_Sim/download.html .*v_sim-(.*).tar.bz2 

$ uscan --watchfile=watchmulti --upstream-version=0 --package=foo --download --repack --no-symlink
foo: Newer version (1.8) available on remote site:
  http://s48.org/1.8/scheme48-1.8.tgz
  (local version is 0)
foo: Successfully downloaded updated package scheme48-1.8.tgz
foo: Newer version (3.4.3) available on remote site:
  http://inac.cea.fr/L_Sim/V_Sim/download/v_sim-3.4.3.tar.bz2
  (local version is 0)
foo: Successfully downloaded updated package v-sim_3.4.3.orig.tar.gz


Regards,

Adam




Information forwarded to debian-bugs-dist@lists.debian.org, Devscripts Devel Team <pkg-devscripts@teams.debian.net>:
Bug#531321; Package devscripts. (Sun, 15 Nov 2009 19:18:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Stefano Zacchiroli <zack@debian.org>:
Extra info received and forwarded to list. Copy sent to Devscripts Devel Team <pkg-devscripts@teams.debian.net>. (Sun, 15 Nov 2009 19:18:04 GMT) Full text and rfc822 format available.

Message #15 received at 531321@bugs.debian.org (full text, mbox):

From: Stefano Zacchiroli <zack@debian.org>
To: pkg-devscripts@teams.debian.net, debian-x@lists.debian.org
Cc: 531321@bugs.debian.org
Subject: Re: uscan, watchfiles and multiple tarballs
Date: Sun, 15 Nov 2009 20:11:08 +0100
[Message part 1 (text/plain, inline)]
[ fully quoting your post to the benefit of the freshly Cc-ed bug log ]

On Sun, Nov 15, 2009 at 06:27:13PM +0100, Julien Cristau wrote:
> with v3 source packages being accepted into the archive I started
> looking at how I could split up the x11-apps and friends tarballs by
> shipping the tarballs from upstream directly, and I'm trying to get
> uscan to download them for me.
> 
> Basically I have tarballs for bitmap, oclock and friends at
> http://xorg.freedesktop.org/releases/individual/app/ each with its
> version number, and I'd like to get, say,
> x11-apps_7.5+1.orig-{bitmap,oclock,...}.tar.bz2
> 
> This would mean that there's no direct mapping from the debian package
> version (in debian/changelog) from/to each upstream tarball version (in
> foo/configure), and I can't seem to make that work with uscan (I can
> make it download the tarballs[1], just not rename them as I'd like).  I'd
> welcome any suggestions (explaining why I'm insane works, too)…

Hi, I hit a similar issue when packaging some Python libraries [1], the
result was filing 531321. With Adam we intended to work on it at last
DebConf, but eventually we were too busy working on other stuff to
actually do that.

My idea back then was actually simpler than yours: simply allow multiple
declarations in debian/watch and kind of use a "always force" mode where
uscan always download everything. Around that you can have some
repacking logics and a dumb diff that checks whether the freshly
repacked tarballs differs from the old one.
What you propose/hint is clearly better.

The main design point to decide upon is where to store the information
about the last downloaded version of each sub-tarball. debian/changelog
does not look like a good idea to me, because the amount of information
can get messy.

How about deciding that the debian/watch* "namespace" is reserved to
uscan? At that point we can imagine a debian/watch.something file which
keeps track of sub-tarball versions and make uscan automatically update
it when downloading new sub-tarballs and/or by a specific invocation.

Any other ideas about where to make uscan store such information?

Cheers.

[1] basically, when you have several related .egg files each of which
    just a few Kb, in such cases having one package per-egg is really
    overkilling

-- 
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..|  .  |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...........| ..: |.... Je dis tu à tous ceux que j'aime
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Devscripts Devel Team <pkg-devscripts@teams.debian.net>:
Bug#531321; Package devscripts. (Fri, 27 Aug 2010 14:48:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Nicholas Bamber <nicholas@periapt.co.uk>:
Extra info received and forwarded to list. Copy sent to Devscripts Devel Team <pkg-devscripts@teams.debian.net>. (Fri, 27 Aug 2010 14:48:06 GMT) Full text and rfc822 format available.

Message #20 received at 531321@bugs.debian.org (full text, mbox):

From: Nicholas Bamber <nicholas@periapt.co.uk>
To: 531321@bugs.debian.org, 531321-subscribe@bugs.debian.org
Cc: debian-perl@lists.debian.org
Subject: my suggestion for multiple upstream tar balls
Date: Fri, 27 Aug 2010 15:38:22 +0100
[Message part 1 (text/plain, inline)]
I was thinking about this. Since packages are supposed to be worth the 
packaging overhead what do you do about a series of very small plugins 
consisting of just a few lines? CPAN is full of such modules and 
libcgi-application-plugins-perl is another good example.

The obvious thing to me would seem to be to store the current versions in
debian/source/versions

This file would consist of lines of the form
URL space version-number  new-line.

Although human readable and part of the packaging source code, it would 
be maintained entirely by uscan and driven by the watch file. uscan 
would not bother to create a versions file if there was only one rule in 
the watch file and the version number in debian/changelog made sense to it.
[nicholas.vcf (text/x-vcard, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Devscripts Devel Team <pkg-devscripts@teams.debian.net>:
Bug#531321; Package devscripts. (Fri, 27 Aug 2010 15:03:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to gregor herrmann <gregoa@debian.org>:
Extra info received and forwarded to list. Copy sent to Devscripts Devel Team <pkg-devscripts@teams.debian.net>. (Fri, 27 Aug 2010 15:03:03 GMT) Full text and rfc822 format available.

Message #25 received at 531321@bugs.debian.org (full text, mbox):

From: gregor herrmann <gregoa@debian.org>
To: Nicholas Bamber <nicholas@periapt.co.uk>
Cc: 531321@bugs.debian.org, debian-perl@lists.debian.org
Subject: Re: my suggestion for multiple upstream tar balls
Date: Fri, 27 Aug 2010 17:00:02 +0200
[Message part 1 (text/plain, inline)]
On Fri, 27 Aug 2010 15:38:22 +0100, Nicholas Bamber wrote:

> The obvious thing to me would seem to be to store the current versions in
> debian/source/versions
> 
> This file would consist of lines of the form
> URL space version-number  new-line.

Take a look at libcatalyst-modules-perl, there's something similar
(tarballs/packages.cfg + scripts)
 
> Although human readable and part of the packaging source code, it
> would be maintained entirely by uscan and driven by the watch file.
> uscan would not bother to create a versions file if there was only
> one rule in the watch file and the version number in
> debian/changelog made sense to it.

uscan integration sounds nice.

Even nicer would be to use the multi-tarball feature of source format
v3, but TTBOMK svn-buildpackage/-inject/-upgrade don't handle it yet.

In any case, improvements to this multi-dist packages would be much
appreciated, currently they are a pain to deal with.

Cheers,
gregor

-- 
 .''`.   http://info.comodo.priv.at/ -- GPG key IDs: 0x8649AA06, 0x00F3CFE4
 : :' :  Debian GNU/Linux user, admin, & developer - http://www.debian.org/
 `. `'   Member of VIBE!AT & SPI, fellow of Free Software Foundation Europe
   `-    NP: Rolling Stones: Crisscrossman
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Devscripts Devel Team <pkg-devscripts@teams.debian.net>:
Bug#531321; Package devscripts. (Fri, 27 Aug 2010 16:03:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Nicholas Bamber <nicholas@periapt.co.uk>:
Extra info received and forwarded to list. Copy sent to Devscripts Devel Team <pkg-devscripts@teams.debian.net>. (Fri, 27 Aug 2010 16:03:06 GMT) Full text and rfc822 format available.

Message #30 received at 531321@bugs.debian.org (full text, mbox):

From: Nicholas Bamber <nicholas@periapt.co.uk>
To: Nicholas Bamber <nicholas@periapt.co.uk>, 531321@bugs.debian.org, debian-perl@lists.debian.org
Subject: Re: my suggestion for multiple upstream tar balls
Date: Fri, 27 Aug 2010 16:53:02 +0100
[Message part 1 (text/plain, inline)]
That libcatalyst-modules-perl stuff surely  deserves its own source format?
I can see some more issues with this multiple sources stuff.
Because you cannot identify a single version number you call it a native 
format.
But making it a native format is surely against the normal Debian 
approach to packaging
where the upstream sources are distributed along side and untouched 
(apart from a  little standardization of the name).

gregor herrmann wrote:
> On Fri, 27 Aug 2010 15:38:22 +0100, Nicholas Bamber wrote:
>
>   
>> The obvious thing to me would seem to be to store the current versions in
>> debian/source/versions
>>
>> This file would consist of lines of the form
>> URL space version-number  new-line.
>>     
>
> Take a look at libcatalyst-modules-perl, there's something similar
> (tarballs/packages.cfg + scripts)
>  
>   
>> Although human readable and part of the packaging source code, it
>> would be maintained entirely by uscan and driven by the watch file.
>> uscan would not bother to create a versions file if there was only
>> one rule in the watch file and the version number in
>> debian/changelog made sense to it.
>>     
>
> uscan integration sounds nice.
>
> Even nicer would be to use the multi-tarball feature of source format
> v3, but TTBOMK svn-buildpackage/-inject/-upgrade don't handle it yet.
>
> In any case, improvements to this multi-dist packages would be much
> appreciated, currently they are a pain to deal with.
>
> Cheers,
> gregor
>
>   

[nicholas.vcf (text/x-vcard, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Devscripts Devel Team <pkg-devscripts@teams.debian.net>:
Bug#531321; Package devscripts. (Fri, 03 Feb 2012 09:42:18 GMT) Full text and rfc822 format available.

Acknowledgement sent to Paul Wise <pabs@debian.org>:
Extra info received and forwarded to list. Copy sent to Devscripts Devel Team <pkg-devscripts@teams.debian.net>. (Fri, 03 Feb 2012 09:42:24 GMT) Full text and rfc822 format available.

Message #35 received at 531321@bugs.debian.org (full text, mbox):

From: Paul Wise <pabs@debian.org>
To: Debian Bug Tracking System <531321@bugs.debian.org>
Subject: Re: [uscan] support for downloading multiple upstream tarballs
Date: Fri, 03 Feb 2012 17:39:24 +0800
[Message part 1 (text/plain, inline)]
usertags 531321 + bittenby
thanks

I recently had a related issue. I wanted to download the current
tarballs of a dpkg-source v3 multi-tarball package. Unfortunately as you
can see below, there is no way to specify which dpkg-source v3 component
name to use for each watch file line. I was able to work around this
easily but it would be nice if this worked in future.

pabs@chianamo ~/devel/debian/games/svn/megaglest-data $ cat debian/watch 
version=3
# The built data is here:
http://sf.net/megaglest/megaglest-data-([\d.]+)\.(?:zip|tar\.gz|tar\.bz2|tar\.xz)
# The data source code is here:
http://sf.net/megaglest/megaglest-data-source-([\d.]+)\.(?:zip|tar\.gz|tar\.bz2|tar\.xz)
pabs@chianamo ~/devel/debian/games/svn/megaglest-data $ uscan --verbose --download-current-version
-- Scanning for watchfiles in .
-- Found watchfile in ./debian
-- In debian/watch, processing watchfile line:
   http://sf.net/megaglest/megaglest-data-([\d.]+)\.(?:zip|tar\.gz|tar\.bz2|tar\.xz)
-- Found the following matching hrefs:
     megaglest-data-3.6.0.tar.xz
     megaglest-data-3.6.0.1.tar.xz
     megaglest-data-3.6.0.3.tar.xz
     megaglest-data-3.6.0.2.tar.xz
     megaglest-data-3.5.2.4.tar.xz
     megaglest-data-3.5.2.3.tar.xz
     megaglest-data-3.5.1.tar.xz
Newest version on remote site is 3.6.0.3, local version is 3.6.0.3
 => Package is up to date
Newest version on remote site is 3.6.0.3, local version is 3.6.0.3
 => Forcing download as requested
-- Downloading updated package megaglest-data-3.6.0.3.tar.xz
-- Successfully downloaded updated package megaglest-data-3.6.0.3.tar.xz
    and renamed it as megaglest-data_3.6.0.3.orig.tar.xz
-- In debian/watch, processing watchfile line:
   http://sf.net/megaglest/megaglest-data-source-([\d.]+)\.(?:zip|tar\.gz|tar\.bz2|tar\.xz)
-- Found the following matching hrefs:
     megaglest-data-source-3.6.0.3.tar.xz
     megaglest-data-source-3.6.0.3.tar.xz
     megaglest-data-source-3.6.0.3.tar.xz
     megaglest-data-source-3.6.0.3.tar.xz
     megaglest-data-source-3.6.0.3.tar.xz
     megaglest-data-source-3.6.0.3.tar.xz
     megaglest-data-source-3.6.0.3.tar.xz
     megaglest-data-source-3.6.0.2.tar.xz
     megaglest-data-source-3.6.0.2.tar.xz
     megaglest-data-source-3.6.0.2.tar.xz
     megaglest-data-source-3.6.0.2.tar.xz
     megaglest-data-source-3.6.0.2.tar.xz
     megaglest-data-source-3.6.0.2.tar.xz
     megaglest-data-source-3.6.0.2.tar.xz
Newest version on remote site is 3.6.0.3, local version is 3.6.0.3
 => Package is up to date
Newest version on remote site is 3.6.0.3, local version is 3.6.0.3
 => megaglest-data_3.6.0.3.orig.tar.xz already in package directory '..'
-- Scan finished

-- 
bye,
pabs

http://wiki.debian.org/PaulWise
[signature.asc (application/pgp-signature, inline)]

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Sun Apr 20 23:55:30 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.