Debian Bug report logs - #466550
Clarify or remove the get-orig-source target specification

version graph

Package: debian-policy; Maintainer for debian-policy is Debian Policy List <debian-policy@lists.debian.org>; Source for debian-policy is src:debian-policy.

Reported by: Alexander Schmehl <alexander@schmehl.info>

Date: Tue, 19 Feb 2008 14:21:05 UTC

Severity: wishlist

Tags: confirmed

Found in version debian-policy/3.7.3.0

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, debian-mentors@lists.debian.org, debian-devel-games@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. Full text and rfc822 format available.

Acknowledgement sent to Alexander Schmehl <alexander@schmehl.info>:
New Bug report received and forwarded. Copy sent to debian-mentors@lists.debian.org, debian-devel-games@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>. Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Alexander Schmehl <alexander@schmehl.info>
To: submit@bugs.debian.org
Subject: Please clarify the get-orig-source target stated in Policy 4.9
Date: Tue, 19 Feb 2008 15:16:54 +0100
Package: debian-policy
Version: 3.7.3.0
Severity: wishlist
X-Debbugs-CC: debian-mentors@lists.debian.org, debian-devel-games@lists.debian.org

Dear policy team,

recently the get-orig-source target of debian/rules has been discussed
on the debian-mentors list (see the threads starting with [1] and [2]).

It seems the get-orig-source specific paragraph of section 4.9 should be
improved to a bit more clearly and answer some open questions, too.

Basically it boils down to two or three open questions:

The first one being, if get-orig-source is intendedn to fetches the most
recent version of the original source upstream wise or if it should
fetch the most recent version debian wise.

If the later is the case and of a package has a version in
experimentatl, should get-orig-source fetch the version of experimental
or from unstable?

And last questions:  Where should tools used in get-orig-source (e.g.
bzip2 or unzip to repacke a tarball) be declared?  As Build-Dependency?
Anywhere else?


Links:
  1: http://lists.debian.org/debian-mentors/2008/02/msg00402.html
  2: http://lists.debian.org/debian-mentors/2008/02/msg00455.html


Yours sincerely,
  Alexander




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. Full text and rfc822 format available.

Acknowledgement sent to Andres Mejia <mcitadel@gmail.com>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. Full text and rfc822 format available.

Message #10 received at 466550@bugs.debian.org (full text, mbox):

From: Andres Mejia <mcitadel@gmail.com>
To: debian-devel-games@lists.debian.org, 466550@bugs.debian.org, debian-mentors@lists.debian.org
Subject: Re: Bug#466550: Please clarify the get-orig-source target stated in Policy 4.9
Date: Tue, 19 Feb 2008 11:01:18 -0500
I would like to add what Russ Allbery wrote.

On Tuesday 19 February 2008 1:47:25 am Russ Allbery wrote:
. . . 
> Personally, I've always read it has emphasizing an entirely different part
> than what people are talking about here.  Rather than focusing on the
> current version bit, I always focused on the "does any necessary
> rearrangement to turn it into the original source tar file format
> described below" bit.  I provide this target only for my packages that
> require repackaging of the upstream source as a way of automating that
> repackaging.
. . .

When considering the phrase in policy "...does any necessary rearrangement to 
turn it into the original source tar file format...", it makes more sense 
when it refers to the original source tar file of the source package in 
Debian (or will be in Debian), and the original source tarball would be at a 
particular version, and thus the get-orig-source should generate that 
particular version of the original source tarball of the source package in 
Debian.

-- 
Regards,
Andres




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. Full text and rfc822 format available.

Acknowledgement sent to Daniel Leidert <daniel.leidert.spam@gmx.net>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. Full text and rfc822 format available.

Message #15 received at 466550@bugs.debian.org (full text, mbox):

From: Daniel Leidert <daniel.leidert.spam@gmx.net>
To: 466550@bugs.debian.org
Cc: ml_debian-mentors <debian-mentors@lists.debian.org>
Subject: Re: Bug#466550: Please clarify the get-orig-source target stated in Policy 4.9
Date: Tue, 19 Feb 2008 18:11:21 +0100
Am Dienstag, den 19.02.2008, 15:16 +0100 schrieb Alexander Schmehl:
> Package: debian-policy
> Version: 3.7.3.0
> Severity: wishlist
> X-Debbugs-CC: debian-mentors@lists.debian.org, debian-devel-games@lists.debian.org
> 
> Dear policy team,
> 
> recently the get-orig-source target of debian/rules has been discussed
> on the debian-mentors list (see the threads starting with [1] and [2]).
> 
> It seems the get-orig-source specific paragraph of section 4.9 should be
> improved to a bit more clearly and answer some open questions, too.
> 
> Basically it boils down to two or three open questions:
> 
> The first one being, if get-orig-source is intendedn to fetches the most
> recent version of the original source upstream wise or if it should
> fetch the most recent version debian wise.

That should be done by uscan invoking "debian/rules get-orig-source" as
suggested several times. This won't make the situation more complicated.
It just requires that any variables necessary to create the tarball
(e.g. upstream version, svn version number, ...) can be overwritten by
uscan (already explained it in this thread)

> If the later is the case and of a package has a version in
> experimentatl, should get-orig-source fetch the version of experimental
> or from unstable?

It should IMO fetch the version by parsing debian/changelog. That's
simple and logical. uscan can overwrite these variables.

> And last questions:  Where should tools used in get-orig-source (e.g.
> bzip2 or unzip to repacke a tarball) be declared?  As Build-Dependency?
> Anywhere else?

I don't think, they should be declared in Build-Depends, because they
are simply not necessary to build the package. No target in debian/rules
invokes get-orig-source during a build and people invoking this target
should be able to check debian/rules, if something goes wrong. But to be
honest: This target normally requires only bzip2/gzip/tar, wget (maybe
we can drop this, if uscan invokes get-orig-source) and rm + sometimes
$VCS-command.

Regards, Daniel





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. Full text and rfc822 format available.

Acknowledgement sent to Kapil Hari Paranjape <kapil@imsc.res.in>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. Full text and rfc822 format available.

Message #20 received at 466550@bugs.debian.org (full text, mbox):

From: Kapil Hari Paranjape <kapil@imsc.res.in>
To: Andres Mejia <mcitadel@gmail.com>
Cc: debian-devel-games@lists.debian.org, 466550@bugs.debian.org, debian-mentors@lists.debian.org
Subject: Re: Bug#466550: Please clarify the get-orig-source target stated in Policy 4.9
Date: Tue, 19 Feb 2008 23:15:49 +0530
[Message part 1 (text/plain, inline)]
Hello,

Perhaps the following elaborate statement can be condensed (once
sufficient cooling has occurred :-))

1. Once pkg_ver.orig.tar.gz enters the Debian archive this is
considered the authoritative Debian version from which all the binary
Debian packages will be built (for that version of the package). A
signature/checksum is used (in the upload and the Sources.gz file) so
as to detect any "contamination".

2. If re-packaging of upstream sources was required in order to create
this .orig.tar.gz, then this should be documented in the copyright
file (with some further explication in README.Debian-source perhaps).

3. Whenever upstream releases a new version, one needs to create a
pkg_nver.orig.tar.gz for the newer version.

In case this is merely a matter of downloading and renaming an
upstream tar.gz, the "uscan" and "uupdate" programs are adequate and
there is no significant need for a get-orig-source target.

In the case when re-packaging has been done as in (2), it is
a non-trivial convenience if these steps are automated by such
a program or target. Such a program further clarifies the statements
in the copyright file and the README.Debian-source file. (Program as
documentation!)

In the last case, someone who wishes to verify the accuracy of the
statements in the copyright file may also wish to re-generate
pkg_ver.orig.tar.gz to compare it with the Debian version. This
can also be provided for to the extent possible.

If there is any reason to suspect that the pkg_ver.orig.tar.gz was
not in fact created as documented then this constitutes a bug whose
severity would depend on the extent of the discrepancy.

Regards,

Kapil.
--
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. Full text and rfc822 format available.

Acknowledgement sent to Alexander Schmehl <tolimar@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. Full text and rfc822 format available.

Message #25 received at 466550@bugs.debian.org (full text, mbox):

From: Alexander Schmehl <tolimar@debian.org>
To: 466550@bugs.debian.org, ml_debian-mentors <debian-mentors@lists.debian.org>
Subject: Re: Bug#466550: Please clarify the get-orig-source target stated in Policy 4.9
Date: Tue, 19 Feb 2008 19:28:51 +0100
[Message part 1 (text/plain, inline)]
Hi!

* Daniel Leidert <daniel.leidert.spam@gmx.net> [080219 18:11]:

> That should be done by uscan invoking "debian/rules get-orig-source" as
> suggested several times.

Thanks for your input, but the question of this supthread is not "How"
but "What" ;)



> > And last questions:  Where should tools used in get-orig-source (e.g.
> > bzip2 or unzip to repacke a tarball) be declared?  As Build-Dependency?
> > Anywhere else?
> I don't think, they should be declared in Build-Depends, because they
> are simply not necessary to build the package. [..]

I agree.  But shouldn't needed packages be documented somewhere?


Yours sincerely,
  Alexander
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. Full text and rfc822 format available.

Acknowledgement sent to Manoj Srivastava <srivasta@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. Full text and rfc822 format available.

Message #30 received at 466550@bugs.debian.org (full text, mbox):

From: Manoj Srivastava <srivasta@debian.org>
To: 466550@bugs.debian.org
Subject: Re: Bug#466550: Please clarify the get-orig-source target stated in Policy 4.9
Date: Thu, 21 Feb 2008 02:31:54 -0600
On Tue, 19 Feb 2008 11:01:18 -0500, Andres Mejia <mcitadel@gmail.com> said: 

> I would like to add what Russ Allbery wrote.
> On Tuesday 19 February 2008 1:47:25 am Russ Allbery wrote:
> . . . 
>> Personally, I've always read it has emphasizing an entirely different
>> part than what people are talking about here.  Rather than focusing
>> on the current version bit, I always focused on the "does any
>> necessary rearrangement to turn it into the original source tar file
>> format described below" bit.  I provide this target only for my
>> packages that require repackaging of the upstream source as a way of
>> automating that repackaging.
> . . .

> When considering the phrase in policy "...does any necessary
> rearrangement to turn it into the original source tar file format...",
> it makes more sense when it refers to the original source tar file of
> the source package in Debian (or will be in Debian), and the original
> source tarball would be at a particular version, and thus the
> get-orig-source should generate that particular version of the
> original source tarball of the source package in Debian.

        While this jells with my recollection of the intent of the
 target, the question now is whether this directive actually makes sense
 as policy at this point.  It obviously does not reflect common
 practice, since the common practice is not to implement this
 target. There is no deep scaffolding that depends on this target, and
 indeed, at this point, there does not appear to be a consensus about
 what this target should do, and whether it is in fact useful.

        My suggestion is that we pare policy down, and remove this
 mostly useless and mostly ignored rule.

        manoj
-- 
Dyslexia means never having to say that you're ysror.
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C




Changed Bug title to `Clarify or remove the get-orig-source target specification' from `Please clarify the get-orig-source target stated in Policy 4.9'. Request was from Russ Allbery <rra@debian.org> to control@bugs.debian.org. (Mon, 17 Mar 2008 06:15:17 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 05 Mar 2009 08:09:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Finney <ben@benfinney.id.au>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 05 Mar 2009 08:09:06 GMT) Full text and rfc822 format available.

Message #37 received at 466550@bugs.debian.org (full text, mbox):

From: Ben Finney <ben@benfinney.id.au>
To: Manoj Srivastava <srivasta@debian.org>
Cc: 466550@bugs.debian.org
Subject: Re: Bug#466550: Please clarify the get-orig-source target stated in Policy 4.9
Date: Thu, 5 Mar 2009 19:06:58 +1100
[Message part 1 (text/plain, inline)]
On 21-Feb-2008, Manoj Srivastava wrote:
>  the question now is whether this directive actually makes sense as
>  policy at this point.  It obviously does not reflect common
>  practice, since the common practice is not to implement this
>  target.

Practice is, I think, changing recently in response to the flowering
of distributed VCSen. Increasingly many packages are now available
from upstream *only* as a VCS branch; no static tarball releases are
available. Yet we must provide a “pristine upstream tarball” for a
Debian source package.

Common practice is to ignore the issue, until someone points out that
Lintian is complaining the package has no ‘debian/watch’ file. Then
the maintainer commonly writes a ‘debian/watch’ file with a static
comment saying “we get the upstream source from such-and-so VCS URL”.

That satisfies Lintian, but the user is left floundering with figuring
out exactly how to get the corresponding source from upstream to
verify Debian's package.


That is a poor substitute for a documented, automated method of
getting a “pristine upstream tarball” of the exact VCS revision from
which the source package was created. I think the ‘get-orig-source’
target is perfectly positioned to be that method in the short term.

All we need is to re-vamp the specification so it means what many in
this discussion want it to mean.

(good sigmonster, have a cookie)

-- 
 \          “That's all very good in practice, but how does it work in |
  `\                                             *theory*?” —anonymous |
_o__)                                                                  |
Ben Finney <ben@benfinney.id.au>
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 05 Mar 2009 09:09:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Finney <ben+debian@benfinney.id.au>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 05 Mar 2009 09:09:03 GMT) Full text and rfc822 format available.

Message #42 received at 466550@bugs.debian.org (full text, mbox):

From: Ben Finney <ben+debian@benfinney.id.au>
To: 466550@bugs.debian.org, Debian BTS control <control@bugs.debian.org>
Subject: Re: Bug#466550: Please clarify the get-orig-source target stated in Policy 4.9
Date: Thu, 5 Mar 2009 20:07:26 +1100
[Message part 1 (text/plain, inline)]
package debian-policy
tags 466550 + patch
thanks

On 19-Feb-2008, Andres Mejia wrote:
> When considering the phrase in policy "...does any necessary
> rearrangement to turn it into the original source tar file
> format...", it makes more sense when it refers to the original
> source tar file of the source package in Debian (or will be in
> Debian), and the original source tarball would be at a particular
> version, and thus the get-orig-source should generate that
> particular version of the original source tarball of the source
> package in Debian.

Here's my attempt:


=== modified file 'policy.sgml'
--- policy.sgml	2009-03-05 08:44:48 +0000
+++ policy.sgml	2009-03-05 09:06:48 +0000
@@ -1907,12 +1907,21 @@
 	    <tag><tt>get-orig-source</tt> (optional)</tag>
 	    <item>
 	      <p>
-		This target fetches the most recent version of the
-		original source package from a canonical archive site
-		(via FTP or WWW, for example), does any necessary
+		This target generates the original source archive for
+		the package, such that its contents exactly match the
+		original source archive used to generate the package
+		for Debian. See the “Original source archive”
+		section, below, for policy details of this file.
+	      </p>
+
+	      <p>
+		The actions for this target fetch the original source
+		package, corresponding to the Debian package version,
+		from a canonical archive site (for example, via FTP,
+		WWW, or a public VCS repository), do any necessary
 		rearrangement to turn it into the original source
-		tar file format described below, and leaves it in the
-		current directory.
+		archive file format described below, and leave it in
+		the current directory.
 	      </p>
 
 	      <p>
@@ -1922,8 +1931,13 @@
 	      </p>
 
 	      <p>
-		This target is optional, but providing it if
-		possible is a good idea.
+		Commonly, upstream developers will make canonical
+		original source archive files for specific versions
+		available for direct public download, and the
+		‘uscan(1)’ tool can automate this task with an
+		appropriate ‘debian/watch’ configuration file. This
+		target is therefore optional, and required only for
+		those cases not satisfied by ‘uscan(1)’.
 	      </p>
 	    </item>
 


-- 
 \     “I hope that after I die, people will say of me: ‘That guy sure |
  `\                            owed me a lot of money’.” —Jack Handey |
_o__)                                                                  |
Ben Finney <ben@benfinney.id.au>
[signature.asc (application/pgp-signature, inline)]

Tags added: patch Request was from Ben Finney <ben+debian@benfinney.id.au> to control@bugs.debian.org. (Thu, 05 Mar 2009 09:09:06 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 05 Mar 2009 14:57:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Charles Plessy <plessy@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 05 Mar 2009 14:57:05 GMT) Full text and rfc822 format available.

Message #49 received at 466550@bugs.debian.org (full text, mbox):

From: Charles Plessy <plessy@debian.org>
To: debian-devel@lists.debian.org
Cc: 466550@bugs.debian.org
Subject: Re: Pristine source from upstream VCS repository
Date: Thu, 5 Mar 2009 23:54:26 +0900
Le Thu, Mar 05, 2009 at 08:39:50PM +1100, Ben Finney a écrit :
> 
> I've proposed a patch to policy (in bug#466550) to bring policy in
> line with this practice.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=466550#42

 	      <p>
-		This target is optional, but providing it if
-		possible is a good idea.
+		Commonly, upstream developers will make canonical
+		original source archive files for specific versions
+		available for direct public download, and the
+		‘uscan(1)’ tool can automate this task with an
+		appropriate ‘debian/watch’ configuration file. This
+		target is therefore optional, and required only for
+		those cases not satisfied by ‘uscan(1)’.


Hi Ben,

at the same time, your patch would make it mandatory to write a get-orig-source
target when uscan(1) can not do the job. Since there are sometimes upstreams
who change the contents of the tarball without changing its name, it means that
it is impossible to be sure that uscan can do the job, and therefore that we
would all need to use complex get-orig-source targets that monitor the contents
of the upstream tarballs. Can you soften your wording to the current "optional"
status ?

Have a nice day,

-- 
Charles Plessy
Tsurumi, Kanagawa, Japan




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Fri, 06 Mar 2009 00:06:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Finney <ben+debian@benfinney.id.au>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Fri, 06 Mar 2009 00:06:03 GMT) Full text and rfc822 format available.

Message #54 received at 466550@bugs.debian.org (full text, mbox):

From: Ben Finney <ben+debian@benfinney.id.au>
To: Charles Plessy <plessy@debian.org>
Cc: debian-devel@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Pristine source from upstream VCS repository
Date: Fri, 6 Mar 2009 11:03:57 +1100
[Message part 1 (text/plain, inline)]
On 05-Mar-2009, Charles Plessy wrote:
> at the same time, your patch would make it mandatory to write a
> get-orig-source target when uscan(1) can not do the job. […] Can you
> soften your wording to the current "optional" status ?

Agreed. I also should have used the standard document markup for
various terms.

Here is an updated patch:


=== modified file 'policy.sgml'
--- policy.sgml	2009-03-05 08:44:48 +0000
+++ policy.sgml	2009-03-05 23:59:38 +0000
@@ -1907,12 +1907,21 @@
 	    <tag><tt>get-orig-source</tt> (optional)</tag>
 	    <item>
 	      <p>
-		This target fetches the most recent version of the
-		original source package from a canonical archive site
-		(via FTP or WWW, for example), does any necessary
+		This target generates the original source archive for
+		the package, such that its contents exactly match the
+		original source archive used to generate the package
+		for Debian. See the “Original source archive”
+		section, below, for policy details of this file.
+	      </p>
+
+	      <p>
+		The actions for this target fetch the original source
+		package, corresponding to the Debian package version,
+		from a canonical archive site (for example, via FTP,
+		WWW, or a public VCS repository), do any necessary
 		rearrangement to turn it into the original source
-		tar file format described below, and leaves it in the
-		current directory.
+		archive file format, and leave it in the current
+		directory.
 	      </p>
 
 	      <p>
@@ -1922,8 +1931,14 @@
 	      </p>
 
 	      <p>
-		This target is optional, but providing it if
-		possible is a good idea.
+		This target is <em>optional</em>. A common reason to
+		forego this target is that the upstream developers
+		make canonical original source archive files for
+		specific versions available for direct public
+		download; in these cases, the package only needs an
+		appropriate <file>debian/watch</file> configuration
+		for <prgn>uscan</prgn> to fetch the original source
+		archive.
 	      </p>
 	    </item>
 


-- 
 \          “Immorality: The morality of those who are having a better |
  `\                                          time.” —Henry L. Mencken |
_o__)                                                                  |
Ben Finney <ben@benfinney.id.au>
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Sat, 07 Mar 2009 19:30:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steve Langasek <vorlon@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Sat, 07 Mar 2009 19:30:05 GMT) Full text and rfc822 format available.

Message #59 received at 466550@bugs.debian.org (full text, mbox):

From: Steve Langasek <vorlon@debian.org>
To: Ben Finney <ben+debian@benfinney.id.au>, Charles Plessy <plessy@debian.org>, debian-devel@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Sat, 7 Mar 2009 11:28:52 -0800
On Fri, Mar 06, 2009 at 11:03:57AM +1100, Ben Finney wrote:

> === modified file 'policy.sgml'
> --- policy.sgml	2009-03-05 08:44:48 +0000
> +++ policy.sgml	2009-03-05 23:59:38 +0000
> @@ -1907,12 +1907,21 @@
>  	    <tag><tt>get-orig-source</tt> (optional)</tag>
>  	    <item>
>  	      <p>
> -		This target fetches the most recent version of the
> -		original source package from a canonical archive site
> -		(via FTP or WWW, for example), does any necessary
> +		This target generates the original source archive for
> +		the package, such that its contents exactly match the
> +		original source archive used to generate the package
> +		for Debian. See the “Original source archive”
> +		section, below, for policy details of this file.
> +	      </p>

Surely, given that this is sgml, you should be referencing "Original source
archive" by id instead of "see below"?

> +
> +	      <p>
> +		The actions for this target fetch the original source
> +		package, corresponding to the Debian package version,

This reads wrong.  What are "actions for this target"?  Should be "This
target fetches [...]" as in the original, or "The target fetches [...]" if
you're trying to avoid repetition.

>  	      <p>
> @@ -1922,8 +1931,14 @@
>  	      </p>
>  
>  	      <p>
> -		This target is optional, but providing it if
> -		possible is a good idea.
> +		This target is <em>optional</em>. A common reason to
> +		forego this target is that the upstream developers
> +		make canonical original source archive files for
> +		specific versions available for direct public
> +		download; in these cases, the package only needs an
> +		appropriate <file>debian/watch</file> configuration
> +		for <prgn>uscan</prgn> to fetch the original source
> +		archive.
>  	      </p>
>  	    </item>

Why is 'optional' emphasized?

I don't like the speculation about developers' reasons for forgoing (not
'foregoing', btw) the target.  I think this is better:

  This target is optional.  In many cases it is not needed because the
  upstream developers make canonical original source archive files for each
  version available for direct public download; in this case, creating a
  <file>debian/watch</file> configuration that tells <prgn>uscan</prgn> how
  to download the original source archive is sufficient.

I also wonder if this ought to be a footnote instead.  I don't think uscan
should be considered normative in policy.

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@ubuntu.com                                     vorlon@debian.org




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Sat, 07 Mar 2009 22:27:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Finney <ben+debian@benfinney.id.au>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Sat, 07 Mar 2009 22:27:07 GMT) Full text and rfc822 format available.

Message #64 received at 466550@bugs.debian.org (full text, mbox):

From: Ben Finney <ben+debian@benfinney.id.au>
To: debian-devel@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Sun, 8 Mar 2009 09:25:16 +1100
[Message part 1 (text/plain, inline)]
On 07-Mar-2009, Steve Langasek wrote:
> On Fri, Mar 06, 2009 at 11:03:57AM +1100, Ben Finney wrote:
> 
> > === modified file 'policy.sgml'
[…]
> > +		for Debian. See the “Original source archive”
> > +		section, below, for policy details of this file.
> > +	      </p>
> 
> Surely, given that this is sgml, you should be referencing "Original
> source archive" by id instead of "see below"?

My SGML-fu is very weak. The existing text also doesn't give a
reference, instead saying “the original source tar file format
described below”.

I've added a reference now.

> > +
> > +	      <p>
> > +		The actions for this target fetch the original source
> > +		package, corresponding to the Debian package version,
> 
> This reads wrong.  What are "actions for this target"?  Should be
> "This target fetches [...]" as in the original, or "The target
> fetches [...]" if you're trying to avoid repetition.

I'm trying to avoid repetition but also being more specific by
following the Make documentation in distinguishing the target (which
is just a label) from the commands run to achieve that target. But I
used the wrong term; changed to “commands for this target”.

> > +		This target is <em>optional</em>. A common reason to
[…]
> 
> Why is 'optional' emphasized?

I misunderstood the “words … used to distinguish the significance of
the various guidelines in this policy document” paragraphs in the
“Scope” section, and thought they needed to be emphasised in order to
have the described effect. That's not true, though, as I see from the
rest of the document. Fixed.

> I don't like the speculation about developers' reasons for forgoing (not
> 'foregoing', btw) the target.
[…]
> I also wonder if this ought to be a footnote instead.  I don't think uscan
> should be considered normative in policy.

Fixed.


Updated patch:


=== modified file 'policy.sgml'
--- policy.sgml	2009-03-05 08:44:48 +0000
+++ policy.sgml	2009-03-07 22:22:45 +0000
@@ -1907,12 +1907,21 @@
 	    <tag><tt>get-orig-source</tt> (optional)</tag>
 	    <item>
 	      <p>
-		This target fetches the most recent version of the
-		original source package from a canonical archive site
-		(via FTP or WWW, for example), does any necessary
+		This target generates the original source archive for
+		the package, such that its contents exactly match the
+		original source archive used to generate the package
+		for Debian.
+	      </p>
+
+	      <p>
+		The commands for this target fetch the original source
+		package, corresponding to the Debian package version,
+		from a canonical archive site (for example, via FTP,
+		WWW, or a public VCS repository), do any necessary
 		rearrangement to turn it into the original source
-		tar file format described below, and leaves it in the
-		current directory.
+		archive file format, and leave it in the current
+		directory. See <ref id="pkg-sourcearchives"> for
+		policy details of the original source archive.
 	      </p>
 
 	      <p>
@@ -1922,8 +1931,14 @@
 	      </p>
 
 	      <p>
-		This target is optional, but providing it if
-		possible is a good idea.
+		This target is optional, and in many cases is
+		unnecessary.<footnote>A common reason to omit this
+		target is that the upstream developers make available,
+		for direct download, canonical original source archive
+		files for each specific version. In these cases, it is
+		sufficient to create a <file>debian/watch</file>
+		configuration for <prgn>uscan</prgn> to fetch the
+		original source archive.</footnote>
 	      </p>
 	    </item>
 


-- 
 \      “If society were bound to invent technologies which could only |
  `\   be used entirely within the law, then we would still be sitting |
_o__)       in caves sucking our feet.” —Gene Kan, creator of Gnutella |
Ben Finney <ben@benfinney.id.au>
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Tue, 10 Mar 2009 00:06:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Gunnar Wolf <gwolf@gwolf.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Tue, 10 Mar 2009 00:06:02 GMT) Full text and rfc822 format available.

Message #69 received at 466550@bugs.debian.org (full text, mbox):

From: Gunnar Wolf <gwolf@gwolf.org>
To: Ben Finney <ben@benfinney.id.au>, Manoj Srivastava <srivasta@debian.org>, 466550@bugs.debian.org
Subject: Re: Bug#466550: Please clarify the get-orig-source target stated in Policy 4.9
Date: Mon, 9 Mar 2009 18:01:22 -0600
Ben Finney dijo [Thu, Mar 05, 2009 at 07:06:58PM +1100]:
> Practice is, I think, changing recently in response to the flowering
> of distributed VCSen. Increasingly many packages are now available
> from upstream *only* as a VCS branch; no static tarball releases are
> available. Yet we must provide a “pristine upstream tarball” for a
> Debian source package.
> 
> Common practice is to ignore the issue, until someone points out that
> Lintian is complaining the package has no ‘debian/watch’ file. Then
> the maintainer commonly writes a ‘debian/watch’ file with a static
> comment saying “we get the upstream source from such-and-so VCS URL”.
> 
> That satisfies Lintian, but the user is left floundering with figuring
> out exactly how to get the corresponding source from upstream to
> verify Debian's package.
> 
> 
> That is a poor substitute for a documented, automated method of
> getting a “pristine upstream tarball” of the exact VCS revision from
> which the source package was created. I think the ‘get-orig-source’
> target is perfectly positioned to be that method in the short term.
> 
> All we need is to re-vamp the specification so it means what many in
> this discussion want it to mean.

FWIW, I am going more or less with this approach for githubredir.d.n
(which makes debian/rules-friendly index pages pointing to tags in
github-hosted projects). Two tarballs generated from the same tag
(that is, from the same commit) will have the same contents, although
their MD5s will be different (and will thus be rejected for an
upload). 

This has rarely been an issue for me... But it might be a bothering
issue. And, yes, an ideal solution would be for uscan to understand
VCS tags as well.

-- 
Gunnar Wolf - gwolf@gwolf.org - (+52-55)5623-0154 / 1451-2244
PGP key 1024D/8BB527AF 2001-10-23
Fingerprint: 0C79 D2D1 2C4E 9CE4 5973  F800 D80E F35A 8BB5 27AF




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Tue, 10 Mar 2009 07:03:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Manoj Srivastava <srivasta@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Tue, 10 Mar 2009 07:03:03 GMT) Full text and rfc822 format available.

Message #74 received at 466550@bugs.debian.org (full text, mbox):

From: Manoj Srivastava <srivasta@debian.org>
To: debian-devel@lists.debian.org
Cc: debian-policy@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Pristine source from upstream VCS repository
Date: Tue, 10 Mar 2009 01:48:46 -0500
On Thu, Mar 05 2009, Russ Allbery wrote:

> Ben Finney <ben+debian@benfinney.id.au> writes:
>
>> It's been brought to my attention that this approach actually conflicts
>> with the above section of policy.
>>
>> Am I right in thinking that the ‘get-orig-source’ target should ignore
>> the version strings in ‘debian/changelog’, and should instead get
>> whatever version is the latest available from upstream?
>
> I think the way that you're using it is more useful (and possible) than
> doing what an exact reading of the current text would indicate, and I do
> the same thing that you're doing.
>

        However, as written, the wording does suggest that the latest
 version  is what will be acquired, and any shift in meaning will make
 currently conforming packages buggy. 

> http://bugs.debian.org/466550 is somewhat related.
>
> For packages with non-trivial rules to generate the upstream source
> tarball used with Debian, it's very difficult or impossible to write a
> future-proofed version of that cdoe that will work with arbitrary future
> versions from upstream.  However, documenting the method used to generate
> the *current* version will let people modify that target as needed to
> package future versions.

        I beg to differ. It would be hard for me to assure that any rule
 run which looks at the debian/changelog version will actually work at
 any time in the future.

        I have upstreams that ship released software tarballs that match
 a pattern I can feed uscan; but older versions are often purged from
 the site quickly. I can, then, use  the pattern to download the latest
 version (perhaps using uscan), and then unpack it, rm -rf the debian
 directory, and repack it, preserving the version number, without much
 hassle.

        Given that at least one version of the software is guaranteed to
 exist, I can craft a generic get-orig0source rule that will work -- but
 if I pay attention to the versoin, the rule will fail just days or
 weeks after upload.

        Making people remove a generic get-orig-source that actually
 gets the latest source package from upstream by making it violate the
 new version of policy would not be a good thing, in my opinion,
 Silenty reverting the original meaning of the target, without a
 transition plan, instead of creating a new target with the new meaning
 is not usually how Debian policy used to work.

        I am wondering which is of more use to the end users as well: I
 can always get the sources of the package I have already on my disk
 from Debian, but getting the latest munged source seems more useful to
 me.


        manoj
-- 
You may have heard that a dean is to faculty as a hydrant is to a
dog. Alfred Kahn
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Wed, 11 Mar 2009 10:33:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Goswin von Brederlow <goswin-v-b@web.de>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Wed, 11 Mar 2009 10:33:02 GMT) Full text and rfc822 format available.

Message #79 received at 466550@bugs.debian.org (full text, mbox):

From: Goswin von Brederlow <goswin-v-b@web.de>
To: debian-devel@lists.debian.org
Cc: debian-policy@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Pristine source from upstream VCS repository
Date: Wed, 11 Mar 2009 11:30:50 +0100
Manoj Srivastava <srivasta@debian.org> writes:

>         I am wondering which is of more use to the end users as well: I
>  can always get the sources of the package I have already on my disk
>  from Debian, but getting the latest munged source seems more useful to
>  me.

Full ACK. The way to get the current upstream source for a debian
package is "apt-get source" or equivalent and then using the
orig.tar.gz. Duplicating this in debian/rules seems wastefull.

>         manoj

MfG
        Goswin




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Wed, 11 Mar 2009 11:36:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Finney <ben+debian@benfinney.id.au>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Wed, 11 Mar 2009 11:36:02 GMT) Full text and rfc822 format available.

Message #84 received at 466550@bugs.debian.org (full text, mbox):

From: Ben Finney <ben+debian@benfinney.id.au>
To: Goswin von Brederlow <goswin-v-b@web.de>, 466550@bugs.debian.org
Cc: debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Wed, 11 Mar 2009 22:33:52 +1100
[Message part 1 (text/plain, inline)]
On 11-Mar-2009, Goswin von Brederlow wrote:
> Manoj Srivastava <srivasta@debian.org> writes:
> 
> >         I am wondering which is of more use to the end users as
> >         well: I can always get the sources of the package I have
> >         already on my disk from Debian, but getting the latest
> >         munged source seems more useful to me.
> 
> Full ACK. The way to get the current upstream source for a debian
> package is "apt-get source" or equivalent and then using the
> orig.tar.gz. Duplicating this in debian/rules seems wastefull.

That's not quite the same thing though. That will get the original
source archive *as in the Debian repository*; while the existing and
proposed ‘get-orig-source’ target gets the original source archive
*from the canonical upstream location itself*.

It's worth asking, then, what is the original purpose for which the
‘get-orig-source’ target specification was inserted into the policy?

* To get some copy of the original source archive. If so, that is
  entirely redundant with making it available in the Debian
  repository. Does anyone think this is what was intended by the
  drafters of that policy clause?

* To get the *latest* version of the package as an original source
  archive, regardless of the Debian version of the package. This is
  largely duplicated by ‘uscan(1)’, but not for all cases.

* To get the original source archive corresponding to the package
  directly from the canonical upstream location. That is the purpose
  of the patch I've submitted to this bug report.

* To do something else. I haven't seen any other options not covered
  here, but that doesn't mean the truth might not be different.

What is our best resource for discovering which of these options is
the actual intent of the ‘get-orig-source’ target when it was inserted
into policy?

-- 
 \        “I took a course in speed waiting. Now I can wait an hour in |
  `\                                 only ten minutes.” —Steven Wright |
_o__)                                                                  |
Ben Finney <ben@benfinney.id.au>
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Wed, 11 Mar 2009 15:42:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Manoj Srivastava <srivasta@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Wed, 11 Mar 2009 15:42:02 GMT) Full text and rfc822 format available.

Message #89 received at 466550@bugs.debian.org (full text, mbox):

From: Manoj Srivastava <srivasta@debian.org>
To: Ben Finney <ben+debian@benfinney.id.au>
Cc: Goswin von Brederlow <goswin-v-b@web.de>, 466550@bugs.debian.org, debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Wed, 11 Mar 2009 10:13:51 -0500
On Wed, Mar 11 2009, Ben Finney wrote:

> On 11-Mar-2009, Goswin von Brederlow wrote:
>> Manoj Srivastava <srivasta@debian.org> writes:
>> 
>> >         I am wondering which is of more use to the end users as
>> >         well: I can always get the sources of the package I have
>> >         already on my disk from Debian, but getting the latest
>> >         munged source seems more useful to me.
>> 
>> Full ACK. The way to get the current upstream source for a debian
>> package is "apt-get source" or equivalent and then using the
>> orig.tar.gz. Duplicating this in debian/rules seems wastefull.
>
> That's not quite the same thing though. That will get the original
> source archive *as in the Debian repository*; while the existing and
> proposed ‘get-orig-source’ target gets the original source archive
> *from the canonical upstream location itself*.


        In most cases, the sources are identical; and even otherwise, it
 is just a matter of looking into debian/copyright and doing a
 wget. This is not a compelling enough case to create a whole new target
 in debian/rules and policy.

        Indeed, the whole rationale for the target, and why it got into
 policy, was for the secific cases where extended munging of upstream
 had to be done, usually, but not restricted to, the cases where
 upstream had DFSG violations and the upstream source could not be
 allowed in the main section of the archive.

        This is what diferentiates is from uscan; indeed, I use uscan in
 the cases where I provide the target, The target unpacks the
 raw upstream source, munges it (by, say, removing a subdir which has
 non-dfsg stuff, or removes the debian dir, applies patches, or whatever
 other processing is required.

        There is no need to do this for the current version; the mungeds
 sources already are an apt-get source away.


        This is not a trivial replacement for uscan watch file with a
 version number in the regexp, which is the trivial way of implementing
 the task of getting the current version from upstream). This target is
 for the developers of munged sources to provide a script to munge the
 latest sources.

        Keep in mind that sometimes an unmunged source will not build
 with the debian/rules  file in the archive;  so it is not just a
 matter of dropping in ./debian into the latest upstream tarball.

        That facility is being dropped silently, and I object to there
 being no transition plan, and nothing to replace the functionality
 currently provided by the target.

        If we move over to the new docbook format (and I have one that
 seems to kinda work, but I got demotivated before I published it), the
 rationalew would have spelled out _why_ we needed such a target, and
 what it was originally designed to be used for.

> It's worth asking, then, what is the original purpose for which the
> ‘get-orig-source’ target specification was inserted into the policy?
>
> * To get some copy of the original source archive. If so, that is
>   entirely redundant with making it available in the Debian
>   repository. Does anyone think this is what was intended by the
>   drafters of that policy clause?

        No.

> * To get the *latest* version of the package as an original source
>   archive, regardless of the Debian version of the package. This is
>   largely duplicated by ‘uscan(1)’, but not for all cases.

        As long as no munging is t be done.

> * To get the original source archive corresponding to the package
>   directly from the canonical upstream location. That is the purpose
>   of the patch I've submitted to this bug report.

        I think this is not useful enough to take over the original
 purpose of the target.  I suggest if you think that looking into
 debian/copyright + wget is too hard, then create a new policy target. 

> * To do something else. I haven't seen any other options not covered
>   here, but that doesn't mean the truth might not be different.

        Indeed. You missed the whole "munge source to make it
 acceptable for debian and allow it to be built with ./debian files"

> What is our best resource for discovering which of these options is
> the actual intent of the ‘get-orig-source’ target when it was inserted
> into policy?

        Ask me. There is institutional knowledge in the post of leading
 the policy development for 10 years. And my memory is not that bad --
 yet, though doubtless senility encroaches.

        manoj
-- 
Under deadline pressure for the next week.  If you want something, it
can wait. Unless it's blind screaming paroxysmally hedonistic...
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Wed, 11 Mar 2009 15:42:14 GMT) Full text and rfc822 format available.

Acknowledgement sent to Manoj Srivastava <srivasta@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Wed, 11 Mar 2009 15:42:14 GMT) Full text and rfc822 format available.

Message #94 received at 466550@bugs.debian.org (full text, mbox):

From: Manoj Srivastava <srivasta@debian.org>
To: Ben Finney <ben+debian@benfinney.id.au>
Cc: Goswin von Brederlow <goswin-v-b@web.de>, 466550@bugs.debian.org, debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Wed, 11 Mar 2009 10:21:14 -0500
Hi,

        The best way to get the exact sources for the current version
 probably should be a  new watch file (watch-current) which has a static
 version number in the regexp, but can use all the other facilities f
 uscan -- wild carded directory, looking thoiugh an index.html page for
 a matching href, and so on.

        This allows us not to reinvent the wheel in policy for uscan,
 does not require every one to do their own uscan, often replicating
 uscan poorly, and allows for simple scripting to grab the watch-current
 file from, say, the pts or packages.d.o and not havce to unpack the old
 source to run make on debian/rules.

        It is simpler for lintian to check, avoids the NIH syndrome,
 does not  silently change the semantics of a policy rule, and continues
 to allow the current target to be used for the original purpose of
 munging raw upstream sources. It also means we will not be doing design
 of the dowloader in policy.

        There was a reason the original policy dictum was vague: we did
 not want to limit the things people do to munge upstream sources to
 make it palatable to Debian.

        Perhaps it is time for me to play a more active role in policy
 again, if Russ is willing to let me back in.


        manoj
-- 
You mean you didn't *know* she was off making lots of little phone
companies?
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Wed, 11 Mar 2009 18:12:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Russ Allbery <rra@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Wed, 11 Mar 2009 18:12:02 GMT) Full text and rfc822 format available.

Message #99 received at 466550@bugs.debian.org (full text, mbox):

From: Russ Allbery <rra@debian.org>
To: Ben Finney <ben+debian@benfinney.id.au>
Cc: Goswin von Brederlow <goswin-v-b@web.de>, 466550@bugs.debian.org, debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Wed, 11 Mar 2009 11:09:19 -0700
Manoj Srivastava <srivasta@debian.org> writes:

>         Perhaps it is time for me to play a more active role in policy
>  again, if Russ is willing to let me back in.

Good heavens, yes.  :)  I've always found your Policy work to be extremely
valuable, and whatever time you're willing to spend on the work is greatly
appreciated.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Wed, 11 Mar 2009 23:45:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Finney <ben+debian@benfinney.id.au>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Wed, 11 Mar 2009 23:45:02 GMT) Full text and rfc822 format available.

Message #104 received at 466550@bugs.debian.org (full text, mbox):

From: Ben Finney <ben+debian@benfinney.id.au>
To: 466550@bugs.debian.org, debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 10:44:29 +1100
[Message part 1 (text/plain, inline)]
On 11-Mar-2009, Manoj Srivastava wrote:
> On Wed, Mar 11 2009, Ben Finney wrote:
> > It's worth asking, then, what is the original purpose for which the
> > ‘get-orig-source’ target specification was inserted into the policy?
>
>         Indeed, the whole rationale for the target, and why it got into
>  policy, was for the secific cases where extended munging of upstream
>  had to be done, usually, but not restricted to, the cases where
>  upstream had DFSG violations and the upstream source could not be
>  allowed in the main section of the archive.
[…]
> 
>         This is not a trivial replacement for uscan watch file with a
>  version number in the regexp, which is the trivial way of implementing
>  the task of getting the current version from upstream). This target is
>  for the developers of munged sources to provide a script to munge the
>  latest sources.
> 
>         Keep in mind that sometimes an unmunged source will not build
>  with the debian/rules  file in the archive;  so it is not just a
>  matter of dropping in ./debian into the latest upstream tarball.
> 
>         That facility is being dropped silently, and I object to there
>  being no transition plan, and nothing to replace the functionality
>  currently provided by the target.

This purpose isn't clear to me at all in the existing policy document;
hence my (and presumably others) misunderstanding of its purpose and
implementing a ‘get-orig-source’ target with a different purpose.

>         Ask me. There is institutional knowledge in the post of leading
>  the policy development for 10 years. And my memory is not that bad --
>  yet, though doubtless senility encroaches.

I would very much like to see a patch from you, submitted to this bug
report, that re-works this policy section to more explicitly state the
intent and purpose of the ‘get-orig-source’ target.

-- 
 \             “To label any subject unsuitable for comedy is to admit |
  `\                                           defeat.” —Peter Sellers |
_o__)                                                                  |
Ben Finney <ben@benfinney.id.au>
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 09:15:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steve Langasek <vorlon@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 09:15:06 GMT) Full text and rfc822 format available.

Message #109 received at 466550@bugs.debian.org (full text, mbox):

From: Steve Langasek <vorlon@debian.org>
To: debian-policy@lists.debian.org, debian-devel@lists.debian.org
Cc: 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 02:05:42 -0700
On Wed, Mar 11, 2009 at 10:13:51AM -0500, Manoj Srivastava wrote:
>         This is what diferentiates is from uscan; indeed, I use uscan in
>  the cases where I provide the target, The target unpacks the
>  raw upstream source, munges it (by, say, removing a subdir which has
>  non-dfsg stuff, or removes the debian dir, applies patches, or whatever
>  other processing is required.

>         There is no need to do this for the current version; the mungeds
>  sources already are an apt-get source away.

For several packages I (co)maintain where I have to munge the upstream
tarball, the standard procedure (inherited from past maintainers) is:

 - increment the version number in the debian packaging
 - call the get-orig-source target

I think it's perfectly reasonable to want the get-orig-source target to give
you a *specified* version of an upstream tarball, rather than the *newest*
version of an upstream tarball.  Packaging a new upstream version doesn't
necessarily mean packaging the latest that uscan can find.

It's also useful for third parties to be able to easily examine the
provenance of specific Debian tarballs.  A get-orig-source target provides a
much more concise description of the Debian changes than examining the diff
between the two tarballs.

So I certainly agree that uscan doesn't obsolete the get-orig-source target,
but I disagree that it's not useful to have such a target generate a tarball
for the 'current' upstream version.

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@ubuntu.com                                     vorlon@debian.org




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 14:15:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Gunnar Wolf <gwolf@gwolf.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 14:15:04 GMT) Full text and rfc822 format available.

Message #114 received at 466550@bugs.debian.org (full text, mbox):

From: Gunnar Wolf <gwolf@gwolf.org>
To: debian-policy@lists.debian.org, debian-devel@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 08:12:57 -0600
Steve Langasek dijo [Thu, Mar 12, 2009 at 02:05:42AM -0700]:
> I think it's perfectly reasonable to want the get-orig-source target to give
> you a *specified* version of an upstream tarball, rather than the *newest*
> version of an upstream tarball.  Packaging a new upstream version doesn't
> necessarily mean packaging the latest that uscan can find.
> 
> It's also useful for third parties to be able to easily examine the
> provenance of specific Debian tarballs.  A get-orig-source target provides a
> much more concise description of the Debian changes than examining the diff
> between the two tarballs.
> 
> So I certainly agree that uscan doesn't obsolete the get-orig-source target,
> but I disagree that it's not useful to have such a target generate a tarball
> for the 'current' upstream version.

Good point you have here - But (and I know it is not being discussed
yet, maybe you want to teleport this thread a couple of years into the
future) I feel this should clearly be an optional target, and the
canonical location for orig.tar.gz files should still be our archive -
Down the other road lies Gentoo's BSD ports' madness, where an
upstream site restructure means packages become unreachable and
insta-FTBFS. 

-- 
Gunnar Wolf - gwolf@gwolf.org - (+52-55)5623-0154 / 1451-2244
PGP key 1024D/8BB527AF 2001-10-23
Fingerprint: 0C79 D2D1 2C4E 9CE4 5973  F800 D80E F35A 8BB5 27AF




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 16:06:10 GMT) Full text and rfc822 format available.

Acknowledgement sent to Manoj Srivastava <srivasta@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 16:06:10 GMT) Full text and rfc822 format available.

Message #119 received at 466550@bugs.debian.org (full text, mbox):

From: Manoj Srivastava <srivasta@debian.org>
To: debian-policy@lists.debian.org
Cc: debian-devel@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 10:31:07 -0500
On Thu, Mar 12 2009, Steve Langasek wrote:

> On Wed, Mar 11, 2009 at 10:13:51AM -0500, Manoj Srivastava wrote:
>>         This is what diferentiates is from uscan; indeed, I use uscan in
>>  the cases where I provide the target, The target unpacks the
>>  raw upstream source, munges it (by, say, removing a subdir which has
>>  non-dfsg stuff, or removes the debian dir, applies patches, or whatever
>>  other processing is required.
>
>>         There is no need to do this for the current version; the mungeds
>>  sources already are an apt-get source away.
>
> For several packages I (co)maintain where I have to munge the upstream
> tarball, the standard procedure (inherited from past maintainers) is:
>
>  - increment the version number in the debian packaging
>  - call the get-orig-source target
>
> I think it's perfectly reasonable to want the get-orig-source target
> to give you a *specified* version of an upstream tarball, rather than
> the *newest* version of an upstream tarball.  Packaging a new upstream
> version doesn't necessarily mean packaging the latest that uscan can
> find.

        Fair enough. But that is not the semantics of the target
 currently: get-orig-source is defined right now to get the  /latest/
 source, and while it is reasonable to have both behaviours, it is not
 necessary to expect both from the same target.

        To recap:
 1) apt-get source is enough to get the latest Debian source from the
    archive (and whet for older sources)
 2) In the absence of munging, uscan, with a watch and watch-current
    files, is adequate to get either the latest or a specific version
    from upstream
 3) It is reasonable to get the latest, or a specific version, from
    upstream, and munge it.

        So, for case 3: get-orig-source has been defined to get the
 latest sources (with munging, if needed). If we want to get a specific
 version, we can:
  a. over-load get-orig-source to take a version number, some how,
     through an env variable, perhaps
  b. create a brand new target, which looks at the env variable, and
     falls back to the version in the changelog.

        I think case a is harder from a policy creation perspective,
 since it should not outlaw currently conforming implementations. The
 new target method can be deployed, tested in the wild, and then made
 into policy when the kinks have been ironed out.

        manoj
-- 
Writing is turning one's worst moments into money. J.P. Donleavy
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 17:27:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Russ Allbery <rra@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 17:27:07 GMT) Full text and rfc822 format available.

Message #124 received at 466550@bugs.debian.org (full text, mbox):

From: Russ Allbery <rra@debian.org>
To: Gunnar Wolf <gwolf@gwolf.org>
Cc: debian-policy@lists.debian.org, debian-devel@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 09:59:50 -0700
Gunnar Wolf <gwolf@gwolf.org> writes:

> Good point you have here - But (and I know it is not being discussed
> yet, maybe you want to teleport this thread a couple of years into the
> future) I feel this should clearly be an optional target, and the
> canonical location for orig.tar.gz files should still be our archive -
> Down the other road lies Gentoo's BSD ports' madness, where an upstream
> site restructure means packages become unreachable and insta-FTBFS.

I certainly agree with this.  I'm not sure if anyone was proposing using
get-orig-source instead of the archive for revisions of the same upstream
version, but I definitely agree that the Debian archive is the canonical
source.

I personally use the same technique that Steve uses for the packages that
I maintain that need to be repacked, and I'm having a failure of
imagination for how I could do it the way that Manoj describes.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 23:03:08 GMT) Full text and rfc822 format available.

Acknowledgement sent to Russ Allbery <rra@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 23:03:08 GMT) Full text and rfc822 format available.

Message #129 received at 466550@bugs.debian.org (full text, mbox):

From: Russ Allbery <rra@debian.org>
To: debian-devel@lists.debian.org
Cc: debian-policy@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 12:22:33 -0700
Manoj Srivastava <srivasta@debian.org> writes:

>  a) Run a upstream version check from cron, which mails me if there are
>     new upstream versions of something I have.
>  b) If there is a new upstream version, cd checked out dir
>     1. No munging required: use uscan --rename --verbose to get the
>        latest source.
>     2. Munging needed. Run get-orig-source to get the latest upstream
>        source via uscan; and munge it as needed to create the
>        orig.tar.gz file

Oh, okay, so your get-orig-source target would internally use uscan.  How
do you tell from that what tarball it downloaded for an automated target?
Would you parse the output of uscan somehow?

>  c) Proceed as per:
>     http://www.golden-gryphon.com/blog/manoj/blog/2009/02/25/A_day_in_the_life_of_a_Debian_hacker/
>
>         Is this so very different from what people do? Some times I do
>  not package every upstream version, if they are coming in rapid
>  succession, or if I find some version unfit for Debian -- but in any
>  case, the majority of the time I want to package the very latest
>  upstream version.

I never use uscan --download; I always download the new upstream source
myself using wget or a web browser or FTP client.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 23:12:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to Bernd Zeimetz <bernd@bzed.de>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 23:12:07 GMT) Full text and rfc822 format available.

Message #134 received at 466550@bugs.debian.org (full text, mbox):

From: Bernd Zeimetz <bernd@bzed.de>
To: srivasta@debian.org, 466550@bugs.debian.org, debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 20:49:19 +0100
Hi,

>         The best way to get the exact sources for the current version
>  probably should be a  new watch file (watch-current) which has a static
>  version number in the regexp, but can use all the other facilities f
>  uscan -- wild carded directory, looking thoiugh an index.html page for
>  a matching href, and so on.

No, please don't just add another watch file just for the sake of it, using
these files is more or less like living in the last century. People are able to
get the current source from the Debian pool, if that is not enough for them,
they should be old enough to be able to click on the upstream homepage link in
the package's description and get the source.

A lot of people, including myself, prefer to pull form the upstream vcs
directly, and work on top of that, using git for example. Using uscan to
retrieve the exact version is often impossible, as it's not trivial to get a
tarball from a specific upstream branch, tag or ref.

I think the way Debian should go is to tell people that they should clone the
developer's git ([.. insert your favourite dvcs here ...]) repository and work
with it, probably requiring to explain how working with the repository works,
which branches are used for what, and so on. At least that would fit *todays*
way of handling packages, at least for a lot of people.

Cheers,

Bernd

-- 
 Bernd Zeimetz                           Debian GNU/Linux Developer
 GPG Fingerprint: 06C8 C9A2 EAAD E37E 5B2C BE93 067A AD04 C93B FF79




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 23:27:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to Bernd Zeimetz <bernd@bzed.de>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 23:27:07 GMT) Full text and rfc822 format available.

Message #139 received at 466550@bugs.debian.org (full text, mbox):

From: Bernd Zeimetz <bernd@bzed.de>
To: debian-devel@lists.debian.org, debian-policy@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 20:55:29 +0100
Manoj Srivastava wrote:
>  a) Run a upstream version check from cron, which mails me if there are
>     new upstream versions of something I have.

What happens if your watch file breaks? Do you check upstream announcements
manually, too?

>  b) If there is a new upstream version, cd checked out dir
>     1. No munging required: use uscan --rename --verbose to get the
>        latest source.
>     2. Munging needed. Run get-orig-source to get the latest upstream
>        source via uscan; and munge it as needed to create the
>        orig.tar.gz file
>  c) Proceed as per:
>     http://www.golden-gryphon.com/blog/manoj/blog/2009/02/25/A_day_in_the_life_of_a_Debian_hacker/
> 
>         Is this so very different from what people do? 

Depends on the package, for the really easy ones I works as described above
probably (didn't read the webpage, though), but as soon as 'munging' or
get-orig-source is needed, I prefer to pull from upstream's repository directly.
Makes live much more easy.

For those cases where a git pull is not enough, I've setup cronjobs which pull
from upstream and push into the according branches in my git, usually named
upstream-svn/upstream-cvs or similar.


Bernd

-- 
 Bernd Zeimetz                           Debian GNU/Linux Developer
 GPG Fingerprint: 06C8 C9A2 EAAD E37E 5B2C BE93 067A AD04 C93B FF79




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 23:30:15 GMT) Full text and rfc822 format available.

Acknowledgement sent to Russ Allbery <rra@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 23:30:15 GMT) Full text and rfc822 format available.

Message #144 received at 466550@bugs.debian.org (full text, mbox):

From: Russ Allbery <rra@debian.org>
To: srivasta@debian.org, 466550@bugs.debian.org, debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 12:57:57 -0700
Bernd Zeimetz <bernd@bzed.de> writes:

> No, please don't just add another watch file just for the sake of it,
> using these files is more or less like living in the last
> century. People are able to get the current source from the Debian pool,
> if that is not enough for them, they should be old enough to be able to
> click on the upstream homepage link in the package's description and get
> the source.
>
> A lot of people, including myself, prefer to pull form the upstream vcs
> directly, and work on top of that, using git for example. Using uscan to
> retrieve the exact version is often impossible, as it's not trivial to
> get a tarball from a specific upstream branch, tag or ref.
>
> I think the way Debian should go is to tell people that they should
> clone the developer's git ([.. insert your favourite dvcs here ...])
> repository and work with it, probably requiring to explain how working
> with the repository works, which branches are used for what, and so
> on. At least that would fit *todays* way of handling packages, at least
> for a lot of people.

Hm, I think I disagree with most of this.

First, I think this new habit (which you don't mention directly but
somewhat allude to) of not making stable formal releases is a very bad one
and I would strongly encourage any of my upstreams to not go down that
path.  The difference may be more psychological than technical, but it's
important to assign a version number on something and push it out the door
and declare it released.  Otherwise, projects have a strong tendency to
drift into perpetual development mode, where it's a crap-shoot whether any
given feature will be working at the moment and often it's quite difficult
to find a point in time when everything is stabilized.  I have one
upstream that does this "just use the VCS" thing and in practice it's
incredibly obnoxious for trying to get their software into Debian.

If there are stable upstream releases, I think that's what the Debian
packaging should be based on.  If you also want to use Git remotes to
track the upstream revision control repository so that you have more
fine-grained metadata, that's great, but I think a lot of clarity and
reproducibility is gained by having upstream release a tarball and by
basing the Debian package on exactly that tarball.  There's a lot to be
said for a clear export and externalization from the VCS that everyone can
synchronize with, regardless of tools.

On the topic of finding the current upstream release, I definitely don't
agree with the idea that the home page link solves this problem.  Some
upstreams have extremely bizarre release processes, poor home pages, no
real home page at all, or make it difficult to figure out just where the
source is at.  Having a watch file that embeds all of the packager's
existing knowledge about how to find the upstream release is very
valuable.

Also, I think you're underestimating the utility of being able to find
exactly the tarball that was used for generating a given Debian package.
It allows independent verification of the package in the archive (useful
in some security scenarios), and it's very important for package
sponsorship where one should not trust the orig.tar.gz provided by the
sponsoree unless you already know the sponsoree well.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 23:30:17 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steve Langasek <vorlon@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 23:30:18 GMT) Full text and rfc822 format available.

Message #149 received at 466550@bugs.debian.org (full text, mbox):

From: Steve Langasek <vorlon@debian.org>
To: debian-devel@lists.debian.org, debian-policy@lists.debian.org
Cc: 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 13:25:03 -0700
On Thu, Mar 12, 2009 at 12:38:24PM -0500, Manoj Srivastava wrote:

>         Is this so very different from what people do? Some times I  do
>  not package every upstream version, if they are coming in rapid
>  succession, or if I find some version unfit for Debian -- but in any
>  case, the majority of the time I want to package the very latest
>  upstream version.

The difference is having a get-orig-source that works for the majority
case (I want to package the very latest), instead of working for all cases
(I want to package upstream version $x, which may or may not be the latest).

I don't see a good way to fit uupdate in with VCS-based packaging, so at
some point you have to manually increment the version number in
debian/changelog to point to the new upstream version you want, yes?  In
that case, it makes sense to me to do this once, then use the changelog
information to pull in the correct upstream tarball via the get-orig-source
target.

(N.B.: I say "it makes sense to me", but in practice the packages I've
inherited hardcode the version to pull in debian/rules rather than parsing
the changelog.  I consider this a minor bug that I just haven't gotten
around to fixing.)

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@ubuntu.com                                     vorlon@debian.org




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 23:33:17 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Finney <ben@benfinney.id.au>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 23:33:17 GMT) Full text and rfc822 format available.

Message #154 received at 466550@bugs.debian.org (full text, mbox):

From: Ben Finney <ben@benfinney.id.au>
To: Gunnar Wolf <gwolf@gwolf.org>, 466550@bugs.debian.org
Cc: debian-policy@lists.debian.org, debian-devel@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Fri, 13 Mar 2009 10:31:17 +1100
[Message part 1 (text/plain, inline)]
On 12-Mar-2009, Gunnar Wolf wrote:
> I feel this should clearly be an optional target, and the canonical
> location for orig.tar.gz files should still be our archive

Yes to both. Thanks for making this explicit in the discussion.

-- 
 \      “Reichel's Law: A body on vacation tends to remain on vacation |
  `\            unless acted upon by an outside force.” —Carol Reichel |
_o__)                                                                  |
Ben Finney <ben@benfinney.id.au>
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 23:36:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Manoj Srivastava <srivasta@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 23:36:05 GMT) Full text and rfc822 format available.

Message #159 received at 466550@bugs.debian.org (full text, mbox):

From: Manoj Srivastava <srivasta@debian.org>
To: debian-devel@lists.debian.org
Cc: debian-policy@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 12:38:24 -0500
On Thu, Mar 12 2009, Russ Allbery wrote:
>
> I personally use the same technique that Steve uses for the packages that
> I maintain that need to be repacked, and I'm having a failure of
> imagination for how I could do it the way that Manoj describes.

        Hmm. Let me see if I can elucidate. Here is my work flow.

 a) Run a upstream version check from cron, which mails me if there are
    new upstream versions of something I have.
 b) If there is a new upstream version, cd checked out dir
    1. No munging required: use uscan --rename --verbose to get the
       latest source.
    2. Munging needed. Run get-orig-source to get the latest upstream
       source via uscan; and munge it as needed to create the
       orig.tar.gz file
 c) Proceed as per:
    http://www.golden-gryphon.com/blog/manoj/blog/2009/02/25/A_day_in_the_life_of_a_Debian_hacker/

        Is this so very different from what people do? Some times I  do
 not package every upstream version, if they are coming in rapid
 succession, or if I find some version unfit for Debian -- but in any
 case, the majority of the time I want to package the very latest
 upstream version.


        manoj
-- 
"The world is coming to an end.  Please log off." Bob Irwin
(birwin@ficc.ferranti.com)
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 23:36:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to Russ Allbery <rra@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 23:36:07 GMT) Full text and rfc822 format available.

Message #164 received at 466550@bugs.debian.org (full text, mbox):

From: Russ Allbery <rra@debian.org>
To: debian-devel@lists.debian.org
Cc: debian-policy@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 13:30:14 -0700
Steve Langasek <vorlon@debian.org> writes:

> (N.B.: I say "it makes sense to me", but in practice the packages I've
> inherited hardcode the version to pull in debian/rules rather than
> parsing the changelog.  I consider this a minor bug that I just haven't
> gotten around to fixing.)

I got into the habit of doing it that way because for some of my packages
there isn't a clear mapping between the Debian version and the upstream
version.  (Tildes may have to be added, for example, and dfsg removed.)  I
ended up doing it the same way everywhere, although I agree that for
simpler cases it would be better to use debian/changelog.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 12 Mar 2009 23:54:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Manoj Srivastava <srivasta@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 12 Mar 2009 23:54:03 GMT) Full text and rfc822 format available.

Message #169 received at 466550@bugs.debian.org (full text, mbox):

From: Manoj Srivastava <srivasta@debian.org>
To: debian-devel@lists.debian.org
Cc: debian-policy@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 18:40:03 -0500
On Thu, Mar 12 2009, Russ Allbery wrote:

> Manoj Srivastava <srivasta@debian.org> writes:
>
>>  a) Run a upstream version check from cron, which mails me if there are
>>     new upstream versions of something I have.
>>  b) If there is a new upstream version, cd checked out dir
>>     1. No munging required: use uscan --rename --verbose to get the
>>        latest source.
>>     2. Munging needed. Run get-orig-source to get the latest upstream
>>        source via uscan; and munge it as needed to create the
>>        orig.tar.gz file
>
> Oh, okay, so your get-orig-source target would internally use uscan.

        It _could_ use uscan. it does not have to be limited to it.

> How do you tell from that what tarball it downloaded for an automated
> target?  Would you parse the output of uscan somehow?

        I just glob for the same pattern as in the watch file, and use
 the last in the lexical sorting, I suppose one could use dpkg
 --compare-versions if one were paranoid enough, and heck, use shell
 sort on the orig tar balls discovered :P

>>  c) Proceed as per:
>>     http://www.golden-gryphon.com/blog/manoj/blog/2009/02/25/A_day_in_the_life_of_a_Debian_hacker/>
>>         Is this so very different from what people do? Some times I do
>>  not package every upstream version, if they are coming in rapid
>>  succession, or if I find some version unfit for Debian -- but in any
>>  case, the majority of the time I want to package the very latest
>>  upstream version.
>
> I never use uscan --download; I always download the new upstream source
> myself using wget or a web browser or FTP client.

        But this is not about our individual work-flows -- it is about
 policy trying hard not to proscrive the work flows _any_ of us use. If
 it turns out there are two sets of processes people follow, I would
 much rather have two mechanisms, with two different sets of semantics,
 rather than have us select one in policy.

        I am beginning to think this whole target is too immature to
 actually be in policy; we are still doing design discussions of this
 feature.

        manoj
-- 
The real problem with hunting elephants is carrying the decoys.
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Fri, 13 Mar 2009 00:03:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Finney <ben@benfinney.id.au>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Fri, 13 Mar 2009 00:03:02 GMT) Full text and rfc822 format available.

Message #174 received at 466550@bugs.debian.org (full text, mbox):

From: Ben Finney <ben@benfinney.id.au>
To: debian-policy@lists.debian.org, debian-devel@lists.debian.org, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Fri, 13 Mar 2009 11:02:20 +1100
[Message part 1 (text/plain, inline)]
On 12-Mar-2009, Manoj Srivastava wrote:
>         To recap:
>  1) apt-get source is enough to get the latest Debian source from the
>     archive (and whet for older sources)

I presume you mean ‘wget’ here. (Apart from ‘apt-get source’, is there
another tool that is *solely* focussed on getting the Debian source
for a package by name?)

>  2) In the absence of munging, uscan, with a watch and watch-current
>     files, is adequate to get either the latest or a specific version
>     from upstream

It's more limited than “in the absence of munging”. None of my
packages currently need munging, but there are some that upstream
doesn't *have* a tarball release for the code I want to package. Those
are the cases that led me to ‘get-orig-source’ in the first place,
since obviously ‘uscan’ can't handle those.

I would re-state this instead as: In the presence of upstream tarball
releases, which don't need munging, at the versions which need to be
packaged, ‘uscan’ is adequate for getting the original source archive.

>  3) It is reasonable to get the latest, or a specific version, from
>     upstream, and munge it.

Yes.

-- 
 \        “With Lisp or Forth, a master programmer has unlimited power |
  `\     and expressiveness. With Python, even a regular guy can reach |
_o__)                               for the stars.” —Raymond Hettinger |
Ben Finney <ben@benfinney.id.au>
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Fri, 13 Mar 2009 00:09:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Finney <ben@benfinney.id.au>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Fri, 13 Mar 2009 00:09:02 GMT) Full text and rfc822 format available.

Message #179 received at 466550@bugs.debian.org (full text, mbox):

From: Ben Finney <ben@benfinney.id.au>
To: Russ Allbery <rra@debian.org>, 466550@bugs.debian.org
Cc: debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Fri, 13 Mar 2009 11:05:42 +1100
[Message part 1 (text/plain, inline)]
On 12-Mar-2009, Russ Allbery wrote:
> Manoj Srivastava <srivasta@debian.org> writes:
> 
> >  b) If there is a new upstream version, cd checked out dir
> >     1. No munging required: use uscan --rename --verbose to get the
> >        latest source.
> >     2. Munging needed. Run get-orig-source to get the latest upstream
> >        source via uscan; and munge it as needed to create the
> >        orig.tar.gz file
> 
> Oh, okay, so your get-orig-source target would internally use uscan.
> How do you tell from that what tarball it downloaded for an
> automated target? Would you parse the output of uscan somehow?

Also, what do you do for the cases where upstream doesn't have a
tarbal (either none at all, or the code you want isn't yet available
as a tarball)?

Or do you (Manoj) not have any upstream packages making original
source available as anything but tarball releases?

> I never use uscan --download; I always download the new upstream
> source myself using wget or a web browser or FTP client.

Why is that? Is there some downside to using ‘uscan --download’? I
would have thought it best to use the automated tool where possible,
if for no other reason than to make sure the automated process will
get the same source you're working with.

-- 
 \     “I was born by Caesarian section. But not so you'd notice. It's |
  `\     just that when I leave a house, I go out through the window.” |
_o__)                                                   —Steven Wright |
Ben Finney <ben@benfinney.id.au>
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Fri, 13 Mar 2009 00:15:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Russ Allbery <rra@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Fri, 13 Mar 2009 00:15:02 GMT) Full text and rfc822 format available.

Message #184 received at 466550@bugs.debian.org (full text, mbox):

From: Russ Allbery <rra@debian.org>
To: Ben Finney <ben@benfinney.id.au>
Cc: 466550@bugs.debian.org, debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 17:13:45 -0700
Ben Finney <ben@benfinney.id.au> writes:
> On 12-Mar-2009, Russ Allbery wrote:

>> I never use uscan --download; I always download the new upstream source
>> myself using wget or a web browser or FTP client.

> Why is that? Is there some downside to using ‘uscan --download’? I would
> have thought it best to use the automated tool where possible, if for no
> other reason than to make sure the automated process will get the same
> source you're working with.

I just personally have never needed it and never found it particularly
useful or interesting.  Getting the right upstream tarball is the least of
the things that I do around packaging new upstream source.  I'm often
packaging new upstream test releases or packaging something in advance of
it being available from upstream's web site, I look through the web site
for restructuring or other information that I need to be aware of, etc.

As Manoj says, this is more about personal workflow than really about what
Policy can talk about.  I guess that I find the current Policy definition
of get-orig-source rather uninteresting and wouldn't bother to implement
something that exactly follows what's there.  I *do* find it useful to
automate the process of stripping an upstream tarball of non-DFSG bits,
and when I first started doing Debian packaging, the examples I looked at
used get-orig-source to do that.  So that's what I started doing as well.

I'm open to the idea that this really isn't the best way of handling it
and we should standardize something other than get-orig-source as the way
of stripping an upstream tarball (such as, for instance, a script in the
debian/ directory that you run on the upstream source tarball, however you
obtained it).  I would rather not have only textual descriptions of what
to do.  It's nice to have it automated and to be able to look at a simple
shell script to see *exactly* what transformations are applied.

But I'm not sure I'd ever personally use a target that downloads the
current upstream source and tries to apply the stripping process that
worked with the last release I packaged, all as one atomic step.  It
doesn't fit my workflow.  (I of course have no objections to standardizing
a way of doing that for people who have different workflows than mine, or
restoring get-orig-source as the correct way of doing that and changing
all my targets to be something else.)

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Fri, 13 Mar 2009 00:24:08 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Finney <ben+debian@benfinney.id.au>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Fri, 13 Mar 2009 00:24:08 GMT) Full text and rfc822 format available.

Message #189 received at 466550@bugs.debian.org (full text, mbox):

From: Ben Finney <ben+debian@benfinney.id.au>
To: Bernd Zeimetz <bernd@bzed.de>, 466550@bugs.debian.org
Cc: debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Fri, 13 Mar 2009 11:20:57 +1100
[Message part 1 (text/plain, inline)]
On 12-Mar-2009, Bernd Zeimetz wrote:
> Hi,
> 
> >         The best way to get the exact sources for the current
> >         version probably should be a  new watch file
> >         (watch-current) which has a static version number in the
> >         regexp

I don't see why this file would be needed, since a watchfile
specifying ‘debian’ as the version already has this effect (according
to the ‘uscan(1)’ manpage).

So I agree with this:

> No, please don't just add another watch file just for the sake of
> it,

but not for Bernd's reasons.

> using these files is more or less like living in the last century.

Given that we need to support tarballs from upstream for the
forseeable future, the existing ‘debian/watch’ files seem a good
solution for that limited scope.

> People are able to get the current source from the Debian pool, if
> that is not enough for them

There are many reasons to want to verify the Debian source package
against the original source archive; for example, security checks,
licensing checks, checking for packaging mistakes, etc.

> they should be old enough

This is rather condescending and judgemental; let's not dismiss as
childish the requirement to do something, without understanding the
reasons first.

> to be able to click on the upstream homepage link in the package's
> description and get the source.

The upstream home page for many packages makes it ridiculously
difficult to get to the original source archive. Some don't have the
original source discoverable from the home page; some don't even have
a home page.

Part of our job as package maintainers is to be an interface between
Debian users and upstream developers; getting the original source as
used by Debian surely counts, since many users want to develop the
package further. If we have to deal with that task more than once, we
should find ways to automate it both for ourselves and our users.

-- 
 \       “I don't care to belong to a club that accepts people like me |
  `\                                        as members.” —Groucho Marx |
_o__)                                                                  |
Ben Finney <ben@benfinney.id.au>
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Fri, 13 Mar 2009 07:36:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steve Langasek <vorlon@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Fri, 13 Mar 2009 07:36:02 GMT) Full text and rfc822 format available.

Message #194 received at 466550@bugs.debian.org (full text, mbox):

From: Steve Langasek <vorlon@debian.org>
To: Bernd Zeimetz <bernd@bzed.de>
Cc: 466550@bugs.debian.org, debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Thu, 12 Mar 2009 21:53:04 -0700
On Thu, Mar 12, 2009 at 08:49:19PM +0100, Bernd Zeimetz wrote:

> No, please don't just add another watch file just for the sake of it, using
> these files is more or less like living in the last century. People are able to
> get the current source from the Debian pool, if that is not enough for them,
> they should be old enough to be able to click on the upstream homepage link in
> the package's description and get the source.

> A lot of people, including myself, prefer to pull form the upstream vcs
> directly, and work on top of that, using git for example. Using uscan to
> retrieve the exact version is often impossible, as it's not trivial to get
> a tarball from a specific upstream branch, tag or ref.

I'm not sure if you're arguing against using get-orig-source this way or
just arguing against a watchfile-like approach, but I would say that's
precisely the case that the get-orig-source target is intended for.  Cases
where generation of the tarballs used as .orig.tar.gz in Debian is
non-trivial are cases where the process of generating these tarballs should
be documented in a machine-automatable manner, whether they're generated by
downloading an existing upstream tarball and munging it, or by pulling from
a particular VCS tag.

In an ideal world, we would have a standard method for recreating a tarball
from upstream that doesn't assume familiarity with any particular VCS, or
familiarity with any particular upstream's tagging conventions.

> I think the way Debian should go is to tell people that they should clone
> the developer's git ([.. insert your favourite dvcs here ...]) repository
> and work with it, probably requiring to explain how working with the
> repository works, which branches are used for what, and so on. At least
> that would fit *todays* way of handling packages, at least for a lot of
> people.

Wrong for the various reasons Russ has already given - and also because
<insert your favorite dvcs here> gives us no common format that all Debian
developers agree to use.  The value in having such targets in policy is so
that developers *other* than the maintainer can rely on them.

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@ubuntu.com                                     vorlon@debian.org




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Fri, 13 Mar 2009 09:51:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Bernd Zeimetz <bzed@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Fri, 13 Mar 2009 09:51:03 GMT) Full text and rfc822 format available.

Message #199 received at 466550@bugs.debian.org (full text, mbox):

From: Bernd Zeimetz <bzed@debian.org>
To: Russ Allbery <rra@debian.org>
Cc: srivasta@debian.org, 466550@bugs.debian.org, debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Fri, 13 Mar 2009 10:49:19 +0100
Russ Allbery wrote:
> Bernd Zeimetz <bernd@bzed.de> writes:
> 
>> No, please don't just add another watch file just for the sake of it,
>> using these files is more or less like living in the last
>> century. People are able to get the current source from the Debian pool,
>> if that is not enough for them, they should be old enough to be able to
>> click on the upstream homepage link in the package's description and get
>> the source.
>>
>> A lot of people, including myself, prefer to pull form the upstream vcs
>> directly, and work on top of that, using git for example. Using uscan to
>> retrieve the exact version is often impossible, as it's not trivial to
>> get a tarball from a specific upstream branch, tag or ref.
>>
>> I think the way Debian should go is to tell people that they should
>> clone the developer's git ([.. insert your favourite dvcs here ...])
>> repository and work with it, probably requiring to explain how working
>> with the repository works, which branches are used for what, and so
>> on. At least that would fit *todays* way of handling packages, at least
>> for a lot of people.
> 
> Hm, I think I disagree with most of this.
> 
> First, I think this new habit (which you don't mention directly but
> somewhat allude to) of not making stable formal releases is a very bad one
> and I would strongly encourage any of my upstreams to not go down that
> path.

I know that some upstream go this way, and I'm not happy about it - but if you
have a sane upstream, there's no problem to import a tag or branch into git (or
whatever you prefer), while keeping the whole development history makes it
*MUCH* more easy to bisect bugs, to release experimental snapshots or just to
retrieve a diff you'd like to add as patch.


> On the topic of finding the current upstream release, I definitely don't
> agree with the idea that the home page link solves this problem.  Some
> upstreams have extremely bizarre release processes, poor home pages, no
> real home page at all, or make it difficult to figure out just where the
> source is at.  Having a watch file that embeds all of the packager's
> existing knowledge about how to find the upstream release is very
> valuable.
> Also, I think you're underestimating the utility of being able to find
> exactly the tarball that was used for generating a given Debian package.
> It allows independent verification of the package in the archive (useful
> in some security scenarios), and it's very important for package
> sponsorship where one should not trust the orig.tar.gz provided by the
> sponsoree unless you already know the sponsoree well.

True, but on the other side, if you and upstream are using git, it is not hard
to verify that the upstream source is still fine. Being able to download a
tarball for sponsoring via a watch file makes my life as sponsor much more easy,
indeed - but I still don't see a reason why we need a second watch file to
retrieve the current tarball. Either the original tarball is in the archive, or,
if it was repackaged, there's no chance to retrieve it with the same md5sum
again. If you're able to find the current tarball by using the normal watch file
and trying to match for the right version, that's fine, but the idea of having a
second watch file is still insane imho.

Cheers,

Bernd

-- 
 Bernd Zeimetz                           Debian GNU/Linux Developer
 GPG Fingerprint: 06C8 C9A2 EAAD E37E 5B2C BE93 067A AD04 C93B FF79




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Mon, 16 Mar 2009 10:49:18 GMT) Full text and rfc822 format available.

Acknowledgement sent to Manoj Srivastava <srivasta@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Mon, 16 Mar 2009 10:49:18 GMT) Full text and rfc822 format available.

Message #204 received at 466550@bugs.debian.org (full text, mbox):

From: Manoj Srivastava <srivasta@debian.org>
To: debian-devel@lists.debian.org
Cc: debian-policy <debian-policy@debian.org>, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Mon, 16 Mar 2009 03:14:25 -0500
On Mon, Mar 16 2009, Ben Finney wrote:

> Manoj Srivastava <srivasta@debian.org> writes:
>
>>         I would not be against a recommendation in policy to implement
>>  direct-from-vcs  upstream tarballs to be created vbia get-orig-source,
>>  and everyone else just use debian/watch and debian/urepack files.
>
> Okay, now I'm officially confused. I don't see how the patch [0] I've
> submitted for this issue does not satisfy what you're saying.
>
> Ideally, I'd like to see you produce a patch for bug#466550 that
> demonstrates what you're saying, so I can see the difference. I can
> understand if that's too much effort though.

        I can see how this discussion could have gotten confusing.

Use cases:
 A) Get upstream version from the Debian archive
 B) Get a specific version from upstream (perhaps to package, or to
    verify the version in the debian archive)
 C) Get the latest version from upstream (usually to package it)
        In cases B and C above, the upstream distribution can be a
 tarball or a VCS.

        Let me see if I can capture the current status again. I am
 starting with a modified form of Kapil's statement early in the report:
1. Once pkg_ver.orig.tar.gz enters the Debian archive this is considered
   the authoritative Debian version from which all the binary Debian
   packages will be built (for that version of the package). A
   signature/checksum is used (in the upload and the Sources.gz file) so
   as to detect any "contamination". apt-get source is enough to get the
   latest Debian source from the archive (and wget for older sources)
2. Whenever upstream releases a new version, one needs to create a
   pkg_nver.orig.tar.gz for the newer version. In most cases this is merely a
   matter of downloading and renaming an upstream tar.gz. 
3. If re-packaging of upstream sources was required in order to create
   this .orig.tar.gz, then this should be documented in the copyright
   file (with some further explication in README.Debian-source
   perhaps). 
4. If upstream distributes tarballs, the "uscan" and "uupdate" programs
   are adequate and there is no significant need for a get-orig-source
   target. 
5. If the upstream distribution is in the form of a VCS, then uscan does
   not cater to it. This seems to be the case where get-orig-source can
   fill a need.



        There are these three variables that govern the logic:
package in Debian already:           Yes/No
Upstream code Mangling Required:     Yes/No
Upstream has  tarballs:              Yes/No
Version to Get:                      Latest/Current

   In tabular/Karnaugh map form (X are the don't care states):
|----------+---------+--------+----------+--------+----------|
| Already  | Version | Has      tarballs | Only     VCS      |
| Packaged | to get  | Mangle | Pristine | Mangle | Pristine |
|----------+---------+--------+----------+--------+----------|
| Yes        latest  | uscan  | uscan    | GOS    | GOS      |
| Yes        current | uscan  | uscan    | GOS    | GOS      |
| No         latest  | uscan  | uscan    | GOS    | GOS      |
| No         current |   X    |   X      |  X     |  X       |
|----------+---------+--------+----------+--------+----------|
        By logic minimization, the answer is clear :)

        While the target was originally designed for cases where we had
 to mangle upstream sources, after this discussion and analysis I am
 coming to the conclusion that uscan has matured to cover all cases
 where upstream distributes a tarball; making the target obsolete. The
 places where we do not have an existing solution is if upstream
 distributes sources _only_ in a VCS.   

        Now, your patch states:
--8<---------------cut here---------------start------------->8---
+		This target generates the original source archive for
+		the package, such that its contents exactly match the
+		original source archive used to generate the package
+		for Debian.

+		The commands for this target fetch the original source
+		package, corresponding to the Debian package version,
+		from a canonical archive site (for example, via FTP,
+		WWW, or a public VCS repository), do any necessary
 		rearrangement to turn it into the original source
+		archive file format, and leave it in the current
+		directory. See <ref id="pkg-sourcearchives"> for
+		policy details of the original source archive.
--8<---------------cut here---------------end--------------->8---

        There are some places where I differ:
 a) You ask this target only refer to the version in the changelog, and
    not the latest version
 b) You ask the file is left in the current directory, instead of ../
    where uscan leaves it
 c) This patch makes  the target work for cases where uscan would be
    enough -- watch files are useful for DEHS and the PTS and stuff, so
    we want to recommend watch files anyway, duplicating uscan in a
    target is not desired.

        I think that the wording for policy should:
  I) Reiterate this target is optional,
 II) Suggest that the target be present only when upstream sources were
     acquired from a VCS
III) suggest that a variable or option called "GOS_version" be
     honored if resent, or else the HEAD of the upstream branch be
     used. The contents of the GOS_version would be something relevant to
     the VCS being used
 IV) suggest that a watch file be used for cases where upstream provides a
     tarball, since this is useful in itself
  V) suggest that upstream mangling scripts be named debian/urepack, and
     if present, should work when invoked as 
      debian/urepack --version <version>  <upstream file name>

        Whew. This was longer than I had hoped for.

        I hope y'all like it.

        Thoughts?

        manoj
-- 
Nice guys finish last, but we get to sleep in. Evan Davis
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Tue, 17 Mar 2009 15:33:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Bill Allombert <Bill.Allombert@math.u-bordeaux1.fr>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Tue, 17 Mar 2009 15:33:03 GMT) Full text and rfc822 format available.

Message #209 received at 466550@bugs.debian.org (full text, mbox):

From: Bill Allombert <Bill.Allombert@math.u-bordeaux1.fr>
To: debian-devel@lists.debian.org, debian-policy <debian-policy@debian.org>, 466550@bugs.debian.org
Subject: Re: Bug#466550: Pristine source from upstream VCS repository
Date: Tue, 17 Mar 2009 16:31:06 +0100
On Mon, Mar 16, 2009 at 03:14:25AM -0500, Manoj Srivastava wrote:
> On Mon, Mar 16 2009, Ben Finney wrote:
> 
> > Manoj Srivastava <srivasta@debian.org> writes:
> >
> >>         I would not be against a recommendation in policy to implement
> >>  direct-from-vcs  upstream tarballs to be created vbia get-orig-source,
> >>  and everyone else just use debian/watch and debian/urepack files.
> >
> > Okay, now I'm officially confused. I don't see how the patch [0] I've
> > submitted for this issue does not satisfy what you're saying.
> >
> > Ideally, I'd like to see you produce a patch for bug#466550 that
> > demonstrates what you're saying, so I can see the difference. I can
> > understand if that's too much effort though.
> 
>         I can see how this discussion could have gotten confusing.
> 
> Use cases:
>  A) Get upstream version from the Debian archive
>  B) Get a specific version from upstream (perhaps to package, or to
>     verify the version in the debian archive)
>  C) Get the latest version from upstream (usually to package it)
>         In cases B and C above, the upstream distribution can be a
>  tarball or a VCS.

There is a use case where get-orig-source is crucial:

  D) Upstream only provides an unversionned set of files on a 
     webserver. In that case, the packagers need to assemble 
     the files and version the result themselves. It is vital
     that such procedure be documented in the form of a script.

Cheers,
-- 
Bill. <ballombe@debian.org>

Imagine a large red swirl here. 




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Wed, 25 Mar 2009 23:03:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Daniel Dickinson <cshore@brucetelecom.com>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Wed, 25 Mar 2009 23:03:02 GMT) Full text and rfc822 format available.

Message #214 received at 466550@bugs.debian.org (full text, mbox):

From: Daniel Dickinson <cshore@brucetelecom.com>
To: 466550@bugs.debian.org, debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: This topic died off; any resolution?
Date: Wed, 25 Mar 2009 19:00:38 -0400
[Message part 1 (text/plain, inline)]
I kind of got lost in this discussion.  Is there a summary and debian
policy and debian reference patch so that those of us who are just
looking to do what we're supposed to do know what we are supposed to do
and how to do it?

Thanks,

Daniel
-- 
And that's my crabbing done for the day.  Got it out of the way early, 
now I have the rest of the afternoon to sniff fragrant tea-roses or 
strangle cute bunnies or something.   -- Michael Devore
GnuPG Key Fingerprint 86 F5 81 A5 D4 2E 1F 1C      http://gnupg.org
[signature.asc (application/pgp-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 26 Mar 2009 02:54:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Russ Allbery <rra@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 26 Mar 2009 02:54:02 GMT) Full text and rfc822 format available.

Message #219 received at 466550@bugs.debian.org (full text, mbox):

From: Russ Allbery <rra@debian.org>
To: Daniel Dickinson <cshore@brucetelecom.com>
Cc: 466550@bugs.debian.org, debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: This topic died off; any resolution?
Date: Wed, 25 Mar 2009 19:53:13 -0700
Daniel Dickinson <cshore@brucetelecom.com> writes:

> I kind of got lost in this discussion.  Is there a summary and debian
> policy and debian reference patch so that those of us who are just
> looking to do what we're supposed to do know what we are supposed to do
> and how to do it?

I think where we're at (with get-orig-source) right now is that uscan has
a feature to run a script after downloading the upstream source.  That
seems like usually the right way of providing repackaging for new upstream
source releases and I think we should find a way to encourage people to
provide such a script.

debian/rules get-orig-source as an interface to run uscan with the
appropriate options seems like it might be a good idea to keep the
interface consistent.  We would need to resolve the question of what
upstream source this should download.  I think I would lean towards having
it download the latest upstream release rather than the one mentioned in
the changelog entry because while I see the reason for doing it the other
way, I think that it wouldn't actually make much difference in workflow.
In the unusual case where one wants to download a very specific upstream
release that isn't the one uscan thinks is latest, one can always download
the tarball and run the repacking script on it by hand; with the repacking
script as a separate script, that shouldn't be too many steps.

That still leaves open the question of what get-orig-source should do with
a package that can't use uscan (unversioned upstream files, multiple
upstream tarballs, upstream that uses only a VCS, etc.).  Currently,
Policy says that it should download and create a tarball of the most
current upstream sources.  I think that makes sense, but it's harder than
doing the most recent blessed sources and might break a lot.  It also
leaves one without an easy way of duplicating exactly the tarball that was
uploaded with the Debian package.

I'm not sure we've reached consensus on the whole discussion, but the
above is my current opinion.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Thu, 26 Mar 2009 05:15:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Finney <ben+debian@benfinney.id.au>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Thu, 26 Mar 2009 05:15:03 GMT) Full text and rfc822 format available.

Message #224 received at 466550@bugs.debian.org (full text, mbox):

From: Ben Finney <ben+debian@benfinney.id.au>
To: 466550@bugs.debian.org
Cc: debian-devel@lists.debian.org, debian-policy@lists.debian.org
Subject: Re: Bug#466550: Clarify or remove the get-orig-source target specification
Date: Thu, 26 Mar 2009 16:13:04 +1100
[Message part 1 (text/plain, inline)]
On 25-Mar-2009, Russ Allbery wrote:
> I think where we're at (with get-orig-source) right now is that
> uscan has a feature to run a script after downloading the upstream
> source.  That seems like usually the right way of providing
> repackaging for new upstream source releases and I think we should
> find a way to encourage people to provide such a script.

This part I agree with; I am glad to learn ‘uscan’ has this
capability, and I wonder why it's not better known.

> debian/rules get-orig-source as an interface to run uscan with the
> appropriate options seems like it might be a good idea to keep the
> interface consistent.

Yes.

> In the unusual case where one wants to download a very specific
> upstream release that isn't the one uscan thinks is latest, one can
> always download the tarball and run the repacking script on it by
> hand; with the repacking script as a separate script, that shouldn't
> be too many steps.

Here my position differs from Russ somewhat.

This case is unusual (in that it's by far not the usual case), but not
rare in my experience. Quite often I've deliberately wanted to build a
specific version of upstream's code that isn't the latest released
version, and it seemed obvious to me (before this discussion) that
the ‘get-orig-source’ target should obey the version string from
‘debian/changelog’ for this purpose.

I admit that I am unable to decide how this use case should be
satisfied. While it's not the normal case, I think it's the one least
able to be satisfied trivially and for *that* reason should be the
normal operation of the ‘get-orig-source’ target.

In other words: getting the original source *for this Debian release
of the package* is what I associate with an operation named
‘get-orig-source’. If “get the latest upstream source” needs
automation, perhaps a hypothetical ‘get-latest-source’ target is
called for.

> That still leaves open the question of what get-orig-source should
> do with a package that can't use uscan (unversioned upstream files,
> multiple upstream tarballs, upstream that uses only a VCS, etc.).
> Currently, Policy says that it should download and create a tarball
> of the most current upstream sources.  I think that makes sense, but
> it's harder than doing the most recent blessed sources and might
> break a lot.  It also leaves one without an easy way of duplicating
> exactly the tarball that was uploaded with the Debian package.

This is made easier, I feel, if ‘get-orig-source’ is defined to get
the original source as referenced by the Debian version number of the
package: just key it to the specific revision identifier for that
upstream version.

-- 
 \     “To punish me for my contempt of authority, Fate has made me an |
  `\                   authority myself.” —Albert Einstein, 1930-09-18 |
_o__)                                                                  |
Ben Finney <ben@benfinney.id.au>
[signature.asc (application/pgp-signature, inline)]

Removed tag(s) patch. Request was from Russ Allbery <rra@debian.org> to control@bugs.debian.org. (Tue, 22 Jun 2010 16:48:02 GMT) Full text and rfc822 format available.

Added tag(s) confirmed. Request was from Jelmer Vernooij <jelmer@debian.org> to control@bugs.debian.org. (Wed, 30 Nov 2011 17:09:05 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Fri, 16 Dec 2011 20:39:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Jelmer Vernooij <jelmer@vernstok.nl>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Fri, 16 Dec 2011 20:39:03 GMT) Full text and rfc822 format available.

Message #233 received at 466550@bugs.debian.org (full text, mbox):

From: Jelmer Vernooij <jelmer@vernstok.nl>
To: 466550@bugs.debian.org
Subject: Fixed in 2.8.0
Date: Fri, 16 Dec 2011 21:37:46 +0100
Version: 2.8.0

This bug was fixed in upstream version 2.8.0.o

bzr-builddeb now warns when using the get-orig-source target in
debian/rules to fetch the currently package upstream tarball.
get-packaged-orig-source should be provided instead.




Reply sent to Jelmer Vernooij <jelmer@vernstok.nl>:
You have taken responsibility. (Fri, 16 Dec 2011 20:54:03 GMT) Full text and rfc822 format available.

Notification sent to Alexander Schmehl <alexander@schmehl.info>:
Bug acknowledged by developer. (Fri, 16 Dec 2011 20:54:03 GMT) Full text and rfc822 format available.

Message #238 received at 466550-done@bugs.debian.org (full text, mbox):

From: Jelmer Vernooij <jelmer@vernstok.nl>
To: 466550-done@bugs.debian.org
Subject: Fixed in 2.8.0
Date: Fri, 16 Dec 2011 21:52:01 +0100
Version: 2.8.0

This bug was fixed in upstream version 2.8.0.o

bzr-builddeb now warns when using the get-orig-source target in
debian/rules to fetch the currently package upstream tarball.
get-packaged-orig-source should be provided instead.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#466550; Package debian-policy. (Fri, 16 Dec 2011 21:21:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steve Langasek <vorlon@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. (Fri, 16 Dec 2011 21:21:03 GMT) Full text and rfc822 format available.

Message #243 received at 466550@bugs.debian.org (full text, mbox):

From: Steve Langasek <vorlon@debian.org>
To: Jelmer Vernooij <jelmer@vernstok.nl>
Cc: 466550@bugs.debian.org
Subject: Re: Fixed in 2.8.0
Date: Fri, 16 Dec 2011 13:19:08 -0800
[Message part 1 (text/plain, inline)]
reopen 466550
thanks

On Fri, Dec 16, 2011 at 09:52:01PM +0100, Jelmer Vernooij wrote:
> Version: 2.8.0

> This bug was fixed in upstream version 2.8.0.o

> bzr-builddeb now warns when using the get-orig-source target in
> debian/rules to fetch the currently package upstream tarball.
> get-packaged-orig-source should be provided instead.

This is a bug filed against debian-policy, though, not against bzr-builddeb.

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@ubuntu.com                                     vorlon@debian.org
[signature.asc (application/pgp-signature, inline)]

Bug No longer marked as fixed in versions 2.8.0 and reopened. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Fri, 16 Dec 2011 21:21:07 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Thu Apr 24 03:00:38 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.