Debian Bug report logs - #654474
Doesn't contain source for waf binary code

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Gerfried Fuchs <rhonda@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: postler: doesn't contain source for waf binary code

Date: Thu, 13 Oct 2011 14:00:08 +0200

Package: postler
Version: 0.1.1-1
Severity: serious

        Hi!

 This was actually found in Ubuntu: https://launchpad.net/bugs/873003

 The included waf script contains binary code in line 161 for which no
source is available, which is a clear policy violation.

 Please include the source for that and actually compile that source and
use the compiled binary data instead of the one that is included now in
the source package.

 Thanks,
Rhonda
-- 
Fühlst du dich mutlos, fass endlich Mut, los      |
Fühlst du dich hilflos, geh raus und hilf, los    | Wir sind Helden
Fühlst du dich machtlos, geh raus und mach, los   | 23.55: Alles auf Anfang
Fühlst du dich haltlos, such Halt und lass los    |

Message #10 received at 645190@bugs.debian.org (full text, mbox, reply):

From: Jakub Wilk <jwilk@debian.org>

To: Gerfried Fuchs <rhonda@debian.org>, 645190@bugs.debian.org

Subject: Re: Bug#645190: postler: doesn't contain source for waf binary code

Date: Thu, 13 Oct 2011 14:11:59 +0200

* Gerfried Fuchs <rhonda@debian.org>, 2011-10-13, 14:00:
>The included waf script contains binary code in line 161 for which no 
>source is available, which is a clear policy violation.

FWIW, the blob _does_ contain (compressed and pickled) source. If you 
run the script (even without any arguments), it will be unpacked to 
./.waf-*/wafadmin/.

-- 
Jakub Wilk

Message #15 received at 645190@bugs.debian.org (full text, mbox, reply):

From: Gerfried Fuchs <rhonda@debian.org>

To: 645190@bugs.debian.org

Subject: Re: Bug#645190: postler: doesn't contain source for waf binary code

Date: Thu, 13 Oct 2011 14:16:56 +0200

* Jakub Wilk <jwilk@debian.org> [2011-10-13 14:11:59 CEST]:
> * Gerfried Fuchs <rhonda@debian.org>, 2011-10-13, 14:00:
> >The included waf script contains binary code in line 161 for which
> >no source is available, which is a clear policy violation.
> 
> FWIW, the blob _does_ contain (compressed and pickled) source. If
> you run the script (even without any arguments), it will be unpacked
> to ./.waf-*/wafadmin/.

 As nice as this might be, but somewhat irrelevant and the wrong way
around: We require source to produce binaries, not the other way round.

 Thanks,
Rhonda
-- 
Fühlst du dich mutlos, fass endlich Mut, los      |
Fühlst du dich hilflos, geh raus und hilf, los    | Wir sind Helden
Fühlst du dich machtlos, geh raus und mach, los   | 23.55: Alles auf Anfang
Fühlst du dich haltlos, such Halt und lass los    |

Message #20 received at 645190@bugs.debian.org (full text, mbox, reply):

From: Gerfried Fuchs <rhonda@deb.at>

To: 645190@bugs.debian.org, 645191@bugs.debian.org

Subject: update on waf binary data

Date: Thu, 13 Oct 2011 15:12:29 +0200

     Hi again,

 it seems that the line 161 is actually a tar.bz2 file that gets
extracted and then used.  Though, first there is some substitution of \r
and \n characters so that the "file" could go on one line.

 IMHO this is not acceptable because there are no tools included or
commandline switches offered with waf (in postler and midori) to
conveniently unpack and repack these part for a.) inspection or b.)
modification, which are required for packages in Debian main.

 From what I understood there seems to be some waf-light that wouldn't
use the mangled tarball included within the script, I would guess that
this is the best way to move forward from here.

 If you really would like to argue that character substitution within
the tarball for embedding it in the waf script is acceptable in
accordance to policy/DFSG without direct tool to unpack/repack it, then
please discuss this on e.g. debian-devel or such, or overrule me and
lower the severity (but please provide understandable reasoning too),
I still believe that this is against our rules.

 Thanks in advance,
Rhonda
-- 
Fühlst du dich mutlos, fass endlich Mut, los      |
Fühlst du dich hilflos, geh raus und hilf, los    | Wir sind Helden
Fühlst du dich machtlos, geh raus und mach, los   | 23.55: Alles auf Anfang
Fühlst du dich haltlos, such Halt und lass los    |

Message #25 received at 645190@bugs.debian.org (full text, mbox, reply):

From: Alexander Reichle-Schmehl <tolimar@debian.org>

To: 645190@bugs.debian.org, 645191@bugs.debian.org

Cc: ftpmaster@debian.org

Subject: waf binary code not DFSG compliant

Date: Tue, 3 Jan 2012 22:12:06 +0100

user ftpmaster@debian.org
reopen 645191

usertags 645190 + waf-unpack
clone 645190 -1 -2 -3 -4 -5 -6 -7 -8 -9 -10 -11 -12 -13 -14 -15 -16 -17 -18 -19 -20 -21 -22 -23 -24 -25 -26 -27 -28 -29 -30 -31 -32 -33 -34 -35 -36 -37 -38 -39 -40 -41 -42 -43 -44 -45 -46 -47 -48 -49 -50 -51 -52 
reassign -1 a2jmidid
reassign -2 composite
reassign -3 ctpl
reassign -4 flowcanvas
reassign -5 geany
reassign -6 geany-plugins
reassign -7 gigolo
reassign -8 gmidimonitor
reassign -9 gnome-python
reassign -10 gnome-python-desktop
reassign -11 gtkimageview
reassign -12 guitarix
reassign -13 hamster-applet
reassign -14 hotssh
reassign -15 isoquery
reassign -16 jackd2
reassign -17 jalv
reassign -18 jcgui
reassign -19 kupfer
reassign -20 ladish
reassign -21 ldb
reassign -22 libdesktop-agnostic
reassign -23 lifeograph
reassign -24 lilv
reassign -25 lv2-extensions-good
reassign -26 lv2core
reassign -27 lv2fil
reassign -28 mda-lv2
reassign -29 mgen
reassign -30 minidjvu
reassign -31 nodejs
reassign -32 ns3
reassign -33 openchange
reassign -34 patchage
reassign -35 pino
reassign -36 radare
reassign -37 raul
reassign -38 samba
reassign -39 samba4
reassign -40 serd
reassign -41 showq
reassign -42 slv2
reassign -43 sord
reassign -44 suil
reassign -45 supercollider
reassign -46 sushi
reassign -47 talloc
reassign -48 tdb
reassign -49 tevent
reassign -50 xiphos
reassign -51 xmms2
reassign -52 zyn
thanks

Hi!

> IMHO this is not acceptable because there are no tools included or
> commandline switches offered with waf (in postler and midori) to
> conveniently unpack and repack these part for a.) inspection or b.)
> modification, which are required for packages in Debian main.

A package in NEW brought this matter to our attention, and after
discussing the issue within the FTP Team, we came to the conclusion that
the submitter of this bug report is correct: packages using waf in this
form do not ship all sources in their prefered form of modification¹.

While the letters of DFSG#2 and the Debian Policy could be fullfilled by
shipping waf in extracted form in the source packages, we would really
love to see the packages moving to a saner build system.

A quick tutorial on how to unpack waf to fulfil our requirements can be
found here: http://wiki.debian.org/UnpackWaf

Best regards,
  Alexander
  for the FTP Team

1: Yes, that phrase originates from the GPL, nevertheless Debian uses it as definiton of "source".

Message #38 received at 654474@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: 654474@bugs.debian.org

Cc: Gerfried Fuchs <rhonda@debian.org>, Luca Falavigna <dktrkranz@debian.org>, Debian Devel <debian-devel@lists.debian.org>

Subject: Doesn't contain source for waf binary code

Date: Fri, 03 Feb 2012 20:33:56 +0100

[Message part 1 (text/plain, inline)]

Hi,

as this issue affects quite a few packages, I'd like to bring this up
for wider discussion.

The issue basically is, that the waf build system uses a python script,
which embeds a bz2 tarball containing further python sources. Those are
unpacked to .waf-*/ when the waf script is executed. More details can be
found at [1].

A few observations/questions:

* We do accept tarball-in-tarball packages. What makes this specific
case different?

* Where exactly is the DFSG violation? The package does ship the source
code even if embedding it in the waf script doesn't make that obvious

but

* Would it be sufficient to document in README.source how a user can
unpack and modify the waf sources (basically what [2] does) and wouldn't
this fulfill the DFSG requirements?


* What about packages which ship both an autotools and waf based build
system and the Debian package uses the one based on autotools?



Michael


[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=645190
[2] http://wiki.debian.org/UnpackWaf


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #45 received at 654474-close@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: 654474-close@bugs.debian.org

Subject: Bug#654474: fixed in hamster-applet 2.91.3+git20120204.b11571c.dfsg-1

Date: Mon, 06 Feb 2012 15:02:16 +0000

Source: hamster-applet
Source-Version: 2.91.3+git20120204.b11571c.dfsg-1

We believe that the bug you reported is fixed in the latest version of
hamster-applet, which is due to be installed in the Debian FTP archive:

hamster-applet_2.91.3+git20120204.b11571c.dfsg-1.debian.tar.gz
  to main/h/hamster-applet/hamster-applet_2.91.3+git20120204.b11571c.dfsg-1.debian.tar.gz
hamster-applet_2.91.3+git20120204.b11571c.dfsg-1.dsc
  to main/h/hamster-applet/hamster-applet_2.91.3+git20120204.b11571c.dfsg-1.dsc
hamster-applet_2.91.3+git20120204.b11571c.dfsg-1_all.deb
  to main/h/hamster-applet/hamster-applet_2.91.3+git20120204.b11571c.dfsg-1_all.deb
hamster-applet_2.91.3+git20120204.b11571c.dfsg.orig.tar.xz
  to main/h/hamster-applet/hamster-applet_2.91.3+git20120204.b11571c.dfsg.orig.tar.xz



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 654474@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Michael Biebl <biebl@debian.org> (supplier of updated hamster-applet package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Mon, 06 Feb 2012 15:52:29 +0100
Source: hamster-applet
Binary: hamster-applet
Architecture: source all
Version: 2.91.3+git20120204.b11571c.dfsg-1
Distribution: unstable
Urgency: low
Maintainer: Debian GNOME Maintainers <pkg-gnome-maintainers@lists.alioth.debian.org>
Changed-By: Michael Biebl <biebl@debian.org>
Description: 
 hamster-applet - time tracking applet for GNOME
Closes: 654474
Changes: 
 hamster-applet (2.91.3+git20120204.b11571c.dfsg-1) unstable; urgency=low
 .
   [ Josselin Mouette ]
   * Replace python-gobject by python-gobject-2.
   * Update repository URL.
 .
   [ Michael Biebl ]
   * New upstream Git snapshot.
   * Repack sources and ship an unpacked waf binary following the instructions
     from http://wiki.debian.org/UnpackWaf. Closes: #654474
   * Remove generated .pyc files during clean.
Checksums-Sha1: 
 7c7f9b073335cb6294e10dd3c95d7564d4493cec 2409 hamster-applet_2.91.3+git20120204.b11571c.dfsg-1.dsc
 3ccc51cfbc4dd9e9672fb4e3466c615147e10296 445712 hamster-applet_2.91.3+git20120204.b11571c.dfsg.orig.tar.xz
 8219bbfa4b24ed8441d66154f603c765740663d4 5014 hamster-applet_2.91.3+git20120204.b11571c.dfsg-1.debian.tar.gz
 9dd253e012f0a83be8d2f114a0313058d6012804 513990 hamster-applet_2.91.3+git20120204.b11571c.dfsg-1_all.deb
Checksums-Sha256: 
 9da824a6ff7035796383cd91b8535e80907477fb9370d7d7a897410035833454 2409 hamster-applet_2.91.3+git20120204.b11571c.dfsg-1.dsc
 1918fcb90c46c8a0cc4619f84f9a5a1505977213320ccadae680b58df5b1cc5d 445712 hamster-applet_2.91.3+git20120204.b11571c.dfsg.orig.tar.xz
 b47478ba4d3cb0a4b4f344a0f6dc17ee840b0ea129fc02b99c9232b414c2b601 5014 hamster-applet_2.91.3+git20120204.b11571c.dfsg-1.debian.tar.gz
 ee88db0079130d28c822843ec123c6c463514d0f44393c320c59810b667f1416 513990 hamster-applet_2.91.3+git20120204.b11571c.dfsg-1_all.deb
Files: 
 d76ae5b77a447d480ffdc089047eb5a5 2409 gnome optional hamster-applet_2.91.3+git20120204.b11571c.dfsg-1.dsc
 1c991e4be9808455703b496bfe7a0dd1 445712 gnome optional hamster-applet_2.91.3+git20120204.b11571c.dfsg.orig.tar.xz
 2ae728702af1425e3e2a24ee1d055605 5014 gnome optional hamster-applet_2.91.3+git20120204.b11571c.dfsg-1.debian.tar.gz
 a9af47ea9675df9b5df6618c37735f3b 513990 gnome optional hamster-applet_2.91.3+git20120204.b11571c.dfsg-1_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBCAAGBQJPL+l9AAoJEGrh3w1gjyLc2OsP/0DMRnDugVZYs4i5oqfjr+cz
9k2p84y52UU2BL46yNid9y9hIf1Nf2nSAi/9L7WHZgFKYBBqVwbiQxVdAMYy+57W
61BwT+UNUB+7cd85KBnUFh0f2tn8OuVZscb4G6KLq4AyuhQMOHHXVCI/EX54JuuQ
hrcrqKP+0BjbsDy8igDvbbxLQe23uHq3zHCebsxEo853FE4Z4N1p1el6dlc9jmWB
VBhYbKJ6dvLTAR0usaJwl7IOFBO/fHD+uAXt1iRUFOX7zD85xj0D0lr8TtlItYXZ
OJqubY1QOZ6/epPqEunTH5FDb9Mz+AvrDwX+sMquW8Ak8uaVzCFcU8JS+2qJfAyD
L7+dzogjobajTGZGsSfoM0WwjV7How/LeyAX5KSXo1pBQBntloCqopY5/U67KaCV
6kly+7h8GlxkADe4GMIWLEYMMoFU5QU8K2Vz7U70QHGmg1ZZBLRtlvnP31X/5qw0
rdidR+U0mRnL2e/h5Euk25JfI6NJlmtdI+D2bvl2El8dlXhN2Jypdq6OO9vaEKw0
H63F6NrhmjneGTU8mmpTnFJmF8IOxaBhQA+5xAofzZoQ0+kfrTEPzxK5Ts58+1x7
i/XnhLvATyf9wJtPCYlqCYowc+ftbD+RlR6ul7KA8a8T4BJok4r1M8883q4eNUqV
57HfmbvDgb+QEzpKaspN
=bYZ2
-----END PGP SIGNATURE-----

Message #50 received at 654474@bugs.debian.org (full text, mbox, reply):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>

To: Michael Biebl <biebl@debian.org>

Cc: 654474@bugs.debian.org, Gerfried Fuchs <rhonda@debian.org>, Luca Falavigna <dktrkranz@debian.org>, Debian Devel <debian-devel@lists.debian.org>

Subject: Re: Doesn't contain source for waf binary code

Date: Tue, 7 Feb 2012 13:16:01 +0000

Michael Biebl writes ("Doesn't contain source for waf binary code"):
> as this issue affects quite a few packages, I'd like to bring this up
> for wider discussion.
> 
> The issue basically is, that the waf build system uses a python script,
> which embeds a bz2 tarball containing further python sources. Those are
> unpacked to .waf-*/ when the waf script is executed. More details can be
> found at [1].

This is quite astonishing enough, but the situation is in fact even
worse than it appears.  I investigated, and my conclusions are:

"waf" is a build system written in Python.  It is normally distributed
in the form of a script called "waf", which the waf authors intend for
upstream authors to include in their upstream distribution tarballs
etc.  The script is a self-extracting archive which, whenever it's
run, extracts the tarball out of itself into a temporary directory and
then passes control to the python code it has just extracted.

This IMO would be bad enough to reject an upstream package, in this
form, in Debian.  After all we want to be able to modify the build
system as well as the package and this approach makes it unreasonably
hard to do so.  And if the build system has some kind of bug we don't
want to have to update dozens of copies embedded in individual source
packages.

But there is more, and worse.

I compared the tarball in waf in postler 0.1.1-1.1 with the upstream
code as obtained from "git clone https://code.google.com/p/waf/".  It
turns out that the tarball embedded in the "waf" script is not the
original "waf" source distribution.  It contains a subset of the
files, and those files it does contain have been processed to remove
comments, whitespace, etc., much like a JavaScript minimisation.  Ie
the "waf" self-extracting archive is generated out of the waf.git
source code by massaging some of the files; modified versions of the
script are supposed to be generated by editing the waf.git
distribution and rerunning its build.

This means that we are distributing files derived from the waf.git
source code, but not the waf.git source code itself.  This is of
course completely unacceptable in Debian.  (It is not a violation of
the copyright on waf itself as waf has a permissive non-copyleft
licence; but will be a breach of the copyright on any GPL'd waf-using
package, because the GPL's requirements extend to the build system.)

I suggest the following fix:

 * Upstream waf should be packaged somehow.  Upstream's declared
   policy of asking packages to ship a copy of waf suggests that there
   won't be much API stability so we will need to encode the waf
   version number in the package name, and we may need to package
   multiple versions of waf.

 * All packages which currently use an included copy of waf should be
   changed to use a system-provided copy of waf instead.

 * We should treat the file "waf" in the root of affected packages as
   we do any other file which is non-DFSG-compliant but which we do
   have permission to redistribute.  Our current practice is to repack
   upstream source archives to remove these files from the Debian
   sources.  (I think this is pointless makework but changing that
   policy is out of scope for this discussion.)

 * It is possible that some upstream "source" packages contain "waf"
   scripts which were generated from modified versions of waf.git.  In
   this case we may discover that those packages cannot be built with
   publicly available versions of waf.git.  

   For those packages, we need to obtain the relevant version of
   waf.git (for example, by hassling the package upstream), or perhaps
   if it's a simple change we can create our own suitable version of
   waf.git by "decompiling" (de-minimising) the supplied "waf" script
   tarball contents (and permanently maintain the forked waf.git).

   If neither of those is possible and the issue can't be worked
   around some other way we will have to move the affected package
   from main to non-free; if the affected upstream package is
   copylefted then we will have no permission to distribute it at all
   and must drop it entirely.

I think this is a release-critical bug for all the affected packages.

Ian.

Message #57 received at 654474@bugs.debian.org (full text, mbox, reply):

From: Alexander Reichle-Schmehl <tolimar@debian.org>

To: Ian Jackson <ijackson@chiark.greenend.org.uk>

Cc: Michael Biebl <biebl@debian.org>, 654474@bugs.debian.org, Gerfried Fuchs <rhonda@debian.org>, Luca Falavigna <dktrkranz@debian.org>, Debian Devel <debian-devel@lists.debian.org>

Subject: Re: Doesn't contain source for waf binary code

Date: Tue, 07 Feb 2012 17:25:48 +0100

Hi!


Am 07.02.2012 14:16, schrieb Ian Jackson:

>  * Upstream waf should be packaged somehow.  Upstream's declared
>    policy of asking packages to ship a copy of waf suggests that there
>    won't be much API stability so we will need to encode the waf
>    version number in the package name, and we may need to package
>    multiple versions of waf.

Sorry, I'm quite busy and don't have time for a longer answer. We'll
follow up on this soonish.

However, regarding that specific point:  waf once was packaged in
Debian.  See <20100227195857.07540195@utumno>
(http://lists.debian.org/debian-devel/2010/02/msg00714.html) for details
about the removal.


Best regards,
  Alexander

Message #62 received at 654474@bugs.debian.org (full text, mbox, reply):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>

To: Alexander Reichle-Schmehl <tolimar@debian.org>

Cc: Michael Biebl <biebl@debian.org>, 654474@bugs.debian.org, Gerfried Fuchs <rhonda@debian.org>, Luca Falavigna <dktrkranz@debian.org>, Debian Devel <debian-devel@lists.debian.org>

Subject: Re: Doesn't contain source for waf binary code

Date: Tue, 7 Feb 2012 18:15:12 +0000

Alexander Reichle-Schmehl writes ("Re: Doesn't contain source for waf binary code"):
> However, regarding that specific point:  waf once was packaged in
> Debian.  See <20100227195857.07540195@utumno>
> (http://lists.debian.org/debian-devel/2010/02/msg00714.html) for details
> about the removal.

Urgh.  Thanks for the pointer.

Ian.

Send a report that this bug log contains spam.