Debian Bug report logs - #645190
postler: doesn't contain source for waf binary code

version graph

Package: postler; Maintainer for postler is (unknown);

Reported by: Gerfried Fuchs <rhonda@debian.org>

Date: Thu, 13 Oct 2011 12:03:04 UTC

Severity: serious

Tags: patch, squeeze-ignore

Found in version postler/0.1.1-1

Fixed in version postler/0.1.1+dfsg-0.1

Done: Luca Falavigna <dktrkranz@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Thu, 13 Oct 2011 12:03:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to Gerfried Fuchs <rhonda@debian.org>:
New Bug report received and forwarded. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Thu, 13 Oct 2011 12:03:41 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Gerfried Fuchs <rhonda@debian.org>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: postler: doesn't contain source for waf binary code
Date: Thu, 13 Oct 2011 14:00:08 +0200
Package: postler
Version: 0.1.1-1
Severity: serious

        Hi!

 This was actually found in Ubuntu: https://launchpad.net/bugs/873003

 The included waf script contains binary code in line 161 for which no
source is available, which is a clear policy violation.

 Please include the source for that and actually compile that source and
use the compiled binary data instead of the one that is included now in
the source package.

 Thanks,
Rhonda
-- 
Fühlst du dich mutlos, fass endlich Mut, los      |
Fühlst du dich hilflos, geh raus und hilf, los    | Wir sind Helden
Fühlst du dich machtlos, geh raus und mach, los   | 23.55: Alles auf Anfang
Fühlst du dich haltlos, such Halt und lass los    |




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Thu, 13 Oct 2011 12:15:19 GMT) Full text and rfc822 format available.

Acknowledgement sent to Jakub Wilk <jwilk@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Thu, 13 Oct 2011 12:15:41 GMT) Full text and rfc822 format available.

Message #10 received at 645190@bugs.debian.org (full text, mbox):

From: Jakub Wilk <jwilk@debian.org>
To: Gerfried Fuchs <rhonda@debian.org>, 645190@bugs.debian.org
Subject: Re: Bug#645190: postler: doesn't contain source for waf binary code
Date: Thu, 13 Oct 2011 14:11:59 +0200
* Gerfried Fuchs <rhonda@debian.org>, 2011-10-13, 14:00:
>The included waf script contains binary code in line 161 for which no 
>source is available, which is a clear policy violation.

FWIW, the blob _does_ contain (compressed and pickled) source. If you 
run the script (even without any arguments), it will be unpacked to 
./.waf-*/wafadmin/.

-- 
Jakub Wilk




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Thu, 13 Oct 2011 13:04:21 GMT) Full text and rfc822 format available.

Acknowledgement sent to Gerfried Fuchs <rhonda@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Thu, 13 Oct 2011 13:04:30 GMT) Full text and rfc822 format available.

Message #15 received at 645190@bugs.debian.org (full text, mbox):

From: Gerfried Fuchs <rhonda@debian.org>
To: 645190@bugs.debian.org
Subject: Re: Bug#645190: postler: doesn't contain source for waf binary code
Date: Thu, 13 Oct 2011 14:16:56 +0200
* Jakub Wilk <jwilk@debian.org> [2011-10-13 14:11:59 CEST]:
> * Gerfried Fuchs <rhonda@debian.org>, 2011-10-13, 14:00:
> >The included waf script contains binary code in line 161 for which
> >no source is available, which is a clear policy violation.
> 
> FWIW, the blob _does_ contain (compressed and pickled) source. If
> you run the script (even without any arguments), it will be unpacked
> to ./.waf-*/wafadmin/.

 As nice as this might be, but somewhat irrelevant and the wrong way
around: We require source to produce binaries, not the other way round.

 Thanks,
Rhonda
-- 
Fühlst du dich mutlos, fass endlich Mut, los      |
Fühlst du dich hilflos, geh raus und hilf, los    | Wir sind Helden
Fühlst du dich machtlos, geh raus und mach, los   | 23.55: Alles auf Anfang
Fühlst du dich haltlos, such Halt und lass los    |




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Thu, 13 Oct 2011 13:15:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Gerfried Fuchs <rhonda@deb.at>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Thu, 13 Oct 2011 13:15:14 GMT) Full text and rfc822 format available.

Message #20 received at 645190@bugs.debian.org (full text, mbox):

From: Gerfried Fuchs <rhonda@deb.at>
To: 645190@bugs.debian.org, 645191@bugs.debian.org
Subject: update on waf binary data
Date: Thu, 13 Oct 2011 15:12:29 +0200
     Hi again,

 it seems that the line 161 is actually a tar.bz2 file that gets
extracted and then used.  Though, first there is some substitution of \r
and \n characters so that the "file" could go on one line.

 IMHO this is not acceptable because there are no tools included or
commandline switches offered with waf (in postler and midori) to
conveniently unpack and repack these part for a.) inspection or b.)
modification, which are required for packages in Debian main.

 From what I understood there seems to be some waf-light that wouldn't
use the mangled tarball included within the script, I would guess that
this is the best way to move forward from here.

 If you really would like to argue that character substitution within
the tarball for embedding it in the waf script is acceptable in
accordance to policy/DFSG without direct tool to unpack/repack it, then
please discuss this on e.g. debian-devel or such, or overrule me and
lower the severity (but please provide understandable reasoning too),
I still believe that this is against our rules.

 Thanks in advance,
Rhonda
-- 
Fühlst du dich mutlos, fass endlich Mut, los      |
Fühlst du dich hilflos, geh raus und hilf, los    | Wir sind Helden
Fühlst du dich machtlos, geh raus und mach, los   | 23.55: Alles auf Anfang
Fühlst du dich haltlos, such Halt und lass los    |




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Tue, 03 Jan 2012 21:30:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Alexander Reichle-Schmehl <tolimar@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Tue, 03 Jan 2012 21:30:06 GMT) Full text and rfc822 format available.

Message #25 received at 645190@bugs.debian.org (full text, mbox):

From: Alexander Reichle-Schmehl <tolimar@debian.org>
To: 645190@bugs.debian.org, 645191@bugs.debian.org
Cc: ftpmaster@debian.org
Subject: waf binary code not DFSG compliant
Date: Tue, 3 Jan 2012 22:12:06 +0100
user ftpmaster@debian.org
reopen 645191

usertags 645190 + waf-unpack
clone 645190 -1 -2 -3 -4 -5 -6 -7 -8 -9 -10 -11 -12 -13 -14 -15 -16 -17 -18 -19 -20 -21 -22 -23 -24 -25 -26 -27 -28 -29 -30 -31 -32 -33 -34 -35 -36 -37 -38 -39 -40 -41 -42 -43 -44 -45 -46 -47 -48 -49 -50 -51 -52 
reassign -1 a2jmidid
reassign -2 composite
reassign -3 ctpl
reassign -4 flowcanvas
reassign -5 geany
reassign -6 geany-plugins
reassign -7 gigolo
reassign -8 gmidimonitor
reassign -9 gnome-python
reassign -10 gnome-python-desktop
reassign -11 gtkimageview
reassign -12 guitarix
reassign -13 hamster-applet
reassign -14 hotssh
reassign -15 isoquery
reassign -16 jackd2
reassign -17 jalv
reassign -18 jcgui
reassign -19 kupfer
reassign -20 ladish
reassign -21 ldb
reassign -22 libdesktop-agnostic
reassign -23 lifeograph
reassign -24 lilv
reassign -25 lv2-extensions-good
reassign -26 lv2core
reassign -27 lv2fil
reassign -28 mda-lv2
reassign -29 mgen
reassign -30 minidjvu
reassign -31 nodejs
reassign -32 ns3
reassign -33 openchange
reassign -34 patchage
reassign -35 pino
reassign -36 radare
reassign -37 raul
reassign -38 samba
reassign -39 samba4
reassign -40 serd
reassign -41 showq
reassign -42 slv2
reassign -43 sord
reassign -44 suil
reassign -45 supercollider
reassign -46 sushi
reassign -47 talloc
reassign -48 tdb
reassign -49 tevent
reassign -50 xiphos
reassign -51 xmms2
reassign -52 zyn
thanks

Hi!

> IMHO this is not acceptable because there are no tools included or
> commandline switches offered with waf (in postler and midori) to
> conveniently unpack and repack these part for a.) inspection or b.)
> modification, which are required for packages in Debian main.

A package in NEW brought this matter to our attention, and after
discussing the issue within the FTP Team, we came to the conclusion that
the submitter of this bug report is correct: packages using waf in this
form do not ship all sources in their prefered form of modification¹.

While the letters of DFSG#2 and the Debian Policy could be fullfilled by
shipping waf in extracted form in the source packages, we would really
love to see the packages moving to a saner build system.

A quick tutorial on how to unpack waf to fulfil our requirements can be
found here: http://wiki.debian.org/UnpackWaf

Best regards,
  Alexander
  for the FTP Team

1: Yes, that phrase originates from the GPL, nevertheless Debian uses it as definiton of "source".





Bug 645190 cloned as bugs 654462, 654463, 654464, 654465, 654466, 654467, 654468, 654469, 654470, 654471, 654472, 654473, 654474, 654475, 654476, 654477, 654478, 654479, 654480, 654481, 654482, 654483, 654484, 654485, 654486, 654487, 654488, 654489, 654490, 654491, 654492, 654493, 654494, 654495, 654496, 654497, 654498, 654499, 654500, 654501, 654502, 654503, 654504, 654505, 654506, 654507, 654508, 654509, 654510, 654511, 654512, 654513. Request was from Alexander Reichle-Schmehl <tolimar@debian.org> to control@bugs.debian.org. (Tue, 03 Jan 2012 21:30:13 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Tue, 03 Jan 2012 21:42:08 GMT) Full text and rfc822 format available.

Acknowledgement sent to Yves-Alexis Perez <corsac@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Tue, 03 Jan 2012 21:42:09 GMT) Full text and rfc822 format available.

Message #32 received at 645190@bugs.debian.org (full text, mbox):

From: Yves-Alexis Perez <corsac@debian.org>
To: Alexander Reichle-Schmehl <tolimar@debian.org>, 645191@bugs.debian.org
Cc: 645190@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: waf binary code not DFSG compliant
Date: Tue, 03 Jan 2012 22:40:28 +0100
[Message part 1 (text/plain, inline)]
On mar., 2012-01-03 at 22:12 +0100, Alexander Reichle-Schmehl wrote:
> user ftpmaster@debian.org
> reopen 645191
> 
> usertags 645190 + waf-unpack
> clone 645190 -1 -2 -3 -4 -5 -6 -7 -8 -9 -10 -11 -12 -13 -14 -15 -16 -17 -18 -19 -20 -21 -22 -23 -24 -25 -26 -27 -28 -29 -30 -31 -32 -33 -34 -35 -36 -37 -38 -39 -40 -41 -42 -43 -44 -45 -46 -47 -48 -49 -50 -51 -52 
> reassign -1 a2jmidid
> reassign -2 composite
> reassign -3 ctpl
> reassign -4 flowcanvas
> reassign -5 geany
> reassign -6 geany-plugins
> reassign -7 gigolo
> reassign -8 gmidimonitor
> reassign -9 gnome-python
> reassign -10 gnome-python-desktop
> reassign -11 gtkimageview
> reassign -12 guitarix
> reassign -13 hamster-applet
> reassign -14 hotssh
> reassign -15 isoquery
> reassign -16 jackd2
> reassign -17 jalv
> reassign -18 jcgui
> reassign -19 kupfer
> reassign -20 ladish
> reassign -21 ldb
> reassign -22 libdesktop-agnostic
> reassign -23 lifeograph
> reassign -24 lilv
> reassign -25 lv2-extensions-good
> reassign -26 lv2core
> reassign -27 lv2fil
> reassign -28 mda-lv2
> reassign -29 mgen
> reassign -30 minidjvu
> reassign -31 nodejs
> reassign -32 ns3
> reassign -33 openchange
> reassign -34 patchage
> reassign -35 pino
> reassign -36 radare
> reassign -37 raul
> reassign -38 samba
> reassign -39 samba4
> reassign -40 serd
> reassign -41 showq
> reassign -42 slv2
> reassign -43 sord
> reassign -44 suil
> reassign -45 supercollider
> reassign -46 sushi
> reassign -47 talloc
> reassign -48 tdb
> reassign -49 tevent
> reassign -50 xiphos
> reassign -51 xmms2
> reassign -52 zyn
> thanks
> 
> Hi!
> 
> > IMHO this is not acceptable because there are no tools included or
> > commandline switches offered with waf (in postler and midori) to
> > conveniently unpack and repack these part for a.) inspection or b.)
> > modification, which are required for packages in Debian main.
> 
> A package in NEW brought this matter to our attention, and after
> discussing the issue within the FTP Team, we came to the conclusion that
> the submitter of this bug report is correct: packages using waf in this
> form do not ship all sources in their prefered form of modification¹.
> 
> While the letters of DFSG#2 and the Debian Policy could be fullfilled by
> shipping waf in extracted form in the source packages, we would really
> love to see the packages moving to a saner build system.
> 
> A quick tutorial on how to unpack waf to fulfil our requirements can be
> found here: http://wiki.debian.org/UnpackWaf
> 
> Best regards,
>   Alexander
>   for the FTP Team
> 
> 1: Yes, that phrase originates from the GPL, nevertheless Debian uses it as definiton of "source".
> 
That still looks to me like a waste of time. waf is a pain to work with,
and the bzip2 part is not really the worse part (technically speaking).

Diverting from upstream (waf as well as the package using it) already
proved painful, so I think the easiest solution would be to just stop
shipping those packages, sadly
-- 
Yves-Alexis
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Wed, 04 Jan 2012 09:00:13 GMT) Full text and rfc822 format available.

Acknowledgement sent to Yves-Alexis Perez <corsac@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Wed, 04 Jan 2012 09:00:20 GMT) Full text and rfc822 format available.

Message #37 received at 645190@bugs.debian.org (full text, mbox):

From: Yves-Alexis Perez <corsac@debian.org>
To: Alexander Reichle-Schmehl <tolimar@debian.org>, 645191@bugs.debian.org
Cc: 645190@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: waf binary code not DFSG compliant
Date: Wed, 04 Jan 2012 09:58:55 +0100
[Message part 1 (text/plain, inline)]
On mar., 2012-01-03 at 22:12 +0100, Alexander Reichle-Schmehl wrote:
> A package in NEW brought this matter to our attention, and after
> discussing the issue within the FTP Team, we came to the conclusion that
> the submitter of this bug report is correct: packages using waf in this
> form do not ship all sources in their prefered form of modification¹.

And out of curiosity, how is that different from tarball-in-tarball (or
even just tarballs, fwiw) or sources packages containing compressed
data?
> 
> While the letters of DFSG#2 and the Debian Policy could be fullfilled by
> shipping waf in extracted form in the source packages, we would really
> love to see the packages moving to a saner build system.
> 
> A quick tutorial on how to unpack waf to fulfil our requirements can be
> found here: http://wiki.debian.org/UnpackWaf
> 


-- 
Yves-Alexis
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Wed, 04 Jan 2012 09:12:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Enrico Tröger <enrico.troeger@uvena.de>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Wed, 04 Jan 2012 09:12:06 GMT) Full text and rfc822 format available.

Message #42 received at 645190@bugs.debian.org (full text, mbox):

From: Enrico Tröger <enrico.troeger@uvena.de>
To: Yves-Alexis Perez <corsac@debian.org>
Cc: Alexander Reichle-Schmehl <tolimar@debian.org>, 645191@bugs.debian.org, 645190@bugs.debian.org
Subject: Re: Bug#645191: waf binary code not DFSG compliant
Date: Wed, 4 Jan 2012 09:07:49 +0000
On Tue, 03 Jan 2012 22:40:28 +0100, Yves-Alexis wrote:

Hi,

/me is one of the upstream developers of Geany and Gigolo also using
Waf.


>> A quick tutorial on how to unpack waf to fulfil our requirements can
>> be found here: http://wiki.debian.org/UnpackWaf
>> 
>> Best regards,
>>   Alexander
>>   for the FTP Team
>> 
>> 1: Yes, that phrase originates from the GPL, nevertheless Debian
>> uses it as definiton of "source".
>> 
>That still looks to me like a waste of time. waf is a pain to work
>with, and the bzip2 part is not really the worse part (technically
>speaking).

Yves-Alexis, again the pointless discussion about Waf?
Me and some other developers consider Waf as a way saner build system
than autotools. Other people do not.

Anyway, I don't feel like discussing this again and again.


>Diverting from upstream (waf as well as the package using it) already
>proved painful, so I think the easiest solution would be to just stop
>shipping those packages, sadly

That'd be really, really sadly.

Did anyone already brought this issue to Waf upstream to see whether
they would like to help on this issue, e.g. by adding a command line
switch to unpack and repack the waf binary?

Regards,
Enrico

-- 
Not sent from my smartphone.




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Wed, 04 Jan 2012 09:15:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Yves-Alexis Perez <corsac@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Wed, 04 Jan 2012 09:15:08 GMT) Full text and rfc822 format available.

Message #47 received at 645190@bugs.debian.org (full text, mbox):

From: Yves-Alexis Perez <corsac@debian.org>
To: Enrico Tröger <enrico.troeger@uvena.de>
Cc: Alexander Reichle-Schmehl <tolimar@debian.org>, 645191@bugs.debian.org, 645190@bugs.debian.org
Subject: Re: Bug#645191: waf binary code not DFSG compliant
Date: Wed, 04 Jan 2012 10:12:23 +0100
[Message part 1 (text/plain, inline)]
On mer., 2012-01-04 at 09:07 +0000, Enrico Tröger wrote:
> On Tue, 03 Jan 2012 22:40:28 +0100, Yves-Alexis wrote:
> 
> Hi,
> 
> /me is one of the upstream developers of Geany and Gigolo also using
> Waf.
> 
> 
> >> A quick tutorial on how to unpack waf to fulfil our requirements can
> >> be found here: http://wiki.debian.org/UnpackWaf
> >> 
> >> Best regards,
> >>   Alexander
> >>   for the FTP Team
> >> 
> >> 1: Yes, that phrase originates from the GPL, nevertheless Debian
> >> uses it as definiton of "source".
> >> 
> >That still looks to me like a waste of time. waf is a pain to work
> >with, and the bzip2 part is not really the worse part (technically
> >speaking).
> 
> Yves-Alexis, again the pointless discussion about Waf?

Well, I wasn't the one opening it, I'm tired too :)

> Me and some other developers consider Waf as a way saner build system
> than autotools. Other people do not.

Agreed.

> 
> Anyway, I don't feel like discussing this again and again.

Agreed :/
> 
> 
> >Diverting from upstream (waf as well as the package using it) already
> >proved painful, so I think the easiest solution would be to just stop
> >shipping those packages, sadly
> 
> That'd be really, really sadly.

Agreed, again.
> 
> Did anyone already brought this issue to Waf upstream to see whether
> they would like to help on this issue, e.g. by adding a command line
> switch to unpack and repack the waf binary?

Well, last time something was ported to waf upstream, it wasn't exactly
nicely welcomed (see
http://article.gmane.org/gmane.linux.debian.devel.general/149572)

And anyway, the unpack part is done automagically at build time already,
which is not satisfying for ftp-masters apparently: aiui they want the
unpack to be done at packaging time, and so to have a repack done at
every release (Alexander, feel free to correct me if I'm wrong, I might
have misinterpreted the wiki page).

Regards,
-- 
Yves-Alexis
[signature.asc (application/pgp-signature, inline)]

Added tag(s) squeeze-ignore. Request was from Alexander Reichle-Schmehl <tolimar@debian.org> to control@bugs.debian.org. (Tue, 07 Feb 2012 15:36:02 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Fri, 09 Mar 2012 20:45:12 GMT) Full text and rfc822 format available.

Message #52 received at 645190@bugs.debian.org (full text, mbox):

From: Carsten Hey <carsten@debian.org>
To: 645190@bugs.debian.org, 645191@bugs.debian.org, 654468@bugs.debian.org
Subject: Re: update on waf binary data
Date: Fri, 9 Mar 2012 21:42:07 +0100
[ I'm sending this to the two bugs Rhonda sent the mail I reply to and
  an additional bug tagged wontfix to avoid spamming all affected bugs ]

* Gerfried Fuchs [2011-10-13 15:12 +0200]:
>  it seems that the line 161 is actually a tar.bz2 file that gets
> extracted and then used.  Though, first there is some substitution of \r
> and \n characters so that the "file" could go on one line.
>
>  IMHO this is not acceptable because there are no tools included or
> commandline switches offered with waf (in postler and midori) to
> conveniently unpack and repack these part for a.) inspection or b.)
> modification, which are required for packages in Debian main.

Exactly regenerating tarballs is, similar to regenerating man pages that
contain a date, possible but not that easy.  Ignoring this non-relevant
difference of regenerated tarballs, I was able to regenerate an exact
copy of the waf script:

  $ rm -rf midori-0.4.3
  $ dpkg-source -x midori_0.4.3-1.dsc >/dev/null 2>&1
  $ cd midori-0.4.3
  $ sed < waf -e '1,/^#==>$/ d' -e '/^#<==$/ d' | tr -d '\n' | sed -e 's/.//' -e 's/#[*]/\n/g' -e 's/#%/\r/g' > waf.orig.tar.bz2
  $ tar tjf waf.orig.tar.bz2
  wafadmin/Logs.py
  wafadmin/Constants.py
  wafadmin/py3kfixes.py
  ...
  $ (sed -n < waf -e '1,/^#==>$/ p'; echo REPLACED BY ENCODED TAR.BZ2; sed -n < waf -e '/^#<==$/ p') > debian/waf.tmpl
  $ wc -c debian/waf.tmpl
  4097 debian/waf.tmpl
  $ (sed -n < debian/waf.tmpl -e '1,/^#==>$/ p'; printf '#'; perl -pe < waf.orig.tar.bz2 's/\n/#*/g; s/\r/#%/g;'; echo; sed -n < debian/waf.tmpl  -e '/^#<==$/ p') > waf.regen
  $ md5sum waf waf.regen
  eca3f4738d809c42cecad2e9ec39a1cc  waf
  eca3f4738d809c42cecad2e9ec39a1cc  waf.regen

I assume that it should be possible to develop a DFSG conforming
solution based on above hack.  The requirements to sed extend POSIX's
specifications, but given that it could be replaced with perl and we use
GNU sed in Debian this shouldn't be a problem.

Carsten




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Fri, 09 Mar 2012 20:54:04 GMT) Full text and rfc822 format available.

Message #55 received at 645190@bugs.debian.org (full text, mbox):

From: Carsten Hey <carsten@debian.org>
To: 645190@bugs.debian.org, 645191@bugs.debian.org, 654468@bugs.debian.org
Subject: Re: update on waf binary data
Date: Fri, 9 Mar 2012 21:51:19 +0100
* Carsten Hey [2012-03-09 21:42 +0100]:
>   $ (sed -n < waf -e '1,/^#==>$/ p'; echo REPLACED BY ENCODED TAR.BZ2; sed -n < waf -e '/^#<==$/ p') > debian/waf.tmpl

Instead of '/^#<==$/ p' it should be '/^#<==$/,$ p' (this occurs
multiple times all around).  Since the matched line is also the last one
in midori's waf script, it currently does not make a difference for
midori.




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Sat, 10 Mar 2012 15:33:08 GMT) Full text and rfc822 format available.

Acknowledgement sent to Yves-Alexis Perez <corsac@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Sat, 10 Mar 2012 15:33:08 GMT) Full text and rfc822 format available.

Message #60 received at 645190@bugs.debian.org (full text, mbox):

From: Yves-Alexis Perez <corsac@debian.org>
To: Carsten Hey <carsten@debian.org>, 645191@bugs.debian.org
Cc: 645190@bugs.debian.org, 654468@bugs.debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Sat, 10 Mar 2012 16:30:59 +0100
[Message part 1 (text/plain, inline)]
On ven., 2012-03-09 at 21:42 +0100, Carsten Hey wrote:
> [ I'm sending this to the two bugs Rhonda sent the mail I reply to and
>   an additional bug tagged wontfix to avoid spamming all affected bugs ]

Can't this be helpful to others?
> 
> * Gerfried Fuchs [2011-10-13 15:12 +0200]:
> >  it seems that the line 161 is actually a tar.bz2 file that gets
> > extracted and then used.  Though, first there is some substitution of \r
> > and \n characters so that the "file" could go on one line.
> >
> >  IMHO this is not acceptable because there are no tools included or
> > commandline switches offered with waf (in postler and midori) to
> > conveniently unpack and repack these part for a.) inspection or b.)
> > modification, which are required for packages in Debian main.
> 
> Exactly regenerating tarballs is, similar to regenerating man pages that
> contain a date, possible but not that easy.  Ignoring this non-relevant
> difference of regenerated tarballs, I was able to regenerate an exact
> copy of the waf script:
> 
>   $ rm -rf midori-0.4.3
>   $ dpkg-source -x midori_0.4.3-1.dsc >/dev/null 2>&1
>   $ cd midori-0.4.3
>   $ sed < waf -e '1,/^#==>$/ d' -e '/^#<==$/ d' | tr -d '\n' | sed -e 's/.//' -e 's/#[*]/\n/g' -e 's/#%/\r/g' > waf.orig.tar.bz2
>   $ tar tjf waf.orig.tar.bz2
>   wafadmin/Logs.py
>   wafadmin/Constants.py
>   wafadmin/py3kfixes.py
>   ...
>   $ (sed -n < waf -e '1,/^#==>$/ p'; echo REPLACED BY ENCODED TAR.BZ2; sed -n < waf -e '/^#<==$/ p') > debian/waf.tmpl
>   $ wc -c debian/waf.tmpl
>   4097 debian/waf.tmpl
>   $ (sed -n < debian/waf.tmpl -e '1,/^#==>$/ p'; printf '#'; perl -pe < waf.orig.tar.bz2 's/\n/#*/g; s/\r/#%/g;'; echo; sed -n < debian/waf.tmpl  -e '/^#<==$/ p') > waf.regen
>   $ md5sum waf waf.regen
>   eca3f4738d809c42cecad2e9ec39a1cc  waf
>   eca3f4738d809c42cecad2e9ec39a1cc  waf.regen
> 
> I assume that it should be possible to develop a DFSG conforming
> solution based on above hack.  The requirements to sed extend POSIX's
> specifications, but given that it could be replaced with perl and we use
> GNU sed in Debian this shouldn't be a problem.
> 
I have to admit I'm not exactly sure what your point is. From where does
the waf command you're using come from?

Regards,
-- 
Yves-Alexis
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Sat, 10 Mar 2012 17:45:03 GMT) Full text and rfc822 format available.

Message #63 received at 645190@bugs.debian.org (full text, mbox):

From: Carsten Hey <carsten@debian.org>
To: Yves-Alexis Perez <corsac@debian.org>
Cc: 645191@bugs.debian.org, 645190@bugs.debian.org, 654468@bugs.debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Sat, 10 Mar 2012 18:43:56 +0100
* Yves-Alexis Perez [2012-03-10 16:30 +0100]:
> On ven., 2012-03-09 at 21:42 +0100, Carsten Hey wrote:
> > [ I'm sending this to the two bugs Rhonda sent the mail I reply to and
> >   an additional bug tagged wontfix to avoid spamming all affected bugs ]
>
> Can't this be helpful to others?

If anybody does not want to remove or patch every waf using package in
Debian, yes, it can be helpful.  A simple script that extracts and
repacks waf and that is documented in Debian.source is unlike an self
extracting python script using various modules something ftpmaster might
accept without shipping the extracted waf source code.  To write such
scripts all one needs to do is to put my hack in a shell script in clean
it up.

> >   $ dpkg-source -x midori_0.4.3-1.dsc >/dev/null 2>&1
> >   $ cd midori-0.4.3
>
> I have to admit I'm not exactly sure what your point is. From where does
> the waf command you're using come from?

It is shipped in the Debian package midori and my point was to show a proof
of concept of an possible solution that would make both happy, ftpmaster
and the maintainers of waf using packages (two RC bugs marked as wontfix
is a good sign that some maintainers currently are not that happy about
the situation).


Regards
Carsten




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Sat, 10 Mar 2012 18:15:03 GMT) Full text and rfc822 format available.

Message #66 received at 645190@bugs.debian.org (full text, mbox):

From: Carsten Hey <carsten@debian.org>
To: Yves-Alexis Perez <corsac@debian.org>, 645191@bugs.debian.org, 645190@bugs.debian.org, 654468@bugs.debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Sat, 10 Mar 2012 19:12:48 +0100
* Carsten Hey [2012-03-10 18:43 +0100]:
> * Yves-Alexis Perez [2012-03-10 16:30 +0100]:
> > I have to admit I'm not exactly sure what your point is. From where does
> > the waf command you're using come from?
>
> ...

Actually I was not using a waf command but instead well known tools
installed on every Debian system to extract the source code from a waf
script and then to regenerate this waf script from the source (that
could have been edited in the preferred form of modification in the
meantime).  tar xf and tar cf were missing though, but we all know how
tar works.

Carsten




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Sat, 10 Mar 2012 19:27:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Yves-Alexis Perez <corsac@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Sat, 10 Mar 2012 19:27:03 GMT) Full text and rfc822 format available.

Message #71 received at 645190@bugs.debian.org (full text, mbox):

From: Yves-Alexis Perez <corsac@debian.org>
To: Carsten Hey <carsten@debian.org>, 645191@bugs.debian.org
Cc: 645190@bugs.debian.org, 654468@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Sat, 10 Mar 2012 20:24:01 +0100
[Message part 1 (text/plain, inline)]
On sam., 2012-03-10 at 19:12 +0100, Carsten Hey wrote:
> * Carsten Hey [2012-03-10 18:43 +0100]:
> > * Yves-Alexis Perez [2012-03-10 16:30 +0100]:
> > > I have to admit I'm not exactly sure what your point is. From where does
> > > the waf command you're using come from?
> >
> > ...
> 
> Actually I was not using a waf command but instead well known tools
> installed on every Debian system to extract the source code from a waf
> script and then to regenerate this waf script from the source (that
> could have been edited in the preferred form of modification in the
> meantime).  tar xf and tar cf were missing though, but we all know how
> tar works.
> 
Yeah, good point. ftpmasters, any comment on this? (see few previous
mail in any of the CC: bugs).

Regards,
-- 
Yves-Alexis
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Thu, 15 Mar 2012 10:15:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Yves-Alexis Perez <corsac@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Thu, 15 Mar 2012 10:15:43 GMT) Full text and rfc822 format available.

Message #76 received at 645190@bugs.debian.org (full text, mbox):

From: Yves-Alexis Perez <corsac@debian.org>
To: 645191@bugs.debian.org
Cc: Carsten Hey <carsten@debian.org>, 645190@bugs.debian.org, 654468@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Thu, 15 Mar 2012 11:11:54 +0100
[Message part 1 (text/plain, inline)]
On sam., 2012-03-10 at 20:24 +0100, Yves-Alexis Perez wrote:
> On sam., 2012-03-10 at 19:12 +0100, Carsten Hey wrote:
> > * Carsten Hey [2012-03-10 18:43 +0100]:
> > > * Yves-Alexis Perez [2012-03-10 16:30 +0100]:
> > > > I have to admit I'm not exactly sure what your point is. From where does
> > > > the waf command you're using come from?
> > >
> > > ...
> > 
> > Actually I was not using a waf command but instead well known tools
> > installed on every Debian system to extract the source code from a waf
> > script and then to regenerate this waf script from the source (that
> > could have been edited in the preferred form of modification in the
> > meantime).  tar xf and tar cf were missing though, but we all know how
> > tar works.
> > 
> Yeah, good point. ftpmasters, any comment on this? (see few previous
> mail in any of the CC: bugs).
> 

Would something like that do:

http://anonscm.debian.org/gitweb/?p=collab-maint/midori.git;a=commitdiff;h=23b89cca6b96266eea166b30ac8d1591ffbf7b2f

Regards,
-- 
Yves-Alexis
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Thu, 15 Mar 2012 19:33:09 GMT) Full text and rfc822 format available.

Message #79 received at 645190@bugs.debian.org (full text, mbox):

From: Carsten Hey <carsten@debian.org>
To: Yves-Alexis Perez <corsac@debian.org>
Cc: 645191@bugs.debian.org, 645190@bugs.debian.org, 654468@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Thu, 15 Mar 2012 20:32:23 +0100
* Yves-Alexis Perez [2012-03-15 11:11 +0100]:
> On sam., 2012-03-10 at 20:24 +0100, Yves-Alexis Perez wrote:
> > On sam., 2012-03-10 at 19:12 +0100, Carsten Hey wrote:
> > > * Carsten Hey [2012-03-10 18:43 +0100]:
> > > Actually I was not using a waf command but instead well known tools
> > > installed on every Debian system to extract the source code from a waf
> > > script and then to regenerate this waf script from the source (that
> > > could have been edited in the preferred form of modification in the
> > > meantime).  tar xf and tar cf were missing though, but we all know how
> > > tar works.
> >
> > Yeah, good point. ftpmasters, any comment on this? (see few previous
> > mail in any of the CC: bugs).
>
> Would something like that do:
>
> http://anonscm.debian.org/gitweb/?p=collab-maint/midori.git;a=commitdiff;h=23b89cca6b96266eea166b30ac8d1591ffbf7b2f

I don't assume this diff would make ftpmaster entirely happy.  I'll send
a NMU diff against midori/sid that adapts the package in a way I hope to
be acceptable by ftpmaster, but I'm not sure if I get to do this this
evening.

I hope to get a comment from ftpmaster after sending the NMU diff.


Regards
Carsten




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Thu, 15 Mar 2012 20:30:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Yves-Alexis Perez <corsac@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Thu, 15 Mar 2012 20:30:02 GMT) Full text and rfc822 format available.

Message #84 received at 645190@bugs.debian.org (full text, mbox):

From: Yves-Alexis Perez <corsac@debian.org>
To: Carsten Hey <carsten@debian.org>, 645191@bugs.debian.org
Cc: 645190@bugs.debian.org, 654468@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Thu, 15 Mar 2012 21:26:54 +0100
[Message part 1 (text/plain, inline)]
On jeu., 2012-03-15 at 20:32 +0100, Carsten Hey wrote:
> > http://anonscm.debian.org/gitweb/?p=collab-maint/midori.git;a=commitdiff;h=23b89cca6b96266eea166b30ac8d1591ffbf7b2f
> 
> I don't assume this diff would make ftpmaster entirely happy.  I'll send
> a NMU diff against midori/sid that adapts the package in a way I hope to
> be acceptable by ftpmaster, but I'm not sure if I get to do this this
> evening.
> 
> I hope to get a comment from ftpmaster after sending the NMU diff.
> 
To be honest, I didn't even wanted to spend any time on this, as I
consider the decision bad. I appreciated your help, so I took few
minutes to cook a debian/rules including your work.

Now, I don't know what improvements you're able to make on this, but
since ftpmasters don't seem very much interested in ways to fix this in
ways which don't divert too much from upstreams, I really don't think
I'll lose more time on this.

Regards,
-- 
Yves-Alexis
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Sat, 17 Mar 2012 01:48:02 GMT) Full text and rfc822 format available.

Message #87 received at 645190@bugs.debian.org (full text, mbox):

From: Carsten Hey <carsten@debian.org>
To: 645191@bugs.debian.org, 645190@bugs.debian.org, 654468@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Sat, 17 Mar 2012 02:45:04 +0100
waf scripts are not cleanly divided into python and data, but instead
the python part contains also two two byte sequences (found using brute
force whilst building the waf script).  My original plan was to ship two
scripts debian/waf-unpack and debian/waf-repack to provide an easy way
to edit the waf sources and document this in README.source. Due to the
above mentioned mix of header information and python source code this
can not be done in a clean way, so there so there is nothing to review
for ftpmaster.

http://bugs.debian.org/660193 (search for the string waf) contains
snippets, based on what Tolimar pointed to in his mail, you just need to
paste into the midori package and some additional notes.  The remaining
part is IMHO to document this in README.source.  One thing I forgot to
mention in my mail to #660193 is that the reason to remove the blob from
the used waf script is to ensure that the unpacked waf source is used.

If requested I could provide a less hackish script to extract the
tarball embedded in a waf script.  It is finished, but it is probably
useless because there is no reliable way to put a new tarball into a waf
script without using ugly hacks or being waf itself.

* Yves-Alexis Perez [2012-03-15 21:26 +0100]:
> To be honest, I didn't even wanted to spend any time on this, as I
> consider the decision bad.

If a security update would require any changes in the packages build
system, using waf the way upstream intended it to be used would cause
the security team a lot of work and reviewing even simple changes
related to the build system would be a mess to review by the release
team during freeze.  Some .jar files also contain their source, should
we in your opinion start to just ship them instead of rebuilding them?
(this was of course a rhetorical question)


Regards
Carsten




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Sat, 17 Mar 2012 08:39:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Yves-Alexis Perez <corsac@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Sat, 17 Mar 2012 08:39:03 GMT) Full text and rfc822 format available.

Message #92 received at 645190@bugs.debian.org (full text, mbox):

From: Yves-Alexis Perez <corsac@debian.org>
To: Carsten Hey <carsten@debian.org>, 645191@bugs.debian.org
Cc: 645190@bugs.debian.org, 654468@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Sat, 17 Mar 2012 09:36:30 +0100
[Message part 1 (text/plain, inline)]
On sam., 2012-03-17 at 02:45 +0100, Carsten Hey wrote:
> waf scripts are not cleanly divided into python and data, but instead
> the python part contains also two two byte sequences (found using brute
> force whilst building the waf script).  My original plan was to ship two
> scripts debian/waf-unpack and debian/waf-repack to provide an easy way
> to edit the waf sources and document this in README.source. Due to the
> above mentioned mix of header information and python source code this
> can not be done in a clean way, so there so there is nothing to review
> for ftpmaster.

I don't completely understand this. What are those two bytes sequences
you bruteforced? Afaict you don't call waf in your snippets (and I
specifically asked you about that).
> 
> http://bugs.debian.org/660193 (search for the string waf) contains
> snippets, based on what Tolimar pointed to in his mail, you just need to
> paste into the midori package and some additional notes.  The remaining
> part is IMHO to document this in README.source.  One thing I forgot to
> mention in my mail to #660193 is that the reason to remove the blob from
> the used waf script is to ensure that the unpacked waf source is used.

Well, in midori diff, I repack and ensure the new one and the old ones
are the same to be sure I don't do anything bad. Now indeed it could be
split in two parts, one run by maintainer, which would then hack in the
waf sources themeselves, and one at build time, which would pick the
extracted sources and make a new waf script.
> 
> If requested I could provide a less hackish script to extract the
> tarball embedded in a waf script.  It is finished, but it is probably
> useless because there is no reliable way to put a new tarball into a waf
> script without using ugly hacks or being waf itself.

I don't understand that either.
> 
> * Yves-Alexis Perez [2012-03-15 21:26 +0100]:
> > To be honest, I didn't even wanted to spend any time on this, as I
> > consider the decision bad.
> 
> If a security update would require any changes in the packages build
> system, using waf the way upstream intended it to be used would cause
> the security team a lot of work and reviewing even simple changes
> related to the build system would be a mess to review by the release
> team during freeze.  Some .jar files also contain their source, should
> we in your opinion start to just ship them instead of rebuilding them?
> (this was of course a rhetorical question)

I'm a security team member so I'm well aware of that. I'm not defending
waf, I *don't* like it, and I already told upstreams about that. Now
it's their choice. And changing the way waf is used at built time is not
supported and might fail in bad ways too, so it's not really helpful to
do things *against* what advised by waf upstream.

And I still consider the decision bad because the source *is* there and
is tunable, even though it's not the easiest way in the world. But
upstream(s) made a choice here, we can disagree (and I do) but at the
end of the day, unless you want to fork, there's not much you can do.

Now, I really think I'm losing my time and yours on those aspects, so
lets keep it on the technical side, because I do find this helpful (and
I'm puzzled about the few questions I asked).

Regards,
-- 
Yves-Alexis
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Fri, 23 Mar 2012 22:42:03 GMT) Full text and rfc822 format available.

Message #95 received at 645190@bugs.debian.org (full text, mbox):

From: Carsten Hey <carsten@debian.org>
To: Yves-Alexis Perez <corsac@debian.org>
Cc: 645191@bugs.debian.org, 645190@bugs.debian.org, 654468@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Fri, 23 Mar 2012 23:39:22 +0100
I think we should drop ftpmaster from CC in further mails.

* Yves-Alexis Perez [2012-03-17 09:36 +0100]:
> On sam., 2012-03-17 at 02:45 +0100, Carsten Hey wrote:
> > waf scripts are not cleanly divided into python and data, but instead
> > the python part contains also two two byte sequences (found using brute
> > force whilst building the waf script).  My original plan was to ship two
> > scripts debian/waf-unpack and debian/waf-repack to provide an easy way
> > to edit the waf sources and document this in README.source. Due to the
> > above mentioned mix of header information and python source code this
> > can not be done in a clean way, so there so there is nothing to review
> > for ftpmaster.
>
> I don't completely understand this. What are those two bytes sequences
> you bruteforced? Afaict you don't call waf in your snippets (and I
> specifically asked you about that).

The bug report mentioned that the waf script contains an embedded
.tar.bz2.  A short look at the script revealed that there is only one
line where it could be embedded and that it replaces a two byte sequence
with \n and an other one with \r.  Finding out that the line's first
character and the line's last do not belong to the .tar.bz2 wasn't that
hard.  All described can be done easily using sed and similar tools
- and the reverse can also easily be done using sed and similar tools.
This is what I did, I put the described in simple commands.  The idea
was to be able to extract the data from the waf script, modify it and
then embed it again into the waf script - just using simple standard
tools.

The above was based on the assumption that these two two byte sequences
are always the same because they can not occur in bzip2 compressed data
for whatever reason.  This assumption was wrong, waf creates the
.tar.bz2 and then tries all possible two byte sequences until if finds
some that do not occur in the .tar.bz2 and then sets the variables C1
and C2 in the python waf template to these values.  If we would do this
in sed and co. we would need to parse python, which doesn't seem to be
an option.

> > http://bugs.debian.org/660193 (search for the string waf) contains
> > snippets, based on what Tolimar pointed to in his mail, you just need to
> > paste into the midori package and some additional notes.  The remaining
> > part is IMHO to document this in README.source.  One thing I forgot to
> > mention in my mail to #660193 is that the reason to remove the blob from
> > the used waf script is to ensure that the unpacked waf source is used.
>
> Well, in midori diff, I repack and ensure the new one and the old ones
> are the same to be sure I don't do anything bad. Now indeed it could be
> split in two parts, one run by maintainer, which would then hack in the
> waf sources themeselves, and one at build time, which would pick the
> extracted sources and make a new waf script.

My point of all this was to provide an easy way to change the source
code, but this can't be accomplished.  You provided a way to extract the
source to be able to review it, but not to change it.

> And changing the way waf is used at built time is not supported and
> might fail in bad ways too, so it's not really helpful to do things
> *against* what advised by waf upstream.

Users might not be advised to use an extracted source, but "#devs use
$WAFDIR" (this is a comment in the waf script shipped in midori), so
using the extracted wafadmin directory isn't unsupported at all.
$WAFDIR is the directory in which the extracted wafadmin directory is
found.

> And I still consider the decision bad because the source *is* there and
> is tunable, even though it's not the easiest way in the world. But
> upstream(s) made a choice here, we can disagree (and I do) but at the
> end of the day, unless you want to fork, there's not much you can do.

Use an trivial build system for trivial projects and ship waf unpacked
for non-trivial packages is what we can do (midori is clearly
non-trivial).  Having an easy command to unpack and repack waf scripts
would have been great, but this is not possible unless we would adapt
the values of C1 and C2 in the waf script (and thus parsing python),
which would lead to an ugly hack.


Regards
Carsten




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Fri, 23 Mar 2012 22:45:05 GMT) Full text and rfc822 format available.

Message #98 received at 645190@bugs.debian.org (full text, mbox):

From: Carsten Hey <carsten@debian.org>
To: Yves-Alexis Perez <corsac@debian.org>, 645191@bugs.debian.org, 645190@bugs.debian.org, 654468@bugs.debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Fri, 23 Mar 2012 23:43:00 +0100
* Carsten Hey [2012-03-23 23:39 +0100]:
> Having an easy command to unpack and repack waf scripts
> would have been great, but this is not possible unless we would adapt
> the values of C1 and C2 in the waf script (and thus parsing python),
> which would lead to an ugly hack.

Alternatively, a template could be used to avoid parsing the waf script,
but this would require way more work to maintain than just shipping the
unpacked wafadmin directory.

Carsten




Added tag(s) pending and patch. Request was from Luca Falavigna <dktrkranz@debian.org> to control@bugs.debian.org. (Sun, 25 Mar 2012 11:13:25 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Sun, 25 Mar 2012 11:16:17 GMT) Full text and rfc822 format available.

Acknowledgement sent to Luca Falavigna <dktrkranz@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Sun, 25 Mar 2012 11:16:45 GMT) Full text and rfc822 format available.

Message #105 received at 645190@bugs.debian.org (full text, mbox):

From: Luca Falavigna <dktrkranz@debian.org>
To: 635423@bugs.debian.org, 645190@bugs.debian.org, 663300@bugs.debian.org
Subject: postler: diff for NMU version 0.1.1+dfsg-0.1
Date: Sun, 25 Mar 2012 12:38:12 +0200
[Message part 1 (text/plain, inline)]
tags 635423 + patch pending
tags 645190 + patch pending
tags 663300 + patch pending
thanks


Dear maintainer,

I've prepared an NMU for postler (versioned as 0.1.1+dfsg-0.1) and
uploaded it to DELAYED/2. Please feel free to tell me if I
should delay it longer.

Regards.
[postler-0.1.1+dfsg-0.1-nmu.diff (text/x-diff, attachment)]

Reply sent to Luca Falavigna <dktrkranz@debian.org>:
You have taken responsibility. (Tue, 27 Mar 2012 11:06:29 GMT) Full text and rfc822 format available.

Notification sent to Gerfried Fuchs <rhonda@debian.org>:
Bug acknowledged by developer. (Tue, 27 Mar 2012 11:06:36 GMT) Full text and rfc822 format available.

Message #110 received at 645190-close@bugs.debian.org (full text, mbox):

From: Luca Falavigna <dktrkranz@debian.org>
To: 645190-close@bugs.debian.org
Subject: Bug#645190: fixed in postler 0.1.1+dfsg-0.1
Date: Tue, 27 Mar 2012 11:02:54 +0000
Source: postler
Source-Version: 0.1.1+dfsg-0.1

We believe that the bug you reported is fixed in the latest version of
postler, which is due to be installed in the Debian FTP archive:

postler_0.1.1+dfsg-0.1.debian.tar.gz
  to main/p/postler/postler_0.1.1+dfsg-0.1.debian.tar.gz
postler_0.1.1+dfsg-0.1.dsc
  to main/p/postler/postler_0.1.1+dfsg-0.1.dsc
postler_0.1.1+dfsg-0.1_amd64.deb
  to main/p/postler/postler_0.1.1+dfsg-0.1_amd64.deb
postler_0.1.1+dfsg.orig.tar.bz2
  to main/p/postler/postler_0.1.1+dfsg.orig.tar.bz2



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 645190@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Luca Falavigna <dktrkranz@debian.org> (supplier of updated postler package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Sun, 25 Mar 2012 12:29:45 +0200
Source: postler
Binary: postler
Architecture: source amd64
Version: 0.1.1+dfsg-0.1
Distribution: unstable
Urgency: low
Maintainer: Devid Antonio Filoni <d.filoni@ubuntu.com>
Changed-By: Luca Falavigna <dktrkranz@debian.org>
Description: 
 postler    - desktop mail client built in vala
Closes: 635423 645190 663300
Changes: 
 postler (0.1.1+dfsg-0.1) unstable; urgency=low
 .
   * Non-maintainer upload.
   * Upstream tarball repacked to provide unpacked waf (Closes: #645190).
   * debian/patches/vala.patch:
     - Cherry-pick patches from upstream to compile with latest vala.
   * debian/control:
     - Build-depend on libwebkitgtk-dev (Closes: #635423).
     - Build-depend on unversioned valac compiler (Closes: #663300)
Checksums-Sha1: 
 d1f9dbf7d937663574cdd964b54a3469a0c88868 1999 postler_0.1.1+dfsg-0.1.dsc
 550d5738d448186dfade8ef7afcc329af2348e12 326142 postler_0.1.1+dfsg.orig.tar.bz2
 c1e1a640f9cab73179f9c8672a5e13e47fdb0312 5321 postler_0.1.1+dfsg-0.1.debian.tar.gz
 a0afd121190ffbcdddaa132b87e4dd266b24bf68 357824 postler_0.1.1+dfsg-0.1_amd64.deb
Checksums-Sha256: 
 2c0c2ec6bdf65abf968afee1d9600e8b2dc58703af19010fd80109c53727a3c2 1999 postler_0.1.1+dfsg-0.1.dsc
 36d17c4cb1728c20c668e138208973a004cf94e8765e95c282d274cbb26f5b3d 326142 postler_0.1.1+dfsg.orig.tar.bz2
 46d3c3e76f8700bedc5ed7b9ac0e6d0288580a35b294cfd2f41d62bdf4d172ff 5321 postler_0.1.1+dfsg-0.1.debian.tar.gz
 6620043c9d81338e023884cafa959391f1058fb2c4f3229ec93e0bfe5e2fc83a 357824 postler_0.1.1+dfsg-0.1_amd64.deb
Files: 
 9f781badb50089a2c1d2488da2694e3f 1999 mail optional postler_0.1.1+dfsg-0.1.dsc
 b3686989905fcdc5a25a7b563ea8d71b 326142 mail optional postler_0.1.1+dfsg.orig.tar.bz2
 22f85970c1e7f23ca8041b3f5bb6e850 5321 mail optional postler_0.1.1+dfsg-0.1.debian.tar.gz
 c157e66ad137f1e98e1da008a70efd18 357824 mail optional postler_0.1.1+dfsg-0.1_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBCAAGBQJPbvVBAAoJEEkIatPr4vMfOGAP/2Ur2HYZP5BXbIqLDq/LeI8S
x73QvkdrSCleQSZR2AYlhB3iIeRhXlt1svBM7L7/TnMWtvQ3xHS9ucN/+O0BAXii
rWR8LIc1M5RP16gyQfXJHDbffEwqJh7tPu9Q+KuacbCIRdc8pGfSJRO8XSe/Sm99
OCy3qhuJ+40A0JPfW5SYYL9cpcuBMDf/S3JukbUjw2CBVXs7t0OUhvAbMGwNDMa/
4e0+MadTSHsocwtM4BQ5eomF8L9qJUkEB4PgMLtmngIblIcqi11L7DFOnrX0kEp+
l+B879NICMjJKnMy/c962r9CIf/BceSuMGhtJX9Km16+OHMcRpG+NTA1vSQPYYnh
E2JdWy842zpLPyIUxEDxnsws5GmHYzR9EyCWOIbW6ZP9XDfBd1bQk4yXrMNEgM6A
hJX+midUVs+0p3XsyzRsxWbjTzERN1xMeKSjkHZCwO/zEjoZc+nfUMSH/z4FxjVT
gx3yjM6108FoRk6UjRiiZVwNif5h5YSIaqAV2akNJR7mailU4/mJ73JfN4xYZMGX
Ubody0xfGqfw1w1ePl6izgXOlzjX1MJlmvU8YtMoAukYYdfIcEiO+MUVbvXNyY9n
W/Ovik4grqhC+VCMMJdfoETdijWEQpkHD7MQr3IyOiw7I1CSJ9RHHWUht+sqdaOu
BVOoodGXJ8mwB7hMkh6V
=YooJ
-----END PGP SIGNATURE-----





Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Sun, 15 Apr 2012 07:21:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Yves-Alexis Perez <corsac@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Sun, 15 Apr 2012 07:21:02 GMT) Full text and rfc822 format available.

Message #115 received at 645190@bugs.debian.org (full text, mbox):

From: Yves-Alexis Perez <corsac@debian.org>
To: Carsten Hey <carsten@debian.org>, 645191@bugs.debian.org
Cc: 645190@bugs.debian.org, 654468@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Sun, 15 Apr 2012 09:18:00 +0200
[Message part 1 (text/plain, inline)]
On ven., 2012-03-23 at 23:39 +0100, Carsten Hey wrote:
> I think we should drop ftpmaster from CC in further mails.

Maybe, since they don't seem to care about this. But considering they
decided to forbid waf, I think they should at least pay attention on how
people is trying to cope with their (imho bad) decision.
> 

> The above was based on the assumption that these two two byte sequences
> are always the same because they can not occur in bzip2 compressed data
> for whatever reason.  This assumption was wrong, waf creates the
> .tar.bz2 and then tries all possible two byte sequences until if finds
> some that do not occur in the .tar.bz2 and then sets the variables C1
> and C2 in the python waf template to these values.  If we would do this
> in sed and co. we would need to parse python, which doesn't seem to be
> an option.

Well, parsing python might not be an option, but what about:

egrep -a "^C[1|2]='..'" waf
C1='#*'
C2='#%'

> 
> > > http://bugs.debian.org/660193 (search for the string waf) contains
> > > snippets, based on what Tolimar pointed to in his mail, you just need to
> > > paste into the midori package and some additional notes.  The remaining
> > > part is IMHO to document this in README.source.  One thing I forgot to
> > > mention in my mail to #660193 is that the reason to remove the blob from
> > > the used waf script is to ensure that the unpacked waf source is used.
> >
> > Well, in midori diff, I repack and ensure the new one and the old ones
> > are the same to be sure I don't do anything bad. Now indeed it could be
> > split in two parts, one run by maintainer, which would then hack in the
> > waf sources themeselves, and one at build time, which would pick the
> > extracted sources and make a new waf script.
> 
> My point of all this was to provide an easy way to change the source
> code, but this can't be accomplished.  You provided a way to extract the
> source to be able to review it, but not to change it.
> 
> > And changing the way waf is used at built time is not supported and
> > might fail in bad ways too, so it's not really helpful to do things
> > *against* what advised by waf upstream.
> 
> Users might not be advised to use an extracted source, but "#devs use
> $WAFDIR" (this is a comment in the waf script shipped in midori), so
> using the extracted wafadmin directory isn't unsupported at all.
> $WAFDIR is the directory in which the extracted wafadmin directory is
> found.

Well, when needed because we need to patch the build script (like for
the hppa issue) we can do that. In the usual case, don't touch it (and
don't break it).

Regards,
-- 
Yves-Alexis
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Sun, 15 Apr 2012 18:36:05 GMT) Full text and rfc822 format available.

Message #118 received at 645190@bugs.debian.org (full text, mbox):

From: Carsten Hey <carsten@debian.org>
To: Yves-Alexis Perez <corsac@debian.org>
Cc: 645191@bugs.debian.org, 645190@bugs.debian.org, 654468@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Sun, 15 Apr 2012 20:32:34 +0200
* Yves-Alexis Perez [2012-04-15 09:18 +0200]:
> On ven., 2012-03-23 at 23:39 +0100, Carsten Hey wrote:
> > I think we should drop ftpmaster from CC in further mails.
>
> Maybe, since they don't seem to care about this.

They provided an IMHO acceptable, but not ideal, way (because there
there does not seem to be an ideal way) to handle this.  I suggested
dropping them from CC because there is nothing relevant yet they could
comment on, which presumably is also the reason they did not comment up
to now ;)


> Well, parsing python might not be an option, but what about:
>
> egrep -a "^C[1|2]='..'" waf
> C1='#*'
> C2='#%'

We need to be able to repack a changed wafadmin directory into an
existing waf script to gain anything.  To repack, C1 and C2 need to be
adapted.  If adapting C1 and C2 is done via regular expressions, it
would fail, possibly without being noticed, if, for example, the
variable names in future waf versions change or if the character ' is
part of this variable and you did not handle this in your regular
expression.  All in all, this is a rather natural approach for this
problem, but it is all but robust.

It could be done using regular expressions, but I assume that the effort
required to ensure that it works correctly and to update it is way more
than the effort to just shipping an unpacked waf in every waf using
package.  Besides this, the probability of unnoticed related errors is
presumably unreasonably high.


A way to handle this that would possibly make everybody happy would
require to convince waf upstream to adapt waf.

As already mentioned, the reason that we are not able to repack waf
scripts in a reasonable way using only essential tools is that waf
scripts are not clearly divided into a data part and an non-data part,
i.e., C1 and C2 contain information that one would expect to be in
a header and not in a script.

If waf script's would instead of the variables C1 and C2 contain
a header like the one below, and would parse the header itself to figure
out which replacements it should do, then tools that unpack and/or
repack waf scripts in a reliable way could easily be written.

  #===
  # Waf-Data-Format: 1.0
  # Waf-Archive-Type: tar.gz
  # Waf-Archive-Base-Directory: wafadmin
  # Waf-Line-Feed-Replacement: ab
  # Waf-Carriage-Return-Replacement: xy
  #==>
  #...
  #<==

If such a header would be used by waf upstream, it would be important
that there is exactly one space between the colon after the field name
and the field's data.  The reason for this is that a replacement string
could begin with a space character.  Introducing a way to escape some
characters would IMO be too over-engineered.  Alternatively, the
(uppercased) hex values could be used instead of the real string, i.e.,
' m' would be written as 206D in the header.

Reasons to brute-force unused sequences instead of simply prefixing all
line feeds and all carriage returns with a numbersign are:

 * Kepp the size of the encoded string as small as possible.  Prefixing
   two of the possible 256 characters would enlarge the encoded string
   on average by 2/256 or 0.78%, given that the compression method is
   reasonable.

 * Some editors do not wrap lines by default.  One could consider
   displaying just one long unwrapped line instead of multiple lines (on
   average size/128 lines) if a waf script is opened in an editor to be
   more beautiful.

 * The data part ends before a line that only contains the string
   '#<=='.  If you would encode an archive of infinite size by the
   described prefixing, it would also contain this line _in_ the data
   part.  A way to fix this it to additionally prefix the equal sign
   with a number sign.  A presumably better way it to interpret the
   semantic of '#<==' as "the data part ends before the _last_ equal
   line in a comment block" and not "... before the _first_ equal line
   ...".

Perl one-liner filters to encode and decode the data part using the
described prefixing are:
    perl -e '$_ = do { local $/ = <> }; s/\n/\n#/sg; s/\r/\r#/sg; print "#", $_, "\n"'
    perl -e '$_ = do { local $/ = <> }; $_ = substr($_, 1, -1); s/\r#/\r/sg; s/\n#/\n/sg; print'

They can be used in the same way as all other filters:
    cat file | filter > result

With this approach, the need for C1 and C2 (or the according header
fields) would vanish.  The header would still be very useful, though.


The remaining non-trivial part, which I will not do since I think the
existing solution (shipping waf unpacked) is ugly but sufficient and
I don't even use waf, is to try to convince waf's upstream to add such
a header.  With such a header and the according scripts, changes between
different Debian revisions would still not be reviewable as easy as
running "zrun interdiff *.diff.gz", but I don't think that this is
a blocker, as long as README.source contains easy recipes for changing
waf and reviewing these changes.


> Well, when needed because we need to patch the build script (like for
> the hppa issue) we can do that.

Being able to do something doesn't necessarily mean that it can be done
in an easy way.


Regards
Carsten


P.S.: Do whatever you want to with this mail's content.  If anything
      in it I wrote (everything that is not quoted from your previous
      mail) is copyrightable, which I doubt, then it is licensed under
      terms of  the practically public domain equivalent license WTFPL
      2.0

P.P.S.: If you want to test if the above can be embedded into a python
        script, set the script's encoding to latin-1, as described in
        PEP 0263 - or just copy the second line of an existing waf
        script.




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Sun, 15 Apr 2012 18:51:11 GMT) Full text and rfc822 format available.

Message #121 received at 645190@bugs.debian.org (full text, mbox):

From: Carsten Hey <carsten@debian.org>
To: Yves-Alexis Perez <corsac@debian.org>, 645191@bugs.debian.org, 645190@bugs.debian.org, 654468@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Sun, 15 Apr 2012 20:48:42 +0200
* Carsten Hey [2012-04-15 20:32 +0200]:
>   #===
>   # Waf-Data-Format: 1.0

    # Waf-Version: 1.8

>   # Waf-Archive-Type: tar.gz
>   # Waf-Archive-Base-Directory: wafadmin
>   # Waf-Line-Feed-Replacement: ab
>   # Waf-Carriage-Return-Replacement: xy
>   #==>
>   #...
>   #<==




Information forwarded to debian-bugs-dist@lists.debian.org, Devid Antonio Filoni <d.filoni@ubuntu.com>:
Bug#645190; Package postler. (Sun, 15 Apr 2012 19:27:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Yves-Alexis Perez <corsac@debian.org>:
Extra info received and forwarded to list. Copy sent to Devid Antonio Filoni <d.filoni@ubuntu.com>. (Sun, 15 Apr 2012 19:27:03 GMT) Full text and rfc822 format available.

Message #126 received at 645190@bugs.debian.org (full text, mbox):

From: Yves-Alexis Perez <corsac@debian.org>
To: Carsten Hey <carsten@debian.org>
Cc: 645191@bugs.debian.org, 645190@bugs.debian.org, 654468@bugs.debian.org, ftpmaster@debian.org
Subject: Re: Bug#645191: update on waf binary data
Date: Sun, 15 Apr 2012 21:25:28 +0200
[Message part 1 (text/plain, inline)]
On dim., 2012-04-15 at 20:32 +0200, Carsten Hey wrote:
> * Yves-Alexis Perez [2012-04-15 09:18 +0200]:
> > On ven., 2012-03-23 at 23:39 +0100, Carsten Hey wrote:

[…]

Ok, I give up.
-- 
Yves-Alexis
[signature.asc (application/pgp-signature, inline)]

Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Mon, 14 May 2012 07:36:29 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Wed Apr 16 23:34:35 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.