Debian Bug report logs - #630982
Automatic build-time dependency generation for jquery.js

version graph

Package: doxygen; Maintainer for doxygen is Matthias Klose <doko@debian.org>; Source for doxygen is src:doxygen.

Reported by: Marcin Owsiany <porridge@debian.org>

Date: Sun, 19 Jun 2011 13:27:01 UTC

Severity: important

Found in version doxygen/1.7.4-2

Fixed in version doxygen/1.8.5-1

Done: Helmut Grohne <helmut@subdivi.de>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Matthias Klose <doko@debian.org>:
Bug#630982; Package doxygen. (Sun, 19 Jun 2011 13:27:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Marcin Owsiany <porridge@debian.org>:
New Bug report received and forwarded. Copy sent to Matthias Klose <doko@debian.org>. (Sun, 19 Jun 2011 13:27:05 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Marcin Owsiany <porridge@debian.org>
To: submit@bugs.debian.org
Subject: Automatic build-time dependency generation for jquery.js
Date: Sun, 19 Jun 2011 14:25:44 +0100
Package: doxygen
Version: 1.7.4-2

I maintain libgadu, which is one of many packages that use doxygen for
generating its API documentation (in this case in libgadu-doc). I'm
writing because of the new lintian check for embedded JavaScript
libraries which also affects libgadu-doc. While this problem affects
more than libgadu-doc, let's focus on this case for sake of simplicity.

I've seen #622147 and #625956, however I feel they do not address the
core of the problem, which is the fact that calling doxygen when
building src:libgadu embeds a copy of jquery.js inside the libgadu-doc
binary package.

It seems that in order to rid of the embedded copy, the libgadu-doc
maintainer would need to hunt down such jquery.js file and replace it
with a symlink to the file provided by libjs-jquery. 

However from libgadu's POV that's not the right approach:
 - libgadu-doc treats doxygen as a service - it points it at some source
   files, and expects a bunch of HTML files in the output directory. The
   exact names or paths of these files are not generally its concern
   (maybe apart from the main index.html). In particular which
   JavaScript libriaries doxygen decides to use for the documentation it
   generates is an implementation detail.
 - It might change between doxygen versions in various ways: the name or
   exact path of the embedded file could change, or it might decide to
   use a completely different jquery implementation altogether. Or might
   even stop using jquery at all! And that's all completely fine, and it
   must not require any action on part of libgadu-doc maintainer.
   Otherwise simple rebuilds against a new doxygen would be unsafe!

I think instead we should use the technology we already have in Debian
for dynamically generating binary package dependencies at build time.

Here's my proposal:

whenever doxygen is used during libgadu-doc build, it should:

1) install symlinks to /usr/share/javascript/jquery/jquery*.js rather
   than outputing the file contents

2) add misc:Depends=libjs-jquery to debian/libgadu-doc.substvars

How this should be implemented is an open question. I don't think the
doxygen binary should be modified to edit Debian build-specific files,
but I could imagine:

 - doxygen emitting a list of jquery.js file paths it created during a
   run,

 - a wrapper for doxygen, which would accept a path to the debian
   directory, and the name of the package being built (libgadu-doc in
   this case). This wrapper would parse the list emitted by doxygen,
   replace files with symlinks, and edit the appropriate .substvars
   file.


Implementing this proposal will obviously require some work, but I think
it will be an overall win given the number of packages which use doxygen
and currently end up with embedded copies of jquery. Moreover, when such
mechanism is implemented, it can probably be used in other packages
whose role is similar to doxygen.

-- 
Marcin Owsiany <porridge@debian.org>             http://marcin.owsiany.pl/
GnuPG: 1024D/60F41216  FE67 DA2D 0ACA FC5E 3F75  D6F6 3A0D 8AA0 60F4 1216




Information forwarded to debian-bugs-dist@lists.debian.org, Matthias Klose <doko@debian.org>:
Bug#630982; Package doxygen. (Sat, 23 Feb 2013 08:30:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Helmut Grohne <helmut@subdivi.de>:
Extra info received and forwarded to list. Copy sent to Matthias Klose <doko@debian.org>. (Sat, 23 Feb 2013 08:30:04 GMT) Full text and rfc822 format available.

Message #10 received at 630982@bugs.debian.org (full text, mbox):

From: Helmut Grohne <helmut@subdivi.de>
To: Marcin Owsiany <porridge@debian.org>, 630982@bugs.debian.org
Subject: Re: Automatic build-time dependency generation for jquery.js
Date: Sat, 23 Feb 2013 09:27:11 +0100
[Message part 1 (text/plain, inline)]
Control: severity -1 important

Raising the severity due to the number of packages affected and the
number of workarounds deployed already.

On Sun, Jun 19, 2011 at 02:25:44PM +0100, Marcin Owsiany wrote:
> Here's my proposal:
> 
> whenever doxygen is used during libgadu-doc build, it should:
> 
> 1) install symlinks to /usr/share/javascript/jquery/jquery*.js rather
>    than outputing the file contents
> 
> 2) add misc:Depends=libjs-jquery to debian/libgadu-doc.substvars

I am attaching a preliminary version of dh_doxygen (bluntly copied from
Jakub Wilk's dh_sphinxdoc). In addition to the above it will also clean
*.md5 and *.map files used to speed up incremental regeneration (which
is useless in binary packages). Note that the substvar is called
${doxygen:Depends} instead. Also ${doxygen:Built-Using} is provided to
meet recent policy changes.

Open issues:

 * Maybe the libjs-jquery dependency should be versioned?
 * Would it be possible for doxygen to ship a doxygen-common package
   shipping files like doxygen.css doxygen.png tabs.css dynsections.js?
   That way we might be able to get rid of the Built-Using field.

Helmut
[dh_doxygen (text/plain, attachment)]

Severity set to 'important' from 'normal' Request was from Helmut Grohne <helmut@subdivi.de> to 630982-submit@bugs.debian.org. (Sat, 23 Feb 2013 08:30:04 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Matthias Klose <doko@debian.org>:
Bug#630982; Package doxygen. (Tue, 26 Feb 2013 21:33:08 GMT) Full text and rfc822 format available.

Acknowledgement sent to Helmut Grohne <helmut@subdivi.de>:
Extra info received and forwarded to list. Copy sent to Matthias Klose <doko@debian.org>. (Tue, 26 Feb 2013 21:33:08 GMT) Full text and rfc822 format available.

Message #17 received at 630982@bugs.debian.org (full text, mbox):

From: Helmut Grohne <helmut@subdivi.de>
To: 625956@bugs.debian.org, 630982@bugs.debian.org, pkg-javascript-devel@lists.alioth.debian.org
Subject: jquery embedding in doxygen
Date: Tue, 26 Feb 2013 22:30:00 +0100
Dear javascript maintainers,

I am writing to you, because I seek help with doxygen. For wheezy I
believe that Mònica Ramírez Arceda's patch is the way to go, so this
mail entirely applies to jessie.

** First embedding of jquery: src:doxygen

The current situation is that doxygen upstream downloads various parts
of jquery in various versions, then obfuscates (or is it called
"compresses"?) the source and stores those parts in their svn. Then they
convert the jquery library into a C header file which is also stored in
their svn. The lack of source for jquery in the sense of "preferred form
for modification" is tracked as #625956. According to upstream svn these
copies are usually generated immediately before releasing a new version
of doxygen.

** Second embedding of jquery: doxygen

The header is compiled to the doxygen binary, so the binary package also
includes a copy of jquery. Once you generate documentation this version
is copied to your documentation tree.

** Third embedding of jquery: reverse build dependencies of doxygen

About 50 packages use doxygen to build their documentation. Unless the
maintainer explicitly replaces the doxygen generated copy of jquery, the
respective package includes it as well.

** So precisely what is copied?

This actually depends on the doxygen version in use. In earlier version
it used to copy jquery 1.3.2. For the current version Mònica Ramírez
Arceda thankfully examined the source and discovered:

jquery 1.7.1 including sizzle
jquery.ScrollTo 1.4.2
jquery hashchange event 1.3
jquery UI 1.8.18
jquery UI Mouse 1.8.18
jquery UI Resizable 1.8.18
jquery UI Widget 1.8.18

Jérémy Lal kindly explained that some parts of this (ScrollTo) are not
currently packaged for Debian, but most is.

** Which embeddings should we solve and how?

For the regular user a doxygen generated tree should be usable
stand-alone. That is doxygen will keep copying jquery during
documentation generation. A debhelper dh_doxygen called from
documentation packages during build could be used to replace these
(thired) copies of the jquery conglomerate by symbolic links to a newly
created doxygen-common package. (See Jakub Wilk's dh_sphinxdoc for a
similar tool.) This raises an important question though: What happens
when upgrading doxygen-common? How to ensure that previously generated
documentation does not break with an upgraded doxygen-common?

Note that if there are backwards-incompatible changes, we have to
rebuild about 50 reverse dependencies of doxygen, and there is no such
thing as binNMU for them, because they are mostly Architecture: all.

Ideally which step should be generating the jquery.js file to be copied
into a documentation tree?
A: Upstream
B: During build of doxygen
C: During the invocation of doxygen

If the answer to the previous question is A: What can we do about the
(first) copy of jquery in the doxygen source package? Reverse
engineering the jqeury components embedded in each new release appears
like a tedious task. In what way can this situation be improved
upstream?

In the other cases we could repack doxygen to remove the jquery files,
but we would still need some kind of upstream support to determine what
to generate.

When creating a doxygen-common package (and we are not in case C),
should that package contain the copy of jquery used to embed into
documentation or should it contain a javascript file loading the
remainders from libjs-something packages?

Note that I do not expect answers to all of these questions. I merely
wrote down the currently open issues and hope for some thoughts
advancing the matter.

Helmut



Information forwarded to debian-bugs-dist@lists.debian.org, Matthias Klose <doko@debian.org>:
Bug#630982; Package doxygen. (Tue, 26 Feb 2013 23:27:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Jonas Smedegaard <dr@jones.dk>:
Extra info received and forwarded to list. Copy sent to Matthias Klose <doko@debian.org>. (Tue, 26 Feb 2013 23:27:05 GMT) Full text and rfc822 format available.

Message #22 received at 630982@bugs.debian.org (full text, mbox):

From: Jonas Smedegaard <dr@jones.dk>
To: Helmut Grohne <helmut@subdivi.de>, 625956@bugs.debian.org, 630982@bugs.debian.org, pkg-javascript-devel@lists.alioth.debian.org
Subject: Re: [Pkg-javascript-devel] jquery embedding in doxygen
Date: Wed, 27 Feb 2013 00:22:52 +0100
[Message part 1 (text/plain, inline)]
Quoting Helmut Grohne (2013-02-26 22:30:00)
> ** First embedding of jquery: src:doxygen
> 
> The current situation is that doxygen upstream downloads various parts 
> of jquery in various versions, then obfuscates (or is it called 
> "compresses"?) the source and stores those parts in their svn. Then 
> they convert the jquery library into a C header file which is also 
> stored in their svn. The lack of source for jquery in the sense of 
> "preferred form for modification" is tracked as #625956. According to 
> upstream svn these copies are usually generated immediately before 
> releasing a new version of doxygen.

I believe the Debian packaging should ignore the prebuild stuff shipped 
by upstream, and during the Debian build mimic same steps as done 
upstream but build-depending on and using Debian-packaged jquery files 
(either uncompressed or compressed - but most likely the best result 
comes from joining all needed files uncompressed and compress them all 
at once).


> ** Second embedding of jquery: doxygen
> 
> The header is compiled to the doxygen binary, so the binary package 
> also includes a copy of jquery. Once you generate documentation this 
> version is copied to your documentation tree.

Ideally doxygen should be patched to not work like that but instead at 
runtime use the Debian-packaged files (then also solving above first 
embedding of jquery).

Assuming that is too difficult, I would suggest do a dirty trick of 
depending versioned on jquery - even if not actually using it for 
anything, and have the versioning autoresolved during build to be 
tightened to current install of it, so as to force requiring a rebuild 
of doxygen every time jquery changes.

(less dirty would be to tighten build-dependency, but that would require 
sourceful rebuild due to Debian Policy not allowing build-dependencies 
to change during build.


> ** Third embedding of jquery: reverse build dependencies of doxygen
> 
> About 50 packages use doxygen to build their documentation. Unless the 
> maintainer explicitly replaces the doxygen generated copy of jquery, 
> the respective package includes it as well.

(see below...)

[snip]

> For the regular user a doxygen generated tree should be usable 
> stand-alone. That is doxygen will keep copying jquery during 
> documentation generation.

Makes sense.


> A debhelper dh_doxygen called from documentation packages during build 
> could be used to replace these (thired) copies of the jquery 
> conglomerate by symbolic links to a newly created doxygen-common 
> package. (See Jakub Wilk's dh_sphinxdoc for a similar tool.) This 
> raises an important question though: What happens when upgrading 
> doxygen-common? How to ensure that previously generated documentation 
> does not break with an upgraded doxygen-common?

I suggest to have doxygen build a doxygenXXX-common binary package which 
depends versioned (as tightly as needed) on appropriate jquery packages, 
with XXX bumped every time the micture of dependent jquery packages 
changes.

...and then have the main doxygen package provide such dh_doxygen tool, 
which both a) replaces javascript files with symlinks and b) adds a 
dependency on the current doxygenXXX-common package.

When doxygenXXX-common is bumped, you will then need to check if any 
package anywhere in the archive actually depend on the old one, and if 
so have your new package also keep creating that old old, or...


> Note that if there are backwards-incompatible changes, we have to 
> rebuild about 50 reverse dependencies of doxygen, and there is no such 
> thing as binNMU for them, because they are mostly Architecture: all.

...file severe bugreports to have those packages be rebuilt (not all 50, 
only the ones actually depending on the particular obsolete 
doxygenXXX-common package).


> Ideally which step should be generating the jquery.js file to be copied
> into a documentation tree?
> A: Upstream
> B: During build of doxygen
> C: During the invocation of doxygen

Ideal to upstream: A

Ideal to distrutors: C - by help of node-uglify.


> If the answer to the previous question is A: What can we do about the 
> (first) copy of jquery in the doxygen source package? Reverse 
> engineering the jqeury components embedded in each new release appears 
> like a tedious task. In what way can this situation be improved 
> upstream?

What I think is reasonable to try persuade upstream to do is leave an 
automated trace of which sources was used for generating the shipped 
JavaScript code, so as to help distributors like us avoid the need for 
reverse engineering.


> When creating a doxygen-common package (and we are not in case C),
> should that package contain the copy of jquery used to embed into
> documentation or should it contain a javascript file loading the
> remainders from libjs-something packages?

When not in case C then we have lost the goal of avoiding code copies 
and I see no point in complicating further by replacing full-copies of 
jquery with some (potentially slower or less reliable) bootstrapping 
code.


Hope that helps.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private
[signature.asc (application/pgp-signature, inline)]

Reply sent to Helmut Grohne <helmut@subdivi.de>:
You have taken responsibility. (Sun, 10 Nov 2013 19:03:09 GMT) Full text and rfc822 format available.

Notification sent to Marcin Owsiany <porridge@debian.org>:
Bug acknowledged by developer. (Sun, 10 Nov 2013 19:03:09 GMT) Full text and rfc822 format available.

Message #27 received at 630982-close@bugs.debian.org (full text, mbox):

From: Helmut Grohne <helmut@subdivi.de>
To: 630982-close@bugs.debian.org
Subject: Bug#630982: fixed in doxygen 1.8.5-1
Date: Sun, 10 Nov 2013 19:00:06 +0000
Source: doxygen
Source-Version: 1.8.5-1

We believe that the bug you reported is fixed in the latest version of
doxygen, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 630982@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Helmut Grohne <helmut@subdivi.de> (supplier of updated doxygen package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Sun, 10 Nov 2013 17:32:18 +0100
Source: doxygen
Binary: doxygen doxygen-latex doxygen-doc doxygen-gui doxygen-dbg
Architecture: source amd64 all
Version: 1.8.5-1
Distribution: unstable
Urgency: low
Maintainer: Matthias Klose <doko@debian.org>
Changed-By: Helmut Grohne <helmut@subdivi.de>
Description: 
 doxygen    - Documentation system for C, C++, Java, Python and other languages
 doxygen-dbg - Debug symbols for doxygen
 doxygen-doc - Documentation for doxygen
 doxygen-gui - GUI configuration tool for doxygen
 doxygen-latex - Documentation system for C, C++, Java, Python and other languages
Closes: 569504 630982 720493
Changes: 
 doxygen (1.8.5-1) unstable; urgency=low
 .
   * doxygen 1.8.5 release.
     + Refresh patches. dot-config.diff had a failed hunk.
     + README got renamed to README.md.
   * Add debug package doxygen-dbg.
   * Demote recommends doxygen-latex from doxygen to suggests. (Closes:
     #720493)
   * LDFLAGS are called QMAKE_LFLAGS_RELEASE (not ...LDFLAGS...),
     this should fix the build log checks and hardening.
   * Install upstream changelog. (Closes: #569504)
   * Shrink doxygen-doc binary package using hardlinks (rdfind).
     + Add lintian-override for package-contains-hardlink
   * Rewrite debian/copyright in DEP5 syntax.
   * Bump standards version to 3.9.5: No changes needed.
   * Explain that jquery embedding is going to stay in README.jquery.
     (Closes: #630982)
   * Add an autopkgtest building the documentation of osmium.
     It managed to crash Doxygen earlier, see #657917.
Checksums-Sha1: 
 3afdd2d1855d433c913c5d9b52db174c031f0668 2233 doxygen_1.8.5-1.dsc
 1fc5ceec21122fe5037edee4c308ac94b59ee33e 6511944 doxygen_1.8.5.orig.tar.gz
 a69e92d55f702176d9e077cf19cf0cff8931f49d 22442 doxygen_1.8.5-1.debian.tar.gz
 adb7180c2941c01b31350f079d3371afe0710cb0 2074600 doxygen_1.8.5-1_amd64.deb
 667a87e1227be2b1050db842196175cc1340f9a6 324880 doxygen-gui_1.8.5-1_amd64.deb
 cdd8a2ee98bdd812b2bd44a19635bab0a46e4f25 7603930 doxygen-dbg_1.8.5-1_amd64.deb
 ebb320967a601c53b0372174b81ae85138b6db16 50050 doxygen-latex_1.8.5-1_all.deb
 dc8d367e7819f1ba7542fd8f55bdd31c38d93d12 1883436 doxygen-doc_1.8.5-1_all.deb
Checksums-Sha256: 
 5dd9ed9a5ea392edd588fd1790efaf67f746c64cf28b9fe6306920a7ca7f7157 2233 doxygen_1.8.5-1.dsc
 243a8b67db12ad68d6ea5b51c6f60dc2cc3a34fa47abf1b5b4499196c3d7cc25 6511944 doxygen_1.8.5.orig.tar.gz
 140aa4a73ce8101ad3532c0b116c09bb6d1814b2de4beae10d2d1b39a7edeae9 22442 doxygen_1.8.5-1.debian.tar.gz
 181992c99ba6157c3fccd8c80a7f51450e4a218d0fe9820303c8efc7ff1afb55 2074600 doxygen_1.8.5-1_amd64.deb
 690130e38db44f9302b6de0406649cbc7e7764ccfac262991026646c53cf86ac 324880 doxygen-gui_1.8.5-1_amd64.deb
 67de815966ec980d409067518929cd596446cce3b2c8f3750db07b79007bbc36 7603930 doxygen-dbg_1.8.5-1_amd64.deb
 bfec19f07e038c9e0cb690b0c437ba31c81105a78ee73db87201b4e69024a141 50050 doxygen-latex_1.8.5-1_all.deb
 22c58067381cd14a6f9bfbd16e59f9ccb555e4ac667779a91e322b48e5c84130 1883436 doxygen-doc_1.8.5-1_all.deb
Files: 
 f78a00dc753c742f4b75c30f47fe5d6e 2233 devel optional doxygen_1.8.5-1.dsc
 db51274568755e2c75c2657e30a78a55 6511944 devel optional doxygen_1.8.5.orig.tar.gz
 e2cba11894c59ac06fedf7cf8b0371a8 22442 devel optional doxygen_1.8.5-1.debian.tar.gz
 ceed907e910165331f3eb162b574fc98 2074600 devel optional doxygen_1.8.5-1_amd64.deb
 94ff532e7c8bf8cfd43845adbecedf6a 324880 devel optional doxygen-gui_1.8.5-1_amd64.deb
 0fcd3351f1fc6a085385bda1fc3e979b 7603930 debug extra doxygen-dbg_1.8.5-1_amd64.deb
 ddf22444a1f64175503e5adbafdf5ab7 50050 devel optional doxygen-latex_1.8.5-1_all.deb
 ed7e902ab72df5afdad11ce21e66408e 1883436 doc optional doxygen-doc_1.8.5-1_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.15 (GNU/Linux)

iQIcBAEBCgAGBQJSf74QAAoJEC0aqs8kRERCLDoP/jxVVjf/4xW5LfXBbuBEREeI
aiOeKkrRyx0DlKGTgoAc3Z9ae2PRNQBe4JFCgtSy1xl9tcbKhrqghIftTxij1nLT
ayVxyESrqCg1G+CnjIstMqB+5+nB8hUtzdjLdNPHYCsR556E5Ft5SlpDfT0YF2yA
BSPb/taKRq/pVcTO6iJmYm42J+RCnluclIPqeVJ3HTvxF6rnPxU8HXI+AF3pq5pQ
z63nyHVX9bi9RszTgNNur0u8RT21AWWSntMGUDKJ5RG+JfTP6ZLTWjtNycUuL06u
0sYs71M00VTnNDtIF5snTCCJPSPz2x8X7P47kK3GmuZaIL8dPFT4MCukY1oq854W
6M3RYZb3sV2PaqVbQsTRj4Pw2v8Oce2ePfI4dKrQNSZ+6H7gGNuy8aU938I1pvAv
5tapNWYis1mCks0KXlTDiMR8PwWicEcPL7C+Ox+3R+PH8jlV3avCvq2OUBMJuNFE
BYv3sdwD//DRWSWq3UauX8myeOkyd+wGnGoXvxYdLvi4ZsCzFEJkXN0jw7TqInxb
Ku7qCh1WFvL7xIRymgmqeQZYLvkO1p1AOP0kGq7ynpxY4zt+l50E8aOGHes5ixXe
Q7AujjjNE3AUCY6/xWvEdWJT/9lcuhal/ChT7zYy6hOrs6CK3Bx3T7z6YLUOcyKS
RTBiZHDxxJu129yFscni
=4+1a
-----END PGP SIGNATURE-----




Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Sun, 15 Dec 2013 07:33:01 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Thu Apr 17 19:37:11 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.