Debian Bug report logs - #895044
debhelper: add support for Ninja with CMake

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Kyle Edwards <kyle.edwards@kitware.com>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: debhelper: add support for Ninja with CMake

Date: Fri, 06 Apr 2018 10:08:24 -0400

Package: debhelper
Version: 11.1.6
Severity: wishlist

Dear Maintainer,

Large projects which use CMake, such as VTK, take a long time to build
with the Makefile generator, and see significant build time improvements
when being built with the Ninja generator instead. Please consider
adding support for using the Ninja generator with CMake. Perhaps
something like:

$ dh --buildsystem=cmakeninja

I realize this may be tricky given how the buildsystems are implemented
in debhelper. I don't know much about Perl, but if it supports multiple
inheritance, would it be possible to make a class like "cmakebase" and
then have "cmake" and "cmakeninja" inherit from both "cmakebase" and
then "makefile" and "ninja" respectively?


-- System Information:
Debian Release: buster/sid
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.15.0-2-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages debhelper depends on:
ii  autotools-dev            20180224.1
ii  dh-autoreconf            17
ii  dh-strip-nondeterminism  0.040-1
ii  dpkg                     1.19.0.5
ii  dpkg-dev                 1.19.0.5
ii  file                     1:5.32-2
ii  libdpkg-perl             1.19.0.5
ii  man-db                   2.8.2-1
ii  perl                     5.26.1-5
ii  po-debconf               1.0.20

debhelper recommends no packages.

Versions of packages debhelper suggests:
pn  dh-make  <none>
pn  dwz      <none>

-- no debconf information

Message #10 received at 895044@bugs.debian.org (full text, mbox, reply):

From: Niels Thykier <niels@thykier.net>

To: 895044@bugs.debian.org, Kyle Edwards <kyle.edwards@kitware.com>

Subject: Re: debhelper: add support for Ninja with CMake

Date: Fri, 06 Apr 2018 14:18:00 +0000

On Fri, 06 Apr 2018 10:08:24 -0400 Kyle Edwards
<kyle.edwards@kitware.com> wrote:
> Package: debhelper
> Version: 11.1.6
> Severity: wishlist
> 
> Dear Maintainer,
> 
> Large projects which use CMake, such as VTK, take a long time to build
> with the Makefile generator, and see significant build time improvements
> when being built with the Ninja generator instead. Please consider
> adding support for using the Ninja generator with CMake. Perhaps
> something like:
> 
> $ dh --buildsystem=cmakeninja
> 
> I realize this may be tricky given how the buildsystems are implemented
> in debhelper. I don't know much about Perl, but if it supports multiple
> inheritance, would it be possible to make a class like "cmakebase" and
> then have "cmake" and "cmakeninja" inherit from both "cmakebase" and
> then "makefile" and "ninja" respectively?
> 
> [...]


Hi Kyle,

Thanks for the suggestion.


How will debhelper know whether cmake will generate a ninja.build and
not a Makefile?  Is that something debhelper decides 100% (via the -G)
parameter?  Or does the project specify a default in CMakeLists.txt and
debhelper should respect that or ...?

If debhelper decides it: How likely is it that a random CMake project
today will work with ninja instead of make?

Thanks,
~Niels

Message #15 received at 895044@bugs.debian.org (full text, mbox, reply):

From: Kyle Edwards <kyle.edwards@kitware.com>

To: Niels Thykier <niels@thykier.net>, 895044@bugs.debian.org

Subject: Re: debhelper: add support for Ninja with CMake

Date: Fri, 6 Apr 2018 10:51:58 -0400

On 04/06/2018 10:18 AM, Niels Thykier wrote:
> On Fri, 06 Apr 2018 10:08:24 -0400 Kyle Edwards
> <kyle.edwards@kitware.com> wrote:
>> Package: debhelper
>> Version: 11.1.6
>> Severity: wishlist
>>
>> Dear Maintainer,
>>
>> Large projects which use CMake, such as VTK, take a long time to build
>> with the Makefile generator, and see significant build time improvements
>> when being built with the Ninja generator instead. Please consider
>> adding support for using the Ninja generator with CMake. Perhaps
>> something like:
>>
>> $ dh --buildsystem=cmakeninja
>>
>> I realize this may be tricky given how the buildsystems are implemented
>> in debhelper. I don't know much about Perl, but if it supports multiple
>> inheritance, would it be possible to make a class like "cmakebase" and
>> then have "cmake" and "cmakeninja" inherit from both "cmakebase" and
>> then "makefile" and "ninja" respectively?
>>
>> [...]
>
> Hi Kyle,
>
> Thanks for the suggestion.
>
>
> How will debhelper know whether cmake will generate a ninja.build and
> not a Makefile?  Is that something debhelper decides 100% (via the -G)
> parameter?  Or does the project specify a default in CMakeLists.txt and
> debhelper should respect that or ...?
>
> If debhelper decides it: How likely is it that a random CMake project
> today will work with ninja instead of make?
>
> Thanks,
> ~Niels

CMake developer here (I work at Kitware). The intention of CMake is to 
generate whatever buildsystem the developer feels most productive with, 
rather than what the project wants. By default, CMake generates 
Makefiles on Unix unless you give it a -G parameter. There's not a 
supported way to set the generator from within CMakeLists.txt, so yes, 
debhelper would be making that decision with -G. But even in a 
hypothetical scenario where a project is setting the generator from 
inside CMakeLists.txt (perhaps through some sort of unsupported hack), 
current debhelper wouldn't be able to handle that, because it's 
expecting a Makefile. If we added "cmakeninja", these hypothetical 
projects which are setting the generator to Ninja could be packaged with 
cmakeninja.

Ninja has been supported by CMake for a while, and most projects tend to 
handle it pretty well. That said, some projects are known to not work 
with Ninja. For those, they can continue to use the classic "cmake" 
buildsystem, which would still generate makefiles, and then projects 
that work with Ninja could use "cmakeninja" if they so desire. This 
change wouldn't impact existing projects unless they explicitly use 
"cmakeninja", because the "cmakeninja" check would come after the 
"cmake" check, which would succeed upon finding CMakeLists.txt.

Kyle

Message #20 received at 895044@bugs.debian.org (full text, mbox, reply):

From: Niels Thykier <niels@thykier.net>

To: Kyle Edwards <kyle.edwards@kitware.com>, 895044@bugs.debian.org

Subject: Re: [debhelper-devel] Bug#895044: debhelper: add support for Ninja with CMake

Date: Fri, 06 Apr 2018 20:15:00 +0000

[Message part 1 (text/plain, inline)]

Control: tags -1 +patch

Kyle Edwards:
> Package: debhelper
> Version: 11.1.6
> Severity: wishlist
> 
> Dear Maintainer,
> 
> Large projects which use CMake, such as VTK, take a long time to build
> with the Makefile generator, and see significant build time improvements
> when being built with the Ninja generator instead. Please consider
> adding support for using the Ninja generator with CMake. Perhaps
> something like:
> 
> $ dh --buildsystem=cmakeninja
> 
> I realize this may be tricky given how the buildsystems are implemented
> in debhelper. I don't know much about Perl, but if it supports multiple
> inheritance, would it be possible to make a class like "cmakebase" and
> then have "cmake" and "cmakeninja" inherit from both "cmakebase" and
> then "makefile" and "ninja" respectively?
> 
> 
> [...]

Ok, I have written a sample patch series (attached) that should make
debhelper support cmake with the ninja backend.

The chosen syntax is:

  dh <target> --buildsystem=cmake+ninja

A review and some testing would be appreciated.

Thanks,
~Niels

[0001-Rewrite-build-system-to-support-a-target-build-syste.patch (text/x-patch, attachment)]

[0002-cmake-Support-ninja-backend.patch (text/x-patch, attachment)]

Message #27 received at 895044@bugs.debian.org (full text, mbox, reply):

From: Kyle Edwards <kyle.edwards@kitware.com>

To: Niels Thykier <niels@thykier.net>, 895044@bugs.debian.org

Subject: Re: [debhelper-devel] Bug#895044: debhelper: add support for Ninja with CMake

Date: Fri, 6 Apr 2018 17:42:13 -0400

On 04/06/2018 04:15 PM, Niels Thykier wrote:
> Control: tags -1 +patch
>
> Kyle Edwards:
>> Package: debhelper
>> Version: 11.1.6
>> Severity: wishlist
>>
>> Dear Maintainer,
>>
>> Large projects which use CMake, such as VTK, take a long time to build
>> with the Makefile generator, and see significant build time improvements
>> when being built with the Ninja generator instead. Please consider
>> adding support for using the Ninja generator with CMake. Perhaps
>> something like:
>>
>> $ dh --buildsystem=cmakeninja
>>
>> I realize this may be tricky given how the buildsystems are implemented
>> in debhelper. I don't know much about Perl, but if it supports multiple
>> inheritance, would it be possible to make a class like "cmakebase" and
>> then have "cmake" and "cmakeninja" inherit from both "cmakebase" and
>> then "makefile" and "ninja" respectively?
>>
>>
>> [...]
> Ok, I have written a sample patch series (attached) that should make
> debhelper support cmake with the ninja backend.
>
> The chosen syntax is:
>
>    dh <target> --buildsystem=cmake+ninja
>
> A review and some testing would be appreciated.
>
> Thanks,
> ~Niels
>

Just tested it, and I like it.

One thing I did notice is that if your project doesn't have tests, you 
have to turn on nocheck when switching to cmake+ninja, because the ninja 
buildsystem fails if no "test" target exists (the makefile buildsystem 
seems to silently ignore the failure.) I don't think this is a 
deal-breaker, but it is something to keep in mind when switching to 
ninja. Perhaps this could be filed as a separate bug.

Thank you for the quick response! I hope to see this in debhelper soon.

Kyle

Message #32 received at 895044@bugs.debian.org (full text, mbox, reply):

From: Niels Thykier <niels@thykier.net>

To: Kyle Edwards <kyle.edwards@kitware.com>, 895044@bugs.debian.org

Subject: Re: [debhelper-devel] Bug#895044: debhelper: add support for Ninja with CMake

Date: Sat, 07 Apr 2018 05:30:00 +0000

Control: tags -1 pending

Kyle Edwards:
> [...]
> 
> Just tested it, and I like it.
> 

Thanks. :)

I am glad it worked. :)

> One thing I did notice is that if your project doesn't have tests, you
> have to turn on nocheck when switching to cmake+ninja, because the ninja
> buildsystem fails if no "test" target exists (the makefile buildsystem
> seems to silently ignore the failure.) I don't think this is a
> deal-breaker, but it is something to keep in mind when switching to
> ninja. Perhaps this could be filed as a separate bug.
> 

Mmm, the reason why make does not fail is that we have a crude (but
functional) way to tell if a makefile target exists.  But we do not have
that for ninja and honestly, I have no idea how we would make it either.

If you have an idea, I would be happy to look at implementing it.

> Thank you for the quick response! I hope to see this in debhelper soon.
> 
> Kyle

Indeed; merged for debhelper 11.2.

Thanks,
~Niels

Message #39 received at 895044-close@bugs.debian.org (full text, mbox, reply):

From: Niels Thykier <niels@thykier.net>

To: 895044-close@bugs.debian.org

Subject: Bug#895044: fixed in debhelper 11.2

Date: Sat, 07 Apr 2018 19:34:21 +0000

Source: debhelper
Source-Version: 11.2

We believe that the bug you reported is fixed in the latest version of
debhelper, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 895044@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Niels Thykier <niels@thykier.net> (supplier of updated debhelper package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Sat, 07 Apr 2018 19:23:32 +0000
Source: debhelper
Binary: debhelper dh-systemd
Architecture: source
Version: 11.2
Distribution: unstable
Urgency: medium
Maintainer: Debhelper Maintainers <debhelper@packages.debian.org>
Changed-By: Niels Thykier <niels@thykier.net>
Description:
 debhelper  - helper programs for debian/rules
 dh-systemd - debhelper add-on to handle systemd unit files - transitional pack
Closes: 894549 894573 894666 894835 894895 895011 895044
Changes:
 debhelper (11.2) unstable; urgency=medium
 .
   [ Niels Thykier ]
   * debhelper.7: Add a ~ to the suggested Build-Depends to ensure
     backports also work for debhelper with single integer versions.
     Thanks to Trent W. Buck for the suggestion.  (Closes: #894666)
   * makefile.pm: Use -Oline with make.  This avoids make mistaking
     a target name for a value for -O and should make build time
     outs less likely for "long" targets.  (Closes: #894573)
   * Dh_Lib.pm: Fix bug that make debhelper trip on packages with
     the version "0".  Thanks to Chris Lamb for reporting the bug
     plus debugging the issue.  (Closes: #894895)
   * Buildsystem.pm: Rewrite to support build systems that generate
     build files for another build system (without using
     inheritance).  This enables generator build systems to have
     multiple backends.
   * cmake.pm: Support ninja as alternative backend (by using the
     build system cmake+ninja).  Thanks to Kyle Edwards for the
     suggestion.  (Closes: #895044)
   * meson.pm: Rewrite as a generator build system with ninja as
     the only backend.
   * debhelper.7: Recommend packagers to use "debian/<pkg>.<file>"
     over "debian/<file>" in most cases.  Thanks to Johannes
     Schauer for the suggestion.
   * dh_usrlocal: Implement a simple guard for directories that
     will likely cause issues in the shell snippets.
   * dh_usrlocal: Use the new rules from Debian Policy 4.1.4 to
     determine the default ownership and mode for directories.
   * d/control: Bump Standards-Versions to 4.1.4.  Beyond the
     dh_usrlocal change listed above, no changes were required.
 .
   [ Nicolas Boulenguez ]
   * dh_installxfonts: Fix typo that causes a misc:Depends on
     non-existing xfont-utils.  (Closes: #894835)
   * dh_installwm.1: Document --all.  (Closes: #895011)
   * dh_usrlocal: Fix bug where the generated prerm script generated
     by dh_usrlocal could remove a directory directly in /usr/local.
     (Closes: #894549)
 .
   [ Mattia Rizzolo ]
   * Lower the version restrictions on dpkg and dpkg-dev.  They are not
     needed anymore, as R³ support is not implied anymore, but requires
     an environment variable to be passed.
Checksums-Sha1:
 71d3f409fbca3c667cfa0809447e658f4de008da 1681 debhelper_11.2.dsc
 9f3ad653d486721186056cc9b897c8b3f682bf97 455608 debhelper_11.2.tar.xz
 9ebc288ca16fd9aaad22de04e3e0f33d9de935ba 4546 debhelper_11.2_source.buildinfo
Checksums-Sha256:
 0da35b90fd65d17ef7a6ddaac472349e92ce6d3ffe6aa9163086add61a062cab 1681 debhelper_11.2.dsc
 bd726051ca3a488a3edaff925d9c5ca45c7a7b20dfcb62a381fb55b93ea2ee66 455608 debhelper_11.2.tar.xz
 6926e624cb4cff32507bf43e21d363a9e3206948ba18e30b96f93205545bd477 4546 debhelper_11.2_source.buildinfo
Files:
 849efbdc1e61126e1c410188f8e01dcd 1681 devel optional debhelper_11.2.dsc
 870ac4fd8278eaccbc3e276c0ba9fe5a 455608 devel optional debhelper_11.2.tar.xz
 28947d1df252254d30a65cb178d4d1e4 4546 devel optional debhelper_11.2_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEsxMaRR2/33ygW0GXBUu7n32AZEIFAlrJG3IACgkQBUu7n32A
ZEJwLg//Wev9BdvkpEVdzKToBn6DWlakupxmRBmVDXyK9DWgdurx7kuFrHrj/SzK
3Yg0PBjhEXuT+XyC773zIiFs43IXpp3bE1zhKF6+zY9wGiE3bhFJ7RKeoKQRUJ/T
ua/5GfezI5s0INmvr49nSb/LiGB66R4juThGfgyRKgnwFHOVGRgbXJ25mrQK5xO7
dosKEDluankJ1XIebKhQMnj41MVByMJqPecTtIZxoeTHTH0AGUMwlotv6xeY0WXz
SOKYUOgucwnj9kuyU0k7gK8jN1TnipZotQa3V5Q4ntVK/g2zg7sH+aPoNyH10GDH
FbZx7O2zgCrdqN3lzOsXGrZD7KShjSNwyjQNjprzbT3zXrEtuYQuSVcQRgBZ/N0F
nfF2nZzz289KSYJuGI9e6gKK/+PewV1UAfTiRa8rzhxjTSOsjVlG1NwEjfBFjXU3
ds894SmCZTvNkE68xmhMDawSXJLrULA6/zQgVNVVWDwUb1TYY+zrY8alCiE3+zNV
oMkR49RHrETE+39f7Ce7PDFCGVt+AhL9+YWigAje1ioJe87dRxRZnept8Uao7+tn
03r2BOW3hnC7IJL/p0yqoMBma+yZn+B/jeK/EwniAlaz2wNNraimmSe4FdW3hv7r
ValqKnhBZeUI+BuBxBOVveOWcJG424txnFWlqYFVCKn+wtVtFmA=
=UO0w
-----END PGP SIGNATURE-----

Message #44 received at 895044@bugs.debian.org (full text, mbox, reply):

From: Kyle Edwards <kyle.edwards@kitware.com>

To: Niels Thykier <niels@thykier.net>, 895044@bugs.debian.org

Subject: Re: [debhelper-devel] Bug#895044: debhelper: add support for Ninja with CMake

Date: Mon, 9 Apr 2018 11:08:22 -0400

On 04/07/2018 01:30 AM, Niels Thykier wrote:
> Thanks. :)
>
> I am glad it worked. :)

My VTK package is currently building with cmake+ninja in sid, and 
leaving cmake+makefile in the dust. I love it. Thank you so much!

> Mmm, the reason why make does not fail is that we have a crude (but
> functional) way to tell if a makefile target exists.  But we do not have
> that for ninja and honestly, I have no idea how we would make it either.
>
> If you have an idea, I would be happy to look at implementing it.
>

There are a couple options I can think of:

1. $ ninja -n test - this does a dry run. If the "test" target exists 
then it succeeds (regardless of whether or not the real target would 
have succeeded), otherwise it fails. Simple and elegant.
2. You could also manually parse build.ninja and its included files. 
Admittedly not as simple, but Ninja is intended to be machine-readable 
and fast, so it might not be as bad as parsing a Makefile.
2a. Alternatively, start by simply grepping build.ninja for "^build[ 
\t]+test:" or something similar. It might not include every corner case, 
but it would be a good start (more than we have right now), and I don't 
think it would yield any false positives. CMake takes a similar approach 
in trying to find #defines in header files - see 
https://gitlab.kitware.com/cmake/cmake/blob/master/Modules/FindBoost.cmake#L1291 
for an example.

Kyle

Send a report that this bug log contains spam.