Debian Bug report logs - #209008
debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Robert Jordens <jordens@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Date: Sat, 06 Sep 2003 22:53:56 +0200

Package: debian-policy
Version: 3.6.1.0
Severity: wishlist

Hi!

I'd like to propose the option "parallel=X" in DEB_BUILD_OPTIONS to allow
packages to build in parallel with X jobs thus enabling faster builds on
SMP machines for packages that support this.

Just setting "$(MAKE)=make -j3" in the environment won't work because
many packages break if built like that.
http://lists.debian.org/debian-policy/2001/debian-policy-200103/msg00368.html


I could find 4 packages that support 3 different interfaces for
accomplishing that.

gcc-snapshot and ardour look for "USE_NJOBS" in the environment and the
figure out themselves how many jobs to use:

ifneq ($(USE_NJOBS),)
   NJOBS := -j$(shell if [ -f /proc/cpuinfo ]; \
            then echo `cat /proc/cpuinfo | grep 'processor' | wc -l`; \
            else echo 1; fi)
endif

openoffice.org does:

# Pararell build - add -PPn into DEB_BUILD_OPTIONS, where n is the number of processes

glibc does:

ifneq (,$(findstring parallel,$(DEB_BUILD_OPTIONS)))
DEB_BUILD_OPTION_PARALLEL = yes
else
DEB_BUILD_OPTION_PARALLEL = no
endif

I propose the following:

--- policy.sgml.orig    2003-09-06 21:58:14.000000000 +0200
+++ policy.sgml 2003-09-06 22:07:04.000000000 +0200
@@ -6325,7 +6325,13 @@
                not be stripped from the binary during installation,
                so that debugging information may be included in the package.
            </item>
          </taglist>
+        <tag>parallel=N</tag>
+        <item>
+        This option means that the package should be built with N jobs in
+        parallel if it can be built in parallel, so that the build process
+        becomes faster on multiprocessor machines.
+        </item>
        </p>
  
        <p>
@@ -6349,6 +6355,19 @@
 ifeq (,$(findstring nostrip,$(DEB_BUILD_OPTIONS)))
 INSTALL_PROGRAM += -s
 endif
+
+ifneq (,$(findstring parallel,$(DEB_BUILD_OPTIONS)))
+    PARALLEL_JOBS := $(shell echo $(DEB_BUILD_OPTIONS) | \
+        sed -e 's/.*parallel=\([0-9]\+\).*/\1/')
+    ifeq ($(DEB_BUILD_OPTIONS),$(PARALLEL_JOBS))
+        PARALLEL_JOBS := $(shell if [ -f /proc/cpuinfo ]; \
+            then echo `cat /proc/cpuinfo | grep 'processor' | wc -l`; \
+            else echo 1; fi)
+    endif
+    NJOBS := -j$(PARALLEL_JOBS)
+endif
+MAKEFLAGS += $(NJOBS)
+
          </example>
        </p>

thus supporting the glibc's interface and the extension of explicitly setting
the number of jobs to be used.
 

	Robert.

-- System Information:
Debian Release: testing/unstable
Architecture: i386
Kernel: Linux schuh 2.5.72 #1 SMP Thu Jun 19 12:51:46 UTC 2003 i686
Locale: LANG=C, LC_CTYPE=en_US.UTF-8

-- no debconf information

Message #12 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@lucas-nussbaum.net>

To: 209008@bugs.debian.org

Subject: Re: Bug#209008: debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Date: Wed, 7 Mar 2007 18:08:52 +0100

Hi,

Bug #209008 proposed to have a common interface to tell packages to do
parallel building (make -j).

Not having this is a PITA for me, since I do rebuilds of the whole
Debian archive on clusters of machines, and my current total build time
is only dependant of the time taken to build the largest packages (see
slides 25-28 of [1] for details).

For some reason, the discussion that happened back in 2003 isn't logged
on the BTS, but can be read in [2]. Everyone seemed to agree on the
proposal, but it was discussed whether this should be implemented as
DEB_BUILD_OPTIONS="parallel=n" or DEB_BUILD_OPTIONS_PARALLEL=n.

It would be great if consensus could be reached early in the lenny release
cycle. A few packages already provide an interface to do parallel
building, and I'd like to work on making all of them switch to the same
interface.

Thank you,

[1] http://www-id.imag.fr/~nussbaum/fosdem07-gridqa-slides.pdf
[2] http://lists.debian.org/debian-policy/2003/09/threads.html#00079
-- 
| Lucas Nussbaum
| lucas@lucas-nussbaum.net   http://www.lucas-nussbaum.net/ |
| jabber: lucas@nussbaum.fr             GPG: 1024D/023B3F4F |

Message #17 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Mike Hommey <mh@glandium.org>

To: Lucas Nussbaum <lucas@lucas-nussbaum.net>, 209008@bugs.debian.org

Subject: Re: Bug#209008: debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Date: Wed, 7 Mar 2007 19:32:16 +0100

On Wed, Mar 07, 2007 at 06:08:52PM +0100, Lucas Nussbaum <lucas@lucas-nussbaum.net> wrote:
> Hi,
> 
> Bug #209008 proposed to have a common interface to tell packages to do
> parallel building (make -j).
> 
> Not having this is a PITA for me, since I do rebuilds of the whole
> Debian archive on clusters of machines, and my current total build time
> is only dependant of the time taken to build the largest packages (see
> slides 25-28 of [1] for details).
> 
> For some reason, the discussion that happened back in 2003 isn't logged
> on the BTS, but can be read in [2]. Everyone seemed to agree on the
> proposal, but it was discussed whether this should be implemented as
> DEB_BUILD_OPTIONS="parallel=n" or DEB_BUILD_OPTIONS_PARALLEL=n.

why not DEB_BUILD_OPTIONS=noparallel, like noopt ?

Mike

Message #22 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@lucas-nussbaum.net>

To: Mike Hommey <mh@glandium.org>, 209008@bugs.debian.org

Subject: Re: Bug#209008: debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Date: Wed, 7 Mar 2007 20:24:28 +0100

On 07/03/07 at 19:32 +0100, Mike Hommey wrote:
> On Wed, Mar 07, 2007 at 06:08:52PM +0100, Lucas Nussbaum <lucas@lucas-nussbaum.net> wrote:
> > Hi,
> > 
> > Bug #209008 proposed to have a common interface to tell packages to do
> > parallel building (make -j).
> > 
> > Not having this is a PITA for me, since I do rebuilds of the whole
> > Debian archive on clusters of machines, and my current total build time
> > is only dependant of the time taken to build the largest packages (see
> > slides 25-28 of [1] for details).
> > 
> > For some reason, the discussion that happened back in 2003 isn't logged
> > on the BTS, but can be read in [2]. Everyone seemed to agree on the
> > proposal, but it was discussed whether this should be implemented as
> > DEB_BUILD_OPTIONS="parallel=n" or DEB_BUILD_OPTIONS_PARALLEL=n.
> 
> why not DEB_BUILD_OPTIONS=noparallel, like noopt ?

Actually DEB_BUILD_OPTIONS="parallel=n" means: "build using n jobs in
parallel", not "don't build in parallel" (this is still the default, of
course)
-- 
| Lucas Nussbaum
| lucas@lucas-nussbaum.net   http://www.lucas-nussbaum.net/ |
| jabber: lucas@nussbaum.fr             GPG: 1024D/023B3F4F |

Message #27 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Bastian Blank <waldi@debian.org>

To: Lucas Nussbaum <lucas@lucas-nussbaum.net>, 209008@bugs.debian.org

Subject: Re: Bug#209008: debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Date: Thu, 8 Mar 2007 11:50:47 +0100

[Message part 1 (text/plain, inline)]

On Wed, Mar 07, 2007 at 06:08:52PM +0100, Lucas Nussbaum wrote:
> Bug #209008 proposed to have a common interface to tell packages to do
> parallel building (make -j).

You can't set a generic value.

For example: A machine with 6 cpus but only 256MiB ram. Building glibc
with -j6 is no problem. Building gcj-* with -j6 is not possible.

Bastian

-- 
You're dead, Jim.
		-- McCoy, "Amok Time", stardate 3372.7

[signature.asc (application/pgp-signature, inline)]

Message #32 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@lucas-nussbaum.net>

To: Bastian Blank <waldi@debian.org>, 209008@bugs.debian.org

Subject: Re: Bug#209008: debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Date: Thu, 8 Mar 2007 13:34:37 +0100

[Message part 1 (text/plain, inline)]

On 08/03/07 at 11:50 +0100, Bastian Blank wrote:
> On Wed, Mar 07, 2007 at 06:08:52PM +0100, Lucas Nussbaum wrote:
> > Bug #209008 proposed to have a common interface to tell packages to do
> > parallel building (make -j).
> 
> You can't set a generic value.
> 
> For example: A machine with 6 cpus but only 256MiB ram. Building glibc
> with -j6 is no problem. Building gcj-* with -j6 is not possible.

Such a machine looks a bit strange (6 CPUs vs 256 MB RAM). I think that
usually, modern SMPs have "enough" memory to handle what all their CPUs
can do. And the "common interface" could still be used to set a sensible,
conservative default.

Also, you say "Building gcj-* with -j6 is not possible". Do you mean
that the build would fail, or just that it would take a lot of time and
cause a lot of swapping ?
-- 
| Lucas Nussbaum
| lucas@lucas-nussbaum.net   http://www.lucas-nussbaum.net/ |
| jabber: lucas@nussbaum.fr             GPG: 1024D/023B3F4F |

[signature.asc (application/pgp-signature, inline)]

Message #37 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Loïc Minier <lool+debian@via.ecp.fr>

To: debian-policy@lists.debian.org, 209008@bugs.debian.org

Subject: Re: Bug#209008: debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Date: Thu, 8 Mar 2007 17:24:05 +0100

On Thu, Mar 08, 2007, Bastian Blank wrote:
> > Bug #209008 proposed to have a common interface to tell packages to do
> > parallel building (make -j).
> You can't set a generic value.
> 
> For example: A machine with 6 cpus but only 256MiB ram. Building glibc
> with -j6 is no problem. Building gcj-* with -j6 is not possible.

 We don't have a way to express RAM usage to build a package, AFAIK the
 only available limit is toexclude some packages on some buildds; we
 could do exactly the same in a world supporting parallel=n.  We could
 also tune the value on a machine like the one you describe.

-- 
Loïc Minier <lool@dooz.org>

Message #42 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Florian Weimer <fw@deneb.enyo.de>

To: Lucas Nussbaum <lucas@lucas-nussbaum.net>

Cc: 209008@bugs.debian.org, Bastian Blank <waldi@debian.org>

Subject: Re: Bug#209008: debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Date: Wed, 14 Mar 2007 20:54:45 +0100

* Lucas Nussbaum:

> Such a machine looks a bit strange (6 CPUs vs 256 MB RAM). I think that
> usually, modern SMPs have "enough" memory to handle what all their CPUs
> can do.

I'm not sure.  512 MB per execution unit is not too uncommen and may
cause problems for C++ packages (or MLton).

Message #47 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@lucas-nussbaum.net>

To: Florian Weimer <fw@deneb.enyo.de>

Cc: 209008@bugs.debian.org, Bastian Blank <waldi@debian.org>

Subject: Re: Bug#209008: debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Date: Wed, 14 Mar 2007 21:03:57 +0100

On 14/03/07 at 20:54 +0100, Florian Weimer wrote:
> * Lucas Nussbaum:
> 
> > Such a machine looks a bit strange (6 CPUs vs 256 MB RAM). I think that
> > usually, modern SMPs have "enough" memory to handle what all their CPUs
> > can do.
> 
> I'm not sure.  512 MB per execution unit is not too uncommen and may
> cause problems for C++ packages (or MLton).

Bastian's example is not 256 MB per CPU, it's 256 MB total.

But on such a system, you could still say: 

  "Well, I want to rebuild several packages, I know some of them might
  cause problems with only 512 MB per WU, so I'll build with parallel=2,
  not parallel=6, just to be on the safe side."

I've never said that I would like the default value for 'parallel' to be
the number of CPUs. This doesn't seem to be reasonable.
-- 
| Lucas Nussbaum
| lucas@lucas-nussbaum.net   http://www.lucas-nussbaum.net/ |
| jabber: lucas@nussbaum.fr             GPG: 1024D/023B3F4F |

Message #52 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Lucas Nussbaum <lucas@lucas-nussbaum.net>, 209008@bugs.debian.org

Subject: Re: Bug#209008: debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Date: Wed, 21 Mar 2007 05:52:05 +0200

On Wed, 2007-03-07 at 18:08:52 +0100, Lucas Nussbaum wrote:
> Bug #209008 proposed to have a common interface to tell packages to do
> parallel building (make -j).

> For some reason, the discussion that happened back in 2003 isn't logged
> on the BTS, but can be read in [2]. Everyone seemed to agree on the
> proposal, but it was discussed whether this should be implemented as
> DEB_BUILD_OPTIONS="parallel=n" or DEB_BUILD_OPTIONS_PARALLEL=n.

The first option needs more code to parse it. And I don't think it's a
good idea to allow it not taking a parameter, that will also increase
substantially the code to set the value, and make it quite unportable
(due to its need to get info from /proc or similar).

The person triggering the build is the one with enough information to
decide what value should be appropriate, and as a counter argument for
packages needing special amounts of memory/cpu we have similar global
overrides for build timeouts in the buildds.

There's also in use in some packages (most notably kernel-package) the
CONCURRENCY_LEVEL variable, but the name seems less consistent.

Personally I'd favour DEB_BUILD_OPTIONS_PARALLEL.

regards,
guillem

Message #57 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Wouter Verhelst <wouter@debian.org>

To: Guillem Jover <guillem@debian.org>, 209008@bugs.debian.org

Cc: Lucas Nussbaum <lucas@lucas-nussbaum.net>

Subject: Re: Bug#209008: debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Date: Wed, 21 Mar 2007 16:25:43 +0100

On Wed, Mar 21, 2007 at 05:52:05AM +0200, Guillem Jover wrote:
> Personally I'd favour DEB_BUILD_OPTIONS_PARALLEL.

Yes, I'll second that. It makes no sense at all to try to have the
package predict how much memory it will need for the build; this amount
of memory will be different depending on the architecture, the exact
versions of build-dependencies that are in use, and a gazillion other
unpredictable variables.

The only person able to figure out which values are sane and which
values aren't is the person running the build, if only by trial and
error. The best anyone else can do is an educated guess, which is not
something I would like in my packages.

-- 
<Lo-lan-do> Home is where you have to wash the dishes.
  -- #debian-devel, Freenode, 2004-09-22

Message #62 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Loïc Minier <lool@dooz.org>

To: 209008@bugs.debian.org

Subject: Honoring MAKEFLAGS instead of adding new env vars?

Date: Wed, 21 Mar 2007 17:57:57 +0100

        Hi,

 Can't we simply allow passing MAKEFLAGS in the env of debian/rules
 instead of defining new vars?

   Bye,
-- 
Loïc Minier

Message #67 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Loïc Minier <lool@dooz.org>, 209008@bugs.debian.org

Subject: Re: Bug#209008: Honoring MAKEFLAGS instead of adding new env vars?

Date: Wed, 21 Mar 2007 23:25:07 +0200

On Wed, 2007-03-21 at 17:57:57 +0100, Loïc Minier wrote:
>  Can't we simply allow passing MAKEFLAGS in the env of debian/rules
>  instead of defining new vars?

That does not help if upstream's build system is not based on make.

regards,
guillem

Message #72 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Marc 'HE' Brockschmidt <he@ftwca.de>

To: Guillem Jover <guillem@debian.org>

Cc: 209008@bugs.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>

Subject: Re: Bug#209008: debian-policy: [PROPOSAL] common interface for parallel building in DEB_BUILD_OPTIONS

Date: Sat, 24 Mar 2007 14:48:04 +0100

[Message part 1 (text/plain, inline)]

Guillem Jover <guillem@debian.org> writes:
> Personally I'd favour DEB_BUILD_OPTIONS_PARALLEL.

Ack. Automated guesses are always a possible source for problems, and in
this special case, the human behind the build process should know best
which number of parallel processes is useful [1]

Marc

Footnotes: 
[1]  The classic problematic case would be a CPU showing two cores due to
     HT and only little RAM, where two parallel g++ runs could easily
     trash the machine, while almost all other builds would show almost no
     improvement.
-- 
Fachbegriffe der Informatik - Einfach erklärt
171: PMPO
       Leistungsabgabe bei spontaner Verbrennung (Arndt Spelten)

[Message part 2 (application/pgp-signature, inline)]

Message #77 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Loïc Minier <lool@dooz.org>

To: 209008@bugs.debian.org

Subject: How should the build logs be handled?

Date: Tue, 10 Apr 2007 15:19:56 +0200

        Hi,

 I added support for DEB_BUILD_OPTIONS_PARALLEL to some big packages
 which are building multiple "flavors" of the software.  First, there's
 a choice of what to parallelize: do you parallelize ./configure runs
 and multiple builds when you have multiple configure + build runs to
 achieve?  Second, I noticed that the output wasn't very easy to read,
 especially with libtool based builds which produce several line of
 output and run multiple commands per object.

 So while I think it's a good thing to add such a support in packages
 for some use cases (perhaps archive rebuilds, repetitive builds of the
 same software after each commit etc.), I wonder how we will deal with
 build logs.

 Should we recommend that such builds have to save their build logs and
 output it at the end of the build?  e.g.:
   make >>debian/build.log || (cat debian/build.log; exit 1)
   cat debian/build.log

 This is what I used in my packages:
    MAKEFLAGS += $(if $(DEB_BUILD_OPTIONS_PARALLEL),-j2 $(DEB_BUILD_OPTIONS_PARALLEL))

 (It wont work with packages calling $(MAKE) -f debian/rules which should
 protect against adding another -j flag in the submake.)

   Bye,
-- 
Loïc Minier
"For subalterns, saying something intelligent is as risky as saying something
 stupid."

Message #82 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Aurelien Jarno <aurelien@aurel32.net>

To: 209008@bugs.debian.org

Subject: Re: How should the build logs be handled?

Date: Wed, 18 Apr 2007 20:33:49 +0200

Hi,

I have just noticed that that things are moving. Nice :)

First of all, I am personally in favor of DEB_BUILD_OPTIONS="parallel=n"
because it is consistent with all other options that can be set when
building a package. If the code to parse it is too long, it could be put
for example in debhelper.

On Tue, Apr 10, 2007 at 03:19:56PM +0200, Loïc Minier wrote:
>  I added support for DEB_BUILD_OPTIONS_PARALLEL to some big packages
>  which are building multiple "flavors" of the software.  First, there's
>  a choice of what to parallelize: do you parallelize ./configure runs
>  and multiple builds when you have multiple configure + build runs to
>  achieve?  Second, I noticed that the output wasn't very easy to read,
>  especially with libtool based builds which produce several line of
>  output and run multiple commands per object.

On glibc we are running configure normally, and then invoking the build
phase with make -j. We still build all flavour sequentially. I can't say
what is the best solution. Building all the flavours sequentially has
the advantage to fail faster if some code does not compile. Not really
useful on the build daemon, but that can be useful locally.

>  So while I think it's a good thing to add such a support in packages
>  for some use cases (perhaps archive rebuilds, repetitive builds of the
>  same software after each commit etc.), I wonder how we will deal with
>  build logs.
> 
>  Should we recommend that such builds have to save their build logs and
>  output it at the end of the build?  e.g.:
>    make >>debian/build.log || (cat debian/build.log; exit 1)
>    cat debian/build.log
> 
>  This is what I used in my packages:
>     MAKEFLAGS += $(if $(DEB_BUILD_OPTIONS_PARALLEL),-j2 $(DEB_BUILD_OPTIONS_PARALLEL))
> 
>  (It wont work with packages calling $(MAKE) -f debian/rules which should
>  protect against adding another -j flag in the submake.)

I should warn about using MAKEFLAGS. A lot of software do support using 
-j for the make all phase, but do not for the make install phase. In the
case of the glibc, using -j for the make install phase was causing a
build failure, rarely but enough often to bother us (and the SRM team).

That's why I would advise to call make -j explicitely in the parts that
have been well tested.

Bye,
Aurelien

-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32@debian.org         | aurelien@aurel32.net
   `-    people.debian.org/~aurel32 | www.aurel32.net

Message #87 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Loïc Minier <lool@dooz.org>

To: Aurelien Jarno <aurelien@aurel32.net>, 209008@bugs.debian.org

Subject: Re: Bug#209008: How should the build logs be handled?

Date: Thu, 19 Apr 2007 10:37:27 +0200

On Wed, Apr 18, 2007, Aurelien Jarno wrote:
> First of all, I am personally in favor of DEB_BUILD_OPTIONS="parallel=n"
> because it is consistent with all other options that can be set when
> building a package. If the code to parse it is too long, it could be put
> for example in debhelper.

 This is a compatibility line to convert parallel=n in DEB_BUILD_OPTIONS
 into DEB_BUILD_OPTIONS_PARALLEL (unless DEB_BUILD_OPTIONS_PARALLEL is
 already set):

 DEB_BUILD_OPTIONS_PARALLEL ?= $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))

 You can then use DEB_BUILD_OPTIONS_PARALLEL as follows:
 MAKEFLAGS += $(if $(DEB_BUILD_OPTIONS_PARALLEL),-j$(DEB_BUILD_OPTIONS_PARALLEL))

 Or obviously any other flags if you prefer not changing the -j flags for
 debian/rules itself or for all sub-make invocations, as you describe
 below.

> I should warn about using MAKEFLAGS. A lot of software do support using 
> -j for the make all phase, but do not for the make install phase. In the
> case of the glibc, using -j for the make install phase was causing a
> build failure, rarely but enough often to bother us (and the SRM team).
> That's why I would advise to call make -j explicitely in the parts that
> have been well tested.

 (Ack, but it's a general problem not limited to make install: the same
 could be said about the main build process which might not be safe for
 make -j, or the testsuite, or debian/rules itself.)

-- 
Loïc Minier

Message #92 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@lucas-nussbaum.net>

To: 209008@bugs.debian.org

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Mon, 25 Jun 2007 05:25:42 +0000

Hi,

I'm trying to summarize this bug:

- generally, everybody find the idea useful.

- the number of parallel jobs should be specified by the developer
  building the package. There's no way to automatically guess the value
  in a sensible way, since it doesn't depend only on the number of CPUs
  but also on the amount of RAM available.

- there's a problem with the logs, since building in parallel could make
  it harder to read them. The common case would be to use make -j, which
  does a good job at keeping things readable. But when another solution
  is used (like OpenOffice.org's build of subprojects in parallel), the
  maintainer has to implement a solution to avoid mixing unrelated
  outputs.

- there is no consensus on the name of the option. Solutions are:
  + use parallel=n (where n is the number of jobs) in DEB_BUILD_OPTIONS
  + use a specific environment variable DEB_BUILD_OPTIONS_PARALLEL

I personally prefer DEB_BUILD_OPTIONS="parallel=n".

DEB_BUILD_OPTIONS_PARALLEL=n is easier to parse, but Loic Minier already
gave us code to parse DEB_BUILD_OPTIONS=parallel=n:
NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))

Switching to several DEB_BUILD_OPTIONS_* options doesn't seem like a
good idea. It will require build tools to be modified to add all of
them, if we want to use them in cases where the environment is not
inherited (think of qemubuilder for example). The benefits are unclear,
and if DEB_BUILD_OPTIONS becomes too difficult to parse, this could
still be moved to debhelper.

Any arguments I missed?
-- 
| Lucas Nussbaum
| lucas@lucas-nussbaum.net   http://www.lucas-nussbaum.net/ |
| jabber: lucas@nussbaum.fr             GPG: 1024D/023B3F4F |

Message #97 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Peter Samuelson <peter@p12n.org>

To: 209008@bugs.debian.org

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Mon, 25 Jun 2007 14:45:51 -0500

[Message part 1 (text/plain, inline)]

[Lucas Nussbaum]
> - the number of parallel jobs should be specified by the developer
>   building the package. There's no way to automatically guess the
>   value in a sensible way, since it doesn't depend only on the number
>   of CPUs but also on the amount of RAM available.

Right.

> - there's a problem with the logs, since building in parallel could
>   make it harder to read them. The common case would be to use make
>   -j, which does a good job at keeping things readable. But when
>   another solution is used (like OpenOffice.org's build of
>   subprojects in parallel), the maintainer has to implement a
>   solution to avoid mixing unrelated outputs.

This problem is worth mentioning, but ultimately I think it's a
maintainer decision.  An out-of-order build log doesn't seem like an
important enough problem to spend too much maintainer time fixing.
When something FTB, it should generally be clear what failed, and if
not, you can always retry without parallel.

> - there is no consensus on the name of the option. Solutions are:
>   + use parallel=n (where n is the number of jobs) in DEB_BUILD_OPTIONS
>   + use a specific environment variable DEB_BUILD_OPTIONS_PARALLEL
> 
> I personally prefer DEB_BUILD_OPTIONS="parallel=n".

+1.  I currently support DEB_BUILD_OPTIONS=-j4 but I'd be happy to
change my rules file to support DEB_BUILD_OPTIONS=parallel=4.  And I
don't like the idea of a proliferation of environment variables either.

> DEB_BUILD_OPTIONS_PARALLEL=n is easier to parse, but Loic Minier already
> gave us code to parse DEB_BUILD_OPTIONS=parallel=n:
> NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))

This will only work if the options in DEB_BUILD_OPTIONS are
space-separated.  I generally comma-separate mine, as you don't have to
quote that to a shell.  This is something Policy should specify but
doesn't.  Since Policy is ambiguous, might I suggest:

  d_b_o:=$(shell echo "$$DEB_BUILD_OPTIONS"|sed 's/[^-_=[:alnum:]]/ /g'|tr a-z- A-Z_)
  $(foreach o, $(d_b_o), $(if $(findstring =,$o),$(eval DEB_BUILD_OPT_$o),$(eval DEB_BUILD_OPT_$o=1)))
  MAKE_-J += $(addprefix -j, $(DEB_BUILD_OPT_PARALLEL))

Then you use $(MAKE_-J) as a macro on specified $(MAKE) lines.  Yes,
the above is 3 lines of complexity, but in return you get this:

  ifndef DEB_BUILD_OPT_NOSTRIP
    INSTALL_FLAGS += -s
  endif

Compare to what Policy suggests:

  ifeq (,$(findstring nostrip,$(DEB_BUILD_OPTIONS)))
    INSTALL_FLAGS += -s
  endif

Maybe it's just me, but in Policy's example, I always have to think for
a moment to figure out when the conditional is true and when it is
false.  (And I have to cut and paste it from somewhere to get the
syntax right.)  Whereas in my example it is immediately obvious.

-- 
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/

[signature.asc (application/pgp-signature, inline)]

Message #102 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: Peter Samuelson <peter@p12n.org>

Cc: 209008@bugs.debian.org

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Wed, 04 Jul 2007 00:31:28 -0700

Okay, I think we have a consensus at this point that this is the right
thing to do and that adding another flag to DEB_BUILD_OPTIONS is the best
way of implementing it.  For the time being, I'd like to keep discussion
of the flag to use separate from the general discussion of how the tags
should be formatted and what Makefile code we should recommend to parse
them.

Now, we need a wording proposal, and it should include a sample
implementation, which we can then later adjust based on the resolution of
Bug#430649.  Here's a first cut; please everyone comment and fine-tune.

--- orig/policy.sgml
+++ mod/policy.sgml
@@ -6466,6 +6466,20 @@
 		not be stripped from the binary during installation,
 		so that debugging information may be included in the package.
 	    </item>
+	    <tag>parallel=n</tag>
+	    <item>
+		This string means that the package should be built using
+		up to <tt>n</tt> parallel processes if the package build
+		system supports this.  It is up to the package maintainer
+		to decide whether the package build times are long enough
+		and the package build system is robust enough to make
+		supporting parallel builds worthwhile.
+		<footnote>
+		    Packages built with <tt>make</tt> can often implement
+		    this by passing the <tt>-jn</tt> option to
+		    <tt>make</tt>.
+		</footnote>
+	     </item>
 	  </taglist>
 	</p>
 
@@ -6490,6 +6504,10 @@
 ifeq (,$(findstring nostrip,$(DEB_BUILD_OPTIONS)))
 INSTALL_PROGRAM += -s
 endif
+ifneq (,$(findstring parallel=,$(DEB_BUILD_OPTIONS)))
+NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
+MAKEFLAGS += -j$(NUMJOBS)
+endif
 	  </example>
 	</p>
 
-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>

Message #107 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Loïc Minier <lool@dooz.org>

To: Russ Allbery <rra@debian.org>, 209008@bugs.debian.org

Cc: Peter Samuelson <peter@p12n.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Wed, 4 Jul 2007 10:56:06 +0200

On Wed, Jul 04, 2007, Russ Allbery wrote:
> +ifneq (,$(findstring parallel=,$(DEB_BUILD_OPTIONS)))
> +NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
> +MAKEFLAGS += -j$(NUMJOBS)
> +endif

 Since this fails for DEB_BUILD_OPTIONS="x=1,parallel=2", please change
 the ifneq to use the same macro as in the jobs computation.  As another
 minor nitpick, I think ":=" would be nicer for NUMJOBS.

ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
NUMJOBS := $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
MAKEFLAGS += -j$(NUMJOBS)
endif

-- 
Loïc Minier

Message #112 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Peter Samuelson <peter@p12n.org>

To: Loïc Minier <lool@dooz.org>, 209008@bugs.debian.org

Cc: Russ Allbery <rra@debian.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Wed, 4 Jul 2007 04:15:05 -0500

[Message part 1 (text/plain, inline)]

[Loïc Minier]
> On Wed, Jul 04, 2007, Russ Allbery wrote:
> > +ifneq (,$(findstring parallel=,$(DEB_BUILD_OPTIONS)))
> > +NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
> > +MAKEFLAGS += -j$(NUMJOBS)
> > +endif
> 
>  Since this fails for DEB_BUILD_OPTIONS="x=1,parallel=2", please change
>  the ifneq to use the same macro as in the jobs computation.
> 
> ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
> NUMJOBS := $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
> MAKEFLAGS += -j$(NUMJOBS)
> endif

$(filter) filters on whitespace, so this still will not work for
"x=1,parallel=2".  My code, posted earlier, handles the (perhaps
common) case of comma-separated DEB_BUILD_OPTIONS.

Also, suggesting that one add this directly to MAKEFLAGS is probably
not great, as a lot of upstream Makefiles may not be -j-safe
everywhere.  This is true of one package I maintain, so I construct a
$(MAKE_-J) and pass it manually to the $(MAKE) targets that are
-j-safe, and not to the ones that aren't.
-- 
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/

[signature.asc (application/pgp-signature, inline)]

Message #117 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Peter Samuelson <peter@p12n.org>

To: Russ Allbery <rra@debian.org>

Cc: 209008@bugs.debian.org

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Wed, 4 Jul 2007 04:22:52 -0500

[Message part 1 (text/plain, inline)]

[Russ Allbery]
> @@ -6466,6 +6466,20 @@
>  		not be stripped from the binary during installation,
>  		so that debugging information may be included in the package.
>  	    </item>
> +	    <tag>parallel=n</tag>
> +	    <item>
> +		This string means that the package should be built using
> +		up to <tt>n</tt> parallel processes if the package build

As a matter of typography, I think the n should be <em> rather than <tt>.

> +		    Packages built with <tt>make</tt> can often implement
> +		    this by passing the <tt>-jn</tt> option to

Here too.  <tt>-j</tt><em>n</em>

> @@ -6490,6 +6504,10 @@
>  ifeq (,$(findstring nostrip,$(DEB_BUILD_OPTIONS)))
>  INSTALL_PROGRAM += -s
>  endif
> +ifneq (,$(findstring parallel=,$(DEB_BUILD_OPTIONS)))
> +NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
> +MAKEFLAGS += -j$(NUMJOBS)
> +endif
>  	  </example>
>  	</p>

As I mentioned in another message, this will only work for a
whitespace-separated DEB_BUILD_OPTIONS value.  Policy unfortunately
doesn't specify the delimiter to be used in DEB_BUILD_OPTIONS, and I
think a lot of people (including myself) use a comma.  I proposed a
makefile snippet earlier that works around this and also provides a
nicer interface for the rest of the makefile.

Aside from those issues, here's a +1 vote from a non-developer on your diff.
-- 
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/

[signature.asc (application/pgp-signature, inline)]

Message #122 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Loïc Minier <lool@dooz.org>

To: Peter Samuelson <peter@p12n.org>, 209008@bugs.debian.org

Cc: Russ Allbery <rra@debian.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Wed, 4 Jul 2007 14:45:35 +0200

On Wed, Jul 04, 2007, Peter Samuelson wrote:
> $(filter) filters on whitespace, so this still will not work for
> "x=1,parallel=2".  My code, posted earlier, handles the (perhaps
> common) case of comma-separated DEB_BUILD_OPTIONS.

 (It is because $(filter ) filters on whitespace that I fixed usage of
 findstring for consistency as the initial version mixed $(findstring )
 and $(filter ); I ignored comma support on purpose as the initial
 version didn't support it.)

 I personally don't care that much about comma support for parallel=n; I
 don't think a lot of people use commas, but the policy syntax will be
 debated in its own bug, so I see no need to impose my view or your in
 this bug.  [I found the comma-aware snippet heavier, but I'll implement
 it if support for commas gets more support (or becomes policy).]

> Also, suggesting that one add this directly to MAKEFLAGS is probably
> not great, as a lot of upstream Makefiles may not be -j-safe
> everywhere.  This is true of one package I maintain, so I construct a
> $(MAKE_-J) and pass it manually to the $(MAKE) targets that are
> -j-safe, and not to the ones that aren't.

 It's a good point to make, perhaps MAKEFLAGS should only be mentionned
 as a possibility instead of mentionning it in the make snippet.

 BTW, MAKEFLAGS also affects the make run for debian/rules, that is if
 you extend MAKEFLAGS, debian/rules must be -j-safe too.

-- 
Loïc Minier

Message #127 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Goswin von Brederlow <brederlo@informatik.uni-tuebingen.de>

To: Peter Samuelson <peter@p12n.org>

Cc: 209008@bugs.debian.org

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Wed, 25 Jul 2007 10:23:03 +0200

Peter Samuelson <peter@p12n.org> writes:

> [Lucas Nussbaum]
>> - the number of parallel jobs should be specified by the developer
>>   building the package. There's no way to automatically guess the
>>   value in a sensible way, since it doesn't depend only on the number
>>   of CPUs but also on the amount of RAM available.
>
> Right.

Which also means the value should differ between package. So one can't
just put one value into the .${SHELL}rc but it needs to be somewhat
specific to the package.

My suggestion therefore remains to query a script about the
parallelism to use given a few hints (language used, rough memory
estimate, ...).

The source would pass the hints to the script, the developer building
the package sets some boundaries in /etc/<script>rc and/or
~/.<script>rc and the script just returns the parallelism to use.

MfG
        Goswin

Message #132 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Matthias Klose <doko@cs.tu-berlin.de>

To: 209008@bugs.debian.org

Cc: Lucas Nussbaum <lucas@lucas-nussbaum.net>, Peter Samuelson <peter@p12n.org>

Subject: Re: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Fri, 27 Jul 2007 16:56:14 +0200

The proposed policy suggests "that the package should be built using
up to <tt>n</tt> parallel processes". Please clarify:

 - The package should not build using more than <n> processes. If
   parallel=1 is used, the build should be sequential. This would
   give buildd maintainers some way to control the build process.

 - What to do if that option is not present? Should a package be
   allowed to build in parallel anyway, determing the number of
   processes itself, or should it be a sequential build?

Message #137 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Wouter Verhelst <wouter@debian.org>

To: Matthias Klose <doko@cs.tu-berlin.de>, 209008@bugs.debian.org

Cc: Lucas Nussbaum <lucas@lucas-nussbaum.net>, Peter Samuelson <peter@p12n.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Fri, 27 Jul 2007 23:35:14 +0200

On Fri, Jul 27, 2007 at 04:56:14PM +0200, Matthias Klose wrote:
>  - What to do if that option is not present? Should a package be
>    allowed to build in parallel anyway, determing the number of
>    processes itself, or should it be a sequential build?

I think it should behave as is currently required then; that is to say,
make it a sequential build.

-- 
<Lo-lan-do> Home is where you have to wash the dishes.
  -- #debian-devel, Freenode, 2004-09-22

Message #142 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@lucas-nussbaum.net>

To: Wouter Verhelst <wouter@debian.org>

Cc: Matthias Klose <doko@cs.tu-berlin.de>, 209008@bugs.debian.org, Peter Samuelson <peter@p12n.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Mon, 30 Jul 2007 18:27:50 +0200

On 27/07/07 at 23:35 +0200, Wouter Verhelst wrote:
> On Fri, Jul 27, 2007 at 04:56:14PM +0200, Matthias Klose wrote:
> >  - What to do if that option is not present? Should a package be
> >    allowed to build in parallel anyway, determing the number of
> >    processes itself, or should it be a sequential build?
> 
> I think it should behave as is currently required then; that is to say,
> make it a sequential build.

Agreed.
-- 
| Lucas Nussbaum
| lucas@lucas-nussbaum.net   http://www.lucas-nussbaum.net/ |
| jabber: lucas@nussbaum.fr             GPG: 1024D/023B3F4F |

Message #147 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Matthias Klose <doko@cs.tu-berlin.de>

To: Lucas Nussbaum <lucas@lucas-nussbaum.net>

Cc: Wouter Verhelst <wouter@debian.org>, 209008@bugs.debian.org, Peter Samuelson <peter@p12n.org>, Loic Minier <lool@dooz.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Sat, 4 Aug 2007 11:23:15 +0200

Lucas Nussbaum writes:
> On 27/07/07 at 23:35 +0200, Wouter Verhelst wrote:
> > On Fri, Jul 27, 2007 at 04:56:14PM +0200, Matthias Klose wrote:
> > >  - What to do if that option is not present? Should a package be
> > >    allowed to build in parallel anyway, determing the number of
> > >    processes itself, or should it be a sequential build?
> > 
> > I think it should behave as is currently required then; that is to say,
> > make it a sequential build.

Updated the Makefile example to work with whitespace and comma
separated values, without using the shell:

ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
  COMMA = ,
  NUMJOBS := $(patsubst parallel=%,%,$(filter parallel=%,$(subst $(COMMA), ,$(DEB_BUILD_OPTIONS))))
  MAKEFLAGS = -j $(NUMJOBS)
endif

Message #152 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Loïc Minier <lool@dooz.org>

To: Matthias Klose <doko@cs.tu-berlin.de>, 209008@bugs.debian.org

Cc: Lucas Nussbaum <lucas@lucas-nussbaum.net>, Wouter Verhelst <wouter@debian.org>, Peter Samuelson <peter@p12n.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Sat, 4 Aug 2007 18:21:59 +0200

On Sat, Aug 04, 2007, Matthias Klose wrote:
> Updated the Makefile example to work with whitespace and comma
> separated values, without using the shell:
> ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))

 I think this doesn't work as the test in the ifneq doesn't match a
 DEB_BUILD_OPTIONS with commas; instead, this is perhaps a simple way to
 support commas between DEB_BUILD_OPTIONS:

, := ,
DEB_BUILD_OPTIONS := $(subst $(,), ,$(DEB_BUILD_OPTIONS))

 and then parse DEB_BUILD_OPTIONS as proposed already:

ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
NUMJOBS := $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
MAKEFLAGS += -j$(NUMJOBS)
endif

 But perhaps the biggest problem is that this would block us from ever
 allowing commas as part of the DEB_BUILD_OPTIONS, for example to pass
 LDFLAGS=-Wl,something (which would work if only spaces are allowed).

-- 
Loïc Minier

Message #157 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Peter Samuelson <peter@p12n.org>

To: Loïc Minier <lool@dooz.org>

Cc: Matthias Klose <doko@cs.tu-berlin.de>, 209008@bugs.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>, Wouter Verhelst <wouter@debian.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Sat, 4 Aug 2007 12:01:57 -0500

[Message part 1 (text/plain, inline)]

[Loïc Minier]
>  But perhaps the biggest problem is that this would block us from ever
>  allowing commas as part of the DEB_BUILD_OPTIONS, for example to pass
>  LDFLAGS=-Wl,something (which would work if only spaces are allowed).

Yeah, that's a good point.  It may be best to amend Policy to require
space-separated options.  This makes command-line usage slightly more
annoying, but I'd support it.  Anything in Policy that clarifies the
syntax of DEB_BUILD_OPTIONS would be a Good Thing.

I still like my makefile snippet, posted earlier, which parses
DEB_BUILD_OPTIONS and sets a DEB_BUILD_OPT_FOO for every "foo" word.
It allows you to use 'ifdef' in the rest of debian/rules, which is much
more natural than ifneq(...) or ifeq(...) with the empty string.
-- 
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/

[signature.asc (application/pgp-signature, inline)]

Message #162 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Matthias Klose <doko@cs.tu-berlin.de>

To: Loïc Minier <lool@dooz.org>

Cc: 209008@bugs.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>, Wouter Verhelst <wouter@debian.org>, Peter Samuelson <peter@p12n.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Sat, 4 Aug 2007 21:19:56 +0200

Loïc Minier writes:
> On Sat, Aug 04, 2007, Matthias Klose wrote:
> > Updated the Makefile example to work with whitespace and comma
> > separated values, without using the shell:
> > ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
> 
>  I think this doesn't work as the test in the ifneq doesn't match a
>  DEB_BUILD_OPTIONS with commas; instead, this is perhaps a simple way to
>  support commas between DEB_BUILD_OPTIONS:
> 
> , := ,
> DEB_BUILD_OPTIONS := $(subst $(,), ,$(DEB_BUILD_OPTIONS))
> 
>  and then parse DEB_BUILD_OPTIONS as proposed already:
> 
> ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
> NUMJOBS := $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
> MAKEFLAGS += -j$(NUMJOBS)
> endif
> 
>  But perhaps the biggest problem is that this would block us from ever
>  allowing commas as part of the DEB_BUILD_OPTIONS, for example to pass
>  LDFLAGS=-Wl,something (which would work if only spaces are allowed).

do it without changing DEB_BUILD_OPTIONS:

, := ,
ifneq (,$(filter parallel=%,$(subst $(,), ,$(DEB_BUILD_OPTIONS))))
  NUMJOBS := $(patsubst parallel=%,%,$(filter parallel=%,$(subst $(,), ,$(DEB_BUILD_OPTIONS))))
  MAKEFLAGS += -j$(NUMJOBS)
endif


but I agree that deprecating the use of comma as a separator in
DEB_BUILD_OPTIONS would be the best thing (or options containing
commata must be separated by whitespace, then simple things like
nocheck,parallel=2,nostrip still work).

  Matthias

Message #167 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Matthias Klose <doko@cs.tu-berlin.de>

To: Loïc Minier <lool@dooz.org>

Cc: 209008@bugs.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>, Wouter Verhelst <wouter@debian.org>, Peter Samuelson <peter@p12n.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Sat, 4 Aug 2007 21:43:21 +0200

Loïc Minier writes:
> On Sat, Aug 04, 2007, Matthias Klose wrote:
> > do it without changing DEB_BUILD_OPTIONS:
> 
>  What's the gain?  I only see the duplication of the $(subst ) foo, and
>  the risk to forget that DEB_BUILD_OPTIONS might be comma-separated
>  later in debian/rules.

your own argument: expect to have commata in some DEB_BUILD_OPTIONS.

Message #172 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Loïc Minier <lool@dooz.org>

To: Matthias Klose <doko@cs.tu-berlin.de>

Cc: 209008@bugs.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>, Wouter Verhelst <wouter@debian.org>, Peter Samuelson <peter@p12n.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Sat, 4 Aug 2007 21:41:08 +0200

On Sat, Aug 04, 2007, Matthias Klose wrote:
> do it without changing DEB_BUILD_OPTIONS:

 What's the gain?  I only see the duplication of the $(subst ) foo, and
 the risk to forget that DEB_BUILD_OPTIONS might be comma-separated
 later in debian/rules.

-- 
Loïc Minier

Message #177 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Loïc Minier <lool@dooz.org>

To: Matthias Klose <doko@cs.tu-berlin.de>

Cc: 209008@bugs.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>, Wouter Verhelst <wouter@debian.org>, Peter Samuelson <peter@p12n.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Sat, 4 Aug 2007 22:21:09 +0200

On Sat, Aug 04, 2007, Matthias Klose wrote:
> your own argument: expect to have commata in some DEB_BUILD_OPTIONS.

 But isn't that in contradiction with splitting arguments over comma?

 Your snippet would be able to parse parallel=n but the trouble would
 come with trying to extract LDFLAGS from such DEB_BUILD_OPTIONS:
    LDFLAGS=-Wl,--as-needed,parallel=2,debug
    LDFLAGS=-Wl,-T,link,parallel=2,debug

 (how would you tell where the LDFLAGS end?)

-- 
Loïc Minier

Message #180 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Robert Millan <rmh@aybabtu.com>

To: Loïc Minier <lool@dooz.org>

Cc: Matthias Klose <doko@cs.tu-berlin.de>, 209008@bugs.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>, Wouter Verhelst <wouter@debian.org>, Peter Samuelson <peter@p12n.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Sun, 2 Sep 2007 12:44:06 +0200

I think you're overengineering.  There are basicaly two use cases:

  a- Developer building a package for a specific purpose (e.g. testing
     an improvement), wanting to speed up this specific build.

  b- Buildd admin wanting to speed up build of the overall archive [1].

(a) can be archieved simply by adding a flag to dpkg-buildpackage, such
that when used it will pass the corresponding -j flag to make when invoking
debian/rules (for consistency, it could also be -j) [2] [4].

(b) can't be archieved easily no matter what we do.  If we make -j NCPUs
implicit, lots of packages will break.  If we design an interface
such that each maintainer has to add cruft to her debian/rules to support
parallel builds, it will take ages anyway untill all maintainers add their
cruft.

OTOH, just by archieving (a) we automaticaly get developers to, in their own
interest, start using this feature and report (non-RC) bugs when it fails [5],
just because they want (like I do) to scratch their own itches.  This will
gradualy improve parallel build support in a true bazaar fashion, untill we
reach a point in which we can declare it a release requirement [3], hence
obtaining (b).

[1] yes, this means build logs become ugly and unreadable, but people can still
    rebuild if they need a sane log, just like they do for nostrip,noopt.
    buildds could even do that when FTBFS is found.  I think it's a reasonable
    price to pay (specialy since the trend for multi-core CPUs is so
    stablished now).

[2] true, this doesn't maximize the benefit for non-make packages, but:
    - it does in the debian/rules part (which is very significant in multi-build
      packages)
    - the small proportion of non-make packages makes this benefit marginal
    - *then* it might be useful to add the proposed cruft, at which point it
      could co-exist with the make -j flag which is needed for debian/rules no
      matter what.

[3] if a package is so braindamaged that parallel builds can't be supported,
    there shouldn't be any problem with explicitly disabling/overriding -j in
    $(MAKE) recursion (does MAKE+=-j1 suffice?)

[4] #355654 would also archieve that, although I think -j would be better.

[5] to us or to upstream;  it doesn't necessarily mean more work in our side.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Message #183 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Robert Millan <rmh@aybabtu.com>

To: Loïc Minier <lool@dooz.org>

Cc: Matthias Klose <doko@cs.tu-berlin.de>, 209008@bugs.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>, Wouter Verhelst <wouter@debian.org>, Peter Samuelson <peter@p12n.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Mon, 3 Sep 2007 12:10:27 +0200

On Sun, Sep 02, 2007 at 12:44:06PM +0200, Robert Millan wrote:
> 
> I think you're overengineering.  There are basicaly two use cases:
> 
>   a- Developer building a package for a specific purpose (e.g. testing
>      an improvement), wanting to speed up this specific build.
> 
>   b- Buildd admin wanting to speed up build of the overall archive [1].
> 
> (a) can be archieved simply by adding a flag to dpkg-buildpackage, such
> that when used it will pass the corresponding -j flag to make when invoking
> debian/rules (for consistency, it could also be -j) [2] [4].

Actually, you might be interested in knowing that this is possible right now,
without any change in dpkg-buildpackage, by using command injection via
gain-root-command:

  dpkg-buildpackage -r"fakeroot make -j 2 -f"

Would still be nice to do this a bit more cleanly ;-).  I'll send a patch to
dpkg maintainers.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Message #186 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Robert Millan <rmh@aybabtu.com>

To: Loïc Minier <lool@dooz.org>

Cc: Matthias Klose <doko@cs.tu-berlin.de>, 209008@bugs.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>, Wouter Verhelst <wouter@debian.org>, Peter Samuelson <peter@p12n.org>

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Mon, 3 Sep 2007 12:37:05 +0200

On Mon, Sep 03, 2007 at 12:10:27PM +0200, Robert Millan wrote:
> 
> Would still be nice to do this a bit more cleanly ;-).  I'll send a patch to
> dpkg maintainers.

See bug #440636

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Message #191 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Peter Samuelson <peter@p12n.org>

To: 209008@bugs.debian.org

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Mon, 3 Sep 2007 09:31:33 -0500

[Message part 1 (text/plain, inline)]

[Robert Millan]
> Actually, you might be interested in knowing that this is possible right now,
> without any change in dpkg-buildpackage, by using command injection via
> gain-root-command:
> 
>   dpkg-buildpackage -r"fakeroot make -j 2 -f"

I doubt that works in the general case, only if the time-consuming and
parallelizable part of the build is done in the binary-* targets
instead of the build-* targets.  (The build and build-* targets are not
run as root.)  Which should be relatively rare.
-- 
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/

[signature.asc (application/pgp-signature, inline)]

Message #194 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Robert Millan <rmh@aybabtu.com>

To: Peter Samuelson <peter@p12n.org>, 209008@bugs.debian.org

Subject: Re: Bug#209008: parallel building: DEB_BUILD_OPTIONS or DEB_BUILD_OPTIONS_PARALLEL

Date: Tue, 4 Sep 2007 18:21:41 +0200

On Mon, Sep 03, 2007 at 09:31:33AM -0500, Peter Samuelson wrote:
> 
> [Robert Millan]
> > Actually, you might be interested in knowing that this is possible right now,
> > without any change in dpkg-buildpackage, by using command injection via
> > gain-root-command:
> > 
> >   dpkg-buildpackage -r"fakeroot make -j 2 -f"
> 
> I doubt that works in the general case, only if the time-consuming and
> parallelizable part of the build is done in the binary-* targets
> instead of the build-* targets.  (The build and build-* targets are not
> run as root.)  Which should be relatively rare.

Ah, right.  I didn't quite think about it.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Message #197 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Robert Millan <rmh@aybabtu.com>

To: 209008@bugs.debian.org

Subject: Fwd: Bug#440636 closed by Raphael Hertzog <hertzog@debian.org> (Bug#440636: fixed in dpkg 1.14.7~newshlib)

Date: Tue, 25 Sep 2007 20:31:09 +0200

FYI

On Tue, Sep 25, 2007 at 07:24:52AM +0000, Debian Bug Tracking System wrote:
>  dpkg (1.14.7~newshlib) experimental; urgency=low
>  [...]
>    * dpkg-buildpackage accepts a -j<n> option now which will set
>      MAKEFLAGS(-j<n>) and DEB_BUILD_OPTIONS(parallel=<n>) accordingly.
>      parallel=<n> in DEB_BUILD_OPTIONS will be passed to MAKEFLAGS as
>      well. Based on an idea by Robert Millan. Closes: #440636

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Message #202 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: 209008@bugs.debian.org, 430649@bugs.debian.org

Subject: New proposed wording for DEB_BUILD_OPTIONS

Date: Sun, 30 Dec 2007 19:23:58 -0800

Okay, here's a revised proposal to address both Bug#209008 (parallel) and
Bug#430649 (DEB_BUILD_OPTIONS parsing).  This proposal does the following:

 * Standardizes on space as the separator for DEB_BUILD_OPTIONS.  This
   only affects what people can pass into debian/rules, not any existing
   packages.  I think all existing packages use parsing methods that would
   work with either, and that's what Policy has previously recommended.  I
   think the argument that we may want to pass values including commas to
   an option is persuasive.

 * Adds parallel=n with wording tweaks based on subsequent discussion,
   including handling the case where the build system supports some
   concurrency but not as much as was requested.  Do people still feel
   that we need to explicitly say that, in the absence of this option,
   packages should be built one process at a time?  Note that supporting
   DEB_BUILD_OPTS at all is only recommended.

 * Moves the whole section about DEB_BUILD_OPTS to a sub-section of the
   section on debian/rules and out of the section on binaries, leaving a
   pointer behind.  There are flags that do things other than affect the
   content of binaries, and I think this is a more intuitive place to look
   for it.

I left the MAKEFLAGS bit in the example for parallel since that example
clearly says that it's only an example and will need modification for the
needs of a specific package.  The way INSTALL is handled in the example
that was already there similarliy won't work for all upstream build
systems.

Comments?  Seconds?

--- orig/policy.sgml
+++ mod/policy.sgml
@@ -1987,6 +1987,104 @@
 	  or system information; the GNU style variables should be
 	  used for that.
 	</p>
+
+	<sect id="debianrules-options">
+	  <heading><file>debian/rules</file> and
+	    <tt>DEB_BUILD_OPTIONS</tt></heading>
+
+	  <p>
+	    Supporting the standardized environment variable
+	    <tt>DEB_BUILD_OPTIONS</tt> is recommended.	This variable can
+	    contain several flags to change how a package is compiled and
+	    built.  Each flag must be in the form <var>flag</var> or
+	    <var>flag</var>=<var>options</var>.	 If multiple flags are
+	    given, they must be separated by whitespace.<footnote>
+	      Some packages support any delimiter, but whitespace is the
+	      easiest to parse inside a makefile and avoids ambiguity with
+	      flag values that contain commas.
+	    </footnote>
+	    <var>flag</var> must consist only of lowercase letters
+	    (<tt>a-z</tt>), numbers (<tt>0-9</tt>), and the characters
+	    <tt>-</tt> and <tt>_</tt> (hyphen and underscore).
+	    <var>options</var> must not contain whitespace.  The same
+	    tag should not be given multiple times with conflicting
+	    values.  Package maintainers may assume that
+	    <tt>DEB_BUILD_OPTIONS</tt> will not contain conflicting tags.
+	  </p>
+
+	  <p>
+	    The meaning of the following tags has been standardized:
+	    <taglist>
+	      <tag>noopt</tag>
+	      <item>
+		  The presence of this tag means that the package should
+		  be compiled with a minimum of optimization.  For C
+		  programs, it is best to add <tt>-O0</tt> to
+		  <tt>CFLAGS</tt> (although this is usually the default).
+		  Some programs might fail to build or run at this level
+		  of optimization; it may be necessary to use
+		  <tt>-O1</tt>, for example.
+	      </item>
+	      <tag>nostrip</tag>
+	      <item>
+		  This tag means that the debugging symbols should not be
+		  stripped from the binary during installation, so that
+		  debugging information may be included in the package.
+	      </item>
+	      <tag>parallel=n</tag>
+	      <item>
+		  This tag means that the package should be built using up
+		  to <tt>n</tt> parallel processes if the package build
+		  system supports this.<footnote>
+		      Packages built with <tt>make</tt> can often implement
+		      this by passing the <tt>-j</tt><var>n</var> option to
+		      <tt>make</tt>.
+		  </footnote>
+		  If the package build system does not support parallel
+		  builds, this string must be ignored.  If the package
+		  build system only supports a lower level of concurrency
+		  than <var>n</var>, the package should be built using as
+		  many parallel processes as the package build system
+		  supports.  It is up to the package maintainer to decide
+		  whether the package build times are long enough and the
+		  package build system is robust enough to make supporting
+		  parallel builds worthwhile.
+	       </item>
+	    </taglist>
+	  </p>
+
+	  <p>
+	    Unknown flags must be ignored by <file>debian/rules</file>.
+	  </p>
+
+	  <p>
+	    The following makefile snippet is an example of how one may
+	    implement the build options; you will probably have to
+	    massage this example in order to make it work for your
+	    package.
+	    <example compact="compact">
+CFLAGS = -Wall -g
+INSTALL = install
+INSTALL_FILE    = $(INSTALL) -p    -o root -g root  -m  644
+INSTALL_PROGRAM = $(INSTALL) -p    -o root -g root  -m  755
+INSTALL_SCRIPT  = $(INSTALL) -p    -o root -g root  -m  755
+INSTALL_DIR     = $(INSTALL) -p -d -o root -g root  -m  755
+
+ifneq (,$(filter noopt,$(DEB_BUILD_OPTIONS)))
+    CFLAGS += -O0
+else
+    CFLAGS += -O2
+endif
+ifeq (,$(filter nostrip,$(DEB_BUILD_OPTIONS)))
+    INSTALL_PROGRAM += -s
+endif
+ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
+    NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
+    MAKEFLAGS += -j$(NUMJOBS)
+endif
+	    </example>
+	  </p>
+	</sect>
       </sect>
 
 <!-- FIXME: section pkg-srcsubstvars is the same as substvars -->
@@ -6439,58 +6537,12 @@
 
 	<p>
 	  Although binaries in the build tree should be compiled with
-	  debugging information by default, it can often be difficult
-	  to debug programs if they are also subjected to compiler
-	  optimization.  For this reason, it is recommended to support
-	  the standardized environment
-	  variable <tt>DEB_BUILD_OPTIONS</tt>.  This variable can
-	  contain several flags to change how a package is compiled
-	  and built.
-	</p>
-
-	<p>
-	  <taglist>
-	    <tag>noopt</tag>
-	    <item>
-		The presence of this string means that the package
-		should be compiled with a minimum of optimization.
-		For C programs, it is best to add <tt>-O0</tt>
-		to <tt>CFLAGS</tt> (although this is usually the
-		default).  Some programs might fail to build or run at
-		this level of optimization; it may be necessary to
-		use <tt>-O1</tt>, for example.
-	    </item>
-	    <tag>nostrip</tag>
-	    <item>
-		This string means that the debugging symbols should
-		not be stripped from the binary during installation,
-		so that debugging information may be included in the package.
-	    </item>
-	  </taglist>
-	</p>
-
-	<p>
-	  The following makefile snippet is an example of how one may
-          implement the build options; you will probably have to
-          massage this example in order to make it work for your
-          package.
- 	  <example compact="compact">
-CFLAGS = -Wall -g
-INSTALL = install
-INSTALL_FILE    = $(INSTALL) -p    -o root -g root  -m  644
-INSTALL_PROGRAM = $(INSTALL) -p    -o root -g root  -m  755
-INSTALL_SCRIPT  = $(INSTALL) -p    -o root -g root  -m  755
-INSTALL_DIR     = $(INSTALL) -p -d -o root -g root  -m  755
-
-ifneq (,$(findstring noopt,$(DEB_BUILD_OPTIONS)))
-CFLAGS += -O0
-else
-CFLAGS += -O2
-endif
-ifeq (,$(findstring nostrip,$(DEB_BUILD_OPTIONS)))
-INSTALL_PROGRAM += -s
-endif
-	  </example>
+	  debugging information by default, it can often be difficult to
+	  debug programs if they are also subjected to compiler
+	  optimization.  For this reason, it is recommended to support the
+	  standardized environment variable <tt>DEB_BUILD_OPTIONS</tt>
+	  (see <ref id="debianrules-options">).  This variable can contain
+	  several flags to change how a package is compiled and built.
 	</p>
 
 	<p>

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>

Message #207 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Robert Millan <rmh@aybabtu.com>

To: Russ Allbery <rra@debian.org>, 209008@bugs.debian.org

Cc: 430649@bugs.debian.org

Subject: Re: Bug#209008: New proposed wording for DEB_BUILD_OPTIONS

Date: Mon, 31 Dec 2007 10:46:04 +0100

On Sun, Dec 30, 2007 at 07:23:58PM -0800, Russ Allbery wrote:
> Okay, here's a revised proposal to address both Bug#209008 (parallel) and
> Bug#430649 (DEB_BUILD_OPTIONS parsing).  This proposal does the following:
> 
>  * Standardizes on space as the separator for DEB_BUILD_OPTIONS.  This
>    only affects what people can pass into debian/rules, not any existing
>    packages.  I think all existing packages use parsing methods that would
>    work with either, and that's what Policy has previously recommended.  I
>    think the argument that we may want to pass values including commas to
>    an option is persuasive.
> 
>  * Adds parallel=n with wording tweaks based on subsequent discussion,
>    including handling the case where the build system supports some
>    concurrency but not as much as was requested.  Do people still feel
>    that we need to explicitly say that, in the absence of this option,
>    packages should be built one process at a time?  Note that supporting
>    DEB_BUILD_OPTS at all is only recommended.
> 
>  * Moves the whole section about DEB_BUILD_OPTS to a sub-section of the
>    section on debian/rules and out of the section on binaries, leaving a
>    pointer behind.  There are flags that do things other than affect the
>    content of binaries, and I think this is a more intuitive place to look
>    for it.
> 
> I left the MAKEFLAGS bit in the example for parallel since that example
> clearly says that it's only an example and will need modification for the
> needs of a specific package.  The way INSTALL is handled in the example
> that was already there similarliy won't work for all upstream build
> systems.
> 
> Comments?  Seconds?

The majority of packages already supports parallel builds by simply passing
the appropiate -j flag to make.  Not that I object to a "parallel" parameter
since it can bring some benefits in very specific situations, but please make
sure this doesn't hinder what is already working.  In particular:

- It'd be good if it required that packages do not disable parallel flags in
  make in case they're already present (dpkg-buildpackage -jN), even if the
  parallel=N parameter is not present at all.

- I think it'd make sense to add a "should" requirement that packages allow any
  amount of parallelisation.  This requirement wouldn't really be excessive, I
  think.  It's just a matter of writing Makefiles properly by defining the
  right targets and dependencies; something that's easily archieveable when
  people remove bad habits like assuming make processes dependencies in a
  particular order, etc.

I can prepare a patch (based on yours) if you like.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Message #212 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: 209008@bugs.debian.org, 430649@bugs.debian.org

Subject: Re: Bug#209008: New proposed wording for DEB_BUILD_OPTIONS

Date: Mon, 31 Dec 2007 11:24:45 -0800

Robert Millan <rmh@aybabtu.com> writes:

> The majority of packages already supports parallel builds by simply
> passing the appropiate -j flag to make.  Not that I object to a
> "parallel" parameter since it can bring some benefits in very specific
> situations, but please make sure this doesn't hinder what is already
> working.  In particular:
>
> - It'd be good if it required that packages do not disable parallel flags in
>   make in case they're already present (dpkg-buildpackage -jN), even if the
>   parallel=N parameter is not present at all.

dpkg-buildpackage -jN adds the parallel=N parameter, so I don't understand
what you're getting at here or why this provision would be useful.

> - I think it'd make sense to add a "should" requirement that packages
>   allow any amount of parallelisation.  This requirement wouldn't really
>   be excessive, I think.  It's just a matter of writing Makefiles
>   properly by defining the right targets and dependencies; something
>   that's easily archieveable when people remove bad habits like assuming
>   make processes dependencies in a particular order, etc.

I think I'm opposed.  The majority of packages don't take long enough to
build to make supporting parallel building that compelling and many (I
would guess most) upstreams never test their packages with parallel builds
and have a wide variety of subtle bugs.  I think this requirement would
put way too much of a burden on Debian maintainers, not to mention making
a huge number of packages insta-buggy (as revealed by recent analysis
posted to debian-devel).

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>

Message #215 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Robert Millan <rmh@aybabtu.com>

To: Russ Allbery <rra@debian.org>, 209008@bugs.debian.org

Cc: 430649@bugs.debian.org

Subject: Re: Bug#209008: New proposed wording for DEB_BUILD_OPTIONS

Date: Mon, 31 Dec 2007 20:38:10 +0100

On Mon, Dec 31, 2007 at 11:24:45AM -0800, Russ Allbery wrote:
> Robert Millan <rmh@aybabtu.com> writes:
> 
> > The majority of packages already supports parallel builds by simply
> > passing the appropiate -j flag to make.  Not that I object to a
> > "parallel" parameter since it can bring some benefits in very specific
> > situations, but please make sure this doesn't hinder what is already
> > working.  In particular:
> >
> > - It'd be good if it required that packages do not disable parallel flags in
> >   make in case they're already present (dpkg-buildpackage -jN), even if the
> >   parallel=N parameter is not present at all.
> 
> dpkg-buildpackage -jN adds the parallel=N parameter, so I don't understand
> what you're getting at here or why this provision would be useful.

Ah, just ignore me on this.  I was under the impression that they used my
patch in dpkg-buildpackage, which just added -jN.  Never mind then.

> > - I think it'd make sense to add a "should" requirement that packages
> >   allow any amount of parallelisation.  This requirement wouldn't really
> >   be excessive, I think.  It's just a matter of writing Makefiles
> >   properly by defining the right targets and dependencies; something
> >   that's easily archieveable when people remove bad habits like assuming
> >   make processes dependencies in a particular order, etc.
> 
> I think I'm opposed.  The majority of packages don't take long enough to
> build to make supporting parallel building that compelling and many (I
> would guess most) upstreams never test their packages with parallel builds
> and have a wide variety of subtle bugs.  I think this requirement would
> put way too much of a burden on Debian maintainers, not to mention making
> a huge number of packages insta-buggy (as revealed by recent analysis
> posted to debian-devel).

I thought the number of affected packages would be small.  Can you point me
to that analisys ?

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Message #220 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: 209008@bugs.debian.org, 430649@bugs.debian.org

Subject: Re: Bug#209008: New proposed wording for DEB_BUILD_OPTIONS

Date: Mon, 31 Dec 2007 12:34:55 -0800

Robert Millan <rmh@aybabtu.com> writes:

> I thought the number of affected packages would be small.  Can you point
> me to that analisys ?

http://lists.debian.org/debian-devel/2007/12/msg00020.html

    204 built BROKEN packages
   1408 FAILED
    230 FAILED, even with regular build
   8986 succeeded
   1014 succeeded, but with jobserver warnings

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>

Message #223 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Robert Millan <rmh@aybabtu.com>

To: Russ Allbery <rra@debian.org>, 209008@bugs.debian.org

Cc: 430649@bugs.debian.org

Subject: Re: Bug#209008: New proposed wording for DEB_BUILD_OPTIONS

Date: Mon, 31 Dec 2007 22:26:28 +0100

On Mon, Dec 31, 2007 at 12:34:55PM -0800, Russ Allbery wrote:
> Robert Millan <rmh@aybabtu.com> writes:
> 
> > I thought the number of affected packages would be small.  Can you point
> > me to that analisys ?
> 
> http://lists.debian.org/debian-devel/2007/12/msg00020.html
> 
>     204 built BROKEN packages
>    1408 FAILED
>     230 FAILED, even with regular build
>    8986 succeeded
>    1014 succeeded, but with jobserver warnings

Thanks for the pointer.  Although that analisys was done with -j3 instead of
-j2.  I wonder if it'd make a difference.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Message #228 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: 209008@bugs.debian.org

Cc: 430649@bugs.debian.org

Subject: DEB_BUILD_OPTIONS and parallel builds

Date: Tue, 04 Mar 2008 19:44:12 -0800

Russ Allbery <rra@debian.org> writes:

> Okay, here's a revised proposal to address both Bug#209008 (parallel)
> and Bug#430649 (DEB_BUILD_OPTIONS parsing).

I have gotten no further feedback on this proposal except for some
discussion of whether packages can currently handle parallel builds.  I
would like to resolve these bugs for the next Policy release, but I don't
want to just commit patches on my own say-so.

Could those reading the Policy list please review the patch at:

    http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=209008#202

and either second or point out problems so that we can resolve this and
close these bugs?

Thanks!

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>

Message #233 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Russ Allbery <rra@debian.org>

Cc: 209008@bugs.debian.org, 430649@bugs.debian.org

Subject: Re: New proposed wording for DEB_BUILD_OPTIONS

Date: Wed, 5 Mar 2008 08:45:20 +0100

[Message part 1 (text/plain, inline)]

On Sun, 30 Dec 2007, Russ Allbery wrote:
> Okay, here's a revised proposal to address both Bug#209008 (parallel) and
> Bug#430649 (DEB_BUILD_OPTIONS parsing).  This proposal does the following:
[...]
> Comments?  Seconds?

Seconded.

Cheers,
-- 
Raphaël Hertzog

Le best-seller français mis à jour pour Debian Etch :
http://www.ouaza.com/livre/admin-debian/

[signature.asc (application/pgp-signature, inline)]

Message #238 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@lucas-nussbaum.net>

To: Raphael Hertzog <hertzog@debian.org>, 430649@bugs.debian.org

Cc: Russ Allbery <rra@debian.org>, 209008@bugs.debian.org

Subject: Re: Bug#430649: New proposed wording for DEB_BUILD_OPTIONS

Date: Wed, 5 Mar 2008 20:54:54 +0100

[Message part 1 (text/plain, inline)]

On 05/03/08 at 08:45 +0100, Raphael Hertzog wrote:
> On Sun, 30 Dec 2007, Russ Allbery wrote:
> > Okay, here's a revised proposal to address both Bug#209008 (parallel) and
> > Bug#430649 (DEB_BUILD_OPTIONS parsing).  This proposal does the following:
> [...]
> > Comments?  Seconds?
> 
> Seconded.

AOL.
-- 
| Lucas Nussbaum
| lucas@lucas-nussbaum.net   http://www.lucas-nussbaum.net/ |
| jabber: lucas@nussbaum.fr             GPG: 1024D/023B3F4F |

[signature.asc (application/pgp-signature, inline)]

Message #243 received at 209008@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: 209008@bugs.debian.org, 430649@bugs.debian.org

Subject: Re: Bug#209008: Bug#430649: New proposed wording for DEB_BUILD_OPTIONS

Date: Sun, 16 Mar 2008 13:24:08 -0700

Lucas Nussbaum <lucas@lucas-nussbaum.net> writes:
> On 05/03/08 at 08:45 +0100, Raphael Hertzog wrote:
>> On Sun, 30 Dec 2007, Russ Allbery wrote:

>>> Okay, here's a revised proposal to address both Bug#209008 (parallel) and
>>> Bug#430649 (DEB_BUILD_OPTIONS parsing).  This proposal does the following:
>> [...]
>>> Comments?  Seconds?

>> Seconded.

> AOL.

Thanks!  I am now applying this to my repository for the next release.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>

Message #252 received at 209008-close@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: 209008-close@bugs.debian.org

Subject: Bug#209008: fixed in debian-policy 3.8.0.0

Date: Wed, 04 Jun 2008 23:32:03 +0000

Source: debian-policy
Source-Version: 3.8.0.0

We believe that the bug you reported is fixed in the latest version of
debian-policy, which is due to be installed in the Debian FTP archive:

debian-policy_3.8.0.0.dsc
  to pool/main/d/debian-policy/debian-policy_3.8.0.0.dsc
debian-policy_3.8.0.0.tar.gz
  to pool/main/d/debian-policy/debian-policy_3.8.0.0.tar.gz
debian-policy_3.8.0.0_all.deb
  to pool/main/d/debian-policy/debian-policy_3.8.0.0_all.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 209008@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Russ Allbery <rra@debian.org> (supplier of updated debian-policy package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Wed, 04 Jun 2008 15:53:27 -0700
Source: debian-policy
Binary: debian-policy
Architecture: source all
Version: 3.8.0.0
Distribution: unstable
Urgency: low
Maintainer: Debian Policy List <debian-policy@lists.debian.org>
Changed-By: Russ Allbery <rra@debian.org>
Description: 
 debian-policy - Debian Policy Manual and related documents
Closes: 65577 186700 209008 250202 291460 367984 379150 392362 403391 422552 430649 431813 440420 442070 452105 455602 458910 473761 475731 480551 481640 481954
Changes: 
 debian-policy (3.8.0.0) unstable; urgency=low
 .
   * Bug fix: "[PROPOSAL] "debian/README.source" file for packages with
     non-trivial source", thanks to Wouter Verhelst, Jörg Sommer, Colin Watson,
     and Junichi Uekawa                                       (Closes: #250202).
   * Bug fix: "[AMENDMENT 11/02/2008] Manual page encoding", thanks to
     Colin Watson                                             (Closes: #440420).
   * Bug fix: "[PROPOSAL] common interface for parallel building in
     DEB_BUILD_OPTIONS", thanks to Loïc Minier, Peter Samuelson, and Robert
     Millan                                                   (Closes: #209008).
   * Bug fix: "Please clarify splitting/syntax of DEB_BUILD_OPTIONS", thanks to
     Loïc Minier, Peter Samuelson, Robert Millan, and Guillem Jover
                                                              (Closes: #430649).
   * Bug fix: "Documentation for Breaks in dpkg", thanks to Ian Jackson
                                                              (Closes: #379150).
   * Bug fix: "support for wrapped Uploaders should now be mandatory"
                                                              (Closes: #431813).
   * Bug fix: "[PROPOSAL] Add should not embed code from other packages",
     thanks to Neil McGovern, Colin Watson, Bill Allombert, Steve Langasek,
     Kurt Roeckx, and others                                  (Closes: #392362).
   * Bug fix: "Homepage field in debian/control undocumented", thanks to
     Mario Iseli                                              (Closes: #452105).
   * Bug fix: "Policy inconsistent with reality: base subsection no longer
     used", thanks to Magnus Holmgren, Bernd Zeimetz, and Colin Watson
                                                              (Closes: #442070).
   * Bug fix: "Inclusion of Apache Software License versions in
     /usr/share/common-licenses", thanks to Barry Hawkins     (Closes: #291460).
   * Bug fix: "[Amended] copyright should include notice if a package is
     not a part of Debian distribution", thanks to Taketoshi Sano
                                                              (Closes: #65577).
   * Bug fix: "scripts as configuration files: should vs. must", thanks to Frank
     Küster                                                   (Closes: #403391).
   * Bug fix: "debconf specification should allow underscores in template
     names", thanks to Colin Watson                           (Closes: #473761).
   * Bug fix: "clarify handling of run-time and compile-time support programs",
     thanks to Goswin Brederlow and Raphael Hertzog           (Closes: #367984).
   * Policy: better document version ranking and empty Debian revisions
     Wording: Russ Allbery <rra@debian.org>
     Seconded: Raphaël Hertzog <hertzog@debian.org>
     Seconded: Manoj Srivastava <srivasta@debian.org>
     Seconded: Guillem Jover <guillem@debian.org>
     Closes: #186700, #458910
   * Policy: remove obsolete app-defaults and Xresources provisions
     Wording: Julien Cristau <jcristau@debian.org>
     Seconded: Russ Allbery <rra@debian.org>
     Closes: #480551
   * Bug fix: "Examples of dpkg frontends should mention apt now", thanks
     to Josh Triplett                                         (Closes: #455602).
   * Bug fix: "Minor typos and wording suggestions", thanks to Michael
     Tautschnig                                               (Closes: #422552).
   * Bug fix: "substvar reference moved from dpkg-source(1) to
     deb-substvars(5)", thanks to Ian Beckwith                (Closes: #475731).
   * Policy: bugs fixed in NMUs are now closed rather than marked fixed
     Wording: Russ Allbery <rra@debian.org> (thanks, Sandro Tosi)
     Closes: #481640
   * Policy: C.1.4, C.1.8: minor typos
     Wording: Sandro Tosi <matrixhasu@gmail.com>
     Closes: #481954
   * Remove the now-obsolete policy-process document.
   * Add an md5sums control file.
   * Add Vcs-Browser and Vcs-Git control fields.
   * Remove build system support for FHS 2.1 and FSSTND, mostly commented out.
   * Remove more temporary files created by the build.
   * Remove the FSSTND license from debian/copyright; no FSSTND files are
     currently part of policy.
   * Update FHS copyright dates in debian/copyright.
   * Standardize the spacing around headings in upgrading-checklist.html.
   * Remove old ChangeLog files and metadata headers in maintainer scripts
     and debian/rules.
Checksums-Sha1: 
 f42b9921908670eb41c04940875084bc07750592 1095 debian-policy_3.8.0.0.dsc
 3eda45d7ca5563bab8bfda93286137071979385c 638655 debian-policy_3.8.0.0.tar.gz
 73680c98bc62507858aa055bcf1f1688a812f5ba 1588552 debian-policy_3.8.0.0_all.deb
Checksums-Sha256: 
 507a048bc7c84039910843e284d8e0e305778224346fd981c6f749176cc79220 1095 debian-policy_3.8.0.0.dsc
 8321b1dddd3ddd55a09539c842084ea05a731265c4c5847997957a552ba1aaa4 638655 debian-policy_3.8.0.0.tar.gz
 6c2083f50ccaa5a2f2d7a89febd320cf3a862b3204157324ffd9b363daac3e58 1588552 debian-policy_3.8.0.0_all.deb
Files: 
 37ff33fb3ccebc4f87e23fd7b91e7859 1095 doc optional debian-policy_3.8.0.0.dsc
 2565d6eaceac0aa2d093538048c1b8ed 638655 doc optional debian-policy_3.8.0.0.tar.gz
 3b153faeec899cdf1199d4d46c5d8859 1588552 doc optional debian-policy_3.8.0.0_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIRyNB+YXjQAr8dHYRAt4NAKDbO1f3BlmKT5SgMVf4AHE2Z7bPTgCffcnI
Kwa3jEGgq+PV6dwiurjmSAc=
=wCDz
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.