Debian Bug report logs - #346248
[regression] make: slow to realize it's unable to build a .SECONDARY target

version graph

Package: make; Maintainer for make is Manoj Srivastava <srivasta@debian.org>; Source for make is src:make-dfsg.

Reported by: Ian Lynagh <igloo@earth.li>

Date: Fri, 6 Jan 2006 16:48:04 UTC

Severity: wishlist

Tags: upstream

Found in version make/3.80-9

Forwarded to https://savannah.gnu.org/bugs/?func=detailitem&item_id=15584

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Manoj Srivastava <srivasta@debian.org>:
Bug#346248; Package make. Full text and rfc822 format available.

Acknowledgement sent to Ian Lynagh <igloo@earth.li>:
New Bug report received and forwarded. Copy sent to Manoj Srivastava <srivasta@debian.org>. Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Ian Lynagh <igloo@earth.li>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: make: 3.80+3.81.b4-1 much slower than 3.80-9 in some situations
Date: Fri, 06 Jan 2006 16:43:37 +0000
[Message part 1 (text/plain, inline)]
Package: make
Version: 3.80-9
Severity: normal


With the attached Makefile, make 3.80+3.81.b4-1 is much slower than
3.80-9 at running "make -wr stage1/ghc-6.4.1" (only a few seconds in the
cutdown case, but more in the real thing - I never waited for it to
terminate to know how much more).


Thanks
Ian


$ dpkg -s make | grep Version
Version: 3.80+3.81.b4-1
$ time make -wr stage1/ghc-6.4.1 
make: Entering directory `/tmp/wibble'
Makefile:20: stage1/profiling/CostCentre.o stage1/profiling/SCCfinal.o stage1/main/Config.o
make: *** No rule to make target `profiling/CostCentre.lhs', needed by `stage1/profiling/CostCentre.o'.  Stop.
make: Leaving directory `/tmp/wibble'

real    0m4.318s
user    0m4.315s
sys     0m0.002s
$


$ dpkg -s make | grep Version
Version: 3.80-9
$ time make -wr stage1/ghc-6.4.1 
make: Entering directory `/tmp/wibble'
Makefile:20: stage1/profiling/CostCentre.o stage1/profiling/SCCfinal.o stage1/main/Config.o
make: *** No rule to make target `profiling/CostCentre.lhs', needed by `stage1/profiling/CostCentre.o'.  Stop.
make: Leaving directory `/tmp/wibble'

real    0m0.008s
user    0m0.006s
sys     0m0.003s
$

-- System Information:
Debian Release: 3.1
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.14-2-amd64-k8
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)

Versions of packages make depends on:
ii  libc6                         2.3.5-8.1  GNU C Library: Shared libraries an

-- no debconf information
[Makefile (text/plain, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Manoj Srivastava <srivasta@debian.org>:
Bug#346248; Package make. Full text and rfc822 format available.

Acknowledgement sent to Ian Lynagh <igloo@earth.li>:
Extra info received and forwarded to list. Copy sent to Manoj Srivastava <srivasta@debian.org>. Full text and rfc822 format available.

Message #10 received at 346248@bugs.debian.org (full text, mbox):

From: Ian Lynagh <igloo@earth.li>
To: 346248@bugs.debian.org
Subject: Re: Bug#346248: make: 3.80+3.81.b4-1 much slower than 3.80-9 in some situations
Date: Sun, 8 Jan 2006 17:17:23 +0000
On Fri, Jan 06, 2006 at 04:43:37PM +0000, Ian Lynagh wrote:
> 
> With the attached Makefile, make 3.80+3.81.b4-1 is much slower than
> 3.80-9 at running "make -wr stage1/ghc-6.4.1" (only a few seconds in the
> cutdown case, but more in the real thing - I never waited for it to
> terminate to know how much more).

I've just tried the real thing, and it's still thinking after >24 hours
of CPU time, so this is effectively going to cause FTBFSs for me
(although technically I think it will eventually terminate).


Thanks
Ian




Information forwarded to debian-bugs-dist@lists.debian.org, Manoj Srivastava <srivasta@debian.org>:
Bug#346248; Package make. Full text and rfc822 format available.

Acknowledgement sent to Justin Pryzby <justinpryzby@users.sourceforge.net>:
Extra info received and forwarded to list. Copy sent to Manoj Srivastava <srivasta@debian.org>. Full text and rfc822 format available.

Message #15 received at 346248@bugs.debian.org (full text, mbox):

From: Justin Pryzby <justinpryzby@users.sourceforge.net>
To: Ian Lynagh <igloo@earth.li>
Cc: 346248@bugs.debian.org, 348633@bugs.debian.org, control@bugs.debian.org
Subject: Re: Bug#348633: make infinite loop (with some recursion??)
Date: Wed, 18 Jan 2006 10:22:38 -0500
severity 346248 important
reassign 348633 ghc6, make
merge 346248 348633
thanks

On Wed, Jan 18, 2006 at 01:33:03PM +0000, Ian Lynagh wrote:
> On Wed, Jan 18, 2006 at 01:33:20AM -0500, Justin Pryzby wrote:
> > 
> > ghc6 compilation sends make into an infinite loop
> 
> This sounds like http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=346248
> 
> I don't believe it's infinite, incidentally.
> 
> > This can run for many minutes..
> 
> For days, IME.
I think it ended after <8 hours for me, running at 2.4GHz, but I can't
be sure because the ssh connection died before this..  /usr/bin/screen
next time.

-- 
Clear skies,
Justin



Severity set to `important'. Request was from Justin Pryzby <justinpryzby@users.sourceforge.net> to control@bugs.debian.org. Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Manoj Srivastava <srivasta@debian.org>:
Bug#346248; Package make. Full text and rfc822 format available.

Acknowledgement sent to Justin Pryzby <justinpryzby@users.sourceforge.net>:
Extra info received and forwarded to list. Copy sent to Manoj Srivastava <srivasta@debian.org>. Full text and rfc822 format available.

Message #22 received at 346248@bugs.debian.org (full text, mbox):

From: Justin Pryzby <justinpryzby@users.sourceforge.net>
To: 346248@bugs.debian.org, 348633@bugs.debian.org, Ian Lynagh <igloo@earth.li>
Cc: control@bugs.debian.org
Subject: /usr/bin/make's 11th hour
Date: Thu, 26 Jan 2006 10:56:53 -0500
reassign 348633 ghc6
severity 348633 serious
retitle  348633 ghc6: effective FTBFS because of a bug in /usr/bin/make
block    348633 with 346248 
thanks

Ian is right, make has now been processing ghc6 for 11 hours
(/tmp/buildd/ghc6-6.4.1/ghc/compiler).  Why did you say, that you
think it eventually terminates?

-- 
Clear skies,
Justin



Information forwarded to debian-bugs-dist@lists.debian.org, Manoj Srivastava <srivasta@debian.org>:
Bug#346248; Package make. Full text and rfc822 format available.

Acknowledgement sent to Ian Lynagh <igloo@earth.li>:
Extra info received and forwarded to list. Copy sent to Manoj Srivastava <srivasta@debian.org>. Full text and rfc822 format available.

Message #27 received at 346248@bugs.debian.org (full text, mbox):

From: Ian Lynagh <igloo@earth.li>
To: Justin Pryzby <justinpryzby@users.sourceforge.net>
Cc: 346248@bugs.debian.org, 348633@bugs.debian.org
Subject: Re: /usr/bin/make's 11th hour
Date: Thu, 26 Jan 2006 16:06:51 +0000
On Thu, Jan 26, 2006 at 10:56:53AM -0500, Justin Pryzby wrote:
> 
> Ian is right, make has now been processing ghc6 for 11 hours
> (/tmp/buildd/ghc6-6.4.1/ghc/compiler).  Why did you say, that you
> think it eventually terminates?

Because the testcase I gave when reporting the bug terminates, and I
believe the problem is just a larger instance of that case.


Thanks
Ian




Blocking bugs added: 346248 Request was from Justin Pryzby <justinpryzby@users.sourceforge.net> to control@bugs.debian.org. Full text and rfc822 format available.

Noted your statement that Bug has been forwarded to https://savannah.gnu.org/bugs/?func=detailitem&item_id=15584. Request was from Manoj Srivastava <srivasta@golden-gryphon.com> to control@bugs.debian.org. Full text and rfc822 format available.

Tags added: upstream Request was from Justin Pryzby <justinpryzby@users.sourceforge.net> to control@bugs.debian.org. Full text and rfc822 format available.

Message #34 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: Boris Kolpackov <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, psmith@gnu.org, boris@kolpackov.net
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Wed, 1 Feb 2006 08:07:01 +0000
Follow-up Comment #1, bug #15584 (project make):

The makefile itself does not give me any idea what might be wrong. Can you
provide a minimal but complete test case that reproduces the problem (I don't
know what stage1/ghc-6.4.1 is).


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #35 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: "Paul D. Smith" <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Wed, 1 Feb 2006 07:32:22 -0500
Follow-up Comment #2, bug #15584 (project make):

Actually, Boris, this makefile IS sufficient to reproduce the problem.  Stick
it in an empty directory.  Run it as shown in the example.  You'll see that
older versions of make fail as shown almost immediately, while the latest CVS
version will do a LOT of work, then fail after a few seconds.

Using -d you can see what's going on: the old version of make will see that
the first prereq cannot be created, then stop:

  Considering target file `stage1/profiling/CostCentre.o'.
   File `stage1/profiling/CostCentre.o' does not exist.
   Looking for an implicit rule for `stage1/profiling/CostCentre.o'.
   Trying pattern rule with stem `profiling/CostCentre'.
   Trying implicit prerequisite `profiling/CostCentre.hs'.
   Trying pattern rule with stem `profiling/CostCentre'.
   Trying implicit prerequisite `profiling/CostCentre.lhs'.
   Found an implicit rule for `stage1/profiling/CostCentre.o'.
    Considering target file `profiling/CostCentre.lhs'.
     File `profiling/CostCentre.lhs' does not exist.
     Looking for an implicit rule for `profiling/CostCentre.lhs'.
     No implicit rule found for `profiling/CostCentre.lhs'.
     Finished prerequisites of target file `profiling/CostCentre.lhs'.
    Must remake target `profiling/CostCentre.lhs'.
make: *** No rule to make target `profiling/CostCentre.lhs', needed by
`stage1/profiling/CostCentre.o'.  Stop.

However, the new version of make does NOT stop and continues to look for
other prerequisites, and gets into some bizarre sort of full-tree walk.  The
full -d output I got was 2297723 _LINES_ long!  The first part, matching the
above, was:

Considering target file `stage1/ghc-6.4.1'.
 File `stage1/ghc-6.4.1' does not exist.
  Looking for an implicit rule for `stage1/profiling/CostCentre.o'.
  Trying pattern rule with stem `profiling/CostCentre'.
  Trying implicit prerequisite `profiling/CostCentre.hs'.
  Trying pattern rule with stem `profiling/CostCentre'.
  Trying implicit prerequisite `profiling/CostCentre.lhs'.
  Found an implicit rule for `stage1/profiling/CostCentre.o'.
   Looking for an implicit rule for `profiling/CostCentre.lhs'.
   No implicit rule found for `profiling/CostCentre.lhs'.
   Looking for an implicit rule for `stage1/utils/Util.hi'.
   Trying pattern rule with stem `Util'.

After make sees that it can't find an implicit rule for one of the prereqs,
instead of stopping it continues with the next prereq.  Then, later on it
gets into some kind of pathological recursion, "Pruning file `Config.hs'."
over and over at different depths.

Unfortunately I didn't have time to get any further into this last night.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #36 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: Boris Kolpackov <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Wed, 1 Feb 2006 20:02:06 +0000
Follow-up Comment #3, bug #15584 (project make):

Seem like my improvement in implicit.c around line 668 uncovered this. Before
make would return from pattern_search with either failure or an implicit rule
for which all implicit prerequisites either exist or can be built. Now, if an
implicit prerequisite is also an explicit prerequisite for this target,
pattern_search skips its normal checks because this prerequisite has to be
built no matter which implicit rule we choose. Sometimes, like in our case,
there may actually be no rule to built such a prerequisite
(profiling/CostCentre.lhs here). Some code in remake.c (either around line
451, or 957, or both) "assumes" that there is a rule for each implicit
prerequisite if pattern_search did not fail. Just need to find that piece of
code and fix it.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #37 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: "Paul D. Smith" <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Wed, 8 Feb 2006 09:30:19 -0500
Update of bug #15584 (project make):

                Severity:              3 - Normal => 5 - Blocker            

    _______________________________________________________

Follow-up Comment #4:

Hm.  So, the skipping doesn't impact which implicit rule matches, right? 
Because, even though a prereq might be explicitly declared it's still the
case that you have to consider it during the SELECTION of implicit rules.  If
there's a prereq pattern in the implicit rule that cannot be satisfied, then
make needs to keep looking for another implicit rule which _does_ match.  I'm
trying to think of a situation where this might make a difference in which
implicit rule was selected.

Is that the failure mode here: in the old code make wouldn't be able to find
any implicit rule because that prerequisite cannot be constructed, while the
new code it does match because of the explicit prereq?

I think this is a regression and needs to be fixed for 3.81.  Since it seemed
like the short-circuit code in implicit.c was just a performance improvement,
I tried just commenting it out but that cause 6 test suite failures (5 in
features/INTERMEDIATE), so it's more involved than that...

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #38 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: Boris Kolpackov <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Wed, 8 Feb 2006 17:30:28 +0000
Update of bug #15584 (project make):

             Assigned to:                    None => bosk                   


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #39 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: Boris Kolpackov <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Wed, 8 Feb 2006 17:49:18 +0000
Follow-up Comment #5, bug #15584 (project make):

It's a bit tricky when it comes to intermediates. Old algo would sometimes
choose subsequent rules because one of the pattern prereqs can only satisfy
as an intermediate (even though it is explicitly mentioned in the non-pattern
rule). Also the new algorithm improves diagnostics: make will print that it
was unable to build foo.h instead of saying that it could not find a rule to
build foo.c. I will take care of this bug.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #40 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: Boris Kolpackov <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Fri, 10 Feb 2006 14:53:29 +0000
Follow-up Comment #6, bug #15584 (project make):

After all, this problem does not appear to be related to my implicit.c
change. It's actually quite obvious from the -d output: both versions
conclude that there is no implicit rule for profiling/CostCentre.lhs and
while 3.80 stops, 3.81 goes on.

Next thing that I found is that if we remove .SECONDARY: from the original
makefile, make 3.81 returns immediately. So it is related to the fact that
all targets are treated as intermediates.

I was trying to figure out what are all the differences between normal and
intermediate targets. GNU make manuall has this strange text:

"The first difference is what happens if the intermediate file does not
exist.  If an ordinary file B does not exist, and `make' considers a target
that depends on B, it invariably creates B and then updates the target from
B.  But if B is an intermediate file, then `make' can leave well enough
alone.  It won't bother updating B, or the ultimate target, unless some
prerequisite of B is newer than that target or there is some other reason to
update that target."

I wonder what those "other reasons" are. Like, does mentioning an
intermediate target as a goal quilify? What about depending on the target
that was mentioned as a goal?

Anyway, the problem appears to be in check_dep() in remake.c. Apparently if
check_dep is called for an intermediate, non-existent file for which there is
no implicit rule and which has intermediate non-existent prereqs (for which
there is no implicit rule, etc.), it will recursively check every prereq and
at the end return 0, without updating anything. That's exactly what happens
in this case. Whether it is correct behaviour or not - I don't know. It
depends on what that piece of the manual above actualy means.
 

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #41 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: "Paul D. Smith" <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Fri, 10 Feb 2006 10:09:10 -0500
Follow-up Comment #7, bug #15584 (project make):

> It won't bother updating B, or the ultimate target, unless some
> prerequisite of B is newer than that target or there is some
> other reason to update that target.

This means if you have a rule like "target: B" and B is intermediate but does
not exist, then target will only be updated if B has to be rebuilt because one
of B's prereqs is newer: B will not be rebuilt just because it does not
exist.

"Some other reason" means if you have a rule "target: B A" in the same
situation above, then target could be rebuilt because of A, even if it
doesn't need to be rebuilt because of B.

Your description of how check_dep() behaves sounds correct on the surface:
you can't know whether B needs to be rebuilt unless you check all its
prerequisites.  However, there's SOMETHING pathalogical going on here: if you
run the test with -d and capture all the output you can see that as the
processing goes along the dep check make performs gets more and more
elaborate and the number of times it does "Pruning file `Config.hs'."
increases almost exponentially.  I can't see any justification for that
behavior offhand... is there one?

I see you qualify the situation by saying there is no implicit rule for the
non-existent file... if there's no implicit or explicit rule for that target
then does it make sense to continue looking at that target's prerequisites? 
Suppose we found one of them and it needed to be rebuilt: what then?  Does
that mean the same thing as an empty rule:

  target : B.x ; @echo cp $< $@
  %.x : %.y
  B.y : ; @echo cp $< $@
  .INTERMEDIATE: B.x

??  Hm.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #42 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: Boris Kolpackov <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Fri, 10 Feb 2006 16:20:59 +0000
Follow-up Comment #8, bug #15584 (project make):

> This means if you have a rule like "target: B" and B is 
> intermediate but does not exist, then target will only be updated 
> if B has to be rebuilt because one of B's prereqs is newer: B 
> will not be rebuilt just because it does not exist.

Then, since none of the targets in this makefile exist and all of them are
intermediate, we shouldn't build anything and should not fail. Or if the
target is also a goal then this logic does not apply? Even if that's the case
we should fail to build stage1/ghc-6.4.1, not profiling/CostCentre.lhs or
stage1/profiling/CostCentre.o?

> "Some other reason" means if you have a rule "target: B A" in 
> the same situation above, then target could be rebuilt because > of A, even
if it doesn't need to be rebuilt because of B.

Ok, got it. Thanks for the clarification.

> Your description of how check_dep() behaves sounds correct on 
> the surface: you can't know whether B needs to be rebuilt 
> unless you check all its prerequisites.

That's definitely not how 3.80 works. So we are talking about some new way of
doing things in 3.81 that caused this behavior.

> However, there's SOMETHING pathalogical going on here: if you
> run the test with -d and capture all the output you can see 
> that as the processing goes along the dep check make performs 
> gets more and more elaborate and the number of times it does 
> "Pruning file `Config.hs'." increases almost exponentially.

I see those except my log is full of "Prunning file `Makefile'.". This seems
to happen multiple times in a row at different nesting levels. I will try to
investigate this.

> I see you qualify the situation by saying there is no implicit 
> rule for the non-existent file... if there's no implicit or 
> explicit rule for that target then does it make sense to 
> continue looking at that target's prerequisites? 

Well, if there is no rule and we don't need to rebuild it then I suppose we
don't fail.

> Suppose we found one of them and it needed to be rebuilt: what
then?

Then we fail. 

For now I am going to assume that what check_dep does is correct and will try
to find out why do we get all those "Prunning..." messages.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #43 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: "Paul D. Smith" <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Fri, 10 Feb 2006 14:28:21 -0500
Follow-up Comment #9, bug #15584 (project make):

Actually, now that I think about it I'm confused.  You _can't_ have a
situation like this:

target : B.x ; @echo cp $< $@
%.x : %.y
B.y : ; @echo cp $< $@

because you can't define an empty pattern rule... an empty pattern rule
deletes the pattern.

So, I don't see how the situation you describe: "an intermediate,
non-existent file for which there is no implicit rule" could ever be
correct... let me think.  I suppose there could be an _explicit_ rule; does
that make a difference?  That could happen because people can declare targets
as intermediate even though they are explicitly mentioned (which normally
would disqualify them from being intermediate).  So, something like this:

target : B.x ; @echo cp $< $@
B.x : B.y
B.y : ; @echo cp $< $@
.INTERMEDIATE: B.x

Offhand this seems legal to me; would this trigger the issue?

If there's no implicit OR explicit rule, then I guess we have one of two
cases: either make should check all the prerequisites of the intermediate
file (B.x above) and if none of them need to be rebuilt, make says "OK".  Or,
make should fail immediately because it doesn't know how to build the
intermediate file (B.x) regardless of whether or not it actually needs to
know.

For explicit rules this is perfectly legal: if there's no command to update a
target then it's just considered "magically updated" without actually doing
anything.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #44 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: Boris Kolpackov <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net, bug-make@gnu.org
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Mon, 13 Feb 2006 15:28:04 +0000
Follow-up Comment #10, bug #15584 (project make):

So I did some mode digging and it appears that make just does what it's
supposed to do. If you look into this makefile carefully you will see the
pattern rule that looks like this %.hi: %.o which pretty much makes every
single .o file depend (directly and inderectly) on all others. Now because
all of the targets in this makefile are intermediate and non-existed, make
dutifully traverses the whole grpath for every .o file and figures out that
it cannot build anyhting. Since it doesn't set any flags like updated or
this_target_is_intermediate_and_does_not_exist_and_i_already_tried_to_build
_it, it does *complete* graph traversal for every .o file it tries to build.
I don't see any way (and don't really want) to fix this in make without doing
a major surgery on of how things work. To summarize, the long time is the
result of three things:

1. Strange desire of the makefile author to have all files intermediate.

2. Pretty much fully-connected dependency graph.

3. Brain-damaged intermediate file logic in GNU make.

I therefore suggest that we leave this corner case alone.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #45 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: "Paul D. Smith" <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net, bug-make@gnu.org
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Mon, 13 Feb 2006 10:59:40 -0500
Follow-up Comment #11, bug #15584 (project make):

Hm.  OK.  I guess the only question is, why does the older version of GNU
make not have this problem?  We must have changed the behavior of
intermediate files or something that caused this change in behavior.  If we
can understand what change we made and that it was legitimate (to fix another
bug) then I'm happy to leave this alone for 3.81.  Thanks for looking into
this one Boris!

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #46 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: Boris Kolpackov <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net, bug-make@gnu.org
Subject: [bug #15584] 3.81.b4 much slower than 3.80 in some situations
Date: Wed, 15 Feb 2006 19:01:29 +0000
Follow-up Comment #12, bug #15584 (project make):

It appears that 3.80 has a bug in handling of .SECONDARY. It does not
actually mark individual targets as intermedite. As a result, check_dep
doesn't treat any target in this makefile as intermediate. In CVS the
following code was added in file.c around line 692:

hash_map (&files, set_intermediate);


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #47 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: "Paul D. Smith" <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net, bug-make@gnu.org
Subject: [bug #15584] 3.81 does way too much work in some pathalogical situations.
Date: Wed, 15 Feb 2006 14:20:25 -0500
Update of bug #15584 (project make):

                Severity:             5 - Blocker => 2 - Minor              
              Item Group:                     Bug => Enhancement            
       Component Version:                    None => CVS                    
                 Summary: 3.81.b4 much slower than 3.80 in some situations =>
3.81 does way too much work in some pathalogical situations.

    _______________________________________________________

Follow-up Comment #13:

OK.  I'm marking this as an enhancement and dropping the Severity level.

Thanks for looking at this Boris.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Message #48 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: Boris Kolpackov <savannah-bounces@gnu.org>
To: 346248-forwarded@bugs.debian.org, Manoj Srivastava <srivasta@debian.org>, Boris Kolpackov <boris@kolpackov.net>, "Paul D. Smith" <psmith@gnu.org>, psmith@gnu.org, boris@kolpackov.net, bug-make@gnu.org
Subject: [bug #15584] 3.81 does way too much work in some pathalogical situations.
Date: Wed, 15 Feb 2006 19:36:00 +0000
Update of bug #15584 (project make):

             Assigned to:                    bosk => None                   


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Severity set to `wishlist'. Request was from Manoj Srivastava <srivasta@golden-gryphon.com> to control@bugs.debian.org. Full text and rfc822 format available.

Message #51 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: Manoj Srivastava <srivasta@golden-gryphon.com>
To: control@bugs.debian.org, 346248-forwarded@bugs.debian.org
Subject: Bad pattern in use
Date: Mon, 20 Mar 2006 15:26:20 -0600
severity 346248  wishlist
thanks

Hi,

        This is the analysis from upstream:
 So I did some mode digging and it appears that make just does what
 it's supposed to do. If you look into this makefile carefully you
 will see the pattern rule that looks like this %.hi: %.o which pretty
 much makes every single .o file depend (directly and inderectly) on
 all others. Now because all of the targets in this makefile are
 intermediate and non-existed, make dutifully traverses the whole
 grpath for every .o file and figures out that it cannot build
 anyhting. Since it doesn't set any flags like updated or
 this_target_is_intermediate_and_does_not_exist_and_i_already_tried_to_build_it,
 it does *complete* graph traversal for every .o file it tries to
 build. I don't see any way (and don't really want) to fix this in
 make without doing a major surgery on of how things work. To
 summarize, the long time is the result of three things:

1. Strange desire of the makefile author to have all files intermediate.

2. Pretty much fully-connected dependency graph.

3. Brain-damaged intermediate file logic in GNU make.

 It appears that 3.80 has a bug in handling of .SECONDARY. It does not
 actually mark individual targets as intermedite. As a result,
 check_dep doesn't treat any target in this makefile as intermediate 

        So, this is being marked as a feature request.

        manoj
-- 
Your own qualities will help prevent your advancement in the world.
Manoj Srivastava     <srivasta@acm.org>    <http://www.golden-gryphon.com/>
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C



Information forwarded to debian-bugs-dist@lists.debian.org, Manoj Srivastava <srivasta@debian.org>:
Bug#346248; Package make. (Tue, 15 Feb 2011 06:57:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Jonathan Nieder <jrnieder@gmail.com>:
Extra info received and forwarded to list. Copy sent to Manoj Srivastava <srivasta@debian.org>. (Tue, 15 Feb 2011 06:57:06 GMT) Full text and rfc822 format available.

Message #56 received at 346248@bugs.debian.org (full text, mbox):

From: Jonathan Nieder <jrnieder@gmail.com>
To: Ian Lynagh <igloo@earth.li>
Cc: 346248@bugs.debian.org
Subject: Re: make: 3.80+3.81.b4-1 much slower than 3.80-9 in some situations
Date: Tue, 15 Feb 2011 00:54:49 -0600
retitle 346248 [regression] make: slow to realize it's unable to build a .SECONDARY target
quit

Ian Lynagh wrote:

> With the attached Makefile, make 3.80+3.81.b4-1 is much slower than
> 3.80-9 at running "make -wr stage1/ghc-6.4.1" (only a few seconds in the
> cutdown case, but more in the real thing - I never waited for it to
> terminate to know how much more).

Just because I'm curious: did ghc ever learn to work around this?




Changed Bug title to '[regression] make: slow to realize it's unable to build a .SECONDARY target' from 'make: 3.80+3.81.b4-1 much slower than 3.80-9 in some situations' Request was from Jonathan Nieder <jrnieder@gmail.com> to control@bugs.debian.org. (Tue, 15 Feb 2011 06:57:08 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Manoj Srivastava <srivasta@debian.org>:
Bug#346248; Package make. (Tue, 15 Feb 2011 14:21:11 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Lynagh <igloo@earth.li>:
Extra info received and forwarded to list. Copy sent to Manoj Srivastava <srivasta@debian.org>. (Tue, 15 Feb 2011 14:21:11 GMT) Full text and rfc822 format available.

Message #63 received at 346248@bugs.debian.org (full text, mbox):

From: Ian Lynagh <igloo@earth.li>
To: Jonathan Nieder <jrnieder@gmail.com>
Cc: 346248@bugs.debian.org
Subject: Re: make: 3.80+3.81.b4-1 much slower than 3.80-9 in some situations
Date: Tue, 15 Feb 2011 13:43:29 +0000
On Tue, Feb 15, 2011 at 12:54:49AM -0600, Jonathan Nieder wrote:
> 
> Ian Lynagh wrote:
> 
> > With the attached Makefile, make 3.80+3.81.b4-1 is much slower than
> > 3.80-9 at running "make -wr stage1/ghc-6.4.1" (only a few seconds in the
> > cutdown case, but more in the real thing - I never waited for it to
> > terminate to know how much more).
> 
> Just because I'm curious: did ghc ever learn to work around this?

Yes:

Mon Feb 27 10:59:39 GMT 2006  Simon Marlow <simonmar@microsoft.com>
  * remove empty .SECONDARY target
  
  This works around a problem with recent versions of GNU make that take
  a long time when all targets are declared intermediate with
  .SECONDARY.  See 
  
    https://savannah.gnu.org/bugs/?func=detailitem&item_id=15584
  
  for discussion of the GNU make issue.

-
-# This line prevents GNU make from deleting any intermediate targets:
-
-.SECONDARY:



Thanks
Ian





Message #64 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: Sol Swords <INVALID.NOREPLY@gnu.org>
To: 346248-forwarded@bugs.debian.org, Sol Swords <sswords@gmail.com>, psmith@gnu.org, boris@kolpackov.net, bug-make@gnu.org
Subject: [bug #15584] 3.81 does way too much work in some pathalogical situations.
Date: Fri, 23 Mar 2012 20:46:08 +0000
Additional Item Attachment, bug #15584 (project make):

File name: BadMakefile                    Size:2 KB


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/





Message #65 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: Sol Swords <INVALID.NOREPLY@gnu.org>
To: 346248-forwarded@bugs.debian.org, Sol Swords <sswords@gmail.com>, psmith@gnu.org, boris@kolpackov.net, bug-make@gnu.org
Subject: [bug #15584] 3.81 does way too much work in some pathalogical situations.
Date: Fri, 23 Mar 2012 20:55:24 +0000
Follow-up Comment #14, bug #15584 (project make):

I ran into this problem with some real work.  I realize it's probably not
common to have such a large tree of secondary/intermediate files, but due to
the nature of our build process it would be nice if that worked without
hitting this performance limitation.

In case it clarifies anything, I've attached a makefile (named BadMakefile)
that replicates the problem.  You can see the pattern; adding another level of
hierarchy approximately doubles the time it takes to "make all".  Also I
predict that if you run this with "make -d" it will produce ~10GB of output.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/





Message #66 received at 346248-forwarded@bugs.debian.org (full text, mbox):

From: "Paul D. Smith" <INVALID.NOREPLY@gnu.org>
To: "Paul D. Smith" <psmith@gnu.org>, 346248-forwarded@bugs.debian.org, Sol Swords <sswords@gmail.com>, boris@kolpackov.net, bug-make@gnu.org
Subject: [bug #15584] 3.81 does way too much work in some pathalogical situations.
Date: Wed, 09 Oct 2013 04:55:58 +0000
Update of bug #15584 (project make):

       Component Version:                     SCM => 3.81                   


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?15584>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Fri Apr 18 09:03:28 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.