Debian Bug report logs - #582745
upstart: Upstart jobs get stuck when "expect daemon" or "expect fork" is wrong.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: David Caldwell <david@porkrind.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: upstart: Upstart jobs get stuck when "expect daemon" or "expect fork" is wrong.

Date: Sun, 23 May 2010 03:22:11 -0700

Package: upstart
Version: 0.6.3-1
Severity: normal


While trying to write some upstart /etc/init .conf files, I noticed that if
I got the "expect daemon/fork" line incorrect then init would get in a
messed up state, causing any subsequent "initctl start" or "initctl stop"
commands to hang on the .conf file in question, even after the bug was
fixed. The only solution I could find was to rename the script. I suspect
rebooting would work to but I'm in a position to reboot right now.

Clearly it was my fault that I got the "expect" line wrong, but even so I
don't think upstart should hang when talking about that job for then
on. There needs to be some way to reset it or not let it get in that state
at all.

Here's a small test case I made. I was unable to get it to hang with just an
"exec" line. I had to put it in a "script" and then call an external
program.

cat > fail.conf <<EOF
expect fork
script
        /bin/true
        sleep 10&
end script
EOF

If I try to start the job now it prints a PID that doesn't match up with the
correct PID. On my system I got:

# initctl start fail
fail start/running, process 20121
# ps aux | grep sleep
root     20122  0.0  0.0   3804   480 ?        S    03:07   0:00 sleep 10

At this point any attempt to "initctl stop fail" will hang. If I Control-C a
hung "stop" and try again it will say it's already stopped, but the state
doesn't show up correctly:

# initctl status fail
fail stop/killed, process 20121

Attempting to start it again also hangs. I couldn't find any way out of this
state except to reboot, or rename the fail.conf file to fail1.conf and
continue testing under that name.

-- System Information:
Debian Release: squeeze/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.33-2-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages upstart depends on:
ii  initscripts                   2.87dsf-10 scripts for initializing and shutt
ii  libc6                         2.10.2-6   Embedded GNU C Library: Shared lib
ii  libdbus-1-3                   1.2.24-1   simple interprocess messaging syst
ii  sysv-rc                       2.87dsf-10 System-V-like runlevel change mech
ii  sysvinit-utils                2.87dsf-10 System-V-like utilities

upstart recommends no packages.

upstart suggests no packages.

-- no debconf information

Message #10 received at 582745@bugs.debian.org (full text, mbox, reply):

From: Alan Givati <alangiv@gmail.com>

To: 582745@bugs.debian.org

Subject: I have been experiencing a similar issue

Date: Mon, 06 Jun 2011 19:57:36 +0300

[Message part 1 (text/plain, inline)]

If there is something bad in a *script* stanza then that job will become 
broken (freezes on start or stop commands) till reboot.  Is there a way 
to "restart" upstart?

[Message part 2 (text/html, inline)]

Message #15 received at 582745@bugs.debian.org (full text, mbox, reply):

From: Paul Makepeace <paulm@paulm.com>

To: 582745@bugs.debian.org

Subject: Any fix in sight?

Date: Fri, 13 Apr 2012 13:31:30 -0700

I've also just run into this issue.

I see this bug has been open for nearly two years, is it going to get
looked at? Is there anything we can do to help?

Message #20 received at 582745@bugs.debian.org (full text, mbox, reply):

From: James Vautin <james.vautin@iplayup.com>

To: 582745@bugs.debian.org

Subject: Me too...

Date: Mon, 7 May 2012 16:32:15 +1000

[Message part 1 (text/plain, inline)]

Throwing up my hand as well!

Upstart, with a particular job, is in limbo (hung start/stop) until I reboot the relevant hosts, which at this time is not an option to maintain our 99.99%!

This seems like a pretty serious bug .. surprised it's been here for 2 years and more people aren't screaming?  Or I'm not sure - perhaps we're all doing seething wrong and there's a way to unstick these jobs without rebooting? :)

[Message part 2 (text/html, inline)]

Message #25 received at 582745@bugs.debian.org (full text, mbox, reply):

From: James Vautin <james@vautin.com>

To: 582745@bugs.debian.org

Subject: Workaround

Date: Mon, 7 May 2012 16:36:18 +1000

[Message part 1 (text/plain, inline)]

I should also have mentioned - my workaround at the moment was to just rename the job.. I can't use the old job name until we reboot I guess :) 

[Message part 2 (text/html, inline)]

Message #30 received at 582745@bugs.debian.org (full text, mbox, reply):

From: Marc Cooper <marc@auxbuss.com>

To: 582745@bugs.debian.org

Subject: Same problem

Date: Wed, 09 May 2012 10:37:04 +0100

Confirmed in version: 0.6.6-4

I just ran into this issue. I'm very grateful you reported it as I could
have spent hours trying to debug it.

I have exactly the symptoms you describe and triaged your debug steps
which I can replicate.

-- 
Best,
Marc

Message #37 received at 582745@bugs.debian.org (full text, mbox, reply):

From: Vladimir Rutsky <rutsky.vladimir@gmail.com>

To: 582745@bugs.debian.org

Subject: Workaround: create process with expected PID

Date: Tue, 19 Feb 2013 23:07:40 +0400

[Message part 1 (text/plain, inline)]

Hello,

I encountered same issue that described in this bug report. I found
workaround by Johan Kiviniemi for this issue:

https://raw.github.com/ion1/workaround-upstart-snafu/master/workaround-upstart-snafu

Usage:

$ wget
https://raw.github.com/ion1/workaround-upstart-snafu/master/workaround-upstart-snafu
$ chmod +x workaround-upstart-snafu
$ ./workaround-upstart-snafu PID

where PID is process ID expected by upstart:

$ initctl status testdaemon
testdaemon stop/killed, process 16413

This is Ruby script and as I see it creates lots of childs using fork,
until process with requested PID created. After this script upstart thinks,
that daemon stopped:

$ initctl status testdaemon
testdaemon stop/waiting

On my machine this script works for a few minutes, which is long, but more
acceptable than reboot in certain conditions.


Best wishes,

Vladimir Rutsky

[Message part 2 (text/html, inline)]

Message #42 received at 582745-done@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 541141-done@bugs.debian.org,582745-done@bugs.debian.org,593348-done@bugs.debian.org,694963-done@bugs.debian.org,711630-done@bugs.debian.org,712763-done@bugs.debian.org,713135-done@bugs.debian.org,732122-done@bugs.debian.org,732123-done@bugs.debian.org,732124-done@bugs.debian.org,732125-done@bugs.debian.org,732126-done@bugs.debian.org,732127-done@bugs.debian.org,732128-done@bugs.debian.org,732130-done@bugs.debian.org,732132-done@bugs.debian.org,732179-done@bugs.debian.org,732756-done@bugs.debian.org,732946-done@bugs.debian.org,732980-done@bugs.debian.org,733287-done@bugs.debian.org,733739-done@bugs.debian.org,733741-done@bugs.debian.org,735853-done@bugs.debian.org,736611-done@bugs.debian.org,737757-done@bugs.debian.org,751649-done@bugs.debian.org,766121-done@bugs.debian.org,770014-done@bugs.debian.org,774450-done@bugs.debian.org,789524-done@bugs.debian.org,

Cc: upstart@packages.debian.org, upstart@packages.qa.debian.org

Subject: Bug#808289: Removed package(s) from unstable

Date: Mon, 11 Jan 2016 05:26:57 +0000

Version: 1.11-5+rm

Dear submitter,

as the package upstart has just been removed from the Debian archive
unstable we hereby close the associated bug reports.  We are sorry
that we couldn't deal with your issue properly.

For details on the removal, please see https://bugs.debian.org/808289

The version of this package that was in Debian prior to this removal
can still be found using http://snapshot.debian.org/.

This message was generated automatically; if you believe that there is
a problem with it please contact the archive administrators by mailing
ftpmaster@ftp-master.debian.org.

Debian distribution maintenance software
pp.
Scott Kitterman (the ftpmaster behind the curtain)

Send a report that this bug log contains spam.