Debian Bug report logs - #915379
anacron.service: should probably use KillMode=process

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Ansgar Burchardt <ansgar@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: anacron.service: should probably use KillMode=process

Date: Mon, 03 Dec 2018 11:23:30 +0100

Package: anacron
Version: 2.3-26
Severity: normal

anacron.service currently uses KillMode=mixed.  It probably should
not.

KillMode=mixed sends SIGTERM to anacron and then SIGKILL to any
processes started by anacron.  The default (KillMode=control-group)
would send SIGTERM to all processes which is probably what one wantes
if processes started by anacron should be stopped when anacron is (for
example during upgrades).

More likely, anacron should probably use KillMode=process.  Then
stopping anacron would only send SIGTERM to anacron itself and leave
the jobs anacron might have started untouched.  (This is what
cron.service uses.)

Ansgar

Message #8 received at 915379-submitter@bugs.debian.org (full text, mbox, reply):

From: Boyuan Yang <byang@debian.org>

To: Ansgar Burchardt <ansgar@debian.org>, 915379-submitter@bugs.debian.org

Subject: Re: Bug#915379: anacron.service: should probably use KillMode=process

Date: Mon, 03 Dec 2018 13:42:58 -0500

[Message part 1 (text/plain, inline)]

Control: tag -1 + moreinfo

在 2018-12-03一的 11:23 +0100，Ansgar Burchardt写道：
> Package: anacron
> Version: 2.3-26
> Severity: normal
> 
> anacron.service currently uses KillMode=mixed.  It probably should
> not.
> 
> KillMode=mixed sends SIGTERM to anacron and then SIGKILL to any
> processes started by anacron.  The default (KillMode=control-group)
> would send SIGTERM to all processes which is probably what one wantes
> if processes started by anacron should be stopped when anacron is (for
> example during upgrades).
> 
> More likely, anacron should probably use KillMode=process.  Then
> stopping anacron would only send SIGTERM to anacron itself and leave
> the jobs anacron might have started untouched.  (This is what
> cron.service uses.)

Hi Ansgar,

I chose KillMode=mixed intentionally. Here's the reason: we want to stop
anacron service elegantly and not to abruptly kill any process inside the
control group. Anacron is accepting SIGUSR1 and interprets it as the request
to exit gracefully; it will wait till all its jobs to finish before it exits
by itself. 90 seconds (a default value provided by systemd) after sending
SIGUSR1, systemd will send SIGKILL to all processes in the control group; this
SIGKILL ensures that jobs are eventually killed if they are not responding. I
believe this is better than abruptly sending SIGTERM to all processes and
definitely better than KillMode=process since it will leak processes out of
systemd's control if the daemon is not handling things properly.

--
Thanks,
Boyuan Yang

[signature.asc (application/pgp-signature, inline)]

Message #15 received at 915379@bugs.debian.org (full text, mbox, reply):

From: Tomas Janousek <tomi@nomi.cz>

To: Boyuan Yang <byang@debian.org>, 915379@bugs.debian.org

Cc: Ansgar Burchardt <ansgar@debian.org>

Subject: Re: Bug#915379: anacron.service: should probably use KillMode=process

Date: Wed, 11 Nov 2020 09:34:16 +0000

Hi Boyuan.

On Mon, Dec 03, 2018 at 01:42:58PM -0500, Boyuan Yang wrote:
> I chose KillMode=mixed intentionally. Here's the reason: we want to stop
> anacron service elegantly and not to abruptly kill any process inside the
> control group. Anacron is accepting SIGUSR1 and interprets it as the request
> to exit gracefully; it will wait till all its jobs to finish before it exits
> by itself.

Unfortunately this isn't entirely true.

Anacron does wait for jobs to finish, but if any job invokes exim4's
/usr/sbin/sendmail to send its result to the user, that sendmail forks the
setuid /usr/sbin/exim4¹ to process the queue in the background and deliver the
mail. Anacron doesn't wait for this forked process, and systemd kills it
immediately, resulting in the mail not being delivered until the queue gets
processed by something else.

This, I believe, is a rather serious issue.

¹) That might be different if another MTA is being used, obviously, but exim4
is still the default MTA that gets installed if any Debian package needs to
deliver (local) mail.

Now setting KillMode=process is one way to fix this. Another is adding an
additional

    ExecStart=/bin/sleep 5
    Type=oneshot

which adds a few seconds for the forked processes to finish before they're
killed.

It would be a bit nicer if systemd's killing had a timeout for these processes
that need to do a bit more work after the main process finished, but it seems
there is not such thing. :-(

> 90 seconds (a default value provided by systemd) after sending SIGUSR1,
> systemd will send SIGKILL to all processes in the control group; this
> SIGKILL ensures that jobs are eventually killed if they are not responding.
> I believe this is better than abruptly sending SIGTERM to all processes and
> definitely better than KillMode=process since it will leak processes out of
> systemd's control if the daemon is not handling things properly.

-- 
Tomáš Janoušek, a.k.a. Pivník, a.k.a. Liskni_si, https://work.lisk.in/

Message #20 received at 915379@bugs.debian.org (full text, mbox, reply):

From: Marc Haber <mh+debian-bugs@zugschlus.de>

To: Tomas Janousek <tomi@nomi.cz>, 915379@bugs.debian.org, 915379-submitter@bugs.debian.org

Cc: Boyuan Yang <byang@debian.org>, Ansgar Burchardt <ansgar@debian.org>

Subject: Re: Bug#915379: anacron.service: should probably use KillMode=process

Date: Fri, 26 Feb 2021 09:08:18 +0100

On Wed, Nov 11, 2020 at 09:34:16AM +0000, Tomas Janousek wrote:
> On Mon, Dec 03, 2018 at 01:42:58PM -0500, Boyuan Yang wrote:
> > I chose KillMode=mixed intentionally. Here's the reason: we want to stop
> > anacron service elegantly and not to abruptly kill any process inside the
> > control group. Anacron is accepting SIGUSR1 and interprets it as the request
> > to exit gracefully; it will wait till all its jobs to finish before it exits
> > by itself.
> 
> Unfortunately this isn't entirely true.
> 
> Anacron does wait for jobs to finish, but if any job invokes exim4's
> /usr/sbin/sendmail to send its result to the user, that sendmail forks the
> setuid /usr/sbin/exim4¹ to process the queue in the background and deliver the
> mail. Anacron doesn't wait for this forked process, and systemd kills it
> immediately, resulting in the mail not being delivered until the queue gets
> processed by something else.

Worse. If the receiving side does post-DATA checking of the message, and
systemd sends SIGKILL to the exim process on the sending side, the
receiving side might continue delivery without the sending side noticing
the confirmation (it's already dead by then). During the next exim queue
run, the message will be delivered a second time.

In this case, 5 seconds of extra wait would probably not be enough.

Greetings
Marc

Message #28 received at 915379@bugs.debian.org (full text, mbox, reply):

From: Tomas Janousek <tomi@nomi.cz>

To: Marc Haber <mh+debian-bugs@zugschlus.de>

Cc: 915379@bugs.debian.org, Boyuan Yang <byang@debian.org>, Ansgar Burchardt <ansgar@debian.org>

Subject: Re: Bug#915379: anacron.service: should probably use KillMode=process

Date: Sat, 25 Dec 2021 20:27:58 +0000

[Message part 1 (text/plain, inline)]

Hi,

On Fri, Feb 26, 2021 at 09:08:18AM +0100, Marc Haber wrote:
>Worse. If the receiving side does post-DATA checking of the message, and
>systemd sends SIGKILL to the exim process on the sending side, the
>receiving side might continue delivery without the sending side noticing
>the confirmation (it's already dead by then). During the next exim queue
>run, the message will be delivered a second time.
>
>In this case, 5 seconds of extra wait would probably not be enough.

This entry in the systemd 250 NEWS gives me hope this might be fixed in 
a nice way eventually:

    * A new service unit file setting ExitType= has been added that
      specifies when to assume a service has exited. By default systemd
      only watches the main process of a service. By setting
      ExitType=cgroup it can be told to wait for the last process in a
      cgroup instead.

I'll probably experiment with it once systemd 250 lands in testing, 
unless someone beats me to it.

-- 
Tomáš "liskin" ("Pivník") Janoušek, https://lisk.in/

[Message part 2 (text/html, inline)]

Message #33 received at 915379@bugs.debian.org (full text, mbox, reply):

From: Tomas Janousek <tomi@nomi.cz>

To: 915379@bugs.debian.org

Cc: Marc Haber <mh+debian-bugs@zugschlus.de>, Boyuan Yang <byang@debian.org>, Ansgar Burchardt <ansgar@debian.org>

Subject: Re: Bug#915379: anacron.service: should probably use KillMode=process

Date: Sun, 23 Jan 2022 12:39:08 +0000

[Message part 1 (text/plain, inline)]

Hi again,

On Sat, Dec 25, 2021 at 08:27:58PM +0000, Tomas Janousek wrote:
>This entry in the systemd 250 NEWS gives me hope this might be fixed 
>in a nice way eventually:
>
>    * A new service unit file setting ExitType= has been added that
>      specifies when to assume a service has exited. By default systemd
>      only watches the main process of a service. By setting
>      ExitType=cgroup it can be told to wait for the last process in a
>      cgroup instead.
>
>I'll probably experiment with it once systemd 250 lands in testing, 
>unless someone beats me to it.

I can confirm that using ExitType=cgroup fixes the issue as well—systemd 
now waits for all processes spawned by anacron to exit.

-- 
Tomáš "liskin" ("Pivník") Janoušek, https://lisk.in/

[Message part 2 (text/html, inline)]

Message #38 received at 915379@bugs.debian.org (full text, mbox, reply):

From: Melvin Vermeeren <vermeeren@vermwa.re>

To: 915379@bugs.debian.org

Subject: Re: Bug#915379: anacron.service: should probably use KillMode=process

Date: Fri, 12 Aug 2022 20:22:29 +0200

[Message part 1 (text/plain, inline)]

Hi,

I ran into this problem today. Automated upgrades with unattended-upgrades 
upgraded debianutils. Then needrestart decided that anacron.service needed to 
be restart.

However, backup cron jobs were running at the time, which take longer than a 
minute or two. After the brief timeout every single process in the chain got 
killed abruptly, without any of the actual cron job tasks even receiving a 
SIGTERM to clean up nicely. Worst of all this means the process does not even 
get a chance to report error/failure, so not mail ends up in the mailbox.

The result (with borgbackup at least) means a lot of manual work to cleanup 
stale repository locks acquired, checking some caches and manually unmount and 
removing the snapshots made for backup purposes.

I strongly feel like stopping/restarting anacron.service should never, ever 
timeout at all. A very long-running (possibly stuck) cron job should result in 
a blocking (or failing) stop action which can then be investigated properly by 
the administrator. Such as event would be a bug in another package and not a 
problem with anacron daemon.

Forcefully killing long-running cron jobs can have severe consequences. In 
today's case it was recoverable but similar cron jobs could also perform 
automated cleanup/pruning tasks in databases, registries, etc, where killing 
is very, very much undesired and effectively as bad as system crash for data 
integrity purposes.

I can think of two ways to improve this.

1. Always let jobs finish cleanly: TimeoutStopSec=infinity
I strongly prefer this option in all cases (desktop/server/...).

2. SIGUSR1 anacron as is the case now, then on timeout SIGTERM to all 
processes in the group, then on timeout again SIGKILL all processes in the 
group. I must admit I don't know how to implement this with systemd services.

Could you share thoughts regarding this issue?

Thanks,

-- 
Melvin Vermeeren
Systems engineer

[signature.asc (application/pgp-signature, inline)]

Message #43 received at 915379@bugs.debian.org (full text, mbox, reply):

From: Lance Lin <LQi254@protonmail.com>

To: Melvin Vermeeren <vermeeren@vermwa.re>, 915379@bugs.debian.org

Subject: Re: Bug#915379: anacron.service: should probably use KillMode=process

Date: Tue, 23 Aug 2022 12:33:11 +0000

[Message part 1 (text/plain, inline)]

Hello Melvin,

Thank you for your report.

> I can think of two ways to improve this.
> 

> 1. Always let jobs finish cleanly: TimeoutStopSec=infinity
> I strongly prefer this option in all cases (desktop/server/...).

Sure, I think this is easy enough to do and it does make sense.

I can push a patch for this.

> 2. SIGUSR1 anacron as is the case now, then on timeout SIGTERM to all
> processes in the group, then on timeout again SIGKILL all processes in the
> group. I must admit I don't know how to implement this with systemd services.

This change is more involved. At present, anacron is in "legacy" status. I recently
picked up the package as part of the cronie transition. cronie is expected to replace
cron/anacron in the future and is actively developed by Fedora. I would suggest we place
major changes/improvements in that project.

How do you think?

Lance

[signature.asc (application/pgp-signature, attachment)]

Message #48 received at 915379@bugs.debian.org (full text, mbox, reply):

From: Melvin Vermeeren <vermeeren@vermwa.re>

To: 915379@bugs.debian.org, Lance Lin <LQi254@protonmail.com>

Subject: Re: Bug#915379: anacron.service: should probably use KillMode=process

Date: Tue, 23 Aug 2022 21:29:40 +0200

[Message part 1 (text/plain, inline)]

Hi Lance,

Thanks for your reply!

On Tuesday, 23 August 2022 14:33:11 CEST Lance Lin wrote:
> > 1. Always let jobs finish cleanly: TimeoutStopSec=infinity
> > I strongly prefer this option in all cases (desktop/server/...).
> 
> Sure, I think this is easy enough to do and it does make sense.
> 
> I can push a patch for this.

Sounds great, I also think this is the best way to solve it. I did some local 
testing already with long-running jobs and can confirm TimeoutStopSec=infinity 
in the [Service] section works perfectly. Anacron will finish its current job 
cleanly (cron.daily etc) and only then stop/restart.

> > 2. SIGUSR1 anacron as is the case now, then on timeout SIGTERM to all
> > processes in the group, then on timeout again SIGKILL all processes in the
> > group. I must admit I don't know how to implement this with systemd
> > services.
> This change is more involved. At present, anacron is in "legacy" status. I
> recently picked up the package as part of the cronie transition. cronie is
> expected to replace cron/anacron in the future and is actively developed by
> Fedora. I would suggest we place major changes/improvements in that
> project.

Makes sense, I fully agree with you. Hotfixing anacron by disabling timeout 
should be all that's needed until the cronie transition is complete.

Cheers,

-- 
Melvin Vermeeren
Systems engineer

[signature.asc (application/pgp-signature, inline)]

Message #53 received at 915379-close@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 915379-close@bugs.debian.org

Subject: Bug#915379: fixed in anacron 2.3-34

Date: Wed, 31 Aug 2022 10:48:58 +0000

Source: anacron
Source-Version: 2.3-34
Done: Lance Lin <LQi254@protonmail.com>

We believe that the bug you reported is fixed in the latest version of
anacron, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 915379@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Lance Lin <LQi254@protonmail.com> (supplier of updated anacron package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Thu, 25 Aug 2022 20:19:37 +0700
Source: anacron
Architecture: source
Version: 2.3-34
Distribution: unstable
Urgency: medium
Maintainer: Lance Lin <LQi254@protonmail.com>
Changed-By: Lance Lin <LQi254@protonmail.com>
Closes: 915379
Changes:
 anacron (2.3-34) unstable; urgency=medium
 .
   [ Melvin Vermeeren ]
   * d/anacron.service: add TimeoutStopSec=infinity (Closes: #915379)
Checksums-Sha1:
 90f93572ad77dcd188110ce84a273ff520a523c7 1807 anacron_2.3-34.dsc
 9a7d4c843117dc45729a39c482e62fc041753021 21188 anacron_2.3-34.debian.tar.xz
 3c89ddf310e5175e8d9dd073ddae8c7c4a7a95fa 6044 anacron_2.3-34_source.buildinfo
Checksums-Sha256:
 0aef2a4a7b7860ee5a07248bcb343f32988c17d46eae97b5d5d84a52c06cecea 1807 anacron_2.3-34.dsc
 ae42974356af5748868e0738919a2ef01ccb6bcdb276f2332523ebc74431e788 21188 anacron_2.3-34.debian.tar.xz
 d9919934430b6ffef27425b8addf62a4046d08e16406e9b43faf619596681499 6044 anacron_2.3-34_source.buildinfo
Files:
 d7a961ba0dcd7d8e1fc92d8602531192 1807 admin optional anacron_2.3-34.dsc
 953c920d7595a7b4e7ed0c6612caaec5 21188 admin optional anacron_2.3-34.debian.tar.xz
 bff51209301882a92053d34707c2dd44 6044 admin optional anacron_2.3-34_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEkjZVexcMh/iCHArDweDZLphvfH4FAmMPN6AACgkQweDZLphv
fH7Hrw/+IxYF+ONSdpdXwErJVsb70frn9ZHz7plG4/PQ2PTlwni81tGye/GiN25S
8uJK8hJGnV+pEFNr81GGQpy7i3p5QJw/6+XdbvjUM4qhumhP1//goYuezlOhrjHG
HtJyD7JUGmOQSK/3jSuMTBNJ0nb22k3OC7pPk9dgdjawExmoIEpMX84+rkSRRyo1
iM82BZ5wPycqRiACouZADRzwjfq2Ph/hjHPzWIKtQ/yGoLwWWStKNVOYEjlyLCMS
Y8YF0LO2weaTDTiqW/aN0q4brrQuKIGfzwoc8XNY9X1W8eoYs9Y+vnVKWYhc4tR5
zD9FO/25JzHZp1Pa/oxUnIC0s42MMMD1Dy7YP+Gg2qnvyyprAUtktQYojQFjoOzc
S+NkCqXfxH/D0e3+ekw9rs62geTog5foD+KwNKfnzFH5zOIMzGwMAQthDdFMRmAJ
8G+DhyTTejASnd1jmeZrcXaejlkemr7qA5JUAMPbNX5C/Xe47oLv0OKdTqx6nejL
7UZCldHUun8usc6BeB42TU+NOKlggRO8GmXZDdc4YB8VlJy2aT2ytWIWKdZT3QTb
b+uCybTnZNZ1rb2Vb1IG2F3fECnKXkimlpDjVawcOjjPZEE3YxmTXW+55S/Hr6G9
focrXj98FlAEmNKrrukJ42QVIDwlaB4OOOHGlOQrZQ5DVPbFMFE=
=2oog
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.