Debian Bug report logs - #711291
php5 cron file can run multiple times in parallel

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: "Alexander E. Patrakov" <patrakov@gmail.com>

To: submit@bugs.debian.org

Subject: php5 cron file can run multiple times in parallel

Date: Thu, 6 Jun 2013 12:36:18 +0600

Package: php5
Version: 5.4.4-14
Severity: normal

Dear Maintainer,

Some bad guys have created a lot of PHP sessions on my server. So many
that it takes more than 30 minutes to remove them for the default PHP5
cron script that runs a fuser process for each session file to be
removed.

While the old instance of the cron job was still running, cron decoded
to spawn another one, further increasing the load average. You could
avoid that by using a lock file and mailing the admin if the script
runs over its own tail.

Of course, it would also be nice to avoid running an expensive fuser
process for each to-be-removed file.

-- System Information:
Debian Release: 7.0
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 3.2.0-4-amd64 (SMP w/8 CPU cores)
Locale: LANG=ru_RU.UTF-8, LC_CTYPE=ru_RU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages php5 depends on:
ii  libapache2-mod-php5  5.4.4-14
ii  php5-common          5.4.4-14

php5 recommends no packages.

php5 suggests no packages.

-- no debconf information

-- 
Alexander E. Patrakov

Message #10 received at 711291@bugs.debian.org (full text, mbox, reply):

From: Ondřej Surý <ondrej@debian.org>

To: "Alexander E. Patrakov" <patrakov@gmail.com>, 711291@bugs.debian.org

Subject: Re: [php-maint] Bug#711291: php5 cron file can run multiple times in parallel

Date: Thu, 6 Jun 2013 10:50:37 +0200

[Message part 1 (text/plain, inline)]

Control: fixed -1 php5/5.5.0~alpha1-1
Control: tags -1 wheezy

Hi Alexander,

you might want to pull cron job:
http://anonscm.debian.org/gitweb/?p=pkg-php/php.git;a=blob;f=debian/php5-common.php5.cron.d;h=8865fddb4c451f38f0d6f1ae722a3082c2eb5f81;hb=refs/heads/debian-experimental

and sessionclean script:
http://anonscm.debian.org/gitweb/?p=pkg-php/php.git;a=blob;f=debian/sessionclean;h=a5f5360d015a4e74199f8c71152ab96f0bd4bb33;hb=refs/heads/debian-experimental

I have already improved the session clean script, but it didn't make it
into wheezy, because of lack of testers with too many session files.

Unfortunatelly we cannot avoid looking at the files, because of #626640.

O.


On Thu, Jun 6, 2013 at 8:36 AM, Alexander E. Patrakov <patrakov@gmail.com>wrote:

> Package: php5
> Version: 5.4.4-14
> Severity: normal
>
> Dear Maintainer,
>
> Some bad guys have created a lot of PHP sessions on my server. So many
> that it takes more than 30 minutes to remove them for the default PHP5
> cron script that runs a fuser process for each session file to be
> removed.
>
> While the old instance of the cron job was still running, cron decoded
> to spawn another one, further increasing the load average. You could
> avoid that by using a lock file and mailing the admin if the script
> runs over its own tail.
>
> Of course, it would also be nice to avoid running an expensive fuser
> process for each to-be-removed file.
>
> -- System Information:
> Debian Release: 7.0
>   APT prefers stable
>   APT policy: (500, 'stable')
> Architecture: amd64 (x86_64)
>
> Kernel: Linux 3.2.0-4-amd64 (SMP w/8 CPU cores)
> Locale: LANG=ru_RU.UTF-8, LC_CTYPE=ru_RU.UTF-8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
>
> Versions of packages php5 depends on:
> ii  libapache2-mod-php5  5.4.4-14
> ii  php5-common          5.4.4-14
>
> php5 recommends no packages.
>
> php5 suggests no packages.
>
> -- no debconf information
>
> --
> Alexander E. Patrakov
>
> _______________________________________________
> pkg-php-maint mailing list
> pkg-php-maint@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-php-maint
>



-- 
Ondřej Surý <ondrej@sury.org>

[Message part 2 (text/html, inline)]

Message #19 received at 711291@bugs.debian.org (full text, mbox, reply):

From: "Alexander E. Patrakov" <patrakov@gmail.com>

To: Ondřej Surý <ondrej@debian.org>

Cc: 711291@bugs.debian.org

Subject: Re: [php-maint] Bug#711291: php5 cron file can run multiple times in parallel

Date: Thu, 6 Jun 2013 15:38:24 +0600

2013/6/6 Ondřej Surý <ondrej@debian.org>:
> Control: fixed -1 php5/5.5.0~alpha1-1
> Control: tags -1 wheezy
>
> Hi Alexander,
>
> you might want to pull cron job:
> http://anonscm.debian.org/gitweb/?p=pkg-php/php.git;a=blob;f=debian/php5-common.php5.cron.d;h=8865fddb4c451f38f0d6f1ae722a3082c2eb5f81;hb=refs/heads/debian-experimental
>
> and sessionclean script:
> http://anonscm.debian.org/gitweb/?p=pkg-php/php.git;a=blob;f=debian/sessionclean;h=a5f5360d015a4e74199f8c71152ab96f0bd4bb33;hb=refs/heads/debian-experimental
>
> I have already improved the session clean script, but it didn't make it into
> wheezy, because of lack of testers with too many session files.
>
> Unfortunatelly we cannot avoid looking at the files, because of #626640.

I see. The new script does avoid a per-session-file process call, and
that should be much better than the current situation. However, I (as
a person who reviews your code) disagree that the bug has been fixed
completely, because it is still possible that two instances of
sessionclean run in parallel from cron and thus uselessly raise the
load average. Please use a lock file (e.g. with flock -w 0
/run/lock/php5.lck prepended to the original sessionclean invocation)
to avoid that issue. As a bonus, this will generate a cron mail
(without raising LA) if the script does run over its own tail.

--
Alexander E. Patrakov

Message #24 received at 711291@bugs.debian.org (full text, mbox, reply):

From: Ondřej Surý <ondrej@debian.org>

To: "Alexander E. Patrakov" <patrakov@gmail.com>

Cc: 711291@bugs.debian.org

Subject: Re: [php-maint] Bug#711291: php5 cron file can run multiple times in parallel

Date: Thu, 6 Jun 2013 12:27:51 +0200

[Message part 1 (text/plain, inline)]

On Thu, Jun 6, 2013 at 11:38 AM, Alexander E. Patrakov
<patrakov@gmail.com>wrote:

> 2013/6/6 Ondřej Surý <ondrej@debian.org>:
> > Control: fixed -1 php5/5.5.0~alpha1-1
> > Control: tags -1 wheezy
> >
> > Hi Alexander,
> >
> > you might want to pull cron job:
> >
> http://anonscm.debian.org/gitweb/?p=pkg-php/php.git;a=blob;f=debian/php5-common.php5.cron.d;h=8865fddb4c451f38f0d6f1ae722a3082c2eb5f81;hb=refs/heads/debian-experimental
> >
> > and sessionclean script:
> >
> http://anonscm.debian.org/gitweb/?p=pkg-php/php.git;a=blob;f=debian/sessionclean;h=a5f5360d015a4e74199f8c71152ab96f0bd4bb33;hb=refs/heads/debian-experimental
> >
> > I have already improved the session clean script, but it didn't make it
> into
> > wheezy, because of lack of testers with too many session files.
> >
> > Unfortunatelly we cannot avoid looking at the files, because of #626640.
>
> I see. The new script does avoid a per-session-file process call, and
> that should be much better than the current situation. However, I (as
> a person who reviews your code) disagree that the bug has been fixed
> completely, because it is still possible that two instances of
> sessionclean run in parallel from cron and thus uselessly raise the
> load average. Please use a lock file (e.g. with flock -w 0
> /run/lock/php5.lck prepended to the original sessionclean invocation)
> to avoid that issue. As a bonus, this will generate a cron mail
> (without raising LA) if the script does run over its own tail.
>

While this might save some cycles if you are cleaning a result of an
onetime action, the locking will in fact make the load worse if it is
result of an ongoing sessions generation.

If you postpone the cron job run by half an hour, the number of sessions
will grow by that half an hour.

Let me model that...

Let's say:
- the number of sessions you can clean in an half an hour is 1.000.000.
- the bad guy generates 5.000.000 session files as one shot thing.
- the normal number of generated sessions per hour is 100.000.

The lock file will cause the files to be cleaned in 3 hours (2.5 hours for
the 5 mio + next run for generated 250.000 normal session file which will
be generated meanwhile).

But if it's not onetime thing, let's say:
- the number of sessions you can clean in an half an hour is 1.000.000.
- the number of sessions of generated per hour is 2.100.000

The lock file will cause the sessions to never clear because:
- first run (ETA +0 min) will run for 1+ hour (making two subsequent ETA
+30 min and +60 min runs to be stopped)
- the number of sessions which will be generated meanwhile is more than
3.150.000
- the next script (ETA +90 min) will have to clean 3 mio+ session files
making it run for ~1.6 hours stopping three subsequent runs
- repeat and rinse until you run out of inodes

So I actually thinks it's better to suffer from temporary load spikes (with
the new script – the old one is crazy, but it took me a while to figure out
how to fix it) than to be DoSed by sessions files not cleaning fast enough.

O.
-- 
Ondřej Surý <ondrej@sury.org>

[Message part 2 (text/html, inline)]

Message #29 received at 711291@bugs.debian.org (full text, mbox, reply):

From: "Pavel V." <pavel2000@ngs.ru>

To: 711291@bugs.debian.org

Subject: Bug#711291, lack of testers with too many session files.

Date: Mon, 24 Feb 2014 13:25:18 +0700

Hi, all.

>you might want to pull cron job:
>http://anonscm.debian.org/gitweb/?p=pkg-php/php.git;a=blob;f=debian/php5-common.php5.cron.d;h=8865fddb4c451f38f0d6f1ae722a3082c2eb5f81;hb=refs/heads/debian-experimental
>
>and sessionclean script:
>http://anonscm.debian.org/gitweb/?p=pkg-php/php.git;a=blob;f=debian/sessionclean;h=a5f5360d015a4e74199f8c71152ab96f0bd4bb33;hb=refs/heads/debian-experimental
>
>I have already improved the session clean script, but it didn't make it
>into wheezy, because of lack of testers with too many session files.

I have found https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=711291#10 while trying to resolve my
problem with "find ... fuser" on directory with 140k sessions and decided to try proposed solution.

New solution works much-much better than old, but while making tests I got the following message:

root@server:/var/lib/php5# [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 180
find: warning: you have specified the -ignore_readdir_race option after a non-option argument -type, but options are not positional
(-ignore_readdir_race affects tests specified before it as well as those specified after it).  Please specify options before other arguments.

I have Debian 7.3 with findutils-4.4.2-4 amd64

After I changed options order, message has gone away. My improved variant is:

# find all files older then maxlifetime
find "${1}" -depth -mindepth 1 -maxdepth 1 -ignore_readdir_race -type f -cmin +${2} -delete

Thanks for your work, maintainers!
Hope my report helps you.

-- 
С уважением,
 Pavel                          mailto:pavel2000@ngs.ru

Message #34 received at 711291@bugs.debian.org (full text, mbox, reply):

From: Ondřej Surý <ondrej@sury.org>

To: "Pavel V." <pavel2000@ngs.ru>, 711291@bugs.debian.org

Subject: Re: [php-maint] Bug#711291: , lack of testers with too many session files.

Date: Tue, 25 Feb 2014 11:11:53 +0100

Hi Pavel.

On Mon, Feb 24, 2014, at 7:25, Pavel V. wrote:
> New solution works much-much better than old

Thanks for the report.

> but while making tests I got the following message:
[...]

This was already reported in 720364 and fixed in 5.5.4+dfsg-1. Somehow
it didn't make it into changelog.

Ondrej
-- 
Ondřej Surý <ondrej@sury.org>
Knot DNS (https://www.knot-dns.cz/) – a high-performance DNS server

Message #43 received at 711291@bugs.debian.org (full text, mbox, reply):

From: Richard Laager <rlaager@wiktel.com>

To: 711291@bugs.debian.org

Cc: ondrej@debian.org

Subject: Re: [php-maint] Bug#711291: php5 cron file can run multiple times in parallel

Date: Mon, 18 Apr 2016 12:26:49 -0500

On Thu, 6 Jun 2013 10:50:37 +0200 Ondřej Surý <ondrej@debian.org> wrote:
> and sessionclean script:
> http://anonscm.debian.org/gitweb/?p=pkg-php/php.git;a=blob;f=debian/sessionclean;h=a5f5360d015a4e74199f8c71152ab96f0bd4bb33;hb=refs/heads/debian-experimental

I haven't reviewed it closely, but at a first glance, this rewritten 
script will probably also solve a problem I'm seeing. With the current 
script, if a process quits after the pidof call, but before `find 
/proc/$pid/fd ...`, find prints this sort of message to stderr:

find: /proc/468/fd: No such file or directory

So that's another plus to getting this change in.

-- 
Richard

Message #48 received at 711291@bugs.debian.org (full text, mbox, reply):

From: Ondřej Surý <ondrej@sury.org>

To: Richard Laager <rlaager@wiktel.com>

Cc: nullnull <711291@bugs.debian.org>

Subject: Re: [php-maint] Bug#711291: php5 cron file can run multiple times in parallel

Date: Mon, 18 Apr 2016 20:03:00 +0200

[Message part 1 (text/plain, inline)]

This is already fixed in current stable Debian release (jessie) and oldstable (wheezy) won't be updated. 

Either pull the updated file by hand and use dpkg-divert to keep it in place or just update to current stable release. 

Cheers, 
-- 
Ondřej Surý 



On 18 Apr 2016 19:32, at 19:32, Richard Laager <rlaager@wiktel.com> wrote:
>On Thu, 6 Jun 2013 10:50:37 +0200 Ondřej Surý <ondrej@debian.org>
>wrote:
>> and sessionclean script:
>>
>http://anonscm.debian.org/gitweb/?p=pkg-php/php.git;a=blob;f=debian/sessionclean;h=a5f5360d015a4e74199f8c71152ab96f0bd4bb33;hb=refs/heads/debian-experimental
>
>I haven't reviewed it closely, but at a first glance, this rewritten 
>script will probably also solve a problem I'm seeing. With the current 
>script, if a process quits after the pidof call, but before `find 
>/proc/$pid/fd ...`, find prints this sort of message to stderr:
>
>find: /proc/468/fd: No such file or directory
>
>So that's another plus to getting this change in.
>
>-- 
>Richard

[Message part 2 (text/html, inline)]

Send a report that this bug log contains spam.