Debian Bug report logs - #316930
Updatedb doesn't run 'find' at the nice priority set in updatedb.conf

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: David Jarvie <software@astrojar.org.uk>

To: submit@bugs.debian.org

Subject: Updatedb doesn't run 'find' at the nice priority set in updatedb.conf

Date: Mon, 4 Jul 2005 22:40:44 +0100

Package: findutils
Version: 4.2.22-1

My /etc/updatedb.conf contains (among other things):

    LOCALUSER="root"
    export LOCALUSER
    NICE=19
    export NICE

and when updatedb is run by cron from /etc/cron.daily/find, it runs at the 
expected nice priority of 19. However, when updatedb runs 'find', find's nice 
level is 0, not 19. Running /etc/cron.daily/find manually as root produces 
the same result.

The output from running it manually and then checking the status with  'ps -eo 
"%p %P %n %a"' was:

28410  4824   0 /bin/sh /etc/cron.daily/find
28412 28410  19 /bin/sh /usr/bin/updatedb
28420 28412  19 /bin/sh /usr/bin/updatedb
28422 28412  19 /usr/bin/sort -z -f
28423 28412  19 /usr/lib/locate/frcode -0
28426 28420   0 /usr/bin/find / -ignore_readdir_race ( -fstype NFS -o -fstype 
nfs -o -fstyp

I'm running Debian testing i386, with kernel 2.6.8, with libc6 2.3.2.ds1-22. 

This bug really makes my system slow down until find finishes - it would be 
really useful to have it fixed.

Message #10 received at 316930@bugs.debian.org (full text, mbox, reply):

From: Andreas Metzler <ametzler@debian.org>

To: software@astrojar.org.uk, 316930@bugs.debian.org

Subject: Re: Bug#316930: Updatedb doesn't run 'find' at the nice priority set in updatedb.conf

Date: Tue, 5 Jul 2005 08:26:47 +0200

On 2005-07-04 David Jarvie <software@astrojar.org.uk> wrote:
> Package: findutils
> Version: 4.2.22-1

> My /etc/updatedb.conf contains (among other things):

>     LOCALUSER="root"
>     export LOCALUSER
>     NICE=19
>     export NICE

> and when updatedb is run by cron from /etc/cron.daily/find, it runs at the 
> expected nice priority of 19. However, when updatedb runs 'find', find's nice 
> level is 0, not 19.
[...]

Hello,
The cause of this is this that since very recently invoking su uses the
PAM module pam_limits.so which in turn resets the nice-level to 0.

http://bugs.debian.org/311058
http://bugs.debian.org/241663

As a workaround you can disable the entry for pam_limits.so in
/etc/pam.d/su. (Gaining that the nice-level is kept, but losing
safeguardlimits imposed on processes run with su.)
               cu andreas
-- 
"See, I told you they'd listen to Reason," [SPOILER] Svfurlr fnlf,
fuhggvat qbja gur juveyvat tha.
Neal Stephenson in "Snow Crash"
                                           http://downhill.aus.cc/

Message #15 received at 316930@bugs.debian.org (full text, mbox, reply):

From: Paul Slootman <paul@debian.org>

To: 316930@bugs.debian.org

Cc: software@astrojar.org.uk

Subject: Re: Bug#316930: Updatedb doesn't run 'find' at the nice priority set in updatedb.conf

Date: Fri, 24 Mar 2006 21:56:24 +0100

On Tue 05 Jul 2005, Andreas Metzler wrote:
> On 2005-07-04 David Jarvie <software@astrojar.org.uk> wrote:
> > Package: findutils
> > Version: 4.2.22-1
> 
> > My /etc/updatedb.conf contains (among other things):
> 
> >     LOCALUSER="root"
> >     export LOCALUSER
> >     NICE=19
> >     export NICE
> 
> > and when updatedb is run by cron from /etc/cron.daily/find, it runs at the 
> > expected nice priority of 19. However, when updatedb runs 'find', find's nice 
> > level is 0, not 19.
> [...]
> 
> Hello,
> The cause of this is this that since very recently invoking su uses the
> PAM module pam_limits.so which in turn resets the nice-level to 0.
> 
> http://bugs.debian.org/311058
> http://bugs.debian.org/241663
> 
> As a workaround you can disable the entry for pam_limits.so in
> /etc/pam.d/su. (Gaining that the nice-level is kept, but losing
> safeguardlimits imposed on processes run with su.)

A workaround could of course be to export the NICE value as something
unique like FINDUTILS_NICE (to avoid unfortunate conflicts with other
things), and then modify /usr/bin/updatedb to add "nice -n
$FINDUTILS_NICE" in the su -c parameter.


Paul Slootman

Message #20 received at 316930@bugs.debian.org (full text, mbox, reply):

From: Arthur Marsh <arthur.marsh@internode.on.net>

To: Debian Bug Tracking System <316930@bugs.debian.org>

Subject: locate: Still experiencing this problem

Date: Wed, 16 Jan 2008 08:47:32 +1030

Package: locate
Version: 4.2.31-4
Followup-For: Bug #316930


Hi, I had find hogging 99.9 percent of CPU time while running 
findutils.updatedb:

 ps -ef|grep find
root      6828  6824  0 07:39 ?        00:00:00 /bin/sh 
/usr/bin/updatedb.findutils
root      6836  6828  0 07:39 ?        00:00:00 /bin/sh 
/usr/bin/updatedb.findutils
nobody    6844  6836  0 07:39 ?        00:00:00 su nobody -s /bin/sh -c 
/usr/bin/find / -ignore_readdir_race      \( -fstype NFS -o -fstype nfs 
-o -fstype nfs4 -o -fstype afs -o -fstype binfmt_misc -o -fstype proc -o 
-fstype smbfs -o -fstype autofs -o -fstype iso9660 -o -fstype ncpfs -o 
-fstype coda -o -fstype devpts -o -fstype ftpfs -o -fstype devfs -o 
-fstype mfs -o -fstype shfs -o -fstype sysfs -o -fstype cifs -o -fstype 
lustre_lite -o -fstype tmpfs -o -fstype usbfs -o -fstype udf -o      
-type d -regex 
'\(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\|\(^/amd$\)\|\(^/alex$\)\|\(^/var/spool$\)\|\(^/sfs$\)\|\(^/media$\)' 
\) -prune -o -print0
nobody    6845  6844 66 07:39 ?        00:23:20 /usr/bin/find / 
-ignore_readdir_race ( -fstype NFS -o -fstype nfs -o -fstype nfs4 -o 
-fstype afs -o -fstype binfmt_misc -o -fstype proc -o -fstype smbfs -o 
-fstype autofs -o -fstype iso9660 -o -fstype ncpfs -o -fstype coda -o 
-fstype devpts -o -fstype ftpfs -o -fstype devfs -o -fstype mfs -o 
-fstype shfs -o -fstype sysfs -o -fstype cifs -o -fstype lustre_lite -o 
-fstype tmpfs -o -fstype usbfs -o -fstype udf -o -type d -regex 
\(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\|\(^/amd$\)\|\(^/alex$\)\|\(^/var/spool$\)\|\(^/sfs$\)\|\(^/media$\) 
) -prune -o -print0
amarsh04  9626  5237  0 08:14 pts/5    00:00:00 grep find
amarsh04@victoria:~$ ps -ef|grep 6824
root      6824  5813  0 07:39 ?        00:00:00 /bin/sh 
/etc/cron.daily/locate
root      6828  6824  0 07:39 ?        00:00:00 /bin/sh 
/usr/bin/updatedb.findutils
amarsh04 10095  5237  0 08:15 pts/5    00:00:00 grep 6824

I have ionice installed.

My /etc/pam.d/su:

$ grep -v \# /etc/pam.d/su

auth       sufficient pam_rootok.so

session       required   pam_env.so readenv=1
session       required   pam_env.so readenv=1 
envfile=/etc/default/locale

session    optional   pam_mail.so nopen

@include common-auth
@include common-account
@include common-session

Do you have any suggestions for logging what is going on and better 
still ensuring that findutils.updatedb doesn't consume cpu time to the 
exclusion of everyting else?



-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 2.6.24-rc7 (SMP w/1 CPU core; PREEMPT)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages locate depends on:
ii  findutils                     4.2.31-4   utilities for finding files--find,
ii  libc6                         2.7-6      GNU C Library: Shared libraries

locate recommends no packages.

-- debconf-show failed

Message #25 received at 316930@bugs.debian.org (full text, mbox, reply):

From: Arthur Marsh <arthur.marsh@internode.on.net>

To: Debian Bug Tracking System <316930@bugs.debian.org>

Subject: locate: updatedb.findutils running with 99.9 percent cpu

Date: Thu, 17 Jan 2008 09:07:37 +1030

Package: locate
Version: 4.2.31-4
Followup-For: Bug #316930


When 

/etc/cron.daily/locate

was running, 99.9 percent of the cpu time was spent in the "find" 
command as user "nobody", leaving everything else unresponsive.

I ran 

sh -x /etc/cron.daily

+ set -e
+ '[' -e /usr/bin/updatedb.findutils ']'
+ FINDOPTIONS=-ignore_readdir_race
+ PRUNEFS='NFS nfs nfs4 afs binfmt_misc proc smbfs autofs iso9660 ncpfs 
coda devpts ftpfs devfs mfs shfs sysfs cifs lustre_lite tmpfs usbfs udf'
+ PRUNEPATHS='/tmp /usr/tmp /var/tmp /afs /amd /alex /var/spool /sfs 
/media'
+ NETPATHS=
+ LOCALUSER=nobody
+ NICE=10
+ IONICE_CLASS=3
+ IONICE_PRIORITY=7
+ '[' -r /etc/updatedb.findutils.cron.local ']'
+ export FINDOPTIONS PRUNEFS PRUNEPATHS NETPATHS LOCALUSER
+ '[' -x /usr/bin/ionice ']'
+ '[' '' = '' ']'
++ uname -r
+ KVER=2.6.24-rc7
+ case "$KVER" in
+ case "$IONICE_CLASS" in
+ priority=
+ getent passwd nobody
+ cd /
+ ionice -c3 nice -n 10 updatedb.findutils


If this is running as user "nobody" the -c3 option shouldn't work.

I'm still at a loss as to how this ends up using 99.9 percent cpu.

ionice is installed here.

-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 2.6.24-rc7 (SMP w/1 CPU core; PREEMPT)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages locate depends on:
ii  findutils                     4.2.31-4   utilities for finding files--find,
ii  libc6                         2.7-6      GNU C Library: Shared libraries

locate recommends no packages.

-- debconf-show failed

Message #30 received at 316930@bugs.debian.org (full text, mbox, reply):

From: Arthur Marsh <arthur.marsh@internode.on.net>

To: Debian Bug Tracking System <316930@bugs.debian.org>

Subject: findutils: /etc/cron.daily/locate had been hand-edited

Date: Thu, 17 Jan 2008 09:47:28 +1030

Package: findutils
Followup-For: Bug #316930


I purged then reinstalled locate and now the update command line run 
from /etc/cron.daily/locate is:

nice -n 10 updatedb.findutils

I must have manually added the "ionice -c3" to /etc/cron.daily/locate in 
an attempt to get ionice working.

updatedb.findutils is running with a nice value of 10 which is ok but 
can still consume too much of the i/o resources and make other 
applications unresponsive.

Any suggestions?


-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 2.6.24-rc7 (SMP w/1 CPU core; PREEMPT)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages findutils depends on:
ii  libc6                         2.7-6      GNU C Library: Shared libraries

findutils recommends no packages.

-- debconf-show failed

Message #35 received at 316930@bugs.debian.org (full text, mbox, reply):

From: Arthur Marsh <arthur.marsh@internode.on.net>

To: Debian Bug Tracking System <316930@bugs.debian.org>

Subject: findutils: more serious - even at nice 10 consumes 99.9 percent cpu

Date: Thu, 17 Jan 2008 10:28:16 +1030

Package: findutils
Version: 4.2.31-4
Followup-For: Bug #316930


After purging and re-installing locate and running

sh -x /etc/cron.daily/locate

which showed:

nice -n 10 updatedb.findutils

and top reporting the find command running with nice level 10, I would 
still get an unresponsive system with 99.9 percent cpu going to find.

At the time, find was searching through a vfat filesystem.

The problem seems to be deeper than the locate utility as no process 
that has a nice level of 10 should bring everything else to a virtual 
halt.

I am now purging locate from this machine as it takes too much contol of 
the machine.

-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 2.6.24-rc7 (SMP w/1 CPU core; PREEMPT)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages findutils depends on:
ii  libc6                         2.7-6      GNU C Library: Shared libraries

findutils recommends no packages.

-- debconf-show failed

Message #40 received at 316930@bugs.debian.org (full text, mbox, reply):

From: "James Youngman" <jay@gnu.org>

To: "Arthur Marsh" <arthur.marsh@internode.on.net>, 316930@bugs.debian.org

Subject: Re: Bug#316930: findutils: more serious - even at nice 10 consumes 99.9 percent cpu

Date: Thu, 17 Jan 2008 13:26:31 +0000

On Jan 16, 2008 11:58 PM, Arthur Marsh <arthur.marsh@internode.on.net> wrote:
> nice -n 10 updatedb.findutils
>
> and top reporting the find command running with nice level 10, I would
> still get an unresponsive system with 99.9 percent cpu going to find.

Find doesn't normally use that much CPU.   2% would be more usual.
Is the CPU being used in user context (find) or in system context (the
kernel)?


> At the time, find was searching through a vfat filesystem.

Could you provide some more details?   Perhaps by using a command like this:


strace -ttt -T -p    (the PID of the find process)


> The problem seems to be deeper than the locate utility as no process
> that has a nice level of 10 should bring everything else to a virtual
> halt.

This is largely true, but my guess is that the find process is still
using a tiny amount of user CPU time, and that the problem is
somethere in how it is interacting with the VFAT filesystem.

James.

Send a report that this bug log contains spam.