Debian Bug report logs - #505718
Please limit grep output, can cause automatic DOS.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Laurent Fousse <laurent@komite.net>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: grep: Ridiculous memory usage

Date: Fri, 14 Nov 2008 16:41:56 +0100

Package: grep
Version: 2.5.1.ds2-6
Severity: important

Hello,

A grep for a simple pattern on a big file results in a similarly big
memory consumption, rendering the machine unusable as it swaps as hell
and/or a memory allocation failure occurs. E.g.:

    $ grep foo /dev/zero
    grep: /dev/zero: Cannot allocate memory

I noticed this problem because of a simple `grep php' test in
chkrootkit (launched from tiger) making my machine unusable.

I can reproduce this problem with 2.5.3~dfsg-6 on amd64.

Regards,

Laurent.

-- System Information:
Debian Release: 4.0
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.18-6-amd64
Locale: LANG=en_US, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF-8)

Versions of packages grep depends on:
ii  libc6                  2.3.6.ds1-13etch7 GNU C Library: Shared libraries

grep recommends no packages.

-- no debconf information

Message #10 received at submit@bugs.debian.org (full text, mbox, reply):

From: "Aioanei Rares" <debian.dev.list@gmail.com>

To: "Laurent Fousse" <laurent@komite.net>, 505718@bugs.debian.org

Cc: "Debian Bug Tracking System" <submit@bugs.debian.org>

Subject: Re: Bug#505718: grep: Ridiculous memory usage

Date: Fri, 14 Nov 2008 17:49:35 +0200

[Message part 1 (text/plain, inline)]

On Fri, Nov 14, 2008 at 5:41 PM, Laurent Fousse <laurent@komite.net> wrote:

> Package: grep
> Version: 2.5.1.ds2-6
> Severity: important
>
> Hello,
>
> A grep for a simple pattern on a big file results in a similarly big
> memory consumption, rendering the machine unusable as it swaps as hell
> and/or a memory allocation failure occurs. E.g.:
>
>    $ grep foo /dev/zero
>    grep: /dev/zero: Cannot allocate memory
>
> I noticed this problem because of a simple `grep php' test in
> chkrootkit (launched from tiger) making my machine unusable.
>
> I can reproduce this problem with 2.5.3~dfsg-6 on amd64.
>
> Regards,
>
> Laurent.
>
> -- System Information:
> Debian Release: 4.0
>  APT prefers stable
>  APT policy: (500, 'stable')
> Architecture: amd64 (x86_64)
> Shell:  /bin/sh linked to /bin/bash
> Kernel: Linux 2.6.18-6-amd64
> Locale: LANG=en_US, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8) (ignored: LC_ALL
> set to en_US.UTF-8)
>
> Versions of packages grep depends on:
> ii  libc6                  2.3.6.ds1-13etch7 GNU C Library: Shared
> libraries
>
> grep recommends no packages.
>
> -- no debconf information
>
>
>
> --
> To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact
> listmaster@lists.debian.org
>
>
Maybe the kernel is at fault?

[Message part 2 (text/html, inline)]

Message #22 received at 505718@bugs.debian.org (full text, mbox, reply):

From: Laurent Fousse <laurent@komite.net>

To: Aioanei Rares <debian.dev.list@gmail.com>

Cc: 505718@bugs.debian.org

Subject: Re: Bug#505718: grep: Ridiculous memory usage

Date: Fri, 14 Nov 2008 17:03:30 +0100

Hello,

* Aioanei Rares [Fri, Nov 14, 2008 at 05:49:35PM +0200]:
> Maybe the kernel is at fault?

The particular argument `/dev/zero/' I supplied to grep in my example
is not necessary, if this is what you mean. I can reproduce it with
any big file. Since /dev/zero is infinite in size, it will eventually
lead to an allocation error.

Laurent.

Message #27 received at 505718@bugs.debian.org (full text, mbox, reply):

From: Nicolas François <nicolas.francois@centraliens.net>

To: Laurent Fousse <laurent@komite.net>, 505718@bugs.debian.org

Subject: Re: Bug#505718: grep: Ridiculous memory usage

Date: Fri, 14 Nov 2008 20:45:57 +0100

On Fri, Nov 14, 2008 at 05:03:30PM +0100, Laurent Fousse wrote:
> Hello,
> 
> * Aioanei Rares [Fri, Nov 14, 2008 at 05:49:35PM +0200]:
> > Maybe the kernel is at fault?
> 
> The particular argument `/dev/zero/' I supplied to grep in my example
> is not necessary, if this is what you mean. I can reproduce it with
> any big file. Since /dev/zero is infinite in size, it will eventually
> lead to an allocation error.

grep is line-based, so I'm not really surprised it eats all the memory
when it receives only zeroes.

What other "big file" did you tested this on?

Can you test if using "LC_ALL=C grep" instead helps or if "grep -F" works?

If you are greping in a binary file, I would recommend using strings +
grep instead.

For chkrootkit, I guess checking for fixed string (at least when we can
expect long greps) is sufficient and would fasten it a lot.

Best Regards,
-- 
Nekral

Message #32 received at 505718@bugs.debian.org (full text, mbox, reply):

From: Laurent Fousse <laurent@komite.net>

To: Nicolas François <nicolas.francois@centraliens.net>

Cc: 505718@bugs.debian.org

Subject: Re: Bug#505718: grep: Ridiculous memory usage

Date: Sat, 15 Nov 2008 11:10:51 +0100

Hello,

* Nicolas François [Fri, Nov 14, 2008 at 08:45:57PM +0100]:
> grep is line-based, so I'm not really surprised it eats all the memory
> when it receives only zeroes.
> 
> What other "big file" did you tested this on?

A 2Gb swap file. It could be that the first newline in this file is
far from the beginning, I've deleted it since.

> For chkrootkit, I guess checking for fixed string (at least when we can
> expect long greps) is sufficient and would fasten it a lot.

It was in fact checking for fixed string when it nearly stopped my
machine, the invocation was `grep php'.

I see that since grep usually prints out matching lines, it needs to
allocate enough memory for the current line. Thanks for pointing that
out. I guess this bug becomes a wishlist bug against chkrootkit.

Regards,

Laurent.

Message #37 received at 505718@bugs.debian.org (full text, mbox, reply):

From: Laurent Fousse <laurent@komite.net>

To: control@bugs.debian.org, 505718@bugs.debian.org

Subject: Re: Bug#505718: grep: Ridiculous memory usage

Date: Mon, 17 Nov 2008 16:45:00 +0100

reassign 505718 chkrootkit
severity 505718 normal
retitle  505718 Please limit grep output, can cause automatic DOS.
thanks

Hello,

While checking e.g. for php based rootkit, the invocation of grep does
not limit the amount of output returned, for example in:

    fileshead="`${find} ${ROOTDIR}tmp ${ROOTDIR}var/tmp ${findargs} -type f -exec head -1 {} \; | grep php 2> /dev/null`"

I had a 2Gb swap file in /tmp and grep kept the current "line" in
memory waiting to see if it would match 'php'. For some reason this
swap file was composed of very long "lines", and my machine was
unreasonably slow because of this grep. Generally speaking, grep's
behaviour here is correct because it needs to store all of the current
line to be able to display it if it matches, but some sort of output
limitation should be used by chkrootkit to avoid a local DOS by the
cron job.

Regards,

Laurent.

Message #48 received at 505718@bugs.debian.org (full text, mbox, reply):

From: Francois Marier <francois@debian.org>

To: 505718@bugs.debian.org

Subject: Clarification on the bug title

Date: Tue, 18 Nov 2008 10:01:38 +1300

Hi Laurent,

I assume that you mean that the output piped to grep (i.e. grep's input)
should be limited (not the output from grep)?

Cheers,
Francois

Message #53 received at 505718@bugs.debian.org (full text, mbox, reply):

From: Francois Marier <francois@debian.org>

To: 505718@bugs.debian.org

Subject: Trucating long lines using cut?

Date: Tue, 18 Nov 2008 13:08:15 +1300

So basically, adding "| cut -b1-1024" before the call to grep should do the
trick?

Francois

Message #58 received at 505718@bugs.debian.org (full text, mbox, reply):

From: Laurent Fousse <laurent.fousse@imag.fr>

To: Francois Marier <francois@debian.org>

Cc: 505718@bugs.debian.org

Subject: Re: Clarification on the bug title

Date: Tue, 18 Nov 2008 10:28:29 +0100

Hello,

* Francois Marier [Tue, Nov 18, 2008 at 10:01:38AM +1300]:
> I assume that you mean that the output piped to grep (i.e. grep's input)
> should be limited (not the output from grep)?

* Francois Marier [Tue, Nov 18, 2008 at 01:08:15PM +1300]:
> So basically, adding "| cut -b1-1024" before the call to grep should do the
> trick?

If you can guarantee that the expected 'php' string will happen in the
first kb of each line, yes.

I really meant to ask grep for a limited output, because if you can't
blame grep for using a lot of memory when it has to store a whole line
unbounded in size, you can expect it to use less memory when you
request a bounded output.

But your solution is likely the best, provided you don't "cut" useful
information.

Regards,

Laurent.

Message #63 received at 505718-done@bugs.debian.org (full text, mbox, reply):

From: Giuseppe Iuculano <giuseppe@iuculano.it>

To: 505718-done@bugs.debian.org, 505718-submitter@bugs.debian.org

Subject: Bug #505718: Please limit grep output, can cause automatic DOS

Date: Thu, 27 Nov 2008 10:20:10 +0100

[Message part 1 (text/plain, inline)]

Source: chkrootkit
Source-Version: 0.48-7


> If you can guarantee that the expected 'php' string will happen in the
> first kb of each line, yes.

Hi,


With chkrootkit 0.48-7 the "suspect PHP files" check was removed:

* debian/patches/nophpcheck.dpatch: Delete the "suspect PHP files" check.
    Not only does it trigger SIGPIPE for file names which contain special
    unescaped characters, the second half is doubtful (it doesn't print any
    filenames and gets confused by binary file contents). (Closes: #479187)


So this issue should be fixed.

Feel free to reopeon if necessary.

Cheers,

Giuseppe.

[signature.asc (application/pgp-signature, attachment)]

Send a report that this bug log contains spam.