Debian Bug report logs - #176201
findutils: xargs enviroment size limited to 20k

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: bob@proulx.com (Bob Proulx)

To: submit@bugs.debian.org

Subject: findutils: xargs enviroment size limited to 20k

Date: Fri, 10 Jan 2003 17:23:22 -0700

Package: findutils
Version: 4.1.7-2.1
Severity: important

The 'xargs' program from findutils complains about environment sizes
larger than 20kb in size.  Instead it spuriously detects a
non-existent error and exits with an error message.  This problem only
exhibits itself when the environment is larger than 20480 characters.

[It is ironical that a program designed to avoid ARG_MAX limits
creates an artifically low limit itself. :-)]

This problem was found through normal use of the system.  Some
programs required a large amount of environment space to hold special
PATH and other environment variables.  When the environment grew to be
larger than 20kb the xargs command stop working even though this was
well within the system ARG_MAX limits.

Here is one method to recreate the problem.  Verify that the current
ARG_MAX is sufficient in size.  Create an environment that is larger
than the current xargs limit.  Then trivially execute xargs.

  getconf ARG_MAX
  131072

  export foo=x
  while [ $(env | wc -c) -lt 20480 ]; do
    export foo="$foo$foo"
  done

  env |wc -c
    23038

  xargs
  xargs: environment is too large for exec

The xargs -s option should be capable of overriding this value and
specifying a different buffer size for xargs.  But it fails to work in
this case.  Here is the result.

  xargs -s 32768
  xargs: environment is too large for exec

Looking at the code and the comments therein it appears that someone
was trying to handle arbitrarily sized arg_max values.  Unfortunately
that goal was not met.

Here is a patch to the code which I believe solves these problems in
the best way.  The man page for xargs concerning the -s option was
also updated to reflect this change.

After implementing this patch the above test case functions as
expected.  The -s max-chars option now functions properly and large
environment sizes up to the system ARG_MAX are possible.  I changed
the error message to better reflect what was happening.  Will that
cause any trouble with translators?  If so that particular change is
not significantly important.

Please let me know if you find any problems.

Bob Proulx <bob@proulx.com>

diff -ru ../findutils-4.1.7.original/debian/changelog ./debian/changelog
--- ../findutils-4.1.7.original/debian/changelog	Mon Jan  6 18:01:41 2003
+++ ./debian/changelog	Fri Jan 10 16:23:41 2003
@@ -1,3 +1,11 @@
+findutils (4.1.7-2.1.1) unstable; urgency=low
+
+  * Fixed bug where xargs would fail to run if a large environment
+    size was present.  Was limited to 20k max.  Now limited by the
+    dynamic system value.
+
+ -- Bob Proulx <bob@proulx.com>  Fri, 10 Jan 2003 15:06:56 -0700
+
 findutils (4.1.7-2.1) unstable; urgency=low
 
   * Non-Maintainer upload.
diff -ru ../findutils-4.1.7.original/xargs/xargs.1 ./xargs/xargs.1
--- ../findutils-4.1.7.original/xargs/xargs.1	Sun Feb  4 13:35:11 1996
+++ ./xargs/xargs.1	Fri Jan 10 16:27:02 2003
@@ -87,8 +87,8 @@
 .I "\-\-max-chars=max-chars, \-s max-chars"
 Use at most \fImax-chars\fR characters per command line, including the
 command and initial arguments and the terminating nulls at the ends of
-the argument strings.  The default is as large as possible, up to 20k
-characters.
+the argument strings.  The default is 20k characters plus the current
+environment size.
 .TP
 .I "\-\-verbose, \-t"
 Print the command line on the standard error output before executing
diff -ru ../findutils-4.1.7.original/xargs/xargs.c ./xargs/xargs.c
--- ../findutils-4.1.7.original/xargs/xargs.c	Mon Jan  6 18:01:41 2003
+++ ./xargs/xargs.c	Fri Jan 10 16:21:41 2003
@@ -268,6 +268,8 @@
   int optc;
   int always_run_command = 1;
   long orig_arg_max;
+  long arg_size;
+  long cur_env_size = env_size (environ);
   char *default_cmd = "/bin/echo";
   int (*read_args) PARAMS ((void)) = read_line;
 
@@ -279,23 +281,14 @@
   bindtextdomain (PACKAGE, LOCALEDIR);
   textdomain (PACKAGE);
 
-  orig_arg_max = ARG_MAX;
+  orig_arg_max = ARG_MAX; 	/* may be defined as sysconf(_SC_ARG_MAX) */
   if (orig_arg_max == -1)
     orig_arg_max = LONG_MAX;
   orig_arg_max -= 2048; /* POSIX.2 requires subtracting 2048.  */
   arg_max = orig_arg_max;
 
-  /* Sanity check for systems with huge ARG_MAX defines (e.g., Suns which
-     have it at 1 meg).  Things will work fine with a large ARG_MAX but it
-     will probably hurt the system more than it needs to; an array of this
-     size is allocated.  */
-  if (arg_max > 20 * 1024)
-    arg_max = 20 * 1024;
-
-  /* Take the size of the environment into account.  */
-  arg_max -= env_size (environ);
-  if (arg_max <= 0)
-    error (1, 0, _("environment is too large for exec"));
+  /* Default to 20 kb above the current environment size. */
+  arg_size = 20 * 1024 + cur_env_size;
 
   while ((optc = getopt_long (argc, argv, "+0e::i::l::n:prs:txP:",
 			      longopts, (int *) 0)) != -1)
@@ -344,7 +337,7 @@
 	  break;
 
 	case 's':
-	  arg_max = parse_num (optarg, 's', 1L, orig_arg_max);
+	  arg_size = parse_num (optarg, 's', 1L, orig_arg_max);
 	  break;
 
 	case 't':
@@ -386,6 +379,15 @@
       argc = 1;
       argv = &default_cmd;
     }
+
+  /* Determine the size of the array to be allocated to hold the
+     arguments.  Do not use arg_max directly since that may be large.
+     Instead use the default value or the caller specified value. */
+  if (arg_size < arg_max)
+    arg_max = arg_size;
+
+  if (cur_env_size > arg_max) /* new arg_max modified by arg_size */
+    error (1, 0, _("environment is too large for max-chars size"));
 
   linebuf = (char *) xmalloc (arg_max + 1);
   argbuf = (char *) xmalloc (arg_max + 1);

Message #12 received at 176201@bugs.debian.org (full text, mbox, reply):

From: James Youngman <jay@gnu.org>

To: 176201@bugs.debian.org

Cc: bob@proulx.com

Subject: Fixed in upstream CVS

Date: Sat, 13 Mar 2004 15:01:47 +0000

I have just fixed this in the upstream CVS code.

Message #17 received at 176201@bugs.debian.org (full text, mbox, reply):

From: Andreas Metzler <ametzler@downhill.at.eu.org>

To: James Youngman <jay@gnu.org>, 176201@bugs.debian.org

Subject: Re: Bug#176201: Fixed in upstream CVS

Date: Sat, 13 Mar 2004 16:54:46 +0100

tags 176201 fixed-upstream
thanks
On 2004-03-13 James Youngman <jay@gnu.org> wrote:
> I have just fixed this in the upstream CVS code.

Thanks for the note.
           cu andreas

Message #34 received at 176201-close@bugs.debian.org (full text, mbox, reply):

From: Andreas Metzler <ametzler@debian.org>

To: 176201-close@bugs.debian.org

Subject: Bug#176201: fixed in findutils 4.1.20-5

Date: Sun, 21 Nov 2004 05:45:05 -0500

Source: findutils
Source-Version: 4.1.20-5

We believe that the bug you reported is fixed in the latest version of
findutils, which is due to be installed in the Debian FTP archive:

findutils_4.1.20-5.diff.gz
  to pool/main/f/findutils/findutils_4.1.20-5.diff.gz
findutils_4.1.20-5.dsc
  to pool/main/f/findutils/findutils_4.1.20-5.dsc
findutils_4.1.20-5_i386.deb
  to pool/main/f/findutils/findutils_4.1.20-5_i386.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 176201@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Andreas Metzler <ametzler@debian.org> (supplier of updated findutils package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.7
Date: Sun, 21 Nov 2004 10:48:23 +0100
Source: findutils
Binary: findutils
Architecture: source i386
Version: 4.1.20-5
Distribution: unstable
Urgency: low
Maintainer: Andreas Metzler <ametzler@debian.org>
Changed-By: Andreas Metzler <ametzler@debian.org>
Description: 
 findutils  - utilities for finding files--find, xargs, and locate
Closes: 176201
Changes: 
 findutils (4.1.20-5) unstable; urgency=low
 .
   * Chuan-kai Lin <cklin@debian.org> has offered to serve as backup
     maintainer. Thanks. Add him to Uploaders.
   * xargs now works eve if the environment is larger than 20KB.
     (Closes: #176201)
Files: 
 4f5eb283e0e2ee5d4c39bffede3f059e 653 base required findutils_4.1.20-5.dsc
 b407c0d31b5c18f820040db6a7d60bfa 14158 base required findutils_4.1.20-5.diff.gz
 d938774c0d90ca23baeecd0dfe6d2a41 151532 base required findutils_4.1.20-5_i386.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBoGnfHTOcZYuNdmMRAv2vAJ4pAh9Oum18LuNFIJHTkdmTumXLjgCdGvkD
EUFSdwngL8HKxysIM0pkqaI=
=AAaH
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.