xargs command length limits

xargs -s option
xargs -n
xargs: insufficient space for argument
xargs parallel
xargs variable
opposite of xargs
xargs sh

I am using jsonlint to lint a bunch of files in a directory (recursively). I wrote the following command:

find ./config/pages -name '*.json' -print0 | xargs -0I % sh -c 'echo Linting: %; jsonlint -V ./config/schema.json -q %;'

It works for most files but some files I get the following error:

Linting: ./LONG_FILE_NAME.json
 return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
  Error: ENOENT, no such file or directory '%'

It appears to fail for long filenames. Is there a way to fix this? Thanks.

Edit 1: Found the problem.

-I replstr

Execute utility for each input line, replacing one or more occurrences of replstr in up to replacements (or 5 if no -R flag is specified) arguments to utility with the entire line of input. The resulting arguments, after replacement is done, will not be allowed to grow beyond 255 bytes; this is implemented by concatenating as much of the argument containing replstr as possible, to the con-structed arguments to utility, up to 255 bytes. The 255 byte limit does not apply to arguments to utility which do not contain replstr, and furthermore, no replacement will be done on utility itself. Implies -x.

Edit 2: Partial solution. Supports longer file names than before but still not as long as I need.

find ./config/pages -name '*.json' -print0 | xargs -0I % sh -c 'file=%; echo Linting: $file; jsonlint -V ./config/schema.json -q $file;'

On BSD like systems (e.g. Mac OS X)

If you happen to be on a mac or freebsd etc. your xargs implementation may support option -J which does not suffer from the argument size limits imposed on option -I.

Excert from manpage

-J replstr
If this option is specified, xargs will use the data read from standard input to replace the first occurrence of replstr instead of appending that data after all other arguments. This option will not effect how many arguments will be read from input (-n), or the size of the command(s) xargs will generate (-s). The option just moves where those arguments will be placed in the command(s) that are executed. The replstr must show up as a distinct argument to xargs. It will not be recognized if, for instance, it is in the middle of a quoted string. Furthermore, only the first occurrence of the replstr will be replaced. For example, the following command will copy the list of files and directories which start with an uppercase letter in the current directory to destdir:
/bin/ls -1d [A-Z]* | xargs -J % cp -Rp % destdir

If you need to refer to the repstr multiple times (*points up* TL;DR -J only replaces first occurrence) you can use this pattern:

echo hi | xargs -J{} sh -c 'arg=$0; echo "$arg $arg"' "{}"
=> hi hi
POSIX compliant method

The posix compliant method of doing this would be to use some other tool, e.g. sed to construct the code you want to execute and then use xargs to just specify the utility. When no repl string is used in xargs the 255 byte limit does not apply. xargs POSIX spec

find . -type f -name '*.json' -print |
  sed "s_^_-c 'file=\\\"_g;s_\$_\\\"; echo \\\"Definitely over 255 byte script..$(printf "a%.0s" {1..255}): \\\$file\\\"; wc -l \\\"\\\$file\\\"'_g" |
  xargs -L1 sh

This of course largely defeats the purpose of xargs to begin with, but can still be used to leverage e.g. parallel execution using xargs -L1 -P10 sh which is quite widely supported, though not posix.

xargs command length limits, In short, no. The long answer: - Find will run the command specified by exec for every match, so if your find turns up 20 files, it will run 20 seperate instances of  I am using jsonlint to lint a bunch of files in a directory (recursively). I wrote the following command: find ./config/pages -name '*.json' -print0 | xargs -0I % sh -c 'echo Linting: %; jsonlint

Use -exec in find instead of piping to xargs.

find ./config/pages -name '*.json' -print0 -exec echo Linting: {} \; -exec jsonlint -V ./config/schema.json -q {} \;

Are there limits to file length piped from find to xargs or using find , --show-limits Display the limits on the command-line length which are imposed by the operating system, xargs' choice of buffer size and the -s option. Pipe the  The xargs command limits the command line length. When the constructed command line runs, the combined Argument and environment lists can not exceed ARG_MAX bytes. Within this constraint, if you do not specify the -n or the -s flags, the default command line length is at least the value specified by LINE_MAX.

The limit on xargs's command line length is imposed by the system (not an environment) variable ARG_MAX. You can check it like:

$ getconf ARG_MAX

Surprisingly, there doesn't not seem to be a way to change it, barring kernel modification.

But even more surprising that xargs by default gets capped to a much lower value, and you can increase with -s option. Still, ARG_MAX is not the value you can set after -s — acc. to man xargs you need to subtract size of environment, plus some "headroom", no idea why. To find out the actual number use the following command (alternatively, using an arbitrary big number for -s will result in a descriptive error):

$ xargs --show-limits 2>&1 | grep "limit on argument length (this system)"
POSIX upper limit on argument length (this system): 2092120

So you need to run … | xargs -s 2092120 …, e.g. with your command:

find ./config/pages -name '*.json' -print0 | xargs -s 2092120 -0I % sh -c 'echo Linting: %; jsonlint -V ./config/schema.json -q %;'

xargs(1) - Linux manual page, In case you haven't, xargs is a command used to execute commands hitting the system limit, xargs has its own limit to the maximum length of  Display the limits on the command-line length that are imposed by the operating system, xargs' choice of buffer size and the -s option. Pipe the input from /dev/null (and perhaps specify --no-run-if-empty) if you don't want xargs to do anything. --exit, -x. Exit if the size (see the -s option) is exceeded.

Things you (probably) didn't know about xargs, It's what it says on the tin: “Maximum length of command we could actually use” is the maximum possible command line length, given the limit  This means that there is an upper limit on the length of input line that xargs will accept when used with the -I option. To work around this limitation, you can use the -s option to increase the amount of buffer space that xargs uses, and you can also use an extra invocation of xargs to ensure that very long lines do not occur.

What is the meaning of xargs show limits output, The limit is sysconf(_SC_ARG_MAX) , including the environment strings and 2048 bytes of headroom. That's the maximum length of the actual strings + the  $ xargs --show-limits Your environment variables take up 1203 bytes POSIX upper limit on argument length (this system): 2093901 POSIX smallest allowable upper limit on argument length (all systems): 4096 Maximum length of command we could actually use: 2092698 Size of command buffer we are actually using: 131072 Execution of xargs will continue

xargs: The command line for command is built up until it reaches a , The xargs command line typically contains the skeleton or template of another sets the maximum allowable size of an argument list to size characters (where  In order to avoid hitting the system limit, xargs has its own limit to the maximum length of the resulting command. If the supplied arguments would cause the invoked command to exceed this built in limit, xargs will split the input and invoke the command repeatedly. This limit defaults to 4096,