better(?) xargs in go
APACHE-2.0 License
fix panic when bash process never started (e.g. because of 'argument list too long')
Published by brentp over 7 years ago
Published by brentp over 7 years ago
Published by brentp over 7 years ago
GARGS_PROCESS_BUFFER
to let user set size of data before a tempfile is used.GARGS_WAIT_MULTIPLIER
to determine how many finished processes will wait for single slow processesPublished by brentp about 8 years ago
Published by brentp about 8 years ago
As of this version gargs
will not read everything into memory as before. It will read up to 1MB. If it does not get an EOF by then it start using a temp-file. This will reduce memory usage.
It also fixes --nlines to be quite useful e.g.:
cat regions.txt | gargs -p 20 -n 10 "bcftools view some.bam {}"
will send 10 regions to each process to amortize the cost of loading the index into memory. Note that the place-holder {}
is specified only once.
It also defaults --sep to "\s+" if --nlines is not specified.
Finally, it adds a --retry argument that takes an integer that indicates the number of times a failed process should be retried. This is nice for transient network errors.
Published by brentp over 8 years ago
remove --shell as an argument and get it from $SHELL from the environment.
Published by brentp over 8 years ago
usage: gargs [--procs PROCS] [--nlines NLINES] [--sep SEP] [--shell SHELL] [--verbose] [--continue-on-error] [--ordered] [--dry-run] COMMAND
positional arguments:
command command to execute
options:
--procs PROCS, -p PROCS
number of processes to use [default: 1]
--nlines NLINES, -n NLINES
number of lines to consume for each command. -s and -n are mutually exclusive. [default: 1]
--sep SEP, -s SEP regular expression split line with to fill multiple template spots default is not to split. -s and -n are mutually exclusive.
--shell SHELL shell to use [default: bash]
--verbose, -v print commands to stderr before they are executed.
--continue-on-error, -c
report errors but don't stop the entire execution (which is the default).
--ordered, -o keep output in order of input; default is to output in order of return which greatly improves parallelization.
--dry-run, -d print (but do not run) the commands (for debugging)
--help, -h display this help and exit
Published by brentp over 8 years ago
Published by brentp over 8 years ago