Bash function to run tasks in parallel and display pretty output as they complete.
MIT License
A Bash function to run tasks in parallel and display pretty output as they complete.
Run three tasks concurrently:
concurrent \
- 'My long task' sleep 10 \
- 'My medium task' sleep 5 \
- 'My short task' sleep 1
Run three tasks sequentially:
concurrent \
- 'My long task' sleep 10 \
- 'My medium task' sleep 5 \
- 'My short task' sleep 1 \
--sequential
Start the medium task after the short task succeeds:
concurrent \
- 'My long task' sleep 10 \
- 'My medium task' sleep 5 \
- 'My short task' sleep 1 \
--require 'My short task' \
--before 'My medium task'
Start the short task after both other tasks succeed:
concurrent \
- 'My long task' sleep 10 \
- 'My medium task' sleep 5 \
- 'My short task' sleep 1 \
--require 'My long task' \
--require 'My medium task' \
--before 'My short task'
Same as above, but shorter:
concurrent \
- 'My long task' sleep 10 \
- 'My medium task' sleep 5 \
- 'My short task' sleep 1 \
--require-all --before 'My short task'
Start the medium task and the long task after the short task succeeds:
concurrent \
- 'My long task' sleep 10 \
- 'My medium task' sleep 5 \
- 'My short task' sleep 1 \
--require 'My short task' \
--before 'My medium task' \
--before 'My long task'
Same as above, but shorter:
concurrent \
- 'My long task' sleep 10 \
- 'My medium task' sleep 5 \
- 'My short task' sleep 1 \
--require 'My short task' --before-all
Run the first two tasks concurrently, and then the second two tasks concurrently, and then the final three tasks concurrently.
concurrent \
- 'Task 1' sleep 3 \
- 'Task 2' sleep 3 \
--and-then \
- 'Task 3' sleep 3 \
- 'Task 4' sleep 3 \
--and-then \
- 'Task 5' sleep 3 \
- 'Task 6' sleep 3 \
- 'Task 7' sleep 3
If your command has a -
argument, you can use a different task delimiter:
concurrent \
+ 'My long task' wget -O - ... \
+ 'My medium task' sleep 5 \
+ 'My short task' sleep 1
You can display extra information at the end of each task's status line by
echoing to fd 3
.
my_task() {
...
echo "(extra info)" >&3
...
}
Take a look at demo.sh
for more involved examples.
If you have a lot of dependencies between tasks, it's generally a good idea to
perform a dry-run to ensure that the tasks are ordered as expected. Set the
CONCURRENT_DRY_RUN
environment variable to perform a dry-run.
By default, concurrent
allows up to 50 concurrently-running tasks.
Set the CONCURRENT_LIMIT
environment variable to override this limit.
A neat trick is to set the limit to 1, essentially forcing a --sequential
run, but with existing tasks between dependencies taken into account.
A limit less than 1 is treated as no limit.
If the number of tasks exceed the terminal height, the "compact display" will
be activated. It can also be explicitly activated by setting the
CONCURRENT_COMPACT
environment variable to anything other than 0
.
In this mode, each task is represented by a single character instead of an entire line. An execution summary is displayed above the tasks.
If the output is not attached to a tty, the "non-interactive" mode will be
activated. It can also be explicitly activated by setting the
CONCURRENT_NONINTERACTIVE
environment variable to anything other than 0
.
In this mode, each task is displayed as soon as it completes. Colors are also disabled.
By default, logs for each task will be created in ./logs/<timestamp>/
.
For example:
$ ls .logs/2016-02-02@00:09:07
0. Creating VM (0).log
1. Creating ramdisk (0).log
2. Enabling swap (0).log
3. Populating VM with world data (1).log
4. Spigot: Pulling docker image for build (1).log
5. Spigot: Building JAR (skip).log
6. Pulling remaining docker images (skip).log
7. Launching services (skip).log
To change this directory, set CONCURRENT_LOG_DIR
before calling concurrent
.
declare -g
)CONCURRENT_NONINTERACTIVE
is non-zero.CONCURRENT_LIMIT
.CONCURRENT_COMPACT
is set), each task will be displayed as a single character instead of a taking up an entire line.command_*
and prereq_*
arrays so that they don't carry into the tasks.CONCURRENT_LOG_DIR
.--and-then
flag for dividing tasks into groups. All tasks in a group run concurrently, but all must complete before the next group may start (inspiration: fooshards on Reddit).kill
is a bash builtin (credit: @ScoreUnder)CONCURRENT_DRY_RUN
environment variable runs sleep 3
instead of actual commands (and prints message).--require-all
and --before-all
flags.--sequential
flag, for when each task requires the previous.$PWD
and $OLDPWD
.$SHELLOPTS
and $BASHOPTS
.-
.