Concurrent Shell Command Queues

Christian Emmer

Nov 6, 2026 · 9 min read

How to run shell commands in parallel while limiting their concurrency.

I had a use case recently where I wanted to speed up some Bash processing that I was doing, but still keep concurrency constrained. So I wondered: is there a straightforward way to create thread pools in Bash? The answer is: yes, but it takes some work to be semi-resilient and portable.

You may ask: why do this in Bash at all? It seems silly to use a language with poor concurrency controls. You'd be right to ask this question, and my answer is: because I have a fondness for Bash, and I wanted the challenge. But you're still probably better off using another language, such as Python.

Here is code that creates a command queue that accepts shell commands to execute. You do not need to know all the commands that you want executed up-front, you can submit commands to it over time.

bash



1# Start this shell's command queue.
2# @param {number} $1 Maximum number of jobs to run at once
3queue_start() {
4  # Validate that the parallelism is a valid limit for xargs
5  if [[ ! ${1:-} =~ ^[1-9][0-9]*$ ]]; then
6    echo "queue_start: size must be a positive integer, got '${1:-}'" >&2
7    return 1
8  fi
9

10  # Close this shell's copy (not the parent's) of an inherited or earlier
11  # command queue's file descriptor
12  if [[ -n ${_queue_fd:-} ]]; then
13    eval "exec ${_queue_fd}>&-"
14    unset _queue_fd _queue_pid
15  fi
16

17  # Find the first unused fd from 3 to 9 (zsh's redirect max)
18  local fd=3
19  while { : <&"${fd}"; } 2>/dev/null; do fd=$(( fd + 1 )); done
20  if [[ ${fd} -gt 9 ]]; then
21    echo "queue_start: no free fd from 3 to 9" >&2
22    return 1
23  fi
24

25  # Make a FIFO queue file in a private temporary directory
26  local dir
27  dir=$(mktemp -d "${TMPDIR:-/tmp}/queue.XXXXXX")
28  mkfifo "${dir}/queue"
29

30  # Find the shell to execute commands (the current shell)
31  local shell
32  shell=$(ps -o comm= -p "$$")
33  shell=${shell#-}
34

35  # Start xargs in the background, reading NUL-separated commands from the
36  # queue file. Job messages are hidden by turning off job control and
37  # discarding the block's stderr. xargs gets the real stderr through a copy on
38  # the queue files's fd, until the file is opened.
39  eval "exec ${fd}>&2"
40  local opts=$-
41  set +m
42  {
43    # shellcheck disable=SC2016
44    xargs -0 -n 1 -P "$1" "${shell}" -c '
45      # Output is captured rather than streamed so that command output stays separate
46      out=$(eval -- "$1" </dev/null 2>&1); rc=$?
47      [ -z "${out}" ] || printf "%s\n" "${out}"
48      # Coerce non-zero exit codes to 1, because a 255 code would stop xargs
49      [ "${rc}" -eq 0 ] || { printf "exit %d: %s\n" "${rc}" "$1" >&2; exit 1;
50    }' _ <"${dir}/queue" 2>&"${fd}" &
51    _queue_pid=$!
52    disown || :
53  } 2>/dev/null
54  if [[ ${opts} == *m* ]]; then set -m; fi
55

56  # Open the queue file for writing, which waits until xargs has opened it for reading
57  # With both ends open, the file can be safely deleted
58  eval "exec ${fd}>\"\${dir}/queue\""
59  _queue_fd=${fd}
60  rm -rf "${dir}"
61

62  # Register an exit handler to terminate this PID's children, ensuring zsh behaves like bash
63  setopt localoptions posixtraps 2>/dev/null || :
64  trap _queue_stop EXIT
65  if [[ $- != *i* ]]; then
66    trap 'exit 129' HUP
67    trap 'exit 130' INT
68    trap 'exit 143' TERM
69  fi
70}
71

72# Queue a command on this shell's command queue.
73# @param {string} $1 Command to run
74queue_submit() {
75  if [[ -z ${_queue_fd:-} ]]; then
76    echo "queue_submit: no command queue started in this shell" >&2
77    return 1
78  fi
79  # Write the command to the queue file, blocking if it is full
80  # Use a printf process so that if xargs has exited, printf will get killed and not this shell
81  if [[ $- == *i* ]]; then
82    # Write the command in the foreground, because an interactive shell won't
83    # have any TERM/HUP/INT traps that could be delayed by a full queue
84    env printf '%s\0' "$1" >&"${_queue_fd}"
85  else
86    # Write the command in the background, so that a full queue doesn't delay any TERM/HUP/INT traps
87    env printf '%s\0' "$1" >&"${_queue_fd}" &
88    wait "$!"
89  fi || {
90    echo "queue_submit: couldn't queue the command; the command queue may have stopped" >&2
91    return 1
92  }
93}
94

95# Close the queue file so xargs finishes the jobs it has, then wait for it to exit
96queue_wait() {
97  if [[ -z ${_queue_fd:-} ]]; then
98    echo "queue_wait: no command queue started in this shell" >&2
99    return 1
100  fi
101  eval "exec ${_queue_fd}>&-"
102  while kill -0 "${_queue_pid}" 2>/dev/null; do sleep 0.1; done
103  unset _queue_fd _queue_pid
104}
105

106# Stop every process below the given PID.
107# @param {number=} $1 PID whose descendants to stop, defaulting to this shell's PID
108_queue_stop() {
109  # Default to this shell's PID, found from a child's parent PID because $$ is
110  # the parent's PID in a subshell
111  local parent=${1:-$(exec sh -c 'echo "${PPID}"')}
112  local pid
113  for pid in $(exec pgrep -P "${parent}"); do
114    # Pause the process so it can't start any new children
115    kill -STOP "${pid}" 2>/dev/null || continue
116    # Terminate the process' descendants, then terminate the process
117    _queue_stop "${pid}"
118    kill -TERM "${pid}" 2>/dev/null || :
119    # Resume the process so it can respond to the termination
120    kill -CONT "${pid}" 2>/dev/null || :
121  done
122}
123

124# Stop all background processes on exit, because they ignore Ctrl-C and don't receive TERM
125if [[ $- != *i* ]]; then
126  trap _queue_stop EXIT
127  trap 'exit 129' HUP
128  trap 'exit 130' INT
129  trap 'exit 143' TERM
130fi

queue_start does the setup, queue_submit submits commands to the queue, and queue_wait waits for the queue to drain before cleaning up.

It works in Bash and Zsh, and it has as much defensive programming as I know how to add, but I'm confident it has failure modes that are difficult to predict or avoid.

Example usage

My original use case had to do with making a large number of web requests, so here's an example using Postman Echo :

● ● ●

shell

$ queue_start 3
$ for n in 1 2 3 4 5; do
    queue_submit "set -o pipefail; \
      curl -sS --fail-with-body --max-time 10 -X POST -H 'Content-Type: application/json' \
        -d '{\"request\": ${n}}' \
        https://postman-echo.com/post | \
      jq -c .json"
  done
$ queue_wait
{"request":1}
{"request":2}
{"request":3}
{"request":4}
{"request":5}

A more deterministic scenario demonstrating that concurrency is respected, only starting command three after command two has completed:

● ● ●

shell

$ queue_start 2
$ queue_submit 'sleep 1; echo first'
$ queue_submit 'echo second'
$ queue_submit 'echo third'
$ queue_wait
second
third
first

Command queues are inherited by subshells:

● ● ●

shell

$ queue_start 2
$ ( queue_submit 'echo "submitted from a subshell"' )
$ queue_submit 'echo "submitted from the main shell"'
$ queue_wait
submitted from a subshell
submitted from the main shell

Different subshells can have their own command queue:

● ● ●

shell

$ (
    queue_start 2
    for n in 1 2 3 4; do queue_submit "sleep 1; echo first queue, job ${n}"; done
    queue_wait
  ) &
$ first=$!
$ (
    queue_start 1
    for n in 1 2; do queue_submit "sleep 1; echo second queue, job ${n}"; done
    queue_wait
  ) &
$ second=$!
$ wait "${first}"
$ wait "${second}"
second queue, job 1
first queue, job 1
first queue, job 2
second queue, job 2
first queue, job 3
first queue, job 4

Queueing each line of a command's output as a command to execute:

● ● ●

shell

$ queue_start 2
$ while IFS= read -r line; do
    queue_submit "${line}"
  done < <(printf 'echo "from command output, line %d"\n' 1 2 3)
$ queue_wait
from command output, line 1
from command output, line 2
from command output, line 3

Queueing each line of a file as a command:

● ● ●

shell

$ queue_start 2
$ while IFS= read -r line; do
    queue_submit "${line}"
  done < commands.txt
$ queue_wait

Known issues

Here are some reasons why you might just want a different programming language:

A shell can only have one queue.
The amount of additional code needed to support multiple is complicated and likely isn't needed in most cases. Subshells can still have their own queues.
Command output is intentionally not streamed.
This prevents interleaving the output of most commands. Output larger than PIPE_BUF may still get interleaved.
queue_wait doesn't return a value indicating any job failure.**
This would require a temporary file or a second file descriptor and some complicated code.
queue_wait has to wait for running processes started after queue_start.
Processes started after queue_start will inherit the queue write file descriptor (which is what allows subshells to work), and xargs won't see the end of the queue until every copy of the write descriptor is closed.
_queue_stop indiscriminately terminates descendants.
Any other backgrounded jobs will be terminated, including ones started before queue_start.

Why not just `xargs -P`?

If you know all commands that you want to execute up-front, then the solution gets far simpler:

● ● ●

shell

$ printf 'echo "from command output, line %d"\0' 1 2 3 | \
    xargs -0 -n 1 -P 2 bash -c
from command output, line 1
from command output, line 2
from command output, line 3

● ● ●

shell

$ tr '\n' '\0' < commands.txt | \
    xargs -0 -n 1 -P 2 bash -c

GNU parallel works similarly, though macOS doesn't ship with it:

● ● ●

shell

$ printf 'echo "from command output, line %d"\n' 1 2 3 | \
    parallel -j 2
from command output, line 1
from command output, line 2
from command output, line 3

shell