Chapter 23. Process Substitution

Piping the stdout of a command into the stdin of another is a powerful technique. But, what if you need to pipe the stdout of multiple commands? This is where process substitution comes in.

Process substitution feeds the output of a process (or processes) into the stdin of another process.

bash$ echo >(true) /dev/fd/63 bash$ echo <(true) /dev/fd/63 bash$ echo >(true) <(true) /dev/fd/63 /dev/fd/62 bash$ wc <(cat /usr/share/dict/linux.words) 483523 483523 4992010 /dev/fd/63 bash$ grep script /usr/share/dict/linux.words | wc 262 262 3601 bash$ wc <(grep script /usr/share/dict/linux.words) 262 262 3601 /dev/fd/63

Bash creates a pipe with two file descriptors, --fIn and fOut--. The stdin of true connects to fOut (dup2(fOut, 0)), then Bash passes a /dev/fd/fIn argument to echo. On systems lacking /dev/fd/<n> files, Bash may use temporary files. (Thanks, S.C.)

Process substitution can compare the output of two different commands, or even the output of different options to the same command.

 bash$ comm <(ls -l) <(ls -al)
 total 12
-rw-rw-r--    1 bozo bozo       78 Mar 10 12:58 File0
-rw-rw-r--    1 bozo bozo       42 Mar 10 12:58 File2
-rw-rw-r--    1 bozo bozo      103 Mar 10 12:58 t2.sh
        total 20
        drwxrwxrwx    2 bozo bozo     4096 Mar 10 18:10 .
        drwx------   72 bozo bozo     4096 Mar 10 17:58 ..
        -rw-rw-r--    1 bozo bozo       78 Mar 10 12:58 File0
        -rw-rw-r--    1 bozo bozo       42 Mar 10 12:58 File2
        -rw-rw-r--    1 bozo bozo      103 Mar 10 12:58 t2.sh

Process substitution can compare the contents of two directories -- to see which filenames are in one, but not the other.

1 diff <(ls $first_directory) <(ls $second_directory)

Some other usages and uses of process substitution:

1 read -a list < <( od -Ad -w24 -t u2 /dev/urandom ) 2 # Read a list of random numbers from /dev/urandom, 3 #+ process with "od" 4 #+ and feed into stdin of "read" . . . 5  6 # From "insertion-sort.bash" example script. 7 # Courtesy of JuanJo Ciarlante.

1 PORT=6881 # bittorrent 2  3 # Scan the port to make sure nothing nefarious is going on. 4 netcat -l $PORT | tee>(md5sum ->mydata-orig.md5) | 5 gzip | tee>(md5sum - | sed 's/-$/mydata.lz2/'>mydata-gz.md5)>mydata.gz 6  7 # Check the decompression: 8  gzip -d<mydata.gz | md5sum -c mydata-orig.md5) 9 # The MD5sum of the original checks stdin and detects compression issues. 10  11 # Bill Davidsen contributed this example 12 #+ (with light edits by the ABS Guide author).

1 cat <(ls -l) 2 # Same as ls -l | cat 3  4 sort -k 9 <(ls -l /bin) <(ls -l /usr/bin) <(ls -l /usr/X11R6/bin) 5 # Lists all the files in the 3 main 'bin' directories, and sorts by filename. 6 # Note that three (count 'em) distinct commands are fed to 'sort'. 7  8  9 diff <(command1) <(command2) # Gives difference in command output. 10  11 tar cf >(bzip2 -c > file.tar.bz2) $directory_name 12 # Calls "tar cf /dev/fd/?? $directory_name", and "bzip2 -c > file.tar.bz2". 13 # 14 # Because of the /dev/fd/<n> system feature, 15 # the pipe between both commands does not need to be named. 16 # 17 # This can be emulated. 18 # 19 bzip2 -c < pipe > file.tar.bz2& 20 tar cf pipe $directory_name 21 rm pipe 22 # or 23 exec 3>&1 24 tar cf /dev/fd/4 $directory_name 4>&1 >&3 3>&- | bzip2 -c > file.tar.bz2 3>&- 25 exec 3>&- 26  27  28 # Thanks, Stéphane Chazelas

Here is a method of circumventing the problem of an echo piped to a while-read loop running in a subshell.

   1 #!/bin/bash
   2 # wr-ps.bash: while-read loop with process substitution.
   3 
   4 # This example contributed by Tomas Pospisek.
   5 # (Heavily edited by the ABS Guide author.)
   6 
   7 echo
   8 
   9 echo "random input" | while read i
  10 do
  11   global=3D": Not available outside the loop."
  12   # ... because it runs in a subshell.
  13 done
  14 
  15 echo "\$global (from outside the subprocess) = $global"
  16 # $global (from outside the subprocess) =
  17 
  18 echo; echo "--"; echo
  19 
  20 while read i
  21 do
  22   echo $i
  23   global=3D": Available outside the loop."
  24   # ... because it does NOT run in a subshell.
  25 done < <( echo "random input" )
  26 #    ^ ^
  27 
  28 echo "\$global (using process substitution) = $global"
  29 # Random input
  30 # $global (using process substitution) = 3D: Available outside the loop.
  31 
  32 
  33 echo; echo "##########"; echo
  34 
  35 
  36 
  37 # And likewise . . .
  38 
  39 declare -a inloop
  40 index=0
  41 cat $0 | while read line
  42 do
  43   inloop[$index]="$line"
  44   ((index++))
  45   # It runs in a subshell, so ...
  46 done
  47 echo "OUTPUT = "
  48 echo ${inloop[*]}           # ... nothing echoes.
  49 
  50 
  51 echo; echo "--"; echo
  52 
  53 
  54 declare -a outloop
  55 index=0
  56 while read line
  57 do
  58   outloop[$index]="$line"
  59   ((index++))
  60   # It does NOT run in a subshell, so ...
  61 done < <( cat $0 )
  62 echo "OUTPUT = "
  63 echo ${outloop[*]}          # ... the entire script echoes.
  64 
  65 exit $?

This is a similar example.

   1 #!/bin/bash
   2 # psub.bash
   3 
   4 # As inspired by Diego Molina (thanks!).
   5 
   6 declare -a array0
   7 while read
   8 do
   9   array0[${#array0[@]}]="$REPLY"
  10 done < <( sed -e 's/bash/CRASH-BANG!/' $0 | grep bin | awk '{print $1}' )
  11 #  Sets the default 'read' variable, $REPLY, by process substitution,
  12 #+ then copies it into an array.
  13 
  14 echo "${array0[@]}"
  15 
  16 exit $?
  17 
  18 # ====================================== #
  19 
  20 bash psub.bash
  21 
  22 #!/bin/CRASH-BANG! done #!/bin/CRASH-BANG!

A reader sent in the following interesting example of process substitution.

1 # Script fragment taken from SuSE distribution: 2  3 # --------------------------------------------------------------# 4 while read des what mask iface; do 5 # Some commands ... 6 done < <(route -n) 7 # ^ ^ First < is redirection, second is process substitution. 8  9 # To test it, let's make it do something. 10 while read des what mask iface; do 11  echo $des $what $mask $iface 12 done < <(route -n) 13  14 # Output: 15 # Kernel IP routing table 16 # Destination Gateway Genmask Flags Metric Ref Use Iface 17 # 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 18 # --------------------------------------------------------------# 19  20 # As Stéphane Chazelas points out, 21 #+ an easier-to-understand equivalent is: 22 route -n | 23  while read des what mask iface; do # Variables set from output of pipe. 24  echo $des $what $mask $iface 25  done # This yields the same output as above. 26  # However, as Ulrich Gayer points out . . . 27  #+ this simplified equivalent uses a subshell for the while loop, 28  #+ and therefore the variables disappear when the pipe terminates. 29  30 # --------------------------------------------------------------# 31  32 # However, Filip Moritz comments that there is a subtle difference 33 #+ between the above two examples, as the following shows. 34  35 ( 36 route -n | while read x; do ((y++)); done 37 echo $y # $y is still unset 38  39 while read x; do ((y++)); done < <(route -n) 40 echo $y # $y has the number of lines of output of route -n 41 ) 42  43 More generally spoken 44 ( 45 : | x=x 46 # seems to start a subshell like 47 : | ( x=x ) 48 # while 49 x=x < <(:) 50 # does not 51 ) 52  53 # This is useful, when parsing csv and the like. 54 # That is, in effect, what the original SuSE code fragment does.