Process Substitution

Advanced Bash-Scripting Guide: An in-depth exploration of the art of shell scripting
Prev |  | Next

# Chapter 23. Process Substitution

Piping the `stdout` of a command into the `stdin` of another is a powerful technique. But, what if you need to pipe the `stdout` of multiple commands? This is where `process substitution` comes in.

Process substitution feeds the output of a process (or processes) into the `stdin` of another process.

Template

Command list enclosed within parentheses

  >(command_list)

  <(command_list)

  Process substitution uses `/dev/fd/<n>` files to send the results of the process(es) within parentheses to another process. [1]

  | There is no space between the the "<" or ">" and the parentheses. Space there would give an error message.

     bash$ echo >(true)
     /dev/fd/63

     bash$ echo <(true)
     /dev/fd/63

     bash$ echo >(true) <(true)
     /dev/fd/63 /dev/fd/62

     bash$ wc <(cat /usr/share/dict/linux.words)
      483523  483523 4992010 /dev/fd/63

     bash$ grep script /usr/share/dict/linux.words | wc
         262     262    3601

     bash$ wc <(grep script /usr/share/dict/linux.words)
         262     262    3601 /dev/fd/63

| Bash creates a pipe with two file descriptors, `--fIn` and `fOut--`. The `stdin` of true connects to `fOut` (dup2(fOut, 0)), then Bash passes a `/dev/fd/fIn` argument to echo. On systems lacking `/dev/fd/<n>` files, Bash may use temporary files. (Thanks, S.C.)

Process substitution can compare the output of two different commands, or even the output of different options to the same command.

     bash$ comm <(ls -l) <(ls -al)
     total 12
    -rw-rw-r--    1 bozo bozo       78 Mar 10 12:58 File0
    -rw-rw-r--    1 bozo bozo       42 Mar 10 12:58 File2
    -rw-rw-r--    1 bozo bozo      103 Mar 10 12:58 t2.sh
            total 20
            drwxrwxrwx    2 bozo bozo     4096 Mar 10 18:10 .
            drwx------   72 bozo bozo     4096 Mar 10 17:58 ..
            -rw-rw-r--    1 bozo bozo       78 Mar 10 12:58 File0
            -rw-rw-r--    1 bozo bozo       42 Mar 10 12:58 File2
            -rw-rw-r--    1 bozo bozo      103 Mar 10 12:58 t2.sh

Process substitution can compare the contents of two directories -- to see which filenames are in one, but not the other.

       1 diff <(ls $first_directory) <(ls $second_directory)

Some other usages and uses of process substitution:

       1 read -a list < <( od -Ad -w24 -t u2 /dev/urandom )
       2 #  Read a list of random numbers from /dev/urandom,
       3 #+ process with "od"
       4 #+ and feed into stdin of "read" . . .
       5
       6 #  From "insertion-sort.bash" example script.
       7 #  Courtesy of JuanJo Ciarlante.

       1 PORT=6881   # bittorrent
       2
       3 # Scan the port to make sure nothing nefarious is going on.
       4 netcat -l $PORT | tee>(md5sum ->mydata-orig.md5) |
       5 gzip | tee>(md5sum - | sed 's/-$/mydata.lz2/'>mydata-gz.md5)>mydata.gz
       6
       7 # Check the decompression:
       8   gzip -d<mydata.gz | md5sum -c mydata-orig.md5)
       9 # The MD5sum of the original checks stdin and detects compression issues.
      10
      11 #  Bill Davidsen contributed this example
      12 #+ (with light edits by the ABS Guide author).

       1 cat <(ls -l)
       2 # Same as     ls -l | cat
       3
       4 sort -k 9 <(ls -l /bin) <(ls -l /usr/bin) <(ls -l /usr/X11R6/bin)
       5 # Lists all the files in the 3 main 'bin' directories, and sorts by filename.
       6 # Note that three (count 'em) distinct commands are fed to 'sort'.
       7
       8
       9 diff <(command1) <(command2)    # Gives difference in command output.
      10
      11 tar cf >(bzip2 -c > file.tar.bz2) $directory_name
      12 # Calls "tar cf /dev/fd/?? $directory_name", and "bzip2 -c > file.tar.bz2".
      13 #
      14 # Because of the /dev/fd/<n> system feature,
      15 # the pipe between both commands does not need to be named.
      16 #
      17 # This can be emulated.
      18 #
      19 bzip2 -c < pipe > file.tar.bz2&
      20 tar cf pipe $directory_name
      21 rm pipe
      22 #        or
      23 exec 3>&1
      24 tar cf /dev/fd/4 $directory_name 4>&1 >&3 3>&- | bzip2 -c > file.tar.bz2 3>&-
      25 exec 3>&-
      26
      27
      28 # Thanks, Stéphane Chazelas

Here is a method of circumventing the problem of an echo piped to a while-read loop running in a subshell.

Example 23-1. Code block redirection without forking

       1 #!/bin/bash
       2 # wr-ps.bash: while-read loop with process substitution.
       3
       4 # This example contributed by Tomas Pospisek.
       5 # (Heavily edited by the ABS Guide author.)
       6
       7 echo
       8
       9 echo "random input" | while read i
      10 do
      11   global=3D": Not available outside the loop."
      12   # ... because it runs in a subshell.
      13 done
      14
      15 echo "\$global (from outside the subprocess) = $global"
      16 # $global (from outside the subprocess) =
      17
      18 echo; echo "--"; echo
      19
      20 while read i
      21 do
      22   echo $i
      23   global=3D": Available outside the loop."
      24   # ... because it does NOT run in a subshell.
      25 done < <( echo "random input" )
      26 #    ^ ^
      27
      28 echo "\$global (using process substitution) = $global"
      29 # Random input
      30 # $global (using process substitution) = 3D: Available outside the loop.
      31
      32
      33 echo; echo "##########"; echo
      34
      35
      36
      37 # And likewise . . .
      38
      39 declare -a inloop
      40 index=0
      41 cat $0 | while read line
      42 do
      43   inloop[$index]="$line"
      44   ((index++))
      45   # It runs in a subshell, so ...
      46 done
      47 echo "OUTPUT = "
      48 echo ${inloop[*]}           # ... nothing echoes.
      49
      50
      51 echo; echo "--"; echo
      52
      53
      54 declare -a outloop
      55 index=0
      56 while read line
      57 do
      58   outloop[$index]="$line"
      59   ((index++))
      60   # It does NOT run in a subshell, so ...
      61 done < <( cat $0 )
      62 echo "OUTPUT = "
      63 echo ${outloop[*]}          # ... the entire script echoes.
      64
      65 exit $?

This is a similar example.

Example 23-2. Redirecting the output of process substitution into a loop.

       1 #!/bin/bash
       2 # psub.bash
       3
       4 # As inspired by Diego Molina (thanks!).
       5
       6 declare -a array0
       7 while read
       8 do
       9   array0[${#array0[@]}]="$REPLY"
      10 done < <( sed -e 's/bash/CRASH-BANG!/' $0 | grep bin | awk '{print $1}' )
      11 #  Sets the default 'read' variable, $REPLY, by process substitution,
      12 #+ then copies it into an array.
      13
      14 echo "${array0[@]}"
      15
      16 exit $?
      17
      18 # ====================================== #
      19
      20 bash psub.bash
      21
      22 #!/bin/CRASH-BANG! done #!/bin/CRASH-BANG!

A reader sent in the following interesting example of process substitution.

       1 # Script fragment taken from SuSE distribution:
       2
       3 # --------------------------------------------------------------#
       4 while read  des what mask iface; do
       5 # Some commands ...
       6 done < <(route -n)
       7 #    ^ ^  First < is redirection, second is process substitution.
       8
       9 # To test it, let's make it do something.
      10 while read  des what mask iface; do
      11   echo $des $what $mask $iface
      12 done < <(route -n)
      13
      14 # Output:
      15 # Kernel IP routing table
      16 # Destination Gateway Genmask Flags Metric Ref Use Iface
      17 # 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
      18 # --------------------------------------------------------------#
      19
      20 #  As Stéphane Chazelas points out,
      21 #+ an easier-to-understand equivalent is:
      22 route -n |
      23   while read des what mask iface; do   # Variables set from output of pipe.
      24     echo $des $what $mask $iface
      25   done  #  This yields the same output as above.
      26         #  However, as Ulrich Gayer points out . . .
      27         #+ this simplified equivalent uses a subshell for the while loop,
      28         #+ and therefore the variables disappear when the pipe terminates.
      29
      30 # --------------------------------------------------------------#
      31
      32 #  However, Filip Moritz comments that there is a subtle difference
      33 #+ between the above two examples, as the following shows.
      34
      35 (
      36 route -n | while read x; do ((y++)); done
      37 echo $y # $y is still unset
      38
      39 while read x; do ((y++)); done < <(route -n)
      40 echo $y # $y has the number of lines of output of route -n
      41 )
      42
      43 More generally spoken
      44 (
      45 : | x=x
      46 # seems to start a subshell like
      47 : | ( x=x )
      48 # while
      49 x=x < <(:)
      50 # does not
      51 )
      52
      53 # This is useful, when parsing csv and the like.
      54 # That is, in effect, what the original SuSE code fragment does.

### Notes

[1] | This has the same effect as a named pipe (temp file), and, in fact, named pipes were at one time used in process substitution.

Prev | Home | Next
Restricted Shells | Up | Functions