Command Substitution

Advanced Bash-Scripting Guide: An in-depth exploration of the art of shell scripting
Prev |  | Next

# Chapter 12. Command Substitution

Command substitution reassigns the output of a command [1] or even multiple commands; it literally plugs the command output into another context. [2]

The classic form of command substitution uses backquotes (`...`). Commands within backquotes (backticks) generate command-line text.

       1 script_name=`basename $0`
       2 echo "The name of this script is $script_name."

The output of commands can be used as arguments to another command, to set a variable, and even for generating the argument list in a for loop.

       1 rm `cat filename`   # "filename" contains a list of files to delete.
       2 #
       3 # S. C. points out that "arg list too long" error might result.
       4 # Better is              xargs rm -- < filename
       5 # ( -- covers those cases where "filename" begins with a "-" )
       6
       7 textfile_listing=`ls *.txt`
       8 # Variable contains names of all *.txt files in current working directory.
       9 echo $textfile_listing
      10
      11 textfile_listing2=$(ls *.txt)   # The alternative form of command substitution.
      12 echo $textfile_listing2
      13 # Same result.
      14
      15 # A possible problem with putting a list of files into a single string
      16 # is that a newline may creep in.
      17 #
      18 # A safer way to assign a list of files to a parameter is with an array.
      19 #      shopt -s nullglob    # If no match, filename expands to nothing.
      20 #      textfile_listing=( *.txt )
      21 #
      22 # Thanks, S.C.

| Command substitution invokes a subshell.

| Command substitution may result in word splitting.
       1 COMMAND `echo a b`     # 2 args: a and b
       2
       3 COMMAND "`echo a b`"   # 1 arg: "a b"
       4
       5 COMMAND `echo`         # no arg
       6
       7 COMMAND "`echo`"       # one empty arg
       8
       9
      10 # Thanks, S.C.
Even when there is no word splitting, command substitution can remove trailing newlines.
       1 # cd "`pwd`"  # This should always work.
       2 # However...
       3
       4 mkdir 'dir with trailing newline
       5 '
       6
       7 cd 'dir with trailing newline
       8 '
       9
      10 cd "`pwd`"  # Error message:
      11 # bash: cd: /tmp/file with trailing newline: No such file or directory
      12
      13 cd "$PWD"   # Works fine.
      14
      15
      16
      17
      18
      19 old_tty_setting=$(stty -g)   # Save old terminal setting.
      20 echo "Hit a key "
      21 stty -icanon -echo           # Disable "canonical" mode for terminal.
      22                              # Also, disable *local* echo.
      23 key=$(dd bs=1 count=1 2> /dev/null)   # Using 'dd' to get a keypress.
      24 stty "$old_tty_setting"      # Restore old setting.
      25 echo "You hit ${#key} key."  # ${#variable} = number of characters in $variable
      26 #
      27 # Hit any key except RETURN, and the output is "You hit 1 key."
      28 # Hit RETURN, and it's "You hit 0 key."
      29 # The newline gets eaten in the command substitution.
      30
      31 #Code snippet by Stéphane Chazelas.

| Using echo to output an unquoted variable set with command substitution removes trailing newlines characters from the output of the reassigned command(s). This can cause unpleasant surprises.
       1 dir_listing=`ls -l`
       2 echo $dir_listing     # unquoted
       3
       4 # Expecting a nicely ordered directory listing.
       5
       6 # However, what you get is:
       7 # total 3 -rw-rw-r-- 1 bozo bozo 30 May 13 17:15 1.txt -rw-rw-r-- 1 bozo
       8 # bozo 51 May 15 20:57 t2.sh -rwxr-xr-x 1 bozo bozo 217 Mar 5 21:13 wi.sh
       9
      10 # The newlines disappeared.
      11
      12
      13 echo "$dir_listing"   # quoted
      14 # -rw-rw-r--    1 bozo       30 May 13 17:15 1.txt
      15 # -rw-rw-r--    1 bozo       51 May 15 20:57 t2.sh
      16 # -rwxr-xr-x    1 bozo      217 Mar  5 21:13 wi.sh

Command substitution even permits setting a variable to the contents of a file, using either redirection or the cat command.

       1 variable1=`<file1`      #  Set "variable1" to contents of "file1".
       2 variable2=`cat file2`   #  Set "variable2" to contents of "file2".
       3                         #  This, however, forks a new process,
       4                         #+ so the line of code executes slower than the above version.
       5
       6 #  Note that the variables may contain embedded whitespace,
       7 #+ or even (horrors), control characters.
       8
       9 #  It is not necessary to explicitly assign a variable.
      10 echo "` <$0`"           # Echoes the script itself to stdout.

       1 #  Excerpts from system file, /etc/rc.d/rc.sysinit
       2 #+ (on a Red Hat Linux installation)
       3
       4
       5 if [ -f /fsckoptions ]; then
       6         fsckoptions=`cat /fsckoptions`
       7 ...
       8 fi
       9 #
      10 #
      11 if [ -e "/proc/ide/${disk[$device]}/media" ] ; then
      12              hdmedia=`cat /proc/ide/${disk[$device]}/media`
      13 ...
      14 fi
      15 #
      16 #
      17 if [ ! -n "`uname -r | grep -- "-"`" ]; then
      18        ktag="`cat /proc/version`"
      19 ...
      20 fi
      21 #
      22 #
      23 if [ $usb = "1" ]; then
      24     sleep 5
      25     mouseoutput=`cat /proc/bus/usb/devices 2>/dev/null|grep -E "^I.*Cls=03.*Prot=02"`
      26     kbdoutput=`cat /proc/bus/usb/devices 2>/dev/null|grep -E "^I.*Cls=03.*Prot=01"`
      27 ...
      28 fi

| Do not set a variable to the contents of a long text file unless you have a very good reason for doing so. Do not set a variable to the contents of a binary file, even as a joke. Example 12-1. Stupid script tricks
       1 #!/bin/bash
       2 # stupid-script-tricks.sh: Don't try this at home, folks.
       3 # From "Stupid Script Tricks," Volume I.
       4
       5 exit 99  ### Comment out this line if you dare.
       6
       7 dangerous_variable=`cat /boot/vmlinuz`   # The compressed Linux kernel itself.
       8
       9 echo "string-length of \$dangerous_variable = ${#dangerous_variable}"
      10 # string-length of $dangerous_variable = 794151
      11 # (Newer kernels are bigger.)
      12 # Does not give same count as 'wc -c /boot/vmlinuz'.
      13
      14 # echo "$dangerous_variable"
      15 # Don't try this! It would hang the script.
      16
      17
      18 #  The document author is aware of no useful applications for
      19 #+ setting a variable to the contents of a binary file.
      20
      21 exit 0
Notice that a buffer overrun does not occur. This is one instance where an interpreted language, such as Bash, provides more protection from programmer mistakes than a compiled language.

Command substitution permits setting a variable to the output of a loop. The key to this is grabbing the output of an echo command within the loop.

Example 12-2. Generating a variable from a loop

       1 #!/bin/bash
       2 # csubloop.sh: Setting a variable to the output of a loop.
       3
       4 variable1=`for i in 1 2 3 4 5
       5 do
       6   echo -n "$i"                 #  The 'echo' command is critical
       7 done`                          #+ to command substitution here.
       8
       9 echo "variable1 = $variable1"  # variable1 = 12345
      10
      11
      12 i=0
      13 variable2=`while [ "$i" -lt 10 ]
      14 do
      15   echo -n "$i"                 # Again, the necessary 'echo'.
      16   let "i += 1"                 # Increment.
      17 done`
      18
      19 echo "variable2 = $variable2"  # variable2 = 0123456789
      20
      21 #  Demonstrates that it's possible to embed a loop
      22 #+ inside a variable declaration.
      23
      24 exit 0

| The $(...) form has superseded backticks for command substitution.
       1 output=$(sed -n /"$1"/p $file)   # From "grp.sh"	example.
       2
       3 # Setting a variable to the contents of a text file.
       4 File_contents1=$(cat $file1)
       5 File_contents2=$(<$file2)        # Bash permits this also.
The $(...) form of command substitution treats a double backslash in a different way than `...`.
     bash$ echo `echo \\`

     bash$ echo $(echo \\)
     \
The $(...) form of command substitution permits nesting. [3]
       1 word_count=$( wc -w $(echo * | awk '{print $8}') )
Or, for something a bit more elaborate . . . Example 12-3. Finding anagrams
       1 #!/bin/bash
       2 # agram2.sh
       3 # Example of nested command substitution.
       4
       5 #  Uses "anagram" utility
       6 #+ that is part of the author's "yawl" word list package.
       7 #  http://ibiblio.org/pub/Linux/libs/yawl-0.3.2.tar.gz
       8 #  http://bash.deta.in/yawl-0.3.2.tar.gz
       9
      10 E_NOARGS=86
      11 E_BADARG=87
      12 MINLEN=7
      13
      14 if [ -z "$1" ]
      15 then
      16   echo "Usage $0 LETTERSET"
      17   exit $E_NOARGS         # Script needs a command-line argument.
      18 elif [ ${#1} -lt $MINLEN ]
      19 then
      20   echo "Argument must have at least $MINLEN letters."
      21   exit $E_BADARG
      22 fi
      23
      24
      25
      26 FILTER='.......'         # Must have at least 7 letters.
      27 #       1234567
      28 Anagrams=( $(echo $(anagram $1 | grep $FILTER) ) )
      29 #          $(     $(  nested command sub.    ) )
      30 #        (              array assignment         )
      31
      32 echo
      33 echo "${#Anagrams[*]}  7+ letter anagrams found"
      34 echo
      35 echo ${Anagrams[0]}      # First anagram.
      36 echo ${Anagrams[1]}      # Second anagram.
      37                          # Etc.
      38
      39 # echo "${Anagrams[*]}"  # To list all the anagrams in a single line . . .
      40
      41 #  Look ahead to the Arrays chapter for enlightenment on
      42 #+ what's going on here.
      43
      44 # See also the agram.sh script for an exercise in anagram finding.
      45
      46 exit $?

Examples of command substitution in shell scripts:

1. Example 11-8

2. Example 11-27

3. Example 9-16

4. Example 16-3

5. Example 16-22

6. Example 16-17

7. Example 16-54

8. Example 11-14

9. Example 11-11

10. Example 16-32

11. Example 20-8

12. Example A-16

13. Example 29-3

14. Example 16-47

15. Example 16-48

16. Example 16-49

### Notes

[1] | For purposes of command substitution, a command may be an external system command, an internal scripting builtin, or even a script function.
[2] | In a more technically correct sense, command substitution extracts the `stdout` of a command, then assigns it to a variable using the = operator.
[3] | In fact, nesting with backticks is also possible, but only by escaping the inner backticks, as John Default points out.
       1 word_count=` wc -w \`echo * | awk '{print $8}'\` `

Prev | Home | Next
Testing and Branching | Up | Arithmetic Expansion